A scene detection benchmark model training method, system and application method

Abstract

The present invention relates to a scene detection benchmark model training method, system and application method. Through the present invention, production and construction projects under construction can be accurately and effectively identified. By checking with the soil and water conservation plans and soil and water conservation prevention and control responsibility scopes that have been entered and reported to the national soil and water conservation supervision and management system, it can be determined whether the production and construction project has been built without approval in violation of regulations and whether the disturbance exceeds the prevention and control responsibility scope, thereby achieving full coverage supervision of production and construction projects.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image processing, and in particular to a scene detection benchmark model training method, system and application method. Background Art

[0002] With the development of computer vision technology and remote sensing image interpretation methods, target detection and semantic segmentation based on high-resolution optical remote sensing images have made significant progress. Detection of special objects, such as buildings, airplanes, and vehicles, has achieved high accuracy. However, the category definitions of these objects are relatively clear, and the diversity within the class is relatively limited. In comparison, complex ground scenes represented by production and construction projects lack widely accepted semantic concept definitions. The images contain a variety of artificial and natural objects, are highly unstructured within the class, have significant differences, and have a small number of training samples. Existing target detection methods are difficult to directly apply to the overall detection of complex ground scenes such as production and construction projects.

[0003] Therefore, how to solve the problem of low overall detection accuracy of complex ground scenes in production and construction projects has become a technical problem that urgently needs to be solved in this field. Summary of the invention

[0004] The purpose of the present invention is to provide a scene detection benchmark model training method, system and application method that can accurately and effectively identify production and construction projects under construction.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A scene detection benchmark model training method, the training method comprising the following steps:

[0007] Obtain original remote sensing images;

[0008] Annotating the production and construction project and the detailed features inside the original remote sensing image, wherein the annotation is a candidate frame;

[0009] Taking the original remote sensing image as input and the annotations of the production and construction projects in the original remote sensing image as labels, a production and construction project detector is trained to obtain a production and construction project detection benchmark model, wherein the production and construction project detector is a Faster RCNN detection network;

[0010] Calculating the information amount of each detailed feature in the sample data set;

[0011] The original remote sensing image is used as input, and the annotations of high-information detailed objects in the original remote sensing image are used as labels to train a detailed object detector to obtain a detailed object detection benchmark model, wherein the detailed object detector is a Faster RCNN detection network, and the high-information detailed objects are detailed objects whose information content is greater than a preset threshold.

[0012] The present invention also provides an application method of a scene detection benchmark model, the application method comprising the following steps:

[0013] Acquisition of remote sensing images;

[0014] The remote sensing image is input into the production and construction project detection benchmark model and the detailed object detection benchmark model respectively, to obtain the preliminary detection results of the actual production and construction project and the preliminary detection results of the actual detailed objects, wherein the production and construction project detection benchmark model and the detailed object detection benchmark model are scene detection benchmark models obtained through the above training; the preliminary detection results of the actual production and construction project are several actual production and construction project candidate frames and their corresponding confidences, and the preliminary detection results of the actual detailed objects are several actual detailed object candidate frames and their corresponding confidences;

[0015] Determine whether there are intersecting candidate frames of actual production and construction projects;

[0016] If so, the intersecting actual production and construction project candidate frames are merged to obtain a number of merged actual production and construction project candidate frames;

[0017] defining the plurality of merged actual production and construction project candidate frames and unmerged actual production and construction project candidate frames as new production and construction project candidate frames;

[0018] Calculate the ratio of the portion where each actual detailed feature candidate frame intersects with the corresponding new production and construction project candidate frame to the corresponding new production and construction project candidate frame;

[0019] The confidence of the final production and construction project candidate frame is obtained according to the ratio, the confidence of the actual detailed feature candidate frame and the confidence of the new production and construction project candidate frame.

[0020] The present invention also provides a scene detection benchmark model training system, the training system comprising:

[0021] The original remote sensing image acquisition module is used to acquire the original remote sensing image;

[0022] A labeling module, used for labeling the production and construction project and the detailed features inside the original remote sensing image, wherein the labeling is a candidate frame;

[0023] A production and construction project detection benchmark model training module is used to use the original remote sensing image as input and the annotations of the production and construction projects in the original remote sensing image as labels to train the production and construction project detector to obtain a production and construction project detection benchmark model, wherein the production and construction project detector is a Faster RCNN detection network;

[0024] An information volume calculation module, used to calculate the information volume of each detailed feature in the sample data set;

[0025] The detailed object detection benchmark model training module is used to take the original remote sensing image as input and the annotations of the high-information detailed objects in the original remote sensing image as labels to train the detailed object detector to obtain a detailed object detection benchmark model, wherein the detailed object detector is a Faster RCNN detection network, and the high-information detailed objects are detailed objects whose information content is greater than a preset threshold.

[0026] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0027] The present invention provides a scene detection benchmark model training method, system and application method. Through the present invention, production and construction projects under construction can be accurately and effectively identified. By checking with the soil and water conservation plans and soil and water conservation prevention and control responsibility scopes that have been entered and reported to the national soil and water conservation supervision and management system, it can be determined whether the production and construction project has been built without approval in violation of regulations and whether the disturbance exceeds the prevention and control responsibility scope, thereby achieving full coverage supervision of production and construction projects. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0029] Figure 1 A flowchart of a scene detection benchmark model training method provided in Example 1 of the present invention;

[0030] Figure 2a Annotate sample drawings for production and construction projects;

[0031] Figure 2b Annotate sample images for detailed features;

[0032] Figure 3 This is the Faster RCNN network framework diagram;

[0033] Figure 4A flowchart of a method for applying a scene detection benchmark model provided in Embodiment 2 of the present invention;

[0034] Figure 5 This is an example diagram of a production and construction project;

[0035] Figure 6 PR curve diagram of the experiment;

[0036] Figure 7 The detection results of Faster RCNN experiment and this method are shown in Figure 2.

[0037] Figure 8 A structural block diagram of a scene detection benchmark model training system provided in Example 3 of the present invention.

[0038] Explanation of symbols:

[0039] 1. Original remote sensing image acquisition module; 2. Labeling module; 3. Production and construction project detection benchmark model training module; 4. Information volume calculation module; 5. Detailed object detection benchmark model training module. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] The purpose of the present invention is to provide a scene detection benchmark model training method, system and application method that can accurately and effectively identify production and construction projects under construction.

[0042] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0043] Embodiment 1:

[0044] See also Figure 1 The present invention provides a scene detection benchmark model training method, the training method comprising the following steps:

[0045] A1: Get the original remote sensing image;

[0046] A2: marking the production and construction project and the detailed features inside it in the original remote sensing image, wherein the marking is a candidate frame;

[0047] A3: Using the original remote sensing image as input and the annotations of the production and construction projects in the original remote sensing image as labels, a production and construction project detector is trained to obtain a production and construction project detection benchmark model, wherein the production and construction project detector is a Faster RCNN detection network;

[0048] A4: Calculate the information amount of each detailed feature in the sample data set;

[0049] A5: The original remote sensing image is used as input, and the annotations of high-information detailed objects in the original remote sensing image are used as labels to train a detailed object detector to obtain a detailed object detection benchmark model, wherein the detailed object detector is a Faster RCNN detection network, and the high-information detailed objects are detailed objects whose information content is greater than a preset threshold.

[0050] In step A2, the production and construction project and its detailed features in the original remote sensing image are marked, specifically including:

[0051] A21: Use the minimum enclosing rectangle to mark production and construction projects;

[0052] A22: Use the minimum bounding rectangle or semantic segmentation method to annotate various relevant detailed features within the production and construction project.

[0053] In the present invention, manual labeling is adopted. For production and construction projects, the minimum circumscribed rectangular frame is labeled based on manual visual judgment. The present invention selects production and construction projects and four kinds of detailed features inside them for labeling, namely bare land, thatched roof, buildings under construction and prefabricated houses. The labeling styles are shown in Figures 2a and 2b. Figure 2b As shown ( Figure 2a A in the middle is a production and construction project. Figure 2b B is bare land, C is covered, D is construction road, E is building under construction, F is completed building and G is prefabricated house). The present invention is based on the 2-meter resolution GF-1 remote sensing image for annotation. The image format is RGB, the scale is 1:1000, the size is 600×600 pixels, and the annotation tool is labelImg. In the annotation results, the number of annotations for production and construction projects, bare land, covered, construction roads, buildings under construction, completed buildings and prefabricated houses are 752, 763, 154, 82, 292, 278 and 372 respectively.

[0054] like Figure 3 As shown, the Faster RCNN detection network includes: a feature extraction module, a region candidate network module, a region of interest pooling module, and a classification and regression network module;

[0055] The feature extraction module is used to extract a feature map of an input image;

[0056] The region candidate network module is used to generate candidate boxes using a deep learning algorithm;

[0057] The region of interest pooling module is used to correspond the candidate box to the corresponding feature map to obtain the feature map of the corresponding candidate box;

[0058] The classification and regression network module is used to regress the boundary of the classification candidate box using the feature map of the candidate box to obtain a detection result.

[0059] In step A4, the information amount of each detailed feature in the sample data set is calculated, which specifically includes:

[0060] The information amount of each detailed feature is calculated by the following formula:

[0061] C i =N i *w i (1)

[0062] In the formula, C i is the amount of information about each detailed feature type, N i The number of detailed features of the i-th type in the sample frame of all production and construction projects, w i is the weight;

[0063]

[0064] Among them, TF i IDF is the frequency of occurrence of the i-th type of detail feature, which is calculated according to the ratio of its occurrence in the marked production and construction project frame to the occurrence of the detail in all marks. i is the inverse file frequency, N i is the number of detailed features of the i-th category, N is the number of all detailed features, Y i is the average area of ​​the i-th type of detailed features, and Y is the average area of ​​production and construction projects.

[0065] According to the average sample area of ​​production and construction projects, the area in the frame is divided into regular grids, and the inverse file frequency IDF i Based on the ratio of the grid data contained in the sample area to the number of grids containing the i-th type of detailed features in the sample area, the logarithm is calculated. In the present invention, six types of detailed features, namely bare land, covered land, construction roads, prefabricated houses, buildings under construction, and completed buildings, are initially selected. According to formula (1), the information amount of the six types of detailed features is calculated as shown in Table 1:

[0066] Table 1 Information content of six types of detailed features

[0067] category Bare Land Cover Construction Road Mobile House Buildings under construction Completed buildings Amount of information 18.81 20.90 9.93 44.82 28.77 8.22

[0068] In theory, all detailed features can be taken into account and relevant detectors of all detailed features can be trained. However, this method has little effect on improving the detection effect of production and construction projects due to the small amount of information of some detailed features, and will increase the complexity of the experiment. When determining the best detailed feature type, the detailed type with high information content is selected based on the amount of information provided by the detailed feature. Therefore, in the present invention, four types of details, namely, prefabricated houses, buildings under construction, tarpaulins and bare land, are selected to characterize production and construction projects, and relevant detailed feature detectors are trained for these four types of detailed features.

[0069] Embodiment 2:

[0070] See also Figure 4 The present invention provides an application method of a scene detection benchmark model, the application method comprising the following steps:

[0071] B1: Acquisition of remote sensing images;

[0072] B2: Input the remote sensing image into the production and construction project detection benchmark model and the detailed object detection benchmark model respectively, and obtain the preliminary detection results of the actual production and construction project and the preliminary detection results of the actual detailed objects, wherein the production and construction project detection benchmark model and the detailed object detection benchmark model are the scene detection benchmark models obtained through training in Example 1; the preliminary detection results of the actual production and construction project are several candidate frames of the actual production and construction project and their corresponding confidences, and the preliminary detection results of the actual detailed objects are several candidate frames of the actual detailed objects and their corresponding confidences;

[0073] B3: Determine whether there are intersecting candidate frames of actual production and construction projects;

[0074] B4: If yes, merge the intersecting actual production and construction project candidate frames to obtain several merged actual production and construction project candidate frames;

[0075] B5: defining the plurality of merged actual production and construction project candidate frames and the unmerged actual production and construction project candidate frames as new production and construction project candidate frames;

[0076] B6: Calculate the proportion of the intersection of each actual detailed feature candidate frame and the corresponding new production and construction project candidate frame to the corresponding new production and construction project candidate frame;

[0077] B7: Obtain the confidence of the final production and construction project candidate frame according to the ratio, the confidence of the actual detailed feature candidate frame and the confidence of the new production and construction project candidate frame.

[0078] In step B4, the intersecting actual production and construction project candidate frames are merged to obtain a plurality of merged actual production and construction project candidate frames, specifically including:

[0079] The confidence corresponding to the merged candidate box of the actual production and construction project is calculated by the following formula:

[0080] Conf3=(Conf1*S1+Conf2*S2) / S3 (3)

[0081] Where S1 is the area of ​​one of the candidate boxes in the intersecting candidate boxes, S2 is the area of ​​the other candidate box in the intersecting candidate box, S3 is the area of ​​the merged candidate box, Conf1 is the confidence of one of the candidate boxes in the intersecting candidate box, Conf2 is the confidence of the other candidate box in the intersecting candidate box, and Conf3 is the confidence corresponding to the merged actual production and construction project candidate box.

[0082] like Figure 5 As shown, the lower left corner mark A is the original green, and B is the original red. Assuming that the ground truth is the green candidate box, the red candidate box intersects in most cases, that is, there is box overlap. Therefore, for the detection results obtained by the production and construction project scene detector, a merge method is adopted to make it consistent with the ground truth green box. The specific implementation method is: assuming that the detection results of the production and construction project detection benchmark model have candidate boxes A and B intersecting, then candidate boxes A and B will be merged into candidate box C, and the area range of candidate box C is the minimum circumscribed rectangle of candidate boxes A and B.

[0083] In step B7, the confidence of the final production and construction project candidate frame is obtained according to the ratio, the confidence of the actual detailed feature candidate frame and the confidence of the new production and construction project candidate frame, which specifically includes:

[0084] The confidence of the final production and construction project candidate box is determined by the following formula:

[0085]

[0086] Where Conf is the confidence of the final candidate box of the production and construction project, γ is a hyperparameter, Confx is the confidence of the new candidate box of the production and construction project, ConFi is the confidence of the i-th detailed feature in the candidate box, i is the i-th detailed feature, and IOP is i is the ratio of the intersection of the i-th detailed feature candidate frame and the corresponding new production and construction project candidate frame to the corresponding new production and construction project candidate frame;

[0087]

[0088] Among them, B1 is the production and construction project candidate frame detected by the production and construction project detection benchmark model, and B2 is the detailed object candidate frame detected by the detailed object detection benchmark model.

[0089] The difficulty in detecting production and construction project scenes is reflected in the difficulty in determining the range of its candidate frames. Another point of its detection difficulty is that the confidence of the detected candidate frames is not accurate enough. Since detailed objects have similar characteristics to general target detection categories, such as single features and clear boundaries, a certain detection effect can be achieved by training detectors based only on visual features. Therefore, detailed objects are obtained by using detailed object detectors to assist production and construction project scene detectors, combining the overall visual features of the production and construction project with the visual features of its internal details to obtain the comprehensive expression score of the production and construction project. According to formula (4), it can be judged how likely the area is to generate a construction project. In theory, the score is greater than 0.

[0090] The application method provided by the present invention is verified by a specific application as follows.

[0091] The present invention conducts experiments on the dataset annotated in Example 1, wherein the production and construction project scene detector uses the production and construction feature as the ground feature marker for training, and the detail feature detector uses four types of detail features, namely bare ground, covered ground, buildings under construction, and prefabricated houses, for training. Both detectors use 70% of the images in the randomly selected dataset as the training set and 30% of the images as the test set. In the experiment of this article, the hardware equipment parameters used for training and testing are shown in Table 2, and the software development environment configuration is shown in Table 3.

[0092] Table 2 Hardware equipment parameters

[0093] hardware model CPU Intel(R)Xeon(R)CPU E5-2665 GPU GeForce RTX 2080Ti Memory DDR4 10G

[0094] Table 3 Software environment configuration

[0095] hardware model operating system Ubuntu 18.04 programming language Python 3.6 Deep Learning Frameworks Pytorch 1.0 Computing Platform CUDA10 cuDNN 7.5.0

[0096] In the present invention, the relevant parameters of the Faster RCNN network are set as follows: the initial learning rate is 0.001, the learning rate is reduced to 1 / 10 of the original after every 10 iterations, and the number of training traversals is 50. The pre-trained ResNet101 is used as the network skeleton, the gradient optimization algorithm is SGD, the momentum is set to 0.9, and the decay coefficient is set to 0.0005.

[0097] The performance evaluation indicators in the present invention use F1 score, precision-recall curve and single category average precision (AP). F1 score is also called balanced F score, which is defined as the harmonic mean of precision and recall, and its formula is:

[0098]

[0099] The average precision of a single category is the area enclosed by the PR curve of that category and the x-axis and y-axis. The PR curve can make a relative assessment of the overall situation of the inspection results of production and construction projects, while the average precision of a single category can relatively objectively evaluate the average quality level of production and construction project inspections. The larger the value, the better the target detection model.

[0100] In general remote sensing object detection, IoU (Intersection over Union) is used as an indicator to evaluate the correctness of a candidate box detection, where IoU is defined as follows:

[0101]

[0102] In the formula, M and M1 are the candidate boxes obtained by the detector and the manually marked ground truth (Ground Truth) respectively. Due to the particularity of the production and construction project, there are cases where a single candidate box corresponds to multiple ground truths. Therefore, using IoU>0.5 is not suitable as a direct indicator for evaluating the correct detection of candidate boxes for production and construction projects. Based on expert knowledge, a one-to-many approach is adopted, that is, a single candidate box corresponds to multiple ground truths that intersect with it, that is, it satisfies Its definition is as follows:

[0103]

[0104] In the formula, M is the candidate box obtained by the detector, M i is the ground truth that intersects with this detector.

[0105] The present invention is based on the same training set and test set. In addition to the experiments of training the production and construction project detector using Faster RCNN and performing post-processing, merging the candidate boxes and adjusting the candidate box scores using the detailed candidate boxes (hereinafter referred to as this method), three groups of experimental controls are also performed, namely, directly training the production and construction project detector using Faster RCNN and Yolo v5x (ref.) (hereinafter referred to as Faster RCNN and Yolo), and only performing post-processing of merging the candidate boxes based on Faster RCNN (hereinafter referred to as variants). Table 4 shows the AP and F1 score comparison results of the four groups of experiments.

[0106] Table 4 AP and F1 score results of four groups of experiments

[0107]

[0108]

[0109] The PR curves of the four groups of experiments are shown in Figure 6 shown.

[0110] As can be seen from Table 4, the AP value and F1 score of this method are higher than those of the other three groups of experiments, indicating that the detection effect of this experiment is the best among the four groups. At the same time, through the PR curve, it can be further found that Yolo's overall performance is lower than that of other groups. At the same time, in the low recall stage (or high precision stage, that is, the left side of the curve), the curve of this method is obviously above the other three groups. This is also the most important improvement of this method compared with other experimental groups. By introducing the score of the detail detector, the reliability of the confidence of the candidate box of the production and construction project is improved, and its detection error rate in the low recall stage is reduced, thereby greatly improving the detection results. The following will classify the production and construction projects according to certain principles, compare the specific detection results of the Faster RCNN experiment and this method, and compare the differences in the detection results of the two experiments and how the detailed features play a role.

[0111] As mentioned above, the detection effect of this method is significantly better than that of FasterRCNN experiment in the low recall stage (left side of PR curve), that is, the false positive rate of this method is lower than that of Faster RCNN in this stage, while in the high recall stage (right side of PR curve), the difference in detection effect is not obvious. The following will analyze the detection results based on the detection results of these two experiments.

[0112] This method is based on the Faster RCNN experiment, adding the merging of candidate boxes and using the scores of detailed object candidate boxes to adjust the scores of production and construction project candidate boxes. We will analyze these two experiments separately.

[0113] Figure 7 These are some of the test results of the two experiments. Each line in the figure is the same image, which is a group of tests. From left to right, they are the Faster RCNN test results, the test results of this method, and the ground truth. In the first two groups of test results, it can be found that it is precisely because the boundaries of the production and construction projects are not clear that the Faster RCNN experiment detects two high-scoring candidate frames, although there is only one ground truth in the figure, resulting in poor results for both candidate frames. The use of candidate frame merging can solve this existing problem well, and can better solve the characteristic of unclear boundaries of production and construction projects. While making the test results and ground truth better consistent, it can also reduce the false detection rate of the test. Generally speaking, because the features of detailed objects are similar to those of general remote sensing objects, with clear features and clear boundaries, directly using Faster RCNN to detect detailed objects will get good test results. When there is a certain detailed frame in the candidate frame, the confidence of the candidate frame will be improved, otherwise it will be reduced. This is also the core idea of ​​the post-processing. In the third group of inspection results, there are false positives, and the confidence of the false positives is relatively high. Since there are no detailed objects detected by the detailed object detector near the false positives, it is easy to see from formula (4) that the confidence of the candidate box in this method is lower than the confidence of the candidate box in the Faster RCNN experiment. This group of results further shows that for the false positives, the confidence of this type of candidate box in this method is lower than that in the Faster RCNN experiment. Although this post-processing step cannot completely eliminate the false positives, it can suppress the confidence of the false positives and improve the confidence of the correct detection box to a certain extent, thereby improving the detection effect as a whole.

[0114] The introduction of detailed ground object detection is not only conducive to improving the confidence of conventional deep learning target detection methods, but also has the function of judging the production and construction project cycle and the implementation of soil and water conservation measures. Figure 7 In the first group of tests, there are large areas of bare land in the production and construction project, and the bare land can be detected by detailed ground objects, which can be used to determine that the production and construction project is in the initial stage of construction. When a building is detected, it can be matched with the buildings under construction and completed buildings based on the characteristics of the building to determine whether the building has been capped and built. At the same time, it can also be determined whether the abandoned soil and slag are strictly implemented in accordance with the relevant requirements and measures for soil and water conservation by checking whether there is thatch in the production and construction project.

[0115] As mentioned above, one of the main characteristics of production and construction projects is the large intra-class differences and the lack of unified visual feature information. The large intra-class differences are mainly reflected in the large differences in the area of ​​production and construction projects and the long construction period of production and construction projects. The visual characteristics of production and construction projects in different periods are very obvious. For example, in the early stage of production and construction projects, their visual characteristics tend to be bare land, while in the later stage of production and construction projects, their visual characteristics tend to be more like buildings. Therefore, according to the size and construction period of production and construction projects, they are classified and the detection performance of Faster RCNN experiments and this method under different categories is analyzed. This is very necessary to understand the degree to which detail detectors and even different details adjust the scores of candidate boxes of production and construction projects. Therefore, according to the size of the area of ​​production and construction projects, the average area of ​​production and construction projects in the sample is used as the boundary to divide production and construction projects into areas larger than the average area and areas smaller than the average area, and production and construction projects are divided into two stages according to the construction period, the early stage and the late stage. The classification standard is that there are no buildings under construction in the early stage of production and construction projects, and there are buildings under construction in the later stage. According to the two classification methods, the IOU value of each type of ground truth and its corresponding detection box is calculated, and the results are shown in Tables 5 and 6.

[0116] Table 5 IOU values ​​of the two groups of experiments classified by production and construction project area

[0117] Smaller than average area Larger than average area Faster RCNN 0.803 0.825 This method 0.819 0.857

[0118] Table 6 IOU values ​​of the two groups of experiments classified by production and construction project cycle

[0119] Early stage Late Faster RCNN 0.820 0.814 This method 0.835 0.826

[0120] It can be seen from Tables 5 and 6 that for both classification methods, the IOU values ​​of this method are higher than those of the Faster RCNN experiment. In the classification based on the area of ​​the production and construction project, the IOU value of this method is improved more in the category of larger than average area compared with the Faster RCNN experiment, that is, in the ground truth with a larger area, this method has the greatest improvement in the detection results through the strategy of merging candidate boxes, which proves that the merged candidate boxes mostly exist in the candidate boxes corresponding to the ground truth with a larger area. In the classification based on the cycle of the production and construction project, it is found that the difference in the improvement of the two categories of this method is not large, indicating that the different cycles of the production and construction projects have little effect on the improvement of the IOU value of this method.

[0121] The weight parameter γ in formula (4) is used to determine the weight of the detailed features, and its value range is 0-1. When γ = 0, the method in this paper is a variant method, that is, the candidate boxes are merged without using the detailed information to adjust the candidate boxes of the production and construction projects. When γ = 1, only the detailed information is used to obtain the score of the candidate box. After searching the parameters on the validation set, the optimal value of γ is 0.7.

[0122] The present invention performs target detection on production and construction projects based on the GF-1 remote sensing image of Wuhan. On the basis of the conventional deep learning target detection method Faster RCNN, considering the complexity of the target scene of the production and construction project, a method of merging the prediction result frame and introducing detailed ground object information inside the production and construction project is proposed to optimize the detection effect. The results show that this method can effectively reduce the false detection rate and improve the consistency between the detection results and the ground reality. The application of this method can accurately and effectively identify the production and construction projects under construction. By checking with the soil and water conservation plans and soil and water conservation prevention and control responsibility scopes that have been entered and reported in the national soil and water conservation supervision and management system, it can be determined whether the production and construction project has violated the regulations and has been built without approval and the disturbance exceeds the prevention and control responsibility scope, thereby achieving full coverage supervision of production and construction projects.

[0123] Introducing detailed landforms into target detection for production and construction projects can, on the one hand, adjust the prediction scores of conventional deep learning target detection methods; on the other hand, it can determine the construction period of production and construction projects (such as whether the building has been topped off and completed) and the implementation of soil and water conservation measures (such as whether abandoned soil and slag are covered) through the characteristics of detailed landforms.

[0124] Embodiment 3:

[0125] See also Figure 8 The present invention provides a scene detection benchmark model training system, the training system comprising:

[0126] The original remote sensing image acquisition module 1 is used to acquire the original remote sensing image;

[0127] A labeling module 2 is used to label the production and construction project and its internal detailed features in the original remote sensing image, wherein the labeling is a candidate frame;

[0128] The production and construction project detection benchmark model training module 3 is used to train the production and construction project detector using the original remote sensing image as input and the annotations of the production and construction projects in the original remote sensing image as labels to obtain a production and construction project detection benchmark model, wherein the production and construction project detector is a Faster RCNN detection network;

[0129] An information volume calculation module 4 is used to calculate the information volume of each detailed feature in the sample data set;

[0130] The detailed object detection benchmark model training module 5 is used to take the original remote sensing image as input, and use the annotations of detailed objects in the original remote sensing image whose information amount is greater than a preset threshold as labels to train the detailed object detector to obtain a detailed object detection benchmark model, wherein the detailed object detector is a Faster RCNN detection network.

[0131] Specifically, the marking module 2 includes:

[0132] The production and construction project annotation submodule is used to annotate the production and construction project using the minimum external rectangular frame;

[0133] The detailed feature annotation submodule is used to annotate various relevant detailed features inside the production and construction project using the minimum bounding rectangle or semantic segmentation method.

[0134] The information volume calculation module is used to calculate the information volume of each detailed feature in the sample data set, specifically including: calculating the information volume of each detailed feature by the following formula:

[0135] C i =N i *w i

[0136] In the formula, C i is the amount of information about each detailed feature type, N i The number of detailed features of the i-th type in the sample frame of all production and construction projects, w i is the weight; TF i is the frequency of occurrence of the i-th type of detailed features, IDF i is the inverse file frequency, N i is the number of detailed features of the i-th category, N is the number of all detailed features, Y i is the average area of ​​the i-th type of detailed features, and Y is the average area of ​​production and construction projects.

[0137] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0138] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A scene detection benchmark model training method, characterized in that: The following steps are involved: Obtain original remote sensing images; Annotating the production and construction project and the detailed features inside the original remote sensing image, wherein the annotation is a candidate frame; Taking the original remote sensing image as input and the annotations of the production and construction projects in the original remote sensing image as labels, a production and construction project detector is trained to obtain a production and construction project detection benchmark model, wherein the production and construction project detector is a Faster RCNN detection network; Calculate the information content of each detailed feature in the sample data set; Taking the original remote sensing image as input, and taking the annotations of high-information-content detailed features in the original remote sensing image as labels, a detailed feature detector is trained to obtain a detailed feature detection benchmark model, wherein the detailed feature detector is a Faster RCNN detection network, and the high-information-content detailed features are detailed features whose information content is greater than a preset threshold; Calculate the information content of each detailed feature in the sample data set, including: The information amount of each detailed feature is calculated by the following formula: C i =N i *w i In the formula, C i is the amount of information about each detailed feature type, N i The number of detailed features of the i-th type in the sample frame of all production and construction projects, w i is the weight; TF i is the frequency of occurrence of the i-th type of detailed features, IDF i is the inverse file frequency, N i is the number of detailed features of the i-th category, N is the number of all detailed features, Y i is the average area of ​​the i-th type of detailed features, and Y is the average area of ​​production and construction projects.

2. The scene detection benchmark model training method according to claim 1, characterized in that: The production and construction project in the original remote sensing image and the detailed features therein are marked, specifically including: Use the minimum bounding rectangle to mark production and construction projects; Use the minimum bounding rectangle or semantic segmentation method to annotate various relevant detailed features inside the production and construction project.

3. The scene detection benchmark model training method according to claim 1, characterized in that: The Faster RCNN detection network includes: a feature extraction module, a region candidate network module, a region of interest pooling module, and a classification and regression network module; The feature extraction module is used to extract a feature map of an input image; The region candidate network module is used to generate candidate boxes using a deep learning algorithm; The region of interest pooling module is used to correspond the candidate box to the corresponding feature map to obtain the feature map of the corresponding candidate box; The classification and regression network module is used to regress the boundary of the classification candidate box using the feature map of the candidate box to obtain a detection result.

4. A method for applying a scene detection benchmark model, characterized in that: The following steps are involved: Acquisition of remote sensing images; The remote sensing image is input into a production and construction project detection benchmark model and a detailed object detection benchmark model respectively to obtain preliminary detection results of actual production and construction projects and preliminary detection results of actual detailed objects, wherein the production and construction project detection benchmark model and the detailed object detection benchmark model are scene detection benchmark models obtained through training according to claim 1; the preliminary detection results of the actual production and construction project are several candidate frames of actual production and construction projects and their corresponding confidences, and the preliminary detection results of actual detailed objects are several candidate frames of actual detailed objects and their corresponding confidences; Determine whether there are intersecting candidate frames of actual production and construction projects; If so, the intersecting actual production and construction project candidate frames are merged to obtain a number of merged actual production and construction project candidate frames; defining the plurality of merged actual production and construction project candidate frames and unmerged actual production and construction project candidate frames as new production and construction project candidate frames; Calculate the ratio of the portion where each actual detailed feature candidate frame intersects with the corresponding new production and construction project candidate frame to the corresponding new production and construction project candidate frame; The confidence of the final production and construction project candidate frame is obtained according to the ratio, the confidence of the actual detailed feature candidate frame and the confidence of the new production and construction project candidate frame.

5. The method for applying the scene detection benchmark model according to claim 4, characterized in that: The actual production and construction project candidate frames that intersect are merged to obtain a plurality of merged actual production and construction project candidate frames, specifically including: The confidence corresponding to the merged actual production and construction project candidate box is calculated by the following formula: Conf3=(Conf1*S1+Conf2*S2) / S3 Where S1 is the area of ​​one of the candidate boxes in the intersecting candidate boxes, S2 is the area of ​​the other candidate box in the intersecting candidate box, S3 is the area of ​​the merged candidate box, Conf1 is the confidence of one of the candidate boxes in the intersecting candidate box, Conf2 is the confidence of the other candidate box in the intersecting candidate box, and Conf3 is the confidence corresponding to the merged candidate box of the actual production and construction project.

6. The method for applying the scene detection benchmark model according to claim 4, characterized in that: According to the ratio, the confidence of the actual detailed feature candidate frame and the confidence of the new production and construction project candidate frame, the confidence of the final production and construction project candidate frame is obtained, which specifically includes: The confidence of the final production and construction project candidate box is determined by the following formula: Where Conf is the confidence of the final candidate box of the production and construction project, γ is a hyperparameter, Confx is the confidence of the new candidate box of the production and construction project, ConFi is the confidence of the i-th detailed feature in the candidate box, i is the i-th detailed feature, and IOP is i is the ratio of the intersection of the i-th detailed feature candidate frame and the corresponding new production and construction project candidate frame to the corresponding new production and construction project candidate frame; where, B1 is the candidate frame of the production and construction project detected by the production and construction project detection benchmark model, and B2 is the candidate frame of the detailed object detected by the detailed object detection benchmark model.

7. A scene detection benchmark model training system, characterized in that: include: The original remote sensing image acquisition module is used to acquire the original remote sensing image; A labeling module, used for labeling the production and construction project and the detailed features inside the original remote sensing image, wherein the labeling is a candidate frame; A production and construction project detection benchmark model training module is used to use the original remote sensing image as input and the annotations of the production and construction projects in the original remote sensing image as labels to train the production and construction project detector to obtain a production and construction project detection benchmark model, wherein the production and construction project detector is a FasterRCNN detection network; The information volume calculation module is used to calculate the information volume of each detailed feature in the sample data set; A detailed object detection benchmark model training module is used to use the original remote sensing image as input and the annotations of high-information detailed objects in the original remote sensing image as labels to train a detailed object detector to obtain a detailed object detection benchmark model, wherein the detailed object detector is a Faster RCNN detection network, and the high-information detailed object is a detailed object whose information content is greater than a preset threshold; The information volume calculation module is used to calculate the information volume of each detailed feature in the sample data set, specifically including: calculating the information volume of each detailed feature by the following formula: C i =N i *w i In the formula, C i is the amount of information about each detailed feature type, N i The number of detailed features of the i-th type in the sample frame of all production and construction projects, w i is the weight; TF i is the frequency of occurrence of the i-th type of detailed features, IDF i is the inverse file frequency, N i is the number of detailed features of the i-th category, N is the number of all detailed features, Y i is the average area of ​​the i-th type of detailed features, and Y is the average area of ​​production and construction projects.

8. The scene detection benchmark model training system according to claim 7, characterized in that: The marking module comprises: The production and construction project annotation submodule is used to annotate the production and construction project using the minimum external rectangular frame; The detailed feature annotation submodule is used to annotate various relevant detailed features within the production and construction project using the minimum bounding rectangle or semantic segmentation method.

Images