A scene classification-assisted target detection and correction method for remote sensing images of mining areas

Abstract

The present invention discloses a scene classification-assisted mining area remote sensing image target detection correction method. The present invention first uses a scene classification algorithm to classify mining area remote sensing images to obtain scene category prior information, and then uses the combination degree of the target and the scene to correct the detection result of the target detection algorithm based on the scene category prior information, thereby improving the accuracy and stability of the target detection algorithm.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing and target detection, and in particular to a scene classification-assisted mining area remote sensing image target detection correction method. Background Art

[0002] Remote sensing image processing technology plays an important role in modern mining exploration and management and has a wide range of application value. In mining areas, target detection technology can help solve a variety of problems. First, by detecting the number, density, and distribution of buildings and vehicles, the reserves of mineral deposits can be estimated and the potential production benefits of mining areas can be evaluated. Secondly, target detection can detect the same area at multiple time points, helping mine managers to understand the changes in resources in a timely manner, optimize mining planning, and improve mining efficiency and productivity. Finally, target detection technology can also be used to regulate mining activities, strengthen the supervision and governance of mining resources, and help the government supervise and manage buildings in mining areas. Existing target detection methods are mainly deep learning methods. Target detection methods based on deep learning can be roughly divided into three categories: single-stage target detection, two-stage target detection, and Transformer-based target detection.

[0003] (1) Single-stage target detection algorithm

[0004] The single-stage target detection algorithm directly calculates the image and generates the detection result. The detection speed is fast, but the detection accuracy is low.

[0005] The classic representative of this type of algorithm is the YOLO series of algorithms, which only needs to process the image once with a neural network to simultaneously predict the location and category, thereby defining the target detection task as an end-to-end regression problem and improving the calculation speed. The workflow of YOLO is as follows:

[0006] For an input image, first scale the image to a uniform size and divide it into several grids. Then use a convolutional neural network to extract image features, and use a fully connected layer to perform regression prediction on each grid. Each grid predicts K boxes, and each box predicts five regression values. Four of the regression values ​​represent the location of the box, and the fifth regression value represents the probability of the box containing an object and the accuracy of the location. Then, for the box containing the object, use a fully connected layer to regress and predict the conditional probability of the object in each category. Therefore, the total number of prediction values ​​output by the convolutional network is N×(K×5+C), where N is the number of grids, K is the number of boxes generated for each grid, and C is the number of categories.

[0007] (2) Two-stage target detection algorithm

[0008] The two-stage target detection algorithm first extracts candidate boxes based on the image, and then performs secondary correction based on the candidate area to obtain the detection result. The detection accuracy is high, but the detection speed is slow. The classic representative of this type of algorithm is Faster RCNN. Due to its excellent performance, Faster RCNN is still a very competitive algorithm in the field of target detection. First, the image is input into CNN to obtain a high-level feature map, which is then passed to RPN. RPN uses a sliding window on the feature map and sets K prior boxes of different sizes and aspect ratios in each window. For each prior box, RPN predicts the probability that the box contains the target object and adjusts the regression amount for the box. Finally, bounding boxes of different shapes and sizes are obtained. After adjusting the size, they are passed to the fully connected layer to obtain the target category and the final prior box.

[0009] (3) Transformer-based methods

[0010] For target detection tasks, theoretically, the relationship between targets is helpful to improve the target detection effect. This type of method mainly uses Transformer to introduce the attention mechanism into the target detection field. DETR is the first representative work to complete the target detection task using Transformer.

[0011] First, DETR uses CNN to extract features of the image, and then adds position encoding commonly used in the NLP field to generate a batch of serialized data. In the encoder stage, the serialized data is sent to the encoder, and the features in the data are extracted using the attention mechanism. In the decoder stage, N random initialization vectors are input, and each object query focuses on different positions of the image. After decoding by the decoder, N vectors are finally generated, each corresponding to a detected target. Finally, these N vectors are input into the neural network to obtain the category and position of each target.

[0012] Due to the relatively low resolution of remote sensing images of mining areas, the diverse types of target objects in mining areas, the great differences in the scales of target objects, the similar colors of targets and backgrounds, and the large number of small targets, the existing target detection methods have problems with false detection and missed detection in mining area target detection.

[0013] In order to improve the accuracy of small target detection, there is a method to extract the enlarged area from the original image and learn more abundant small target features from it. The additional features have a positive contribution to the detection performance, but the selection of the area to be enlarged brings a computational burden and is not suitable for remote sensing images of large areas. There are also studies that propose a fully convolutional network for small target detection, which includes a visual attention mechanism to select the most promising area containing small targets and their context, giving higher weights to relevant parts, while minimizing irrelevant parts and giving them lower weights. However, due to the similar RGB values ​​of various parts in the mining area, this method cannot play a good effect in target detection in remote sensing images of mining areas. Summary of the invention

[0014] In view of this, the present invention provides a scene classification-assisted mining area remote sensing image target detection and correction method, which can effectively improve the detection accuracy of mining area targets.

[0015] The scene classification-assisted mining area remote sensing image target detection and correction method of the present invention comprises:

[0016] Step 1: extract the band information of the mining area remote sensing image and use the classifier to classify the scene; perform morphological post-processing on the classification results to obtain scene category information;

[0017] Step 2, detecting the targets in the remote sensing image and obtaining information of each target;

[0018] Step 3: For each target detected in step 2, the target and its surrounding area are selected and defined as the scene frame of the target; the scene category of the scene frame is determined according to each scene category and its proportion in the scene frame; and the category confidence of the target is corrected based on the scene category of the scene frame;

[0019] Step 4: Filter out the targets whose corrected target category confidence is greater than the set threshold to complete target detection.

[0020] Preferably, in step 1, the RGB band of the mining area remote sensing image is extracted, and the three RGB channels are combined in pairs and normalized to obtain three new composite bands; the mining area remote sensing image is converted into an HSV image to obtain the HSV band; and scene classification is performed based on the RGB band, the composite band and the HSV band.

[0021] Preferably, infrared and microwave bands are also included.

[0022] Preferably, the classifier is a random forest classifier, a maximum likelihood classifier, an XGBoost classifier or a combination thereof.

[0023] Preferably, when the classifier is a combination of classifiers, a voting method is used to fuse the classification results of each classifier in the combination.

[0024] Preferably, in step 2, YOLO, SSD or Faster-RCNN is used for target detection.

[0025] Preferably, the scene category of the scene frame is determined in the following manner:

[0026] Select the two categories with the highest proportion of scene categories in the scene frame, which are the first scene category and the second scene category. If the difference between the proportion of the first scene category and the proportion of the second scene category is greater than or equal to 0.4, the weight of the first scene category is 1, and the weight of the second scene category is 0; if the difference is greater than or equal to 0.2 and less than 0.4, the weight of the first scene category is 0.8, and the weight of the second scene category is 0.2; if the difference is less than 0.2, the weight of the first scene category is 0.6, and the weight of the second scene category is 0.4;

[0027] The category confidence of the target is modified as follows:

[0028] conf′=conf+w1x1+w2x2

[0029] Among them, conf is the target category confidence obtained in step 2; w1 and w2 are the weights of the first scene category and the second scene category respectively; x1 is the corrected change value of the first scene category for the target detection category, and x2 is the corrected change value of the second scene category for the target detection category.

[0030] For the better case, when the target categories are small vehicles, trucks, large vehicles and buildings, and the scene categories are sand, road, building, rock and soil, the correction change values ​​of the scene categories for the target detection categories are shown in Table 1:

[0031] Table 1 Relationship between scene and target correction

[0032]

[0033]

[0034] Preferably, the scene category also includes shadows; when the first scene category is a shadow, only the second scene category is used to correct the target detection category.

[0035] Beneficial effects:

[0036] (1) The present invention first uses a scene classification algorithm to classify the remote sensing images of the mining area to obtain scene category prior information, and then uses the degree of integration between the target and the scene to correct the detection results of the target detection algorithm based on the scene category prior information, thereby improving the accuracy and stability of the target detection algorithm.

[0037] (2) Use the data after pairwise normalization of the RGB bands and the data converted to HSV to expand the data set and improve the accuracy of the scene classification algorithm.

[0038] (3) The classification results of multiple classifiers are combined using an integrated classifier to improve the accuracy of the scene classification algorithm.

[0039] (4) Use the opening and closing operations of morphological post-processing to eliminate noise and redundant information in the classification results, smooth feature edges, and further improve the accuracy of scene category information.

[0040] (5) In the scene frame, not only one scene is considered, but different weights are assigned according to the proportion of different scene categories in the scene frame, which together affect the final target detection result, making the correction result more objective and effective.

[0041] (6) After observing mining area images and conducting multiple experiments, appropriate parameter values ​​were used in the process of correcting target detection results through scene category information, so that there is an independent set of correction rules for different scenes and different targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 The figure is a flow chart of the method of the present invention.

[0043] Figure 2 This is a remote sensing image of a mining area.

[0044] Figure 3 Labeling data for remote sensing imagery object detection.

[0045] Figure 4 The normalized image details are shown.

[0046] Figure 5 The results of scene classification are shown.

[0047] Figure 6 Correction effect display for target detection. DETAILED DESCRIPTION

[0048] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0049] The present invention provides a scene classification-assisted mining area remote sensing image target detection and correction method, which consists of two parts: scene classification and target detection. Among them, the scene classification part is responsible for extracting scene category prior information; the target detection part first uses the target detection method to perform preliminary detection on the mining area remote sensing image target, and then uses the scene category prior information to correct the target initial detection result. Among them, the scene classification part can use the band information of the mining area remote sensing image to complete scene classification based on the classifier to obtain scene information; the band information includes visible light band, infrared band, microwave band, etc.; the classifier can use a random forest classifier, a maximum likelihood classifier, an XGBoost classifier (eXtreme Gradient Boosting, extreme gradient boosting), etc. and a combination of multiple classifiers. The initial target detection can be achieved by using methods such as YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector, single point multi-box detector), and Faster-RCNN (Faster-Region Convolutional Neural Network, faster regional convolutional neural network).

[0050] This embodiment takes the remote sensing image data of the Chuquicamata mining area as an example for explanation. The Maxar satellite remote sensing image dataset provided by the Google EarthEngine Pro platform is used to download images of the Chuquicamata copper mine area in multiple time periods, and convert the format to obtain a TIF remote sensing image dataset, such as Figure 2 As shown. Remote sensing image data has three characteristics: high resolution, wide coverage and real-time update. Maxar has the world's largest commercial high-resolution satellite network, and provides remote sensing images with high resolution, reaching less than 50 cm, including remote sensing images of different categories such as buildings, roads, bridges, rivers, forests, etc., which can capture detailed terrain and land features, and provide coverage in almost all areas, making the data very comprehensive. In addition, Maxar continuously updates its satellite image database and publishes the latest remote sensing images in real time, which enables users to obtain the latest earth image information. In this embodiment, the target detection method adopts YOLOv7, which is a relatively new version of the YOLO series. In the range of 5FPS to 160FPS, its speed and accuracy have exceeded all previous target detectors. In terms of mining area monitoring, it can effectively detect buildings, mining facilities and mining vehicles, improve monitoring efficiency and accuracy, and has broad prospects in mining area remote sensing applications.

[0051] The solution flow chart of this embodiment is as follows Figure 1 As shown, the specific steps include:

[0052] Step 1: extract the band information of the mining area remote sensing image and use the classifier to classify the scene.

[0053] Among them, the band information includes visible light band, infrared band and microwave band. In this embodiment, only the visible light band is extracted, and the RGB band of the TIF image is extracted from the remote sensing image. In order to increase the band information and improve the accuracy of scene classification, this embodiment imitates indicators such as NDVI (Normalized Difference Vegetation Index) and NDBI (Normalized Difference Built-up Index), and normalizes the three RGB bands after combining them in pairs to obtain three composite bands, as shown in formulas (1) to (3). The details of the normalized image are shown in Figure 2. Figure 4 As shown, the edge contour information of scenes such as buildings, roads, and mountains can be clearly seen.

[0054] RG=255×(RG)÷(R+G) (1)

[0055] RB=255×(RB)÷(R+B) (2)

[0056] GB=255×(GB)÷(G+B) (3)

[0057] In addition, the RGB image is converted to an HSV image to obtain three HSV bands. The nine band information (three RGB bands, three composite bands, and three HSV bands) is used to classify the scene. Compared with the classification effect of using only RGB band information, the training set accuracy of the classifier is increased by about 8 percentage points after adding HSV band information and composite band information.

[0058] Scene classification can be achieved based on the classifier. The classifier can be a random forest classifier, a maximum likelihood classifier, an XGBoost classifier (eXtreme Gradient Boosting, extreme gradient boosting), etc.; this embodiment uses a combined classifier, that is, firstly using a random forest classifier, a maximum likelihood classifier, and an XGBoost classifier to classify the scene respectively, and then using a voting method to integrate the three classification results to improve the accuracy of classification.

[0059] Then, the classification results are subjected to morphological post-processing to optimize the classification effect and improve the classification accuracy. In this embodiment, the morphological post-processing uses an open operation followed by a closed operation, which can effectively remove noise, smooth object edges, separate small objects, and restore connectivity between objects, thereby improving the accuracy of the scene classification results. Figure 5 shown.

[0060] Step 2: Use the target detection algorithm to perform preliminary target detection on the remote sensing image of the mining area to obtain the initial target candidate frame.

[0061] The target detection algorithm may adopt algorithms such as YOLO, SSD, and Faster-RCNN. This embodiment adopts YOLOv7.

[0062] Step 3: Based on the scene classification in step 1, the category confidence of the target detection result in step 2 is corrected. The corrected result is as follows: Figure 6 shown.

[0063] The categories of scene classification in step 1 include sand, road, building, rock, shadow and soil, and the categories of object detection in step 2 include large vehicles (such as mining cars and excavators), trucks, small vehicles (such as cars and vans) and buildings.

[0064] There is a certain degree of closeness between the target and the scene in which it appears. Take vehicles and buildings as an example:

[0065] As for vehicles, all vehicles rarely appear on sand. In some cases, they will appear on the soil. The soil here refers to the terrain that appears after the wheels of various vehicles crush the sand, so it is listed as a separate category. Most vehicles will appear on the road, and vehicles will not appear on buildings (targets). Only large vehicles will appear in rocky areas for mining and transportation of ore. When there are no shadow samples, shadows will be identified as roads. Therefore, the scene categories of the present invention also include shadows. Such classification is mainly to prevent supervised classification from confusing shadows and roads, so as to obtain better classification results.

[0066] For buildings, sand, roads, rocks, shadows and soil will appear around the buildings, but the main part of the building (scene) is still composed of the building (target) itself.

[0067] Therefore, the confidence of the target detection result can be corrected according to the scene category that the target is in. The present invention provides a correction relationship corresponding to each scene category and the target detection category, as shown in Table 1.

[0068] Table 1 Relationship between scene and target correction

[0069]

[0070] First, for each target candidate box obtained in step 2, extract the coordinate values ​​x1, x2, y1, y2 of the upper left corner and lower right corner of the target candidate box to obtain the height h and width w of the candidate box. In the scene classification image, the target candidate box and the surrounding area are selected, which is called the scene box. Assuming that the size of the target candidate box is 40*80, the size of the scene box is 60*120.

[0071] Then, the category of the scene frame is determined according to the ratio of the pixel proportion of each scene category in the scene frame in the entire scene frame. This embodiment comprehensively considers the impact of different scene categories appearing in the scene frame and various scene categories on the target category, and only considers the two scene categories with the largest proportion in the scene frame. This is because in actual situations, the possibility of three or more scene categories appearing in a scene frame at the same time is low, usually one or two scene categories. If only one scene category is considered, the target detection accuracy may sometimes be reduced. For example, a vehicle next to a building may be negatively affected and missed. If too many scene categories are considered, manually setting too many correction parameters will be counterproductive and reduce the correction effect.

[0072] Specifically, firstly, the two scene categories with the highest proportion in the scene frame are selected, which are called the first scene category and the second scene category. Then, the scene category of the scene frame is determined according to the difference between the proportion of the first scene category and the proportion of the second scene category, as shown in Table 2.

[0073] Table 2. Corresponding weights of category proportions

[0074]

[0075] For example, if the sand category accounts for more than 90% of the scene frame, then the scene frame is classified as the sand category; if the sand category accounts for 50% and the road category accounts for 40%, then these two scene categories will be given different weights, 0.6 and 0.4 respectively, and will affect the target detection results in the scene frame.

[0076] The weight of the first scene category is recorded as w1, the weight of the second scene category is recorded as w2, the correction change value of the first scene category to the target detection category is x1, the correction change value of the second scene category to the target detection category is x2, the preliminary detection category confidence of the target detection candidate box is conf, and the corrected target category confidence conf′ is:

[0077] conf′=conf+w1x1+w2x2 (4)

[0078] Based on formula (4), the confidence level of target detection can be corrected by different scene categories. Next, we will discuss the correction methods of target detection algorithm using scene category information in each specific case:

[0079] (1) Sand as the first scene

[0080] There are generally no vehicles and few buildings on the sand. When the sand is used as the first scene category, the solution for target detection correction is shown in Table 3.

[0081] Table 3 Correction plan when sand is used as the first scenario

[0082]

[0083]

[0084] (2) Road as the first scene

[0085] Most vehicles appear on the road, and there are also roads around buildings. When the road is used as the first scene category, the solution for target detection correction is shown in Table 4.

[0086] Table 4 Correction scheme when road is used as the first scenario

[0087]

[0088] (3) Architecture as the first scene

[0089] Various vehicle types will not be detected in the building scene, while the target detection results of buildings will appear in the building scene. When the building is used as the first scene category, the solution for target detection correction is shown in Table 5.

[0090] Table 5 Correction plan when building is used as the first scene

[0091]

[0092] (4) Rocks as the first scene

[0093] In the rock scene, only large vehicles (excavators, mining cars) will appear, and no cars, trucks, or buildings will appear. When rock is used as the first scene category, the solution for target detection correction is shown in Table 6.

[0094] Table 6 Correction scheme when rock is used as the first scene

[0095]

[0096] (5) Soil as the first scene

[0097] Since the vehicle can drive in the soil scene, and the scene frame of the building target should mainly contain the building scene, the confidence of some building targets can be reduced. When soil is used as the first scene category, the solution for target detection correction is shown in Table 7.

[0098] Table 7 Modification scheme when soil is used as the first scenario

[0099]

[0100] (6) Shadow as the First Scene

[0101] The shadow category is mainly used to avoid confusion with the road category during classification. Buildings and vehicles will produce shadows, so when the shadow category is used as the first category, the second category is used to correct the target detection results, as shown in Table 8.

[0102] Table 8 Correction scheme when shadow is used as the first scene

[0103]

[0104] The scene frame range is delineated according to the target detection candidate frame, and then the weight is set according to the category ratio of the scene in the scene frame according to the result of scene classification. The corrected target detection candidate frame category confidence is calculated according to the scheme in the above table.

[0105] Step 4: Perform threshold judgment based on the corrected target detection candidate box category confidence, and finally output an output image with a target detection box and confidence greater than the set threshold.

[0106] The final effect comparison of the present invention is shown in Table 9. It can be seen that the present invention improves the detection accuracy of the YOLOv7 target detection algorithm for each type of target on the mine remote sensing image dataset annotated by itself.

[0107] Table 9 Comparison results of modified experiments

[0108]

[0109] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A scene classification-assisted mining area remote sensing image target detection and correction method, characterized in that: include: Step 1, extract the band information of the mining area remote sensing image and use the classifier to classify the scene; Perform morphological post-processing on the classification results to obtain scene category information; Step 2, detecting the targets in the remote sensing image and obtaining information of each target; Step 3: For each target detected in step 2, the target and its surrounding area are selected and defined as the scene frame of the target; the scene category of the scene frame is determined according to each scene category and its proportion in the scene frame; and the category confidence of the target is corrected based on the scene category of the scene frame; The scene category of the scene frame is determined in the following manner: Select the two categories with the highest proportion of scene categories in the scene frame, which are the first scene category and the second scene category. If the difference between the proportion of the first scene category and the proportion of the second scene category is greater than or equal to 0.4, the weight of the first scene category is 1, and the weight of the second scene category is 0; if the difference is greater than or equal to 0.2 and less than 0.4, the weight of the first scene category is 0.8, and the weight of the second scene category is 0.2; if the difference is less than 0.2, the weight of the first scene category is 0.6, and the weight of the second scene category is 0.4; The category confidence of the target is modified as follows: conf'′=conf+W1X1+W2X2 Wherein, conf is the target category confidence obtained in step 2; w1 and w2 are the weights of the first scene category and the second scene category respectively; x1 is the correction change value of the first scene category to the target detection category, and x2 is the correction change value of the second scene category to the target detection category; Among them, when the target categories are small vehicles, trucks, large vehicles and buildings, and the scene categories are sand, road, building, rock and soil, the correction change values ​​of the scene categories for the target detection categories are shown in Table 1: Table 1 Relationship between scene and target correction Step 4: Filter out the targets whose corrected target category confidence is greater than the set threshold to complete target detection.

2. The method according to claim 1, characterized in that In the step 1, the RGB band of the mining area remote sensing image is extracted, and the three RGB channels are combined in pairs and normalized to obtain three new composite bands; the mining area remote sensing image is converted into an HSV image to obtain an HSV band; and scene classification is performed based on the RGB band, the composite band and the HSV band.

3. The method according to claim 2, characterized in that It also includes infrared and microwave bands.

4. The method according to claim 1, 2 or 3, characterized in that: The classifier is a random forest classifier, a maximum likelihood classifier, an XGBoost classifier or a combination thereof.

5. The method according to claim 4, characterized in that When the classifier is a combination of classifiers, the voting method is used to fuse the classification results of each classifier in the combination.

6. The method according to claim 1, characterized in that In step 2, YOLO, SSD or Faster-RCNN is used for target detection.

7. The method according to claim 1, characterized in that The scene category also includes shadows; when the first scene category is a shadow, only the second scene category is used to correct the target detection category.

Images