Multi-scale target detection method and device based on confidence saliency analysis
By introducing a significant confidence analysis loss function and multi-scale information interaction module in deep convolutional neural networks, the problem of low accuracy in remote sensing image object detection is solved, and higher detection accuracy and recognition effect are achieved.
Patent Information
- Application Number
- CN202210758801.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-30
- Publication Date
- 2024-12-20
- Estimated Expiration
- 2042-06-30
AI Technical Summary
The existing remote sensing image object detection method has low accuracy when dealing with complex backgrounds and high noise environments, especially because there is room for improvement in recall and accuracy of single-stage detectors, and the class imbalance problem during training is more prominent. .
Using a multi-scale object detection method based on confidence significance analysis, a deep convolutional neural network is established, including batch normalized leakage module, residual block and multi-scale information interaction module, using confidence cross-entropy loss and significance loss function. , perform position regression and classification training, and non-maximum suppression is used during the training process to obtain the final target detection results.
The accuracy of remote sensing image object detection is significantly improved, and the impact of low confidence candidate boxes is suppressed by significantly analyzing the confidence level loss function, which improves cross-base recognition performance.
Smart Images
Figure CN115019213B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to remote sensing image target detection, and in particular to a multi-scale target detection method and device based on confidence saliency analysis. Background Art
[0002] Remote sensing image target detection and recognition is one of the most basic and essential tasks in the field of optical remote sensing image processing. Due to its wide coverage, long range, and high execution efficiency, it has important application value in both military and civilian fields. In the military field, remote sensing image data can be used to collect, analyze, and process intelligence and reconnaissance. Secondly, in the civilian field, it can achieve good results in many applications from landform analysis, vegetation distribution to disaster monitoring, land use planning, urban traffic monitoring, etc. However, remote sensing images have defects such as a large field of view, high background complexity, special viewing angle, and small target size, which makes remote sensing images provide more areas of interest while having more complex background noise information, which brings huge challenges to the target detection task.
[0003] In recent years, with the continuous development of deep networks, the excellent performance of deep convolutional neural networks in image-related tasks has attracted more and more attention, and using them in remote sensing images for target detection and recognition tasks has gradually become a mainstream trend. Compared with traditional machine learning target detection methods, deep learning algorithms can automatically extract target-related features, while machine learning algorithms require manual design of features. The huge labor cost greatly limits the application generalization of related algorithms. More importantly, in some scenarios, the features extracted by deep learning algorithms can more accurately and representatively represent the information of related targets than manually designed features.
[0004] In recent years, researchers have proposed many effective methods based on deep learning to automatically detect remote sensing targets. For example, the representative work of two-stage target detection, R-CNN (Region-CNN), obtains pre-selected boxes that may be image targets through sliding window search, and then uses a convolutional network to extract the corresponding task features from the pre-selected box image. Its main network structure includes a selective search module, a feature extraction network, a classifier, and a regressor. In the detection process of R-CNN, a large number of overlapping pre-selected boxes are generated in the sliding window search, which greatly increases the computational complexity of the feature extraction network. To solve this problem, the Fast R-CNN algorithm was proposed. From the basic principle level, it still uses the basic structure of the traditional R-CNN algorithm, but it uses a selective search method to obtain pre-selected boxes that meet actual needs instead of sliding search of all positions, which greatly reduces the number of candidate boxes. At the same time, the regressor and classifier are merged so that they are in the same network and trained at the same time, which reduces its complexity and greatly improves the detection accuracy.
[0005] Compared with the two two-stage detectors mentioned above, the researchers further proposed a single-stage detector. The single-stage detector does not require a candidate region generation stage, but directly treats all locations on the image as potential candidates, and also attempts to classify each region of interest as background or target object. However, judging from the results of the various single-stage detectors proposed at this stage on detection tasks, there is a lot of room for improvement in both recall and accuracy. This is mainly because setting candidate boxes on the entire feature map usually generates a large number of easily classified negative samples, which causes class imbalance problems during training. However, there is currently little research on this issue, and the overall progress is relatively slow. Summary of the invention
[0006] Purpose of the invention: In view of the problems existing in the prior art, the present invention provides a multi-scale target detection method and device based on confidence significance analysis with higher accuracy.
[0007] Technical solution: The multi-scale target detection method based on confidence saliency analysis of the present invention comprises:
[0008] (1) obtaining a remote sensing image target detection database based on aerial photography, wherein the database includes a plurality of remote sensing images and target positions and type labels in the remote sensing images;
[0009] (2) Preprocessing the remote sensing images in the database;
[0010] (3) establishing a deep convolutional neural network, wherein the deep convolutional neural network includes a batch normalization leakage module DBL (Darknet-Batchnorm-Leaky), a first residual block, a second residual block, a third residual block, a fourth residual block, a fifth residual block, a first multi-scale information interaction module, a second multi-scale information interaction module, a first multi-scale detection and recognition module, a second scale detection and recognition module, and a third scale detection and recognition module, which are sequentially connected from front to back;
[0011] (4) The preprocessed images and their corresponding labels are input as samples into the convolutional neural network for training. The loss function used in the training is the sum of the position regression loss, the confidence cross entropy loss, the classification cross entropy loss, and the significance loss calculated using the confidence;
[0012] (5) After preprocessing, the remote sensing image to be identified is used as a sample input into the trained deep convolutional neural network, and non-maximum suppression is performed to obtain the final target detection result.
[0013] The further step (2) specifically includes: using the cv2 toolkit of python to preprocess the remote sensing images in the database.
[0014] Furthermore, the batch normalization leakage module DBL (Darknet-Batchnorm-Leaky) in step (3) includes a convolutional layer and a batch normalization layer connected in sequence.
[0015] Furthermore, the multi-scale information interaction module in step (3) includes a DBL and an upsampling layer connected in sequence.
[0016] Furthermore, in step (3), the first multi-scale detection and recognition module, the second scale detection and recognition module and the third scale detection and recognition module have the same structure, and all include six DBLs connected in sequence and a single convolutional layer, but the feature scales are reduced by 1 / 2 in sequence.
[0017] Furthermore, each neuron in the deep convolutional network uses a linear correction unit leakyReLu as an activation function.
[0018] Furthermore, the loss function L in step (4) total for:
[0019] L totar =αL scn +βL
[0020]
[0021] L scn =max{0, margin-(α high -αlow )}
[0022] Where, L scn is the significance loss, L is the traditional target detection loss function including detection loss function and location loss function, α and β are the hyperparameters for adjusting the influence of target detection loss function and confidence significance loss function, and λ coord With λ noobj To adjust the coefficient of the influence of positive and negative samples in the target detection task, S 2 is the number of grids divided into images at a single scale, B is the number of candidate boxes generated for each grid, and Is the mark of whether the jth candidate box of the i-th grid contains the target, if so, The value is 1 otherwise it is 0. On the contrary, x i is the true horizontal coordinate of the target center, is the horizontal coordinate of the target center output by the network, y i is the true ordinate of the target center, is the vertical coordinate of the target center output by the network, is the actual width of the target, is the target width of the network output, is the target true height, is the target height output by the network, classes is the total number of target categories, c is the corresponding category, is the confidence of the jth candidate box in the i-th grid, is the corresponding true label, is the predicted probability of the jth candidate box in the i-th grid on the c-th category, is the corresponding true label, The value is 1 in the cth category and 0 in other cases; α high and α low They respectively represent the average values of the groups with confidences higher than the preset value and lower than the preset value corresponding to different candidate boxes at the same position of feature maps of different scales. Margin is a preset constant, which represents the minimum difference between the two groups of average values.
[0023] The multi-scale target detection device based on confidence significance analysis described in the present invention includes a processor and a computer program stored in a memory and executable on the processor. When the processor executes the program, the above method is implemented.
[0024] Beneficial effect: Compared with the prior art, the present invention has the significant advantage of having a higher accuracy rate. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1It is a flow chart of a multi-scale target detection method based on confidence significance analysis provided by the present invention;
[0026] Figure 2 It is a schematic diagram of a deep convolutional neural network;
[0027] Figure 3 It is the detailed structure of the feature extraction network of the multi-scale object detection network;
[0028] Figure 4 This is a comparison of experimental results of multi-scale networks with confidence significance analysis in target detection and recognition experiments. DETAILED DESCRIPTION
[0029] This embodiment provides a multi-scale target detection method based on confidence saliency analysis, such as Figure 1 As shown, the following steps are included:
[0030] (1) Obtaining a remote sensing image target detection database based on aerial photography, wherein the database includes a plurality of remote sensing images and target positions and type labels in the remote sensing images.
[0031] (2) Preprocess the remote sensing images in the database.
[0032] Among them, the preprocessing specifically uses Python's cv2 toolkit to preprocess the remote sensing images in the database.
[0033] (3) Establishing a deep convolutional neural network, the deep convolutional neural network includes a DBL, a first residual block, a second residual block, a third residual block, a fourth residual block, a fifth residual block, a first multi-scale information interaction module, a second multi-scale information interaction module, a first multi-scale detection and recognition module, a second scale detection and recognition module and a third scale detection and recognition module connected in sequence from front to back.
[0034] like Figure 2 As shown, the batch normalization leakage module DBL (Darknet-Batchnorm-Leaky) includes sequentially connected convolutional layers and batch normalization layers. The multi-scale information interaction module includes sequentially connected DBL and upsampling layers. In step (3), the first multi-scale detection and recognition module, the second scale detection and recognition module and the third scale detection and recognition module have the same structure, all including six sequentially connected DBLs and a single convolutional layer, but the feature scale is reduced by 1 / 2 in sequence. The neurons of each convolutional layer in the deep convolutional network use the linear correction unit leakyReLu as the activation function, and the outputs of all fully connected layer neurons use Dropout=0.5 to prevent overfitting.
[0035] like Figure 3As shown in the figure, the five residual blocks in the network all have two convolution kernels with different numbers of local receptive field sizes of 1*1 and 3*3 and corresponding batch processing. The stride of all convolution kernels is set to 1, and the edge zero padding strategy is used after convolution to keep the feature map size unchanged. For the first residual block, 2 convolution kernels are set; for the second residual block, 4 convolution kernels are set; for the third and fourth residual blocks, 16 convolution kernels are set; for the last residual block, 8 convolution kernels are set; for the last three different scales of detection and recognition networks, each contains 6 convolution kernels, and the feature dimension of the final output of each scale is reduced to half of the previous scale.
[0036] (4) The preprocessed images and their corresponding labels are input as samples into the convolutional neural network for training. The loss function used in training is the sum of position regression loss, confidence cross entropy loss, classification cross entropy loss and significance loss calculated using confidence.
[0037] The loss function L total for:
[0038] L totar =aL scn +βL
[0039]
[0040] L scn =max{0, margin-(α high -α low )}
[0041] Where, L scn is the significance loss, L is the traditional target detection loss function including detection loss function and location loss function, α and β are the hyperparameters for adjusting the influence of target detection loss function and confidence significance loss function, and λ coord With λ noobj To adjust the coefficient of the influence of positive and negative samples in the target detection task, S 2 is the number of grids divided into images at a single scale, B is the number of candidate boxes generated for each grid, and Is the mark of whether the jth candidate box of the i-th grid contains the target, if so, The value is 1 otherwise it is 0. On the contrary, x i is the true horizontal coordinate of the target center, is the horizontal coordinate of the target center output by the network, y i is the true ordinate of the target center, is the vertical coordinate of the target center output by the network, is the actual width of the target, is the target width of the network output, is the target true height, is the target height output by the network, classes is the total number of target categories, c is the corresponding category, is the confidence of the jth candidate box in the i-th grid, is the corresponding true label, and its value criteria are the same as same, is the predicted probability of the jth candidate box in the i-th grid on the c-th category, is the corresponding true label, The value is 1 in the cth category and 0 in other cases; α high and α low They represent the average values of the confidence levels of the groups with higher and lower confidence levels corresponding to different candidate boxes at the same position in feature maps of different scales. Margin is a preset constant that represents the minimum difference between the two groups of average values. This loss comprehensively describes the accuracy of the predicted category position, and then the loss function is reduced during training through the back propagation (BP) algorithm. By minimizing L scn The gap between the candidate boxes with higher confidence and the candidate boxes with lower confidence can be widened, thus suppressing the influence of the candidate boxes with low confidence.
[0042] (5) After preprocessing, the remote sensing image to be identified is used as a sample input into the trained deep convolutional neural network, and non-maximum suppression is performed to obtain the final target detection result.
[0043] Training optimizer configuration:
[0044] The training optimizer uses the Adaptive Moment Estimation (Adam) algorithm with a correction factor momentum (Nesterov Momentum) to calculate the loss function and continuously update the weights. Nesterov Momentum simulates the concept of momentum in physics to suppress the oscillation of the gradient direction and speed up the convergence. If the historical gradient is consistent with the current gradient direction, the Momentum term increases, otherwise it decreases; and the Nesterov term adds a correction when the gradient is updated to prevent the gradient from updating too fast. At the same time, the gradient update is more flexible. The iterative process is as follows:
[0045] m t =β1m t-1 +(1-β1)g t-1
[0046]
[0047] Where η represents the learning rate, which is set to 10 in the experiment -3 , in addition, the weight decay is set to 0.92.
[0048] This embodiment also provides a multi-scale target detection device based on confidence significance analysis, including a processor and a computer program stored in a memory and executable on the processor. When the processor executes the program, the above method is implemented.
[0049] In order to verify the present invention, a simulation was conducted and two categories of aircraft and ships were selected from the DOTA2.0 remote sensing image database. In order to verify the effectiveness and necessity of the confidence significance analysis loss function in the present invention, an ablation experiment was conducted on the selected database. In the experiment, MAP (mean Average Precision), F1-Score, Precision and Recall were selected as evaluation indicators. The results are shown in Figure 2. Figure 4 As shown, it can be observed that the confidence significance analysis loss function in the present invention has achieved excellent recognition results in remote sensing target detection and recognition tasks. The confidence significance analysis loss function draws on the idea of self-healing network and processes low-confidence candidate boxes separately; in addition, the designed loss function can constrain the confidence of candidate boxes at the same position of networks of different scales during training, suppress the influence of low-confidence candidate boxes on the training process, and plays a key role in improving cross-database recognition performance.
[0050] The above disclosure is only a preferred embodiment of the present invention, which cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.
Claims
1. A multi-scale target detection method based on confidence saliency analysis, characterized in that The method includes: (1) obtaining a remote sensing image target detection database based on aerial photography, wherein the database includes a plurality of remote sensing images and target positions and type labels in the remote sensing images; (2) Preprocessing the remote sensing images in the database; (3) establishing a deep convolutional neural network, wherein the deep convolutional neural network includes a batch normalization leakage module, a first residual block, a second residual block, a third residual block, a fourth residual block, a fifth residual block, a first multi-scale information interaction module, a second multi-scale information interaction module, a first multi-scale detection and recognition module, a second scale detection and recognition module, and a third scale detection and recognition module, which are sequentially connected from front to back; (4) The preprocessed image and its corresponding label are input as samples into the convolutional neural network for training. The loss function used in the training is the sum of the position regression loss, the confidence cross entropy loss, the classification cross entropy loss and the significance loss calculated using the confidence. The loss function L total for: L total =αL scn +βL L scn =max{0,margin-(α high -a low )} Where, L scn is the significance loss, L is the traditional target detection loss function, α and β are the hyperparameters that adjust the influence of the target detection loss function and the confidence significance loss function, and λ coord With λ noobj To adjust the coefficient of the influence of positive and negative samples in the target detection task, S 2 is the number of grids divided into images at a single scale, B is the number of candidate boxes generated for each grid, and Is the mark of whether the jth candidate box of the i-th grid contains the target, if so, The value is 1 otherwise it is 0. On the contrary, x i is the true horizontal coordinate of the target center, is the horizontal coordinate of the target center output by the network, yi is the true vertical coordinate of the target center, is the vertical coordinate of the target center output by the network, is the actual width of the target, is the target width of the network output, is the target true height, is the target height output by the network, classes is the total number of target categories, c is the corresponding category, is the confidence of the jth candidate box in the i-th grid, is the corresponding true label, is the predicted probability of the jth candidate box in the i-th grid on the c-th category, is the corresponding true label, The value is 1 in the cth category and 0 in other cases; α high and α low They represent the average values of the confidence levels of the groups with higher confidence levels than the preset values and the groups with confidence levels lower than the preset values at the same position of different candidate boxes in feature maps of different scales. Margin is a preset constant indicating the minimum difference between the average values of the two groups. (5) After preprocessing, the remote sensing image to be identified is used as a sample input into the trained deep convolutional neural network, and non-maximum suppression is performed to obtain the final target detection result.
2. The multi-scale target detection method based on confidence saliency analysis according to claim 1, characterized in that: Step (2) specifically includes: The remote sensing images in the database are preprocessed using the Python cv2 toolkit.
3. The multi-scale target detection method based on confidence saliency analysis according to claim 1, characterized in that: The batch normalization leakage module in step (3) includes a convolutional layer and a batch normalization layer connected in sequence.
4. The multi-scale target detection method based on confidence saliency analysis according to claim 1, characterized in that: The multi-scale information interaction module in step (3) includes a batch normalization leakage module and an upsampling layer connected in sequence.
5. The multi-scale target detection method based on confidence saliency analysis according to claim 1, characterized in that: In step (3), the first multi-scale detection and recognition module, the second scale detection and recognition module and the third scale detection and recognition module have the same structure, and all include six batch normalization leakage modules connected in sequence and a single convolutional layer, but the feature scale is reduced by 1 / 2 in sequence.
6. The multi-scale target detection method based on confidence saliency analysis according to claim 1, characterized in that: Each neuron in the deep convolutional neural network uses a linear correction unit leakyReLu as an activation function.
7. A multi-scale target detection device based on confidence saliency analysis, comprising a processor and a computer program stored in a memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.