Small target detection method based on multi-modal remote sensing image

Abstract

The invention discloses a small target detection method based on a multi-modal remote sensing image, and the method comprises the steps: carrying out the detection of a small target through an end-to-end target detection network based on a convolutional neural network and the same, complementary and multi-level features of a target in the multi-modal remote sensing image; the detection precision of the small target of the remote sensing image is improved while the related features in the multi-mode remote sensing image are fully utilized. According to the method, the same and complementary characteristics of registered multi-modal remote sensing images are fully utilized, the multi-modal target detection method for small targets is provided on the basis of the characteristics that the small targets of the remote sensing images in actual engineering are small in size and weak in characteristic, the multi-modal input images are fused at the pixel level, and the multi-modal target detection accuracy is improved. Target features are extracted layer by layer through a convolutional neural network, multi-scale target features are fused in a cross-level mode, fine features are enhanced through a weighted convolution attention mechanism, the overall identification capability is improved, an end-to-end target detection network is formed, and the purpose of high-precision and high-speed detection of a small target of a remote sensing image is achieved. Experiments show that the recall ratio and precision ratio of small target detection of the remote sensing image can be improved while the multi-mode remote sensing image can be effectively utilized.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and machine learning, and specifically relates to a small target detection method based on multimodal remote sensing images. Background Art

[0002] With the development of remote sensing image target detection, small target detection has gradually become a research focus. Small targets in remote sensing images have weak features and small size. Weak features are mainly reflected in unclear contours, unprominent texture features, and high similarity with the adjacent background; small size is manifested in the small number of pixels occupied in the image, only dozens of pixels. Therefore, it is difficult to extract enough effective features in remote sensing images, and small target detection becomes difficult.

[0003] Multimodal image fusion technology extracts information from different sensors to complement each other, making the information of the fused image more comprehensive and reliable, thereby improving the accuracy of subsequent image processing and decision-making tasks. Its core idea is to select the best data without interference, extract it effectively, eliminate redundant information, enhance robustness, and improve recognition accuracy.

[0004] In recent years, deep learning has shown great advantages in target detection. Traditional methods require clear features, have poor flexibility, are not robust, and are not suitable for infrared and full-color images. Deep learning can learn features from massive data sets and extract shallow surface features and deep semantic features. When there are sufficient samples, the trained network model has strong generalization ability and can cope with target detection in complex environments. For targets with clear features and large sizes, conventional methods such as Faster R-CNN, SSD, and YOLO can achieve high accuracy. However, in order to effectively use multimodal data to accurately detect small targets, conventional methods can no longer meet the needs. It is necessary to propose key technologies to improve the detection accuracy of small targets in multimodal detection tasks based on the characteristics of small targets. Summary of the invention

[0005] In order to solve the problem of low detection accuracy of small targets in multimodal remote sensing images by conventional target detection methods and meet the requirements of detecting targets under complex conditions in target detection tasks, the present invention proposes a small target detection method based on multimodal remote sensing images.

[0006] The present invention proposes a small target detection method based on multimodal remote sensing images. The method is based on the YOLOv5 algorithm convolutional neural network, the same and complementary features of targets in multimodal images, and the multi-level features of small targets. Through an end-to-end target detection network, the small target detection accuracy of multimodal remote sensing images is improved. The specific process of the detection method is:

[0007] Step 1: Input the multimodal remote sensing image into the multimodal fusion module, fuse the multimodal images and obtain the fused multimodal feature map;

[0008] Step 2: Input the feature map obtained in step 1 into the convolutional neural network, and the backbone feature extraction network samples the remote sensing image to form multi-level features of the image. At the same time, a weighted convolution attention module is added to the backbone feature extraction network;

[0009] Step 3: Input the feature map obtained in step 2 into the BSS-FPN feature pyramid for feature fusion. Double upsampling and downsampling operations are performed on some levels of the multi-level features of the image, and the upper and lower level features of adjacent feature extraction layers and cross-scale feature layers are fused to extract fused features;

[0010] Step 4: Perform three-branch prediction; finally, use the non-maximum suppression algorithm NMS to obtain the multimodal small target detection results.

[0011] Furthermore, the step 1 is specifically as follows:

[0012] The multimodal remote sensing image with small targets is first input into the multimodal fusion module (the multimodal images input into this module are respectively processed along two identical branches, and only one of the branches is taken as an example here). After the input image is subjected to the SE attention mechanism and two 1×1 convolutional layers to extract features, it is multiplied pixel by pixel with the feature map before convolution, and then added pixel by pixel with the input image to obtain a weighted image, and then the weighted image is subjected to a 3×3 convolution operation to obtain the final feature map. The final feature map is added to the final feature map obtained from the other branch on the channel, and the feature map is finally input into the network through the SE attention module.

[0013] Furthermore, the step 2 is specifically as follows:

[0014] The remote sensing image with small targets is input into the convolutional neural network, and the backbone feature extraction network continuously downsamples the remote sensing image four times, with the downsampling multiple being twice, thereby extracting the shallow to deep features of the image, and the corresponding level number is p i (i=4,6,9), forming multi-level features of the image.

[0015] Furthermore, the step 3 is specifically as follows:

[0016] The feature map obtained in step 2 is input into the BSS-FPN feature pyramid for feature fusion. 9 Double upsampling is performed, and cross-scale fusion is performed on adjacent level features and cross-scale feature layers to achieve feature fusion. First, BSS-FPN obtains input feature maps from features at different levels, including p 4 、p6 、p 9 . p 9 The feature map is doubled in size through a nearest neighbor upsampling operation. Next, p 6 Feature map and upsampled p 9 The feature map is at node p 12 Then, p 12 Features generate new feature map p 14 In the second-level feature fusion process, p 4 Feature map and upsampled p 6 The feature map is at node p 17 The upsampled feature map p 14 Fusion is performed to generate a new feature map p 17 . p 17 The features are convolved to generate a new feature map p 19 . Feature map p 19 At node p 20 In the feature map p 14 And the feature map p 6 Perform cross-level fusion. Feature map p 20 After convolution at node p 23 In the feature map p 10 Perform cross-level fusion. The fused features are passed to p 18 、p 21 and p 24 These features are convolutionally processed and used in three detection layers to output the location and category information of the target.

[0017] The feature fusion is expressed by the following formula:

[0018]

[0019] p 23 =p 10 +p 20 ↓ 2×

[0020] This method enables the network to maintain high-resolution feature details, thereby ensuring the accuracy of small object detection. At the same time, it provides a wider sensory field to determine the detection accuracy of regular-sized objects. When combining data of different scales in the backbone network, up- and down-sampling with equal feature resolution is adopted. Direct connections are established between the initial input nodes and output nodes of the same feature, further enriching the feature set without increasing expenditure.

[0021] Furthermore, the step 4 is specifically as follows:

[0022] After convolution processing, these features are used in three detection layers to output the location and category information of the target. The features of regular targets and small targets are obtained. The small target branch feature map with prominent features is used for small target prediction, and finally the non-maximum suppression (NMS) is used to obtain the small target detection result.

[0023] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0024] The present invention is applicable to small target detection tasks in multimodal remote sensing images, solves the problem of low detection accuracy of existing target detection methods when performing multimodal small target detection tasks, and effectively improves the recall and precision rates of multimodal small targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the overall process of the target detection method of the present invention;

[0026] Figure 2 This is an example of a multimodal remote sensing image with small targets;

[0027] Figure 3 Schematic diagram of a multimodal fusion module in the target detection method of the present invention;

[0028] Figure 4 Schematic diagram of a weighted convolutional attention module in the target detection method of the present invention;

[0029] Figure 5 It is an example diagram of the BSS-FPN feature pyramid structure in the target detection method of the present invention;

[0030] Figure 6 This is an example diagram of the results of small target detection in remote sensing images in the target detection method of the present invention. DETAILED DESCRIPTION

[0031] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings:

[0032] This paper proposes a small target detection method based on multimodal remote sensing images. This method uses the YOLOv5 convolutional neural network, combines the same and complementary features of targets in multimodal images and the multi-level features of small targets, and improves the detection accuracy of small targets in multimodal remote sensing images through an end-to-end target detection network. The network structure diagram is shown in Figure 1 The present invention can effectively utilize multimodal remote sensing images, accurately detect small targets therein, and significantly improve the recall and precision of small target detection.

[0033] The specific process of the small target detection method based on multimodal remote sensing images is as follows:

[0034] Step 1: First input the multimodal remote sensing image with small targets into the multimodal fusion module (the multimodal images input into this module are respectively processed along two identical branches, and only one of the branches is taken as an example here). The structure diagram of this module is shown in Figure 3 As shown. After the input image is extracted through the SE attention mechanism and two 1×1 convolutional layers, it is multiplied pixel by pixel with the feature map before convolution, and then added pixel by pixel with the input image to obtain a weighted image. The weighted image is then convolved 3×3 to obtain the final feature map. The final feature map is added to the final feature map obtained from the other branch on the channel, and the feature map is finally input to the network through the SE attention module.

[0035] Step 2: Input the feature map obtained in step 1 into the convolutional neural network, and use the backbone feature extraction network to continuously downsample the remote sensing image four times, with the downsampling multiple being twice, thereby extracting the shallow to deep features of the image, and the corresponding level number is p i (i=4,6,9), forming multi-level image features. At the same time, a weighted convolution attention module is added to the backbone feature extraction network. The module structure is shown in the figure Figure 4 This module extracts information from the compressed feature map of each channel, so that the feature layer with stronger semantic information has a higher weight;

[0036] Step 3: Input the feature map obtained in step 2 into the BSS-FPN feature pyramid for feature fusion. 9 Double upsampling is performed, and cross-scale fusion is performed on adjacent level features and cross-scale feature layers to achieve feature fusion. First, BSS-FPN obtains input feature maps from features at different levels, including p 4 、p 6 、p 9 . p 9 The feature map is doubled in size through a nearest neighbor upsampling operation. Next, p 6 Feature map and upsampled p 9 The feature map is at node p 12 Then, p 12 Feature convolution generates a new feature map p 14 In the second-level feature fusion process, p 4 Feature map and upsampled p 6 The feature map is at node p 17 The upsampled feature map p 14 Fusion is performed to generate a new feature map p 17 . p 17 The features are convolved to generate a new feature map p 19 . Feature map p 19 At node p20 In the feature map p 14 And the feature map p 6 Perform cross-level fusion. Feature map p 20 After convolution at node p 23 In the feature map p 10 Perform cross-level fusion. The fused features are passed to p 18 、p 21 and p 24 These features are convolutionally processed and used in three detection layers to output the location and category information of the target.

[0037] The feature fusion is expressed by the following formula:

[0038]

[0039] p 23 =p 10 +p 20 ↓ 2×

[0040] Step 4: After convolution processing, these features are used in three detection layers to output the location and category information of the target. The features of regular targets and small targets are obtained. The small target branch feature map with prominent features is used for small target prediction, and finally the non-maximum suppression (NMS) is used to obtain the small target detection result.

[0041] The specific example of small target detection in multimodal remote sensing images is as follows:

[0042] Figure 2 This is an example of a multimodal remote sensing image with small targets. As can be seen from the figure, the characteristics of small targets in multimodal remote sensing images include occupying few pixels, having small differences in spectral characteristics and background, possibly having temporal dynamic characteristics, unclear texture and shape, being easily masked by complex backgrounds, and being susceptible to noise and interference. By combining multiple sensor data, effective background modeling, and advanced image processing and data fusion methods, the detection and recognition capabilities of these small targets can be enhanced.

[0043] Application step 1: First input the multimodal remote sensing image with small targets into the multimodal fusion module (the multimodal images input into this module are respectively processed along two identical branches, and only one of the branches is taken as an example here). After the input image is subjected to the SE attention mechanism and two 1×1 convolutional layers to extract features, it is multiplied pixel by pixel with the feature map before convolution, and then added pixel by pixel with the input image to obtain a weighted image, and then the weighted image is subjected to a 3×3 convolution operation to obtain the final feature map. The final feature map is added to the final feature map obtained from the other branch on the channel, and the feature map is formed through the SE attention module to form the final input network feature map;

[0044] Application step 2: Input the feature map obtained in step 1 into the convolutional neural network, and use the backbone feature extraction network to continuously downsample the remote sensing image four times, with the downsampling multiple being twice, thereby extracting the shallow to deep features of the image, and the corresponding level number is p i (i=4,6,9), forming multi-level image features. At the same time, a weighted convolution attention module is added to the backbone feature extraction network. This module extracts information from the compressed feature map of each channel, so that the feature layer with stronger semantic information has a higher weight;

[0045] Application step 3: Input the feature map obtained in step 2 into the BSS-FPN feature pyramid for feature fusion. 9 Double upsampling is performed, and cross-scale fusion is performed on adjacent level features and cross-scale feature layers to achieve feature fusion. First, BSS-FPN obtains input feature maps from features at different levels, including p 4 、p 6 、p 9 . p 9 The feature map is doubled in size through a nearest neighbor upsampling operation. Next, p 6 Feature map and upsampled p 9 The feature map is at node p 12 Then, p 12 Feature convolution generates a new feature map p 14 In the second-level feature fusion process, p 4 Feature map and upsampled p 6 The feature map is at node p 17 The upsampled feature map p 14 Fusion is performed to generate a new feature map p 17 . p 17 The features are convolved to generate a new feature map p 19 . Feature map p 19 At node p 20 In the feature map p 14 And the feature map p 6 Perform cross-level fusion. Feature map p 20 After convolution at node p 23 In the feature map p 10 Perform cross-level fusion. The fused features are passed to p 18 、p 21 and p 24 These features are convolutionally processed and used in three detection layers to output the location and category information of the target.

[0046] The feature fusion is expressed by the following formula:

[0047]

[0048] p 23 =p 10 +p 20 ↓ 2×

[0049] Application step 4: After convolution processing, these features are used in three detection layers to output the location and category information of the target. The features of regular targets and small targets are obtained. The small target branch feature map with prominent features is used for small target prediction, and finally the non-maximum suppression (NMS) is used to obtain the small target detection result.

[0050] Examples of specific test results include Figure 6 The specific implementation methods described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A small target detection method based on multimodal remote sensing images, characterized in that: The method uses a pixel-level multimodal fusion module to fuse target features in multimodal remote sensing images, extracts target features layer by layer through a convolutional neural network, fuses multi-scale target features across layers, and uses a weighted convolutional attention mechanism to enhance subtle features and improve overall recognition capabilities, forming an end-to-end target detection network to achieve accurate detection of small targets in tasks based on multimodal remote sensing images.

2. The small target detection method based on multimodal remote sensing images according to claim 1, characterized in that: The specific process of the detection method is: Step 1: Input the multimodal remote sensing image into the pixel-level multimodal fusion module, perform pixel-level fusion on the multimodal image and obtain the fused multimodal feature map; Step 2: Input the multimodal feature map obtained in step 1 into the backbone feature extraction network of the convolutional neural network, and the backbone feature extraction network samples the multimodal feature map to form multi-level features of the image; a weighted convolutional attention module is included in the backbone feature extraction network, and the weighted convolutional attention module extracts information by compressing each channel of the multimodal feature map to enhance the weight of the feature layer with strong semantic information; Step 3: Input the feature map obtained in step 2 into the BSS-FPN feature pyramid for feature fusion, perform double upsampling and downsampling operations on some levels of the multi-level features, fuse the upper and lower level features of adjacent feature extraction layers and cross-scale feature layers, and extract fused features; Step 4: Use the feature maps of the three branches to predict the target; finally, use the Non-Maximum Suppression (NMS) algorithm to obtain the target detection result.

3. The small target detection method based on multimodal remote sensing images according to claim 2 is characterized in that: In step 1, the pixel-level multimodal fusion module is specifically: First, the multimodal remote sensing image is input into the pixel-level multimodal fusion module. The image after feature extraction by the SE attention mechanism layer and the convolution layer is multiplied pixel by pixel with the feature map before the input convolution layer, and then added pixel by pixel with the input multimodal remote sensing image to obtain a weighted image. The weighted image is then subjected to a second convolution operation to obtain the first final feature map. The first final feature map is added to the final feature map obtained by another branch in the multimodal fusion module, and then passed through the SE attention mechanism layer to form a second final feature map.

4. The small target detection method based on multimodal remote sensing images according to claim 2, characterized in that: In step 2, the weighted convolution attention module is specifically: Input the second final feature map, first generate the average pooling feature map F through global average pooling and global maximum pooling gap And the maximum pooling feature map F gmp ; The average pooling feature map F is pooled by weight factors α and (1-α) gap And the maximum pooling feature map F gmp Combining to generate weighted feature maps; Then, the weighted feature map is passed through a shared multilayer perceptron to generate channel weights F c , the channel weight F c Acting on the second final feature map to generate a weighted channel attention map; Then, the weighted channel attention is again subjected to global maximum pooling and global average pooling, and concatenated along the channel dimension, and the spatial weight F is generated through the convolution layer. sw ; Finally, the spatial weight F sw Acting on the weighted channel attention map, we get the weighted convolutional attention module map M WCAM .

5. The small target detection method based on multimodal remote sensing images according to claim 3 is characterized in that: In step 3, the BSS-FPN feature pyramid structure is specifically as follows: The upper and lower level features of adjacent feature extraction layers and cross-scale feature layers are fused to extract the fused features as follows: The weighted convolutional attention module graph M obtained in step 2 WCAM Input the BSS-FPN feature pyramid for feature fusion, and double upsample the p9 output by the backbone network; Features at adjacent levels and features at different scales are fused across scales in the feature pyramid.

6. The small target detection method based on multimodal remote sensing images according to claim 5, characterized in that: First, the BSS-FPN feature pyramid includes feature p4 feature map, p6 feature map, and p9 feature map; The p9 feature map is upsampled by the nearest neighbor interpolation method to double its size. The p6 feature map and the upsampled p9 feature map are placed at node p 12 Feature fusion is performed at p 12 Feature convolution generates a new feature map p 14 ; The p4 feature map is fused with the upsampled p6 feature map at the node p 17 At the same time as the upsampled p 14 The feature maps are fused to generate a new feature map p 17 , p 17 The features are convolved to generate a new p 19 Feature map; p 19 The feature map is at node p 20 In p 14 The feature map is fused with the p6 feature map across levels; 20 Feature convolution at node p 23 Place and p 10 The feature maps are fused across levels; the fused features are passed to p 18 Feature map, p 21 Feature map and feature map p 24 ; The p 18 Feature map, p 21 Feature map and feature map p 24 After convolution processing, they are used in three detection layers to output the location and category information of the target; The feature fusion is expressed by the following formula: p 23 =p 10 +p 20 ↓ 2× 。

Images