Remote sensing small target detection method based on multi-scale feature fusion and context information enhancement

Abstract

The application discloses a kind of multiscale feature fusion and context information enhancement's remote sensing small target detection method, including feature fusion module and global context perception module, feature fusion module is fused to multiscale feature, wherein the channel weighting module can optimize the fusion process, provide more effective feature map for subsequent global context information extraction;Global context perception module can carry out global context information extraction, suppress background information in feature map, enhance the characteristics of small target.The method extracts remote sensing small target feature by neural network, channel weighting fusion target feature, global context modeling enhances target feature and predicts target, forms end-to-end target detection network, completes the accurate detection of small target in remote sensing image.Experiments show that the application realizes the improvement of remote sensing small target detection precision by channel weighting feature fusion, global context information enhancement.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and machine learning technology, specifically to a remote sensing small target detection method that combines multi-scale feature fusion and contextual information enhancement. Background Technology

[0002] In recent years, optical remote sensing technology has developed rapidly and has been widely used in traffic monitoring, search and rescue, security, and military fields. Significant progress has also been made in research on methods for detecting small targets. Small targets typically refer to targets with a pixel area of ​​less than 32×32 pixels. Small target detection requires the localization and classification of small targets.

[0003] With the rapid development of deep learning, convolutional neural networks have achieved remarkable results in the field of object detection. Common object detection algorithms are divided into two-stage object detection algorithms and single-stage object detection algorithms. Typical representatives of two-stage object detection algorithms are the R-CNN series algorithms, while typical representatives of single-stage object detection algorithms are the YOLO series algorithms and the SSD series algorithms.

[0004] Remote sensing systems suffer from uncontrollable observation conditions due to the movement of the observation platform, are susceptible to atmospheric interference, and are subject to complex observation scenarios. These factors make it difficult to distinguish between targets and backgrounds. Furthermore, remote sensing images typically have a large field of view, and targets of interest are often small and feature-poor in these images, making them difficult to detect. Therefore, a remote sensing method for detecting small targets that can suppress complex background information and enhance the features of small targets has become an urgent need. Summary of the Invention

[0005] To address the problem of low accuracy in remote sensing small target detection caused by the above-mentioned situation, this invention aims to provide a new remote sensing small target detection method based on multi-scale feature fusion and context information enhancement. This method can suppress background information and enhance small target features through a feature fusion module and a global context awareness module, thereby improving the accuracy of small target detection.

[0006] To address the aforementioned technical problems, this invention provides a remote sensing small target detection method based on multi-scale feature fusion and contextual information enhancement. The method comprises the following steps:

[0007] Step 1) Extract features from the input remote sensing image through the backbone network, and enhance the features of the feature maps at different scales to initially form a multi-scale feature map;

[0008] Step 2) Perform upsampling and downsampling operations on some levels of the multi-scale feature map, and perform channel-weighted fusion on adjacent level feature maps to obtain the multi-scale feature map after feature fusion;

[0009] Step 3) Extract global context information from the multi-scale feature map after feature fusion to further form a feature map for prediction;

[0010] Step 4) Input the predicted feature map into the YOLO Head prediction network to complete the prediction of the location and category of the small target. After non-maximum suppression processing, the final small target detection result is obtained.

[0011] Furthermore, step 2) specifically includes the following steps:

[0012] Step 2.1: Dimensionally reduce the high-level feature map X4 to obtain feature map X4', then upsample feature map X4' to obtain feature map of the same scale as X3, and then weightedly fuse this feature map with the X3 channel to obtain feature map X3';

[0013] Upsample the low-level feature map X3' to obtain a feature map of the same scale as X2, and then weight-fuse this feature map with the X2 channel to obtain the feature map X2';

[0014] Step 2.2: Downsample the feature map X2' through a convolution operation to obtain a feature map of the same scale as X3'. Then, weightedly fuse this feature map with the channels of X3 and X3' to obtain X3”.

[0015] The feature map X3” is downsampled through a convolution operation to obtain a feature map of the same scale as X4'. This feature map is then weighted and fused with the X4' channel to obtain X4".

[0016] Furthermore, in step 3), the extraction of global contextual information from the multi-scale feature map after feature fusion includes the following steps:

[0017] Step 3.1: Input the feature maps X2', X3”, and X4” after multi-scale feature fusion into the global context awareness module. The global context awareness module contains three branches. The first branch uses global average pooling and global max pooling to extract global information. The second branch is the Value branch, which is used to generate the linear transformation result of the feature map value tensor. The third branch is the QK branch, which is used to generate the product of the feature map query tensor and the key tensor.

[0018] Step 3.2: Multiply the output matrices of the first branch and the second branch to obtain the global channel context information; multiply the output matrices of the third branch and the second branch to obtain the global spatial context information.

[0019] Step 3.3: Obtain the extended Hadamard product of global channel context information and global spatial context information, and use the product as the output of the global context-aware module.

[0020] Furthermore, the calculation expression for the multi-scale feature map after feature fusion is as follows:

[0021]

[0022] X3”=CSP{CRC[CB S(X3’),X3,CBS(X2’,stride=2)]}

[0023] X4”=CSP{CRC[X4’,CBS(X3”,stride=2)]}

[0024] In the above formula, The symbol represents upsampling operation, CBS represents 3×3 convolution, batch normalization and SiLU activation function, CRC represents channel weighted fusion operation, and CSP represents local bridging operation.

[0025] Furthermore, the expression for the output of the global context-aware module is:

[0026]

[0027]

[0028] In the above formula, P i j , Let N represent the input and output of the j-th pixel in the i-th feature map, respectively. i ω represents the total number of pixels in the i-th layer feature map. v ,ω qk represents the linear transformation matrix of the Value branch and the QK branch, respectively. avg(·) and max(·) represent the global average pooling (GAP) and global maximum pooling (GMP), respectively. GAP and GMP guide the selection of channels with important information in the feature map.

[0029] Furthermore, in the channel weighting module, all channels of the feature maps are first concatenated, and then multiplied by normalized trainable weights. The number of weights is the same as the total number of channels, as shown in the following formula:

[0030]

[0031] In the above formula, ω j Let ∑ represent the training weights of the j-th channel, m represent the total number of channels after concatenation of the feature maps, and ∑ represent the training weights of the j-th channel. m ω m This represents the sum of all training weights, and ε represents a constant. To avoid the denominator being 0, ε is set to 0.0001.

[0032] The beneficial effects of this invention are: it can suppress background information and enhance small target features through the feature fusion module and the global context awareness module, thereby improving the accuracy of remote sensing small target detection. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0034] Figure 2 Here is a structural diagram of the feature fusion module;

[0035] Figure 3 This is a structural diagram of the global context-aware module. Detailed Implementation

[0036] Now combined Figures 1 to 3 The technical solution of this invention is described in full. The following description is merely a part of the embodiments of this invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0037] The remote sensing small target detection method provided by this invention, which combines multi-scale feature fusion and contextual information enhancement, includes the following steps:

[0038] (1) The input remote sensing images are processed by the backbone network to extract and enhance features, and feature maps with scales of 160×160, 80×80 and 40×40 are initially formed.

[0039] (2) Multi-scale features are fused through the feature fusion module to fully extract information of small targets, and the feature maps of adjacent levels are fused using the channel weighting module to optimize the fusion process of multi-scale features.

[0040] (3) Through the global context awareness module, global context information is extracted from the multi-scale feature map after feature fusion, making full use of global context information, suppressing background information, enhancing small target features, and further forming a feature map for prediction.

[0041] (4) The YOLO Head prediction network is used to predict the location and category of small targets. After non-maximum suppression, the final small target detection result is obtained.

[0042] Preferably, in step (2), the multi-scale feature fusion steps are as follows:

[0043] (1) Reduce the dimensionality of the high-level feature map X4 to obtain feature map X4', then upsample the feature map X4' to obtain a feature map of the same scale as X3, and then use the channel weighting module to obtain feature map X3'.

[0044] (2) Upsample the low-level feature map X3' to obtain a feature map of the same scale as X2, and then use the feature map and X2 through the channel weighting module to obtain feature map X2';

[0045] (3) Downsample the feature map X2' through a convolution operation to obtain a feature map of the same scale as X3'. Then, use the feature map, X3, and X3' through a channel weighting module to obtain X3”.

[0046] (4) Downsample the feature map X3” through a convolution operation to obtain a feature map of the same scale as X4'. Then, use the channel weighting module to obtain X4” from the feature map X3”.

[0047] Preferably, in step (3), the global context information extraction steps are as follows:

[0048] (1) Input the feature maps X2', X3”, and X4” after multi-scale feature fusion into the global context awareness module. This module contains three branches. The first branch uses global average pooling (GAP) and global maximum pooling (GMP) to extract global information. The second branch generates the linear transformation result Value of the feature map value tensor. The third branch generates the product QK of the feature map query tensor and the key tensor.

[0049] (2) Multiply the first branch by the second branch to obtain the global channel context information, multiply the third branch by the second branch to obtain the global spatial context information, and the three branches merge into two branches;

[0050] (3) Obtain the extended Hadamard product of global channel context information and global spatial context information, and use the product as the output of the global context-aware module.

[0051] Example 1

[0052] 1. Feature Fusion Module

[0053] High-level feature maps and low-level feature maps contain different information. Low-level feature maps contain more information such as the shape and location of small targets, while high-level feature maps contain more semantic information. Multi-scale feature fusion can enhance the features of small targets.

[0054] The feature fusion module improves upon the BiFPN neck network by constructing a channel-weighted module and modifying the original BiFPN neck network structure to output feature maps at three scales for detection. The structure of the feature fusion module is as follows: Figure 2 As shown. The input to the feature fusion module includes low-level feature maps X2 (160×160) and X3 (80×80), and high-level feature map X4 (40×40).

[0055] The feature fusion process from deep to shallow layers is as follows: First, the enhanced feature map X4 is dimensionality-reduced to obtain feature map X4'. Then, X4' is upsampled to obtain a feature map with the same scale as feature map X3. The two feature maps are fused using a channel weighting module, and the fused feature map is processed by the CSP module to obtain X3'. The above operation is repeated on feature map X3' to obtain feature map X2'. Feature maps X2', X3', and X4' realize the flow of semantic information of small targets from deep to shallow layers. The fusion of features from shallow to deep layers is similar to the fusion from deep to shallow layers. Feature map X2' is downsampled through convolution to obtain a feature map of the same scale as X3'. The two feature maps and X3' are then fused through a channel weighting module, and then processed by a CSP module to obtain X3'". This process is repeated to fuse X3' with the higher-level feature map X4' to obtain X4'. Feature maps X3' and X4' enable the flow of semantic information of small targets from shallow to deep layers. In the process of obtaining feature map X3', a skip link is added, which fuses feature map X3' with other feature maps. This allows for the fusion of richer features without increasing computational cost. The output of the feature fusion module is feature maps X2', X3', and X4'. The calculation process of the feature fusion module can be expressed as the following formula.

[0056]

[0057] X3”=CSP{CRC[CB S(X3’),X3,CBS(X2’,stride=2)]}

[0058] X4”=CSP{CRC[X4’,CBS(X3”,stride=2)]}

[0059] In the above formula, The symbol represents upsampling operation, CBS represents 3×3 convolution, batch normalization and SiLU activation function, CRC represents channel weighted fusion operation, and CSP represents local bridging operation.

[0060] To enhance small target features in multi-scale feature maps and fully utilize features from different channels during feature map fusion, a channel weighting module was constructed within the feature fusion module. This module reweights the channels of the feature maps, as illustrated in the diagram below. Figure 2 The lower half is shown. The strategy of the channel-weighted module is to first concatenate all channels of the feature maps, and then multiply them by normalized trainable weights. The number of weights is the same as the total number of channels, as shown in the following formula.

[0061]

[0062] In the above formula, ω jLet ∑ represent the training weights of the j-th channel, m represent the total number of channels after concatenation of the feature maps, and ∑ represent the training weights of the j-th channel. m ω m The sum of all training weights is represented by ε, which is a constant set to 0.0001 to avoid a denominator of 0. The structure of the feature fusion module and its channel weighting module optimize the multi-scale feature fusion process for small targets, providing more efficient feature maps for subsequent global context information extraction.

[0063] 2. Global Context Awareness Module

[0064] After the feature map has been processed by the feature fusion module, local information about the small target has been obtained. Therefore, it is now more necessary to obtain global context information. Obtaining global context information can suppress useless background information and improve the model's ability to distinguish between the target and the background.

[0065] Inspired by GCNet and SCP, the global context-aware module consists of three branches, as shown in the diagram below. Figure 3 As shown. The first branch uses global average pooling (GAP) and global max pooling (GMP) to obtain global information. The second branch uses convolution to obtain the linear transformation result Value of the feature map value tensor. The third branch uses convolution to obtain the product QK of the feature map query tensor and key tensor. Then, matrix multiplication is performed on the outputs of the first and second branches to obtain channel global context information, and matrix multiplication is performed on the outputs of the third and second branches to obtain spatial global context information. Finally, the feature map output by the global context-aware module is obtained by performing extended Hadamard multiplication on these two branches. The process of obtaining global context information can be represented by the following formula.

[0066]

[0067]

[0068] In the above formula, P i j , Let N represent the input and output of the j-th pixel in the i-th feature map, respectively. i ω represents the total number of pixels in the i-th layer feature map. v ,ω qk represents the linear transformation matrix of the Value branch and the QK branch, respectively. avg(·) and max(·) represent GAP and GMP, respectively. GAP and GMP can guide the feature map to select channels with important information, which enables the global context-aware module to obtain contextual information about the channel dimension.

[0069] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A remote sensing small target detection method based on multi-scale feature fusion and contextual information enhancement, characterized in that, The method is based on a neural network and includes the following steps: Step 1) Extract features from the input remote sensing image through the backbone network, and enhance the features of the feature maps at different scales to initially form a multi-scale feature map; Step 2) Perform upsampling and downsampling operations on some levels of the multi-scale feature map, and perform channel-weighted fusion on adjacent level feature maps to obtain the multi-scale feature map after feature fusion; Step 3) Extract global context information from the multi-scale feature map after feature fusion to further form a feature map for prediction; Step 4) Input the predicted feature map into the YOLO Head prediction network to complete the prediction of the location and category of the small target. After non-maximum suppression processing, the final small target detection result is obtained.

2. The remote sensing small target detection method with multi-scale feature fusion and contextual information enhancement as described in claim 1, characterized in that, Step 2) specifically includes the following steps: Step 2.1: Dimensionally reduce the high-level feature map X4 to obtain feature map X4', then upsample feature map X4' to obtain feature map of the same scale as X3, and then weightedly fuse this feature map with the X3 channel to obtain feature map X3'; Upsample the low-level feature map X3' to obtain a feature map of the same scale as X2, and then weight-fuse this feature map with the X2 channel to obtain the feature map X2'; Step 2.2: Downsample the feature map X2' through a convolution operation to obtain a feature map of the same scale as X3'. Then, weightedly fuse this feature map with the channels of X3 and X3' to obtain X3”. The feature map X3” is downsampled through a convolution operation to obtain a feature map of the same scale as X4'. This feature map is then weighted and fused with the X4' channel to obtain X4".

3. The remote sensing small target detection method with multi-scale feature fusion and contextual information enhancement as described in claim 1, characterized in that, Step 3) involves extracting global contextual information from the multi-scale feature map after feature fusion, which includes the following steps: Step 3.1: Input the feature maps X2', X3”, and X4” after multi-scale feature fusion into the global context awareness module. The global context awareness module contains three branches. The first branch uses global average pooling and global max pooling to extract global information. The second branch is the Value branch, which is used to generate the linear transformation result of the feature map value tensor. The third branch is the QK branch, which is used to generate the product of the feature map query tensor and the key tensor. Step 3.2: Multiply the output matrices of the first branch and the second branch to obtain the global channel context information; multiply the output matrices of the third branch and the second branch to obtain the global spatial context information. Step 3.3: Obtain the extended Hadamard product of global channel context information and global spatial context information, and use the product as the output of the global context-aware module.

4. The remote sensing small target detection method with multi-scale feature fusion and contextual information enhancement as described in claim 2, characterized in that, The calculation expression for the multi-scale feature map after feature fusion is as follows: X3”=CSP{CRC[CB S(X3’),X3,CBS(X2’,stride=2)]} X4”=CSP{CRC[X4’,CBS(X3”,stride=2)]} In the above formula, The symbol represents upsampling operation, CBS represents 3×3 convolution, batch normalization and SiLU activation function, CRC represents channel weighted fusion operation, and CSP represents local bridging operation.

5. The remote sensing small target detection method with multi-scale feature fusion and contextual information enhancement as described in claim 3, characterized in that, The expression for the output of the global context-aware module is: In the above formula, P i j , Let N represent the input and output of the j-th pixel in the i-th feature map, respectively. i ω represents the total number of pixels in the feature map of the i-th layer. v ,ω qk represents the linear transformation matrix of the Value branch and the QK branch, respectively. avg(·) and max(·) represent the global average pooling (GAP) and global maximum pooling (GMP), respectively. GAP and GMP guide the selection of channels with important information in the feature map.

6. The remote sensing small target detection method with multi-scale feature fusion and contextual information enhancement as described in claim 4, characterized in that, In the channel weighting module, all channels of the feature maps are first concatenated, and then multiplied by normalized trainable weights. The number of weights is the same as the total number of channels, as shown in the following formula: In the above formula, ω j Let ∑ represent the training weights of the j-th channel, m represent the total number of channels after concatenation of the feature maps, and ∑ represent the training weights of the j-th channel. m ω m This represents the sum of all training weights, and ε represents a constant. To avoid the denominator being 0, ε is set to 0.0001.

Images