Real-time multi-source remote sensing image small target detection method adopting super-resolution assisted reasoning

Abstract

The invention relates to the field of remote sensing image intelligent interpretation, in particular to a super-resolution technology and a multi-source remote sensing image small target detection method, which not only can effectively improve the precision of multi-source remote sensing image small target detection, but also can reduce the complexity of a model, so that the model is lighter. The method comprises the following steps: constructing a lightweight pixel-level multi-source image fusion method in combination with a channel attention mechanism, aiming at paying attention to important features in different modes and reducing fusion of redundant features; according to the method, a super-resolution auxiliary reasoning branch with edge feature enhancement is provided, a target texture and a high-resolution feature map with richer details are reconstructed, and the extraction and representation capability of a backbone network on high-resolution features is enhanced; in the detection process, medium-size and large-size anchor frames are removed, and a special anchor frame for small target detection is newly added, so that the detection precision is improved, and the reasoning time of the model is shortened.

Description

Technical Field

[0001] The present invention relates to the field of intelligent interpretation of remote sensing images, and in particular to super-resolution technology and a method for detecting small targets in multi-source remote sensing images. Background Art

[0002] Remote sensing images contain a wealth of ground object information. By extracting different target ground object features in the image, various targets can be accurately identified and located, which plays a vital role in the intelligent processing and analysis of remote sensing images. At present, remote sensing technology is widely used in urban planning and construction, resource and environmental testing, smart agriculture, military reconnaissance and other fields, and has important civil and military application value. With the continuous development of sensor technology, more and more remote sensing satellites are continuously launched, bringing a large amount of remote sensing image data. The rich image resources have brought new opportunities and challenges to the intelligent interpretation of remote sensing images. As one of the important research contents in the field of remote sensing, remote sensing image target detection accurately identifies and detects targets by extracting image features. Therefore, studying remote sensing image target detection methods is of great significance to broadening the application scope of remote sensing images.

[0003] The essential reasons why small targets in remote sensing images are difficult to detect can be divided into two points: First, from the imaging principle, it can be seen that the ground sample distance (GSD) between different images is different, which leads to large differences in target size within the same category. Under the same GSD conditions, small-sized targets occupy a smaller proportion of image pixels for different categories of targets, and lack the texture and detail features required for target detection. They are easily confused by the complex background of remote sensing scenes. Therefore, compared with medium and large-sized targets, they are difficult to detect; secondly, in the current target detection framework based on deep learning technology, the model extracts features through multiple layers of convolution, and the features of small targets appear more in shallow feature maps. The features of small targets are easily partially lost or completely disappear in the deep feature maps obtained by multiple convolutions. Therefore, how to solve the problem of small target feature loss during convolution is another difficulty.

[0004] At the same time, the reconstruction network in the field of super resolution (SR) is directly applied to the field of remote sensing. Due to the complex image background, the feature reconstruction is not obvious. At the same time, SR reconstruction focuses more on global feature reconstruction and cannot effectively enhance the features of small targets in the image. In addition, the method of fusing different image features to improve detection accuracy has a complex fusion method, high model calculation overhead and poor detection results. Therefore, the reasonable use of SR technology can effectively improve the accuracy of small target detection in remote sensing images.

[0005] Therefore, the present invention discloses a real-time multi-source remote sensing image small target detection method using super-resolution assisted reasoning. The main contents are as follows: First, a pixel-level fusion method combined with a channel attention mechanism is proposed. By capturing global channel features, more features in visible light and infrared images that are beneficial to the detection task are focused on, and the fusion of redundant features is reduced. Secondly, a super-resolution assisted reasoning branch combined with boundary enhancement is constructed to enhance the backbone network's ability to extract and represent high-resolution features of small targets in multi-source remote sensing images. Finally, in order to reduce the amount of calculation and accelerate the reasoning time of the model, by removing the medium and large-sized anchor frames in the YOLO architecture, anchor frames for small targets are added, and the number of feature layer fusion layers is reduced. Under the condition of reducing the number of parameters, the IR and RGB image features are fully fused to improve the accuracy of small target detection. Summary of the invention

[0006] In view of this, the present invention proposes a real-time multi-source remote sensing image small target detection method using super-resolution assisted reasoning, which includes three aspects: multi-source feature fusion, feature extraction and super-resolution reconstruction feature map, and target detection.

[0007] In the first aspect, the present invention proposes a multi-source feature fusion method, the steps are as follows:

[0008] S1: Use 1×1 convolution to extract features from the input image and generate a feature matrix of fixed size.

[0009] S2: Use the channel attention mechanism to capture global channel information, pay more attention to the features of different modal channel features that have gains for the target area, and reduce the fusion of redundant features.

[0010] S3: Integrate the extracted features of different modalities channel by channel, and use the attention mechanism and splicing method to fuse the features.

[0011] The second aspect is feature extraction and super-resolution reconstruction of feature maps. The steps are as follows:

[0012] S4: YOLOv8s method is used to extract model features, and features are gradually extracted through stacked Fusion, C2f, and SPPF modules.

[0013] S5: The backbone network uses the FPN+PANET method to extract the features to achieve multi-scale feature fusion, so that the model can focus on more targets with large scale differences.

[0014] S6: The features extracted by the backbone network are sent to the constructed ES-WDSR branch for feature reconstruction. This branch combines the Sobel edge feature extraction operator to reconstruct the edge features of the target, reconstruct the high-resolution feature map, and merge it with the backbone network features again to improve the extraction and representation capabilities of the backbone network and high-resolution features.

[0015] The third aspect is target detection. The steps are as follows:

[0016] S7: Preliminary detection of the target, and calculation of the loss value using the loss function.

[0017] S8: Back propagation, gradient update, further training until the loss function converges.

[0018] S9: Get the optimal weight, classify and locate the target.

[0019] S10: End.

[0020] This method is advanced and scientific. By combining super-resolution technology, it can effectively reconstruct more small target features and improve the accuracy of small target detection in multi-source remote sensing images. The results show that this method has a high detection accuracy and reduces the computational overhead of the model, and has good application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only schematic diagrams of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0022] Figure 1 For flow chart

[0023] Figure 2 The network framework diagram for multi-source remote sensing image target detection

[0024] Figure 3 Fusion module diagram

[0025] Figure 4 ES-WDSR decoder diagram

[0026] Figure 5 The vehicle target detection results of multi-source remote sensing images

[0027] Figure 6 Pedestrian target detection results DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] The following is the multi-source feature fusion part:

[0030] Step 1: Use 1×1 convolution to extract features from the input image to generate a fixed-size feature matrix. Specifically, the input is IR and RGB remote sensing images, and 1×1 convolution is first used to extract image features. The formula is:

[0031] F RGB =Conv(I RGB ), F IR =Conv(I IR ) (1)

[0032] In the formula, I RGB and I IR The input visible light image RGB and near infrared image IR are respectively, Conv is a conventional 1×1 convolution, F RGB and F IR These are the extracted visible light and near-infrared features, respectively.

[0033] Step 2: Use the channel attention mechanism to capture global channel information, pay more attention to the features of different modal channel features that have gains for the target area, and reduce the fusion of redundant features.

[0034] A RGB =CA(F RGB ), A IR =CA(F IR ) (2)

[0035] In the formula, CA is the channel attention mechanism, A RGB and A IR It is the feature after the input feature is processed by channel attention.

[0036] Step 3: Integrate the extracted features of different modalities channel by channel, and use the attention mechanism and splicing method to fuse the features. The calculation method is shown in formula (3) (4):

[0037] F fus1 =F RGB +A RGB , F fus2 =F IR +A IR (3)

[0038] F ful =CA(Concat(F fus1 , F fus2 )) (4)

[0039] In the formula, Concat is the concatenation method, F ful It is the fusion feature obtained after the splicing features are processed by the ECA method.

[0040] The following is the feature extraction and super-resolution reconstruction feature map part:

[0041] Step 4: Use the YOLOv8s method to extract model features, and gradually extract feature F through stacked Fusion, C2f, and SPPF modules.

[0042] Step 5: The backbone network uses the FPN+PANET method to extract the features to achieve multi-scale feature fusion, so that the model can focus on more targets with large scale differences.

[0043] Step 6: The feature F extracted by the backbone network is sent to the constructed ES-WDSR branch to reconstruct the feature. This branch combines the Sobel edge feature extraction operator to reconstruct the edge features of the target, reconstruct the high-resolution feature map, and merge it with the backbone network features again to improve the backbone network's ability to extract and represent high-resolution features.

[0044] The soft threshold calculation method is shown in formula (5):

[0045]

[0046] In the formula, F Soft is the output feature, F is the input feature, τ is the threshold, the features in the interval [-τ,τ] are set to 0, the features greater than τ are subtracted from τ, and the features less than -τ are added to τ, that is, the features with absolute values ​​below a certain threshold are set to zero, and the other features are also adjusted towards zero, thus achieving a shrinkage effect, thereby removing redundant features. Specifically, after convolution of the super-resolution branch residual module, the features are input to the soft threshold module, and the soft thresholds of each channel are obtained using formulas (6)(7)(8). First, in the sub-network, the absolute value of the input feature is taken, and after global average pooling, the feature map is adjusted to Secondly, the processed features are passed through the full connection and RBL layers to obtain Z, and the result is activated by Sigmoid nonlinearity to obtain α; finally, the result of global average pooling is averaged and multiplied by α to obtain the soft threshold τ, and formula (1) is used to perform thresholding processing with the original input.

[0047] Z=FC(RBL(GAP(|F|)) (6)

[0048]

[0049] τ=α·average(GAP(F)) (8)

[0050] Where GAP is the global average pooling method, |F| is the absolute value of the input feature F, RBL is a set of three operations, namely LeakyReLU activation function, BathNormLize batch normalization and Linear linear layer, and average is the averaging method. The improved super-resolution branch ensures that the thresholds of all channels are positive and avoids the situation where all outputs are zero. In addition, the soft threshold module of the neck network stack can adaptively generate an independent set of thresholds for each group of samples, reducing the impact of inconsistent noise caused by different backgrounds in the samples, thereby effectively reducing the weight of each channel.

[0051] The edge feature extraction method uses the classic first-order edge detection operator Sobel operator, which uses the gradient information of the image to detect the edge in the image. In the Sobel operator, there are two convolution kernels, which are used to calculate the gradient of the image in the horizontal and vertical directions respectively. The forms of these two convolution kernels are as follows:

[0052] Horizontal gradient direction

[0053] Vertical gradient direction Where M x is the convolution kernel in the horizontal gradient direction, M y is the vertical convolution kernel.

[0054] First, the Sobel horizontal convolution kernel is applied to the processed grayscale image to obtain the horizontal gradient, and then the Sobel vertical convolution kernel is used to extract the vertical gradient information. Finally, the edge strength and direction of each pixel in the image are calculated by combining the gradient values ​​on the horizontal gradient and the vertical gradient, so as to extract the edge features, and fuse the edge features with the features processed by the soft threshold network to enhance the edge features of the target in the reconstruction network, as shown in formulas (11)(12)(13)(14):

[0055] G Sobel_x =F input ×M x (11)

[0056] G Sobel_y =F input ×M y (12)

[0057]

[0058] F Out =F Soft +F Sobel (14)

[0059] In the formula, G Sobel_x , G Sobel_yare the gradient values ​​on the horizontal gradient and vertical gradient respectively, F input is the feature extracted by the encoder, F Sobel is the extracted gradient feature, F Soft is the feature obtained by the soft threshold method, F Out F Soft Output features after fusing edge features.

[0060] The following is the target detection part:

[0061] Step 7: Preliminary detection of the target, using the loss function to calculate the loss value.

[0062] Step 8: Back propagation, gradient update, further training until the loss converges.

[0063] In the present invention, the overall network loss L is detected. all It consists of two parts: target detection loss L o and the super-resolution branch loss L sr ,Right now:

[0064] L all =a1L o +a2L sr (15)

[0065] Where a1 and a2 are the modulation coefficients of the two tasks, which are used to balance the two training tasks.

[0066] L sr =||SR-X||1 (16)

[0067] In the formula, SR is the reconstructed feature image and X is the input image.

[0068] Object detection loss function L o It consists of three parts: positioning loss L loc , confidence loss L obj and classification loss L cls ,Right now:

[0069]

[0070] Where l is the two scales of the detection head, a l , b l 、c l are the weights of the detection head, and λ loc , obj , cls They are the modulation coefficients of the three parts, which are used to adjust the weights of positioning loss, confidence loss and classification loss for the target detection task.

[0071] Step 9: Get the optimal weight, classify and locate the target.

[0072] Step 10: End.

[0073] The above description of the disclosed embodiments enables professionals and technicians in the field to implement or use the present invention. Various modifications to these embodiments will be obvious to professionals and technicians in the field, and the general principles defined in this solution can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown in this solution, but will conform to the widest range consistent with the principles and novel features disclosed in the present invention. It should be noted that for ordinary technicians in this technical field, without departing from the technical principles of the present invention, several improvements and modifications can also be made, and these improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A real-time multi-source remote sensing image small target detection method using super-resolution assisted reasoning, including three parts: multi-source feature fusion, feature extraction and super-resolution reconstruction feature map, and target detection: The steps of multi-source feature fusion are as follows: S1: Use 1×1 convolution to extract features from the input image and generate a feature matrix of fixed size. S2: Use the channel attention mechanism to capture global channel information, pay more attention to the features of different modal channel features that have gains for the target area, and reduce the fusion of redundant features. S3: Integrate the extracted features of different modalities channel by channel, and use the attention mechanism and splicing method to fuse the features. The steps of feature extraction and super-resolution reconstruction feature map are as follows: S4: The YOLOv8s algorithm is used to extract model features, and features are gradually extracted through stacked Fusion, C2f, and SPPF modules. S5: The backbone network uses the FPN+PANET method to extract the features to achieve multi-scale feature fusion, so that the model can focus on more targets with large scale differences. S6: The features extracted by the backbone network are sent to the constructed ES-WDSR branch for feature reconstruction. This branch combines the Sobel edge feature extraction operator to reconstruct the edge features of the target, reconstruct the high-resolution feature map, and merge it with the backbone network features again to improve the extraction and representation capabilities of the backbone network and high-resolution features. The target detection steps are as follows: S7: Preliminary detection of the target, using the loss function to calculate the loss value. S8: Back propagation, gradient update, further training until the loss converges. S9: Get the optimal weight, classify and locate the target. S10: End.

2. The method for detecting small targets in multi-source remote sensing images using super-resolution-assisted reasoning in real time according to claim 1, characterized in that: In step S7, the target detection model weights generated in steps S1 to S6 are applied.

3. A real-time multi-source remote sensing image small target detection method using super-resolution assisted reasoning according to claim 1 and claim 2, characterized in that: In step S9, the final target classification and positioning are completed.

Images