Small-scale surface target detection method in remote sensing images based on adaptive multi-level fusion

Through the remote sensing image detection method with adaptive multi-level fusion, the problem of small-scale target feature extraction and false alarm is solved, and the detection performance of high detection rate and low false alarm rate is improved, which is suitable for weak target detection in complex scenarios.

CN115641507BActive Publication Date: 2024-11-05HARBIN INST OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202211387533.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-11-07
Publication Date
2024-11-05
Estimated Expiration
2042-11-07

Smart Images

  • Figure CN115641507B_ABST
    Figure CN115641507B_ABST
Patent Text Reader

Abstract

The invention discloses a remote sensing image small-scale surface target detection method based on adaptive multi-level fusion, the method comprising the following steps: step 1: extracting shallow and deep multi-level feature maps of an input image using a backbone feature extraction network, with the downsampling levels being 4, 8, 16, and 32 times respectively; step 2: using a multi-level feature extraction architecture with adaptive fusion weights to achieve fusion of features with different downsampling levels in step 1; step 3: selecting high-resolution feature layers with 4 and 8 times downsampling levels after fusion to predict target position and category information, and obtaining the final detection result. The method can achieve effective fusion of semantic and structural information in different levels, improve the network's feature extraction and detection and positioning capabilities for small-scale targets, effectively reduce the interference of false alarm sources in the scene on target detection, and thus achieve high detection rate and low false alarm rate detection of small-scale targets in remote sensing images.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection and recognition, and relates to a detection method for small-scale surface targets in remote sensing images, and specifically to a detection method for small-scale surface targets in remote sensing images based on adaptive multi-level feature fusion. Background Art

[0002] As a key technology in the field of remote sensing image interpretation, remote sensing image target detection can effectively classify and accurately locate the region of interest or target instance by extracting the differential features between the target and the background in aerospace remote sensing data, and plays a key role in military and civilian applications such as timely rescue at sea, intelligent traffic management, and real-time regional monitoring. The continuous improvement of remote sensing image resolution makes it possible to detect and identify smaller-scale man-made objects such as ships and vehicles.

[0003] Early detection methods required artificially designed image feature extraction operators for different types of targets. Common features for small-scale surface targets such as vehicles include geometric contour features, texture edge features, symmetry features, etc. However, such artificially designed features are easily affected by factors such as lighting, scale, and color, and have limited ability to characterize target characteristics, and are only applicable to target detection in specific scenarios. With the improvement of hardware computing power and the surge in data samples, target detection methods based on neural networks can learn more robust and strong characterization features of targets through adaptive mining of image features. However, due to the special bird's-eye imaging method of remote sensing, imaging resolution constraints, and imaging link degradation, small-scale targets such as ships and vehicles lack detailed information such as geometric texture and edge contours in images. At the same time, there are a large number of false alarm sources similar to target morphology, texture, and other characteristics in complex and diverse scenes. These factors increase the difficulty of extracting effective features of small-scale remote sensing targets. Therefore, it is urgent to closely combine the actual application needs and carry out research on fast and accurate small-scale target detection methods. Summary of the invention

[0004] In view of the difficulties such as small geometric size and weak texture characteristics of small-scale surface targets in remote sensing images and many false alarm sources of suspected targets in complex scenes, the present invention provides a remote sensing image small-scale surface target detection method based on adaptive multi-level fusion. The method can achieve effective fusion of semantic and structural information in different levels, improve the network's feature extraction and detection and positioning capabilities for small-scale targets, effectively reduce the interference of false alarm sources in the scene on target detection, thereby achieving high detection rate and low false alarm rate detection of small-scale targets in remote sensing images, and can effectively support the detection of weak small targets in complex scenes.

[0005] The objective of the present invention is achieved through the following technical solutions:

[0006] A remote sensing image small-scale surface target detection method based on adaptive multi-level fusion includes the following steps:

[0007] Step 1: Use the backbone feature extraction network to extract the shallow and deep multi-level feature maps of the input image, with the downsampling levels being 4, 8, 16, and 32 times respectively;

[0008] Step 2: Use a multi-level feature extraction architecture with adaptive fusion weights to fuse features of different downsampling levels in step 1;

[0009] Step 3: Select the fused high-resolution feature layer with downsampling levels of 4 and 8 times to predict the target position and category information to obtain the final detection result.

[0010] Compared with the prior art, the present invention has the following advantages:

[0011] (1) The present invention proposes a detection method for small-scale surface targets in remote sensing images, which can effectively solve the problems of small-scale targets having small geometric scales and lacking texture features, as well as the presence of a large number of false alarm sources with similar shape and texture characteristics to the target in complex scenes, which lead to weak target effective feature extraction capability and poor detection performance. The method can be applied to small-scale target detection in scenes with complex ground interference and densely arranged targets.

[0012] (2) The present invention proposes a multi-level feature extraction architecture with adaptive fusion weights, which can achieve effective fusion of semantic and structural information at different levels, improve the network's feature extraction and detection and positioning capabilities for small-scale targets, and effectively reduce the interference of false alarm sources in the scene on target detection.

[0013] (3) The present invention can effectively improve the detection performance in scenes with densely arranged targets by adopting the strategy of only using high-level feature layers for target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a flowchart of small-scale surface target detection in remote sensing images based on adaptive multi-level fusion;

[0015] Figure 2 For the true value and detection results of small-scale ship targets;

[0016] Figure 3 The true value and detection results for small-scale vehicle targets. DETAILED DESCRIPTION

[0017] The technical solution of the present invention is further described below in conjunction with the accompanying drawings, but is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention should be included in the protection scope of the present invention.

[0018] The present invention provides a remote sensing image small-scale surface target detection method based on adaptive multi-level fusion, such as Figure 1 As shown, the specific implementation steps of the method are as follows:

[0019] Step 1: Use the backbone feature extraction network to extract the shallow and deep multi-level feature maps of the input image, with the downsampling levels being 4, 8, 16, and 32 times respectively. The specific steps are as follows:

[0020] For the input remote sensing image, the backbone feature extraction network is used to extract the shallow and deep multi-level feature maps in the remote sensing image, and the feature maps with downsampling levels of 4, 8, 16, and 32 times are extracted respectively, which are recorded as P2, P3, P4, and P5 levels.

[0021] In this step, the backbone feature extraction networks that can be selected include but are not limited to typical networks such as CspDarknet53, ResNet, and DenseNet.

[0022] Step 2: Use a multi-level feature extraction architecture with adaptive fusion weights to fuse features of different downsampling levels in step 1. The specific steps are as follows:

[0023] The fusion passes high-level semantics and low-level positioning information through top-down and bottom-up paths respectively. In the top-down fusion path, for the feature map F of the nth level n , whose length, width, and number of channels are recorded as (H, W, C), and it is necessary to fuse the feature map F of the n+1th deeper level n+1 , its length, width, and number of channels are recorded as (H / 2, W / 2, C).

[0024] Step 2-1: Use the CARAFE upsampling operator to upsample the deep-level feature map to achieve aggregation of semantic information. The specific steps are as follows:

[0025] In order to make the shallow level feature map F n and deep level feature map F n+1 The length and width dimensions of F are kept consistent, and the CARAFE upsampling operator is used to n+1 Upsampling is performed to aggregate semantic information and obtain high-level feature maps Its length, width, and number of channels are recorded as (H, W, C). The upsampling methods that can be selected include but are not limited to interpolation, deconvolution, CARAFE operator, etc.

[0026] Step 2-2: Shallow feature map correction based on deep feature supervision. The specific steps are as follows:

[0027] Since the low-level feature extraction of convolutional neural networks is not sufficient, there will be a lot of noise, which affects the fusion of effective features. Therefore, high-level feature maps are used to extract the features of the convolutional neural network. As a supervision to the shallow level feature map F n Fine-tune, correct the noisy low-level features through high-level semantic features, and process them through Sigmoid nonlinear activation function Get the corrected weight map and apply it to the shallow feature map F n On the top, we get the low-level feature map Low-level feature maps It can be calculated by the following formula:

[0028]

[0029] Where σ(·) is the Sigmoid activation function.

[0030] Step 2-3: Based on the spatial and channel feature saliency combined with the broadcast mechanism, point-by-point fusion weights are adaptively generated to achieve the fusion of deep and shallow features.

[0031] After obtaining the upsampled high-level features and the corrected low-level features, we can start to fuse the two. However, the importance of feature maps of each channel in different levels for characterizing the characteristics of small-scale targets is not exactly the same. If the features of different levels are directly fused, the channels rich in scene semantic information may serve as interference to cover up the features of small-scale targets. Based on the idea of ​​feature saliency, by assigning different weights to features at each level, their importance in the training and reasoning process is adaptively adjusted. The specific steps are as follows:

[0032] Step 2-3-1: The high-level and low-level feature maps are concatenated along the channel dimension. The specific steps are as follows:

[0033] Concatenate low-level feature maps along the channel dimension and high-level feature maps Get the concatenated feature map This is used for subsequent channel saliency and spatial saliency generation.

[0034] Step 2-3-2: Generate channel saliency based on spatial feature map pooling and fully connected neural network. The specific steps are as follows:

[0035] First, we aggregate through spatial pooling The global information in the channel is then used to learn the dependencies between channels through a fully connected neural network, which is recorded as:

[0036]

[0037] In the formula, AvePool(·) is the average pooling, fFCN (·) is the fully connected layer, ω c is the generated channel saliency, with dimension (1,1,C).

[0038] Step 2-3-3: Generate spatial saliency based on 1×1 convolution and 3×3 convolution. The specific steps are as follows:

[0039] Adjusted by 1×1 convolution Channel, and then use 3×3 convolution to aggregate channel dimension information to generate spatial saliency, denoted as:

[0040]

[0041] In the formula, conv 3×3 and conv 1×1 Represent 3×3 convolution and 1×1 convolution respectively, ω s is the generated spatial saliency with dimension (H, W, 1).

[0042] Step 2-3-4: Generate point-by-point fusion weights based on the broadcast mechanism. The specific steps are as follows:

[0043] The dimensions of channel saliency and spatial saliency are different, but they can be added together through the broadcast mechanism to generate point-by-point saliency of dimension (H, W, C). The fusion weight ω can be obtained by using the Sigmoid nonlinear activation function, which is denoted as:

[0044] ω=σ(ω c +ω s );

[0045] Step 2-3-5: Based on the adaptively generated fusion weights, weighted fusion of deep-level features and shallow-level features is performed. The specific steps are as follows:

[0046] Based on the generated adaptive fusion weights, the low-level feature maps And upsample high-level feature maps Fusion is performed to obtain the feature fusion result of the nth level The fusion formula is as follows:

[0047]

[0048] Step 3: Select the fused high-resolution feature layer with downsampling levels of 4 and 8 times to predict the target position and category information to obtain the final detection result.

[0049] In order to improve the detection performance of dense scenes, only high-resolution prediction layers P2 and P3 with downsampling multiples of 4 and 8 are used to predict the target position and category information. The prediction methods include but are not limited to the target information regression method of YOLO.

[0050] Figure 2 , Figure 3 The true value and detection results for small-scale ships and small-scale vehicles are given respectively in complex scenarios such as false alarm source interference and dense arrangement. As can be seen from the figure, this method can detect all targets and there is no false alarm.

Claims

1. A method for small-scale surface target detection in remote sensing images based on adaptive multi-level fusion. Features The method comprises the following steps: Step 1: Use the backbone feature extraction network to extract the shallow and deep multi-level feature maps of the input image, with the downsampling levels being 4, 8, 16, and 32 times respectively; Step 2: Use a multi-level feature extraction architecture with adaptive fusion weights to fuse features of different downsampling levels in step 1. The specific steps are as follows: The fusion transfers high-level semantics and low-level positioning information through top-down and bottom-up paths respectively. In the top-down fusion path, for the feature map F of the nth level n , whose length, width, and number of channels are recorded as (H, W, C), and it is necessary to fuse the feature map F of the n+1th deeper level n+1 , its length, width, and number of channels are recorded as (H / 2, W / 2, C); Step 2-1: Use the CARAFE upsampling operator to upsample the deep-level feature map to achieve aggregation of semantic information. The specific steps are as follows: F n+1 Upsampling is performed to aggregate semantic information and obtain high-level feature maps Its length, width, and number of channels are recorded as (H, W, C); Step 2-2: Shallow feature map correction based on deep feature supervision. The specific steps are as follows: High-level feature maps As a supervision to the shallow level feature map F n Fine-tune, correct the noisy low-level features through high-level semantic features, and process them through Sigmoid nonlinear activation function Get the corrected weight map and apply it to the shallow feature map F n On the top, we get the low-level feature map Step 2-3: Based on the spatial and channel feature saliency combined with the broadcast mechanism, adaptively generate point-by-point fusion weights to achieve the fusion of deep and shallow features. The specific steps are as follows: Step 2-3-1: High-level and low-level feature maps are concatenated along the channel dimension. The specific steps are as follows: Concatenate low-level feature maps along the channel dimension and high-level feature maps Get the concatenated feature map F n C , for subsequent channel saliency and spatial saliency generation; Step 2-3-2: Generate channel saliency based on spatial feature map pooling and fully connected neural network. The specific steps are as follows: First, aggregate F through spatial pooling n C The global information in the channel is then used to learn the dependencies between channels through a fully connected neural network, which is recorded as: ωc=f FCN (AvePool(Fn C )); In the formula, AvePool(·) is the average pooling, f FCN (·) is the fully connected layer, ω c is the generated channel saliency, with dimension (1,1,C); Step 2-3-3: Generate spatial saliency based on 1×1 convolution and 3×3 convolution. The specific steps are as follows: Adjust F through 1×1 convolution n C Channel, and then use 3×3 convolution to aggregate channel dimension information to generate spatial saliency, denoted as: ωs=conv 3×3 (conv 1×1 (Fn C )); In the formula, conv 3×3 and conv 1×1 Represent 3×3 convolution and 1×1 convolution respectively, ω s is the generated spatial saliency, with dimension (H, W, 1); Step 2-3-4: Generate point-by-point fusion weights based on the broadcast mechanism. The specific steps are as follows: The dimensions of channel saliency and spatial saliency are different, but they can be added together through the broadcast mechanism to generate point-by-point saliency of dimension (H, W, C). The fusion weight ω can be obtained by using the Sigmoid nonlinear activation function, which is denoted as: ω=σ(ω c +oh s ); Step 2-3-5: Based on the adaptively generated fusion weights, weighted fusion of deep-level features and shallow-level features is performed. The specific steps are as follows: Based on the generated adaptive fusion weights, the low-level feature maps And upsample high-level feature maps Fusion is performed to obtain the feature fusion result of the nth level Step 3: Select the fused high-resolution feature layer with downsampling levels of 4 and 8 times to predict the target position and category information to obtain the final detection result.

2. The remote sensing image small-scale surface target detection method based on adaptive multi-level fusion according to claim 1, Features The specific steps of step 1 are as follows: For the input remote sensing image, the backbone feature extraction network is used to extract the shallow and deep multi-level feature maps in the remote sensing image, and the feature maps with downsampling levels of 4, 8, 16, and 32 times are extracted respectively, which are recorded as P2, P3, P4, and P5 levels.

3. The remote sensing image small-scale surface target detection method based on adaptive multi-level fusion according to claim 1 or 2, Features The backbone feature extraction network is CspDarknet53, ResNet or DenseNet.

4. The remote sensing image small-scale surface target detection method based on adaptive multi-level fusion according to claim 1, Features In step 2-1, the upsampling method is interpolation, deconvolution or CARAFE operator.

5. The remote sensing image small-scale surface target detection method based on adaptive multi-level fusion according to claim 1, Features The low-level feature map Calculated by the following formula: Where σ(·) is the Sigmoid activation function.

6. The remote sensing image small-scale surface target detection method based on adaptive multi-level fusion according to claim 1, Features In the steps 2-3-5, the low-level feature map And upsample high-level feature maps The formula for fusion is as follows:

7. The remote sensing image small-scale surface target detection method based on adaptive multi-level fusion according to claim 1, Features In step 3, the prediction method is the target information regression method of YOLO.