An infrared dim small target detection method based on sparse attention and multi-scale feature fusion

Abstract

The present application relates to the field of infrared image weak and small target detection, specifically to a kind of sparse attention and multi-scale feature fusion infrared small target detection network framework, feature map group is introduced in network downsampling, perception, enhance the weight of small target feature map group, improve the representation ability of small target;In the skip connection, the full connection layer is introduced, the similar background is inhibited from global receptive field, so as to improve the contrast of target and background;The long-range relationship between small target and background is learned by using Biformer module, effectively making up for the lack of detail texture features, improving the expression ability of features;Adopt adaptive asymmetric fusion module, fully excavate the shallow spatial features and deep semantic features of infrared weak and small targets, and fuse multi-scale context information from high and low level features. Complete the pixel classification of real target and background noise, and obtain the real target detection result.

Description

Technical Field

[0001] This invention relates to the field of infrared image weak target detection, specifically to an infrared weak target detection method that combines sparse attention and multi-scale feature fusion. Background Technology

[0002] Infrared target detection utilizes the difference in thermal radiation between the target and the background in infrared imaging to locate the target, and it has wide applications in military, security monitoring, and early warning systems. Compared with ordinary visible light targets, infrared targets are imaged at greater distances, resulting in a smaller proportion of the target in the image and a lack of obvious texture, shape, and structural features. Furthermore, infrared radiation energy attenuates with increasing distance, making the target easily obscured by background clutter and noise. The target sparsity also leads to a severe imbalance between the target and the background, posing a challenge to detection.

[0003] Methods for infrared weak target detection can be broadly categorized into model-driven mathematical modeling methods and data-driven deep learning methods. Model-driven methods include those based on filtering, human visual systems, and low-rank sparse matrices. These methods utilize prior knowledge for modeling but have limitations when handling complex scenes. For example, methods based on the infrared background consistency assumption, while simple in principle, are prone to high false alarm rates in complex scenes; methods based on sparse matrices and low-rank matrix recovery are computationally complex and easily mistake noise for targets when noise levels are high, increasing the false detection rate; methods based on the human visual system, while considering the contrast between the target and its neighborhood, suffer from low computational efficiency and issues such as the target not standing out in salient images and incomplete background clutter suppression in complex backgrounds. These model-driven methods consider the physical and imaging characteristics of the target when designing detection algorithms, but inevitably rely on some assumptions and handcrafted features, which may not be robust enough in complex backgrounds. Therefore, further exploration of more effective methods is needed to improve the accuracy and robustness of infrared weak target detection.

[0004] Data-driven methods do not require prior knowledge or hand-crafted features and have training frameworks based on large amounts of accurate data, thus achieving higher detection accuracy and lower false positive rates. Therefore, this invention employs a data-driven approach and designs a deep learning network to achieve accurate detection of small infrared targets in complex backgrounds. Summary of the Invention

[0005] To address the challenges of low detection rate and high false alarm rate caused by the small pixel proportion and lack of detailed texture features in infrared weak targets under complex backgrounds, this paper proposes an infrared weak target detection network based on sparse attention and multi-scale feature fusion. This network is used to detect weak targets in infrared images under complex backgrounds, effectively improving the detection accuracy and robustness.

[0006] This invention employs the following technical solution: It utilizes an infrared weak target detection network based on sparse attention and multi-scale feature fusion to accurately detect infrared weak targets in various complex backgrounds. The process includes the following steps: First, the input image is input to a preprocessing module for preprocessing to normalize it. The preprocessed input image then enters the first feature extraction module. The feature map extracted by the first feature extraction module is then input to the next feature extraction module. A feature extraction module incorporating segmentation attention extracts features at different scales. The feature maps extracted by the feature extraction module at different scales are input to an information enhancement module. The enhanced feature map then enters an attention module, where sparse attention is introduced to learn the long-range relationship between the target and the background. The feature map output by the last attention module is input to the last upsampling module. The high-level features sampled by the last upsampling module and the low-level features output by the penultimate attention module... The features are input into the last fusion module. The feature map output by the last fusion module is input into the penultimate upsampling module. The high-level features sampled by the penultimate upsampling module and the low-level features output by the penultimate attention module are input into the penultimate fusion module. The feature map output by the penultimate fusion module is input into the penultimate upsampling module. The high-level features sampled by the penultimate upsampling module and the low-level features output by the penultimate attention module are input into the penultimate fusion module, and so on. The feature map output by the second fusion module is input into the first upsampling module. The high-level features sampled by the first upsampling module and the low-level features output by the first attention module are input into the penultimate fusion module. The fusion module fuses the high-level and low-level features. The feature map output by the first fusion module is finally output as a binary image of the detection result through the output head module. The combination of the above modules constitutes an infrared weak target detection network based on sparse attention and multi-scale feature fusion. It uses a public dataset and performs data augmentation operations including rotation, cropping, symmetry, and adding random noise. The network is trained iteratively multiple times, and the output is constrained by a loss function to obtain the final network model for prediction. The network input is a single frame of infrared image to be detected, and the output is the detection result containing the real target.

[0007] The preprocessing module standardizes the input image. First, batch normalization is performed on the input image to adjust its brightness and contrast, improving model stability and convergence speed. This reduces internal covariate shifts, mitigating the vanishing gradient problem during training and enhancing the model's generalization ability. After batch normalization, a convolutional layer transforms the single-channel 256×256×1 input image into a richer 256×256×16 image, introducing more effective information and contextual relationships to capture more complex features and structures, thus improving overall detection performance. Another batch normalization layer normalizes the output of the convolutional layer, further stabilizing network training and accelerating convergence. Finally, a ReLU activation function is applied to increase non-linearity and enhance the model's expressive power.

[0008] The feature extraction module employs the Resnest module, which introduces a segmentation attention mechanism on top of ResNet. By dividing the channels into multiple subgroups, the network can more effectively learn representations of different features. First, grouped convolutions are used to divide the feature map into k×r feature map groups according to channels. Then, r feature map groups are treated as a single group and subjected to average pooling. This results in different channel weights for each feature map group, increasing the weight of feature map groups containing small objects, thereby improving the representation ability of small object features.

[0009] The information enhancement module includes convolutional layers, fully connected layers, normalization layers, and batch normalization layers. The feature map first passes through a convolutional layer to fuse different information; then it passes through a fully connected layer to map the input 256×256×C dimensional vector to a 64×C dimensional vector, reducing the number of parameters and computational complexity. Normalization ensures stable data distribution across different layers of the network. Another fully connected layer maps the 64×C dimensional vector back to a 256×256×C dimensional vector for better utilization of feature information. Convolutional layers further process the features. Finally, a batch normalization layer enhances the network's stability and convergence speed. The introduction of fully connected layers in the information enhancement module enhances the model's non-linear expressive power. Each neuron in a fully connected layer is connected to all neurons in the previous layer, enabling the network to learn more complex feature mappings, thereby improving its ability to extract abstract features. Meanwhile, fully connected layers possess a global receptive field, with each neuron having weights for all locations in the input feature map. By learning global features, fully connected layers can suppress redundant structures in the background, reducing false alarm rates. Fully connected layers also enhance the network's ability to perceive differences between small targets and the background, allowing the network to focus more on key regions and improving the accuracy of target detection.

[0010] The attention module employs the Biformer mechanism, a sparse attention mechanism that computes the correlations between inputs in the feature sequence, ensuring the model effectively handles the importance of global contextual information in locating the target within complex backgrounds. Therefore, the multi-layered features obtained from ResNest are processed by the Biformer module to compute self-attention, fully utilizing global features and establishing long-range dependencies on the target. This block-based approach reduces computational complexity and improves model efficiency. Furthermore, the Biformer attention mechanism can better capture long-range dependencies, thereby enhancing model performance.

[0011] Traditional fusion modules typically employ top-down or bottom-up mechanisms to integrate low-level and high-level features within deep learning frameworks. Top-down modulation methods can embed more accurate semantic information from high-level features, but they are prone to losing small infrared targets in complex backgrounds as network depth increases. Bottom-up modulation can enrich high-level features with spatial details of low-level features, highlighting deep infrared small targets, but it may weaken the semantic information of the target, thereby reducing detection accuracy. To combine top-down global attention with bottom-up local attention, an adaptive asymmetric fusion module is proposed to exchange multi-scale contextual information from low and high levels.

[0012] The upsampling module uses nearest-neighbor upsampling to expand the depth feature map from W×H×2C to 2W×2H×C, and then fuses and merges it into the shallow feature map point by point, finally outputting a feature map of size 2W×2H×C.

[0013] The output head module first uses convolutional layers to extract spatial features from the feature map, capturing local features in the image and further enhancing the network's perceptual capabilities. Then, a normalization layer is introduced to normalize the feature map, accelerating network convergence and improving stability, while adjusting the numerical range of the feature map to enhance generalization performance. Next, the ReLU activation function is applied for a non-linear transformation, pruning negative values ​​to encourage the network to learn more complex feature representations and improve non-linear expressive capabilities. After ReLU activation, convolutional layers are used again for deeper feature extraction, further optimizing the network's abstract representation of the target and enhancing sensitivity to complex targets. Finally, the output head module uses the Sigmoid activation function to convert the feature map into a binary image of the target, mapping the value of each pixel to the range of 0 to 1, and outputting the probability of the target's presence.

[0014] In order to reduce the number of parameters, the feature map output by the first information enhancement module is directly fed into the first fusion module without passing through the attention module in the above-mentioned infrared weak target detection network design method based on sparse attention and multi-scale feature fusion. That is, there is no first attention module in the network, and the input feature map is used as the low-level feature input into the first fusion module.

[0015] The aforementioned infrared weak target detection network design method based on sparse attention and multi-scale feature fusion obtains the input source image and label image through the following process: The IRSTD-1K (1000 images) and NUDT-SIRST (1327 images) datasets are used as the base datasets, containing targets of different categories and sizes, as well as complex backgrounds such as sky, ocean, buildings, and fields, making detection challenging. Due to the small dataset size, network training is prone to instability, insufficient generalization ability, and overfitting. To address these issues, the data was augmented, including rotation, horizontal flipping, random flipping, cropping, and expansion, resulting in a total of 6328 images. The input image size was adjusted to 256×256, and the dataset was divided into training, validation, and test sets in a 7:2:1 ratio.

[0016] To address the challenges of low pixel count and lack of detailed texture features in infrared weak targets against complex backgrounds, leading to difficulties in feature extraction, low detection rates, and high false alarm rates, this invention proposes a U-shaped infrared weak target detection network based on sparse attention and multi-scale feature fusion. By designing an end-to-end infrared weak target segmentation network, robust detection of real targets is achieved. The network learns the long-range relationship between small targets and the background, effectively compensating for the lack of detailed texture features and improving the contour description performance of real targets. This provides more prior information for subsequent recognition and tracking operations. Attached Figure Description

[0017] Figure 1 This is a diagram of the network model structure of the present invention.

[0018] Figure 2 This is a structural diagram of different modules of the present invention.

[0019] Figure 3 This is a structural diagram of the feature fusion module of the present invention.

[0020] Figure 4 (a) is the original input image of the ocean background infrared in an embodiment of the present invention.

[0021] Figure 4 (b) is the label image corresponding to the marine background source image in the embodiment of the present invention.

[0022] Figure 4 (c) is the output image of the marine background detection network in an embodiment of the present invention.

[0023] Figure 5 (a) is the original infrared input image of the field background in an embodiment of the present invention.

[0024] Figure 5 (b) is the label image corresponding to the field background source image in the embodiment of the present invention.

[0025] Figure 5 (c) is the output image of the field background detection network in the embodiment of the present invention.

[0026] Figure 6 (a) is the original infrared input image of the cloud background in an embodiment of the present invention.

[0027] Figure 6 (b) is the label image corresponding to the cloud background source image in the embodiment of the present invention.

[0028] Figure 6 (c) is the output image of the cloud background detection network in the embodiment of the present invention. Detailed Implementation

[0029] The present invention will be further described below with reference to the accompanying drawings and specific embodiments:

[0030] Reference Figure 1 The infrared weak target detection network design method based on sparse attention and multi-scale feature fusion in this embodiment includes the following steps:

[0031] Step 1: Input an infrared image to be detected, X∈R H×W×1 ;

[0032] Step 2: Send the infrared image X to be detected into the preprocessing module to convert the single-channel 256×256×1 input image into a richer 256×256×16 image.

[0033] X'=σ(BN(Conv(BN(X)))) (1)

[0034] X and X' represent the infrared image to be detected and the output image, respectively; BN(g) represents the data normalization operation; Conv(g) represents the convolution operation; σ represents the ReLU activation function.

[0035] Step 3: After preprocessing, the image enters the feature extraction module. After passing through feature extraction module 1, the size of the image feature map remains unchanged, only the number of channels changes to 256×256×32. After passing through feature extraction modules 2, 3, and 4, feature maps of three scales, 128×128×64, 64×64×128, and 32×32×256, are extracted respectively.

[0036] Step 4: Enhance the extracted feature maps at different scales to learn the overall features, suppress redundant structures in the background, reduce the false alarm rate, and improve the ability to perceive differences between small targets and the background, thus focusing more on key areas.

[0037] Step 5: The augmented feature maps undergo a dynamic query attention mechanism to filter out irrelevant key-value pairs at a coarse region level, retaining a small subset of relevant regions. Fine-grained token-to-token attention is then applied to these regions. To reduce the number of parameters, attention calculation is performed only on the feature maps after augmentations 2, 3, and 4. First, region partitioning and input mapping are performed. Given input features X″∈R... H×W×C First, it is divided into S×S distinct regions, where each region contains One eigenvector. That is, X″ becomes Then, Q, K, and V are obtained through linear mapping.

[0038] Q = X r W q (2)

[0039] K = X r W k (3)

[0040] V = X r W v (4)

[0041] Among them, W q W k W v ∈R C×C These are the projected weights for query, key, and value, respectively.

[0042] By constructing a directed graph, we can find the regions that each given region should participate in; first, we calculate the average values ​​of Q and K in each region, and then obtain... Then calculate Q. R K R The adjacency matrix of inter-region correlation:

[0043] A r =Q r (K r ) T (5)

[0044] Then, only the top k connections for each region are retained to prune the correlation graph. There is a routing index matrix. Save the indexes of the first k joins row by row.

[0045] I r =topkIndexA r (6)

[0046] I r The i-th row contains the indexes of the first k most relevant regions of the i-th region.

[0047] Using the area-to-area routing index matrix I r It computes fine-grained token-to-token attention. For each Q in region i, it focuses on the aggregated K and V of k routing regions.

[0048] K g =gather(K,I) r ),V g =gather(V,I) r (7)

[0049] Among them, K g V g It is a tensor of aggregated key-value pairs, and then attention operations are applied to the aggregated KV pairs:

[0050] O = Attention(Q,K) g V g )+LCE(V) (8)

[0051] In the formula, Attention(g) represents attention calculation; LCE(g) represents depthwise convolution parameterization.

[0052] Step 6: The upsampling module uses nearest neighbor upsampling to expand the depth feature map, making its size twice as large.

[0053] Step 7: The output of attention module 3 enters upsampling module 3. The high-level features of upsampling module 3 and the low-level features after passing through attention module 2 are input into fusion module (AFM module) 3. The output of fusion module 3 enters upsampling module 2. The high-level features of upsampling module 2 and the low-level features after passing through attention module 1 are input into fusion module 2. The output of fusion module 2 enters upsampling module 1. The high-level features of upsampling module 1 and the low-level features after passing through information enhancement module 1 are input into fusion module 1. The high-level features contain a lot of semantic information, and important channels are selected through channel attention; the low-level features contain target location information, and a point attention mechanism is used to preserve the spatial detail information of small infrared targets.

[0054] g pa (X)=δ(W2σ(W1X)) (9)

[0055] g ca (X)=δ(W2σ(W1P(X))) (10)

[0056] After summing and fusing the features, respectively, g pa and g ca Constraints are applied to address the issue of feature mismatch caused by individual constraints. and ⊙ represent the corresponding unit multiplication and the corresponding vector tensor multiplication, respectively. δ is the Sigmoid activation function, and σ is the ReLU activation function.

[0057]

[0058] Step 8: Model Training Parameter Settings: Model training was conducted on a Windows server with a 16GB NVIDIA RTX A4000 graphics card, and the testing software was PyCharm 2021.1. The training set, validation set, and test set consisted of 4429, 1266, and 633 images respectively, with an image size of 256×256. The total number of training epochs was 100, the learning rate was initialized to 0.01, the Adam optimizer was selected as the optimization function, and the network framework was PyTorch 2.1.

[0059] Step 9: Model Training Process: The infrared weak target detection network based on feature compensation proposed in this invention is an end-to-end network model. Every 16 images are fed into the network model as a batch for iterative training. The softIoU loss and focal loss functions are used to calculate the loss between the network output and the corresponding label, and then the network parameters are updated through gradient backpropagation. After each round of training, a validation round is performed to avoid overfitting or underfitting. The trained network model is tested using a test set, and the test results are evaluated using evaluation metrics (signal-to-noise ratio gain, background suppression factor, intersection-over-union ratio, precision, recall, and F-metric).

[0060] Step 10: Image Detection Process: Input the image to be detected into the trained model for detection. The detection results are as follows: Figure 4 (c) Figure 5 (c) Figure 6 As shown in (c).

Claims

1. A method for designing an infrared small target detection network that combines sparse attention and multi-scale feature fusion, characterized in that: Includes the following steps: First, the input image is fed into a preprocessing module for normalization. The preprocessed image then enters the first feature extraction module. The feature map extracted by the first feature extraction module is then fed into the next feature extraction module, which uses a feature extraction module incorporating segmentation attention to extract features at different scales. These feature maps at different scales are then fed into an information enhancement module. The enhanced feature maps are then fed into an attention module, which introduces sparse attention to learn the long-range relationships between the target and the background. The feature map output from the last attention module is fed into the last upsampling module. The high-level features sampled by the last upsampling module and the low-level features output from the penultimate attention module are then fed into the last fusion module. The feature map output from the last fusion module is then fed into the penultimate upsampling module, which samples high-level features and the penultimate attention module outputs low-level features. The features are input into the penultimate fusion module. The feature map output from the penultimate fusion module is input into the penultimate upsampling module. The high-level features sampled by the penultimate upsampling module and the low-level features output from the penultimate attention module are input into the penultimate fusion module. This process continues, with the feature map output from the second fusion module being input into the first upsampling module. The high-level features sampled by the first upsampling module and the low-level features output from the first attention module are input into the penultimate fusion module. The fusion module fuses the high and low-level features. The feature map output from the first fusion module is finally output as a binary image of the detection result through the output head module. The combination of the above modules constitutes an infrared weak target detection network based on sparse attention and multi-scale feature fusion. A public dataset is used and data augmentation is performed. The network is trained through multiple iterations, and the output of the network is constrained by a loss function to obtain the final network model for prediction. The feature extraction module uses the Resnest module, which introduces a segmentation attention mechanism on top of ResNet; the attention module uses the Biformer attention mechanism.

2. The design method for an infrared small target detection network based on sparse attention and multi-scale feature fusion according to claim 1, characterized in that: The preprocessing module transforms the single-channel input image into a richer image through convolutional layers, capturing more complex features and structures in the image, introducing more information and contextual associations, and providing richer input for the subsequent feature extraction module; The feature extraction module uses the Resnest module, which introduces a segmentation attention mechanism on the basis of ResNet. By dividing the channels into multiple subgroups, the network can learn more effective feature representations between different subgroups, thereby improving the network's ability to capture different features. The information enhancement module introduces a fully connected layer, which has a global receptive field. Each neuron has weights for all locations in the input feature map. By learning the overall features, redundant structures in the background are suppressed, the false alarm rate is reduced, and the ability to perceive differences between small targets and the background is improved, making the network more able to focus on key regions. The attention module employs a sparse attention mechanism, which can calculate the correlation between inputs in the feature sequence, ensuring that the model can handle the importance of global contextual information in complex backgrounds for target localization; at the same time, it reduces model parameters to meet real-time requirements. The feature fusion module combines top-down global attention and bottom-up local attention, and proposes an adaptive asymmetric fusion module for exchanging multi-scale contextual information at low and high levels.

3. The design method for an infrared small target detection network based on sparse attention and multi-scale feature fusion according to claim 2, characterized in that: The feature extraction module uses grouped convolution to divide the feature map into k×r feature map groups according to the channels. Then, the r feature map groups are treated as a group and average pooled, so that the channel weights of each feature map group are different. This increases the weight of the feature map groups containing small targets and improves the representation ability of small target features.

4. The design method for an infrared small target detection network based on sparse attention and multi-scale feature fusion according to claim 3, characterized in that: The attention module adopts the Biformer attention mechanism. By introducing a bidirectional attention mechanism, the Biformer attention mechanism can better capture the dependencies between different positions in the sequence. It divides the sequence into multiple small blocks, performs self-attention calculation within each small block, and then captures the dependencies between different small blocks through cross-attention calculation.

5. The design method for an infrared small target detection network based on sparse attention and multi-scale feature fusion according to claim 4, characterized in that: The feature fusion module inputs the high-level features from the upsampling module and the low-level features from the attention module into the AFM module. The high-level features contain a lot of semantic information and are selected by channel attention to select important channels. The low-level features contain target location information and are used by point attention mechanism to retain the spatial detail information of infrared small targets, thereby exchanging multi-scale contextual information between low and high levels.

6. The design method for an infrared small target detection network based on sparse attention and multi-scale feature fusion according to claim 5, characterized in that: Data augmentation includes operations such as rotation, cropping, symmetry, and adding random noise.

7. The design method for an infrared small target detection network based on sparse attention and multi-scale feature fusion according to claim 1, 2, 3, 4, 5, or 6, characterized in that: To reduce the number of parameters, the feature map output by the first information enhancement module is directly fed into the first fusion module, and this input feature map is used as the low-level feature in the first fusion module.

8. The design method for an infrared small target detection network based on sparse attention and multi-scale feature fusion according to claim 1, 2, 3, 4, 5, or 6, characterized in that: The IRSTD-1K and NUDT-SIRST datasets were used as the base datasets, which contain different categories, different sizes of objects, and different complex backgrounds such as sky, ocean, buildings, and fields. The data was augmented, including rotation, left and right flipping, random flipping, cropping, and the input image size was adjusted to 256×256. The dataset was divided into training set, validation set and test set in a 7:2:1 ratio.

Images