An end-to-end detection method for small targets based on infrared dual-band images

Abstract

The present invention discloses an end-to-end detection method for small and dim targets based on infrared dual-band images. The method constructs a dual-band infrared small and dim target detection model, which includes multiple layers of sequentially connected feature fusion units. Each feature fusion unit processes a medium-wave information parameter and a long-wave information parameter respectively through two dynamic attention modules to obtain respective process features one. The two process features one are processed respectively through the fusion module to obtain fusion features. The fusion features are summed with the results of the feature fusion units in the previous layer, and the sum is processed through the dynamic attention module to obtain the output features of the feature fusion unit. The target detection model reuses the Attention module multiple times, simplifies the entire detection model, and significantly improves the performance of the algorithm. By enhancing the global modeling capability of the model, the algorithm has better performance in dual-band infrared small and dim target detection.

Description

Technical Field

[0001] The present invention belongs to the field of target detection in digital image processing, and in particular to an end-to-end detection method for small and dim targets based on infrared dual-band images. Background Art

[0002] Infrared small target detection technology is widely used in various infrared search and warning systems, playing a vital role in both civilian and military applications. However, due to the long-range camera shooting and interference from noise and clutter, infrared targets often appear as small pixels and dim features, making single-band infrared small target detection challenging. Compared to single-band infrared imagery, dual-band infrared detection offers the following advantages: ① Improved immunity to complex background interference: long-wave infrared has greater penetration through smoke and dust, resulting in better image quality; ② Long-wave infrared is more sensitive to ambient temperature changes, enabling enhanced target detection in low-temperature environments; and ③ Dual-band fusion can yield richer target details, improving resolution of small targets. Therefore, dual-band infrared small target detection technology presents a promising future. However, limited data acquisition and other factors have resulted in limited research on target detection and recognition tasks involving dual-band infrared.

[0003] Current research on multi-band image fusion detection primarily focuses on the visible light / infrared domain. Technical approaches can be categorized into two approaches: fusion-first, detection-later, and end-to-end detection models. End-to-end detectors offer new possibilities for addressing the problem of detecting small, dim targets in dual-band infrared. However, compared to visible light / infrared detection, detection is not only difficult due to the small size of the target and the lack of distinct features like texture and scale. The dual-band imaging characteristics also exacerbate this challenge. Specifically, when a target appears bright in one band and dark in another, fusion can cause these features to cancel each other out, reducing detection performance or even leading to detection failure. Inaccurate target feature acquisition and insufficient and irrational fusion of multi-band image features are the primary reasons for the poor performance of dual-band infrared dim target detection. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide an end-to-end detection method for small targets based on infrared dual-band images, so as to solve the problems of inaccurate acquisition of small target characteristics, insufficient fusion of multi-band image characteristics and unreasonableness in the prior art.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] An end-to-end detection method for small targets based on infrared dual-band images includes the following steps:

[0007] S1, obtain medium-wave infrared images and long-wave infrared images of small targets, extract medium-wave information parameters from the medium-wave infrared image, and extract long-wave information parameters from the long-wave infrared image;

[0008] S2, inputting the medium-wave information parameters and the long-wave information parameters into the dual-band infrared small target detection model;

[0009] The dual-band infrared dim target detection model includes multiple layers of sequentially connected feature fusion units. Each layer of feature fusion units processes the medium-wave information parameters and the long-wave information parameters respectively through two dynamic attention modules to obtain respective process features one. The two process features one are processed by the fusion module to obtain fusion features. The fusion features are added to the results of the feature fusion units of the previous layer, and the sum is processed by the dynamic attention module to obtain the output features of the feature fusion units. The output features of the feature fusion units of each layer are the input features of the feature fusion units of the next layer. The feature fusion units of the last layer output the output features of the dual-band infrared dim target detection model.

[0010] S3, after processing the output features of the dual-band infrared small target detection model through the segmentation task head, detects the small target.

[0011] A further improvement of the present invention is:

[0012] Preferably, the number of medium-wave information parameters and the number of long-wave information parameters obtained in S1 match the number of feature fusion units in the dual-band infrared small target detection model.

[0013] Preferably, in S2, a linear encoding unit is provided before each of the feature fusion units.

[0014] Preferably, in S2, the dynamic attention module divides the input features evenly to obtain X features and Y features; the X features and Y features are processed by the MA feature extraction module and the PTA feature extraction module respectively to obtain process features 2 and 3; the process features 2 and 3 are multiplied by the corresponding dynamic parameters respectively, and the two products are added to the corresponding X features and Y features respectively; the two sums are merged in the channel dimension to obtain a merged feature image; the merged feature image is mapped to a preset dimension through a multilayer perceptron to obtain an output feature.

[0015] Preferably, a module including SSM is introduced into the MA feature extraction module.

[0016] Preferably, the PTA feature extraction module first encodes the input features and normalizes the features, then samples the input feature map with an interval of S in the spatial dimension using the normalized features, and obtains the summed features by adding the sampling results to the input features. After the summed results are processed by encoding and normalization, the processed results are passed through MLP mapping, and the MLP mapping results are added to the summed features to obtain process feature three.

[0017] Preferably, the fusion module normalizes the two process features one respectively, cross-fuses the normalized results through the dynamic attention module, and adds each fusion result to the corresponding process feature one to obtain the sum result of the medium-wave image and the sum result of the long-wave image; merges the two sum results in the channel dimension and performs Fourier fusion processing at the same time; the merged result is sequentially normalized and MLP mapped, and the MLP mapping result is added to the merged result and the Fourier fusion result to obtain the fusion result.

[0018] Preferably, the process of cross-fusing the normalized processing results through the dynamic attention module is: after processing the two process features 1 respectively through the MA feature extraction module and the PTA feature extraction module, process features 2 and 3 are obtained, process features 2 and 3 are multiplied by the corresponding dynamic parameters respectively, the two products are added to the corresponding X features and Y features respectively, the two sums are merged in the channel dimension to obtain a merged feature image, and the merged feature image is mapped to a preset dimension through a multi-layer perceptron to obtain an output feature.

[0019] Preferably, the Fourier fusion is to first perform Fourier transform on the two summed results to obtain corresponding amplitude components and phase components; use the amplitude components and phase components of the medium wave image as the dominant image features to obtain the real and imaginary parts of the high-order fusion features, add the real and imaginary parts of the high-order fusion features, and then perform inverse Fourier transform to obtain the Fourier fusion processing result.

[0020] Preferably, the MLP mapping consists of convolution, batch normalization and ReLU activation functions.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] The present invention discloses an end-to-end detection method for small and dim targets based on infrared dual-band images. The method constructs a dual-band infrared small and dim target detection model, which fully utilizes the advantages of dual-band detection in improving target recognizability and enhancing contrast. The model design includes multiple layers of sequentially connected feature fusion units and adopts multiple reused Attention modules, thereby simplifying the structure of the entire detection model. This design not only improves the performance of the algorithm, but also enhances the model's adaptability to complex backgrounds and environmental conditions. By enhancing the global modeling capability of the model, the algorithm demonstrates excellent performance in dual-band infrared small and dim target detection. This method fully utilizes the fusion advantages of dual-band data in real-time monitoring and dynamic target recognition, ensuring that small and dim targets can be effectively identified under various climatic conditions. Overall, the present invention provides an innovative solution for the application of infrared dual-band detection technology, which can better cope with challenges in actual scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a schematic diagram of the infrared dual-band small target detection algorithm framework;

[0024] Figure 2 Schematic diagram of the dynamic attention module;

[0025] Figure 3 Schematic diagram of the Patch Transformer Attention module;

[0026] Figure 4 Schematic diagram of the Fourier fusion module;

[0027] Figure 5 Schematic diagram of the test results. DETAILED DESCRIPTION

[0028] Hereinafter, the terms "first," "second," "third," and "fourth" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the technical features indicated. Thus, a feature identified as "first," "second," "third," or "fourth" may explicitly or implicitly include one or more of such features.

[0029] The co-shooting method provided in the embodiments of the present application can be applied to terminal devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). The embodiments of the present application do not impose any restrictions on the specific types of terminal devices.

[0030] It should be noted that the terms "first," "second," and the like in the description and drawings of the present invention are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatus.

[0031] See also Figure 1 The first aspect of the present invention discloses an end-to-end detection method for small and weak targets based on infrared dual-band images. The main attributes of infrared small and weak targets are "weak" and "small". The so-called "weak" means that the grayscale value of the target is lower than the surrounding background in the infrared image, that is, the contrast is relatively low. The so-called "small" means that the area occupied by the target on the image is small, which is specifically reflected in the image as the number of pixels occupied by the target is very small. Generally speaking, in an image with a pixel size of 256×256, a target with a signal-to-noise ratio of less than 4 and a pixel size of less than 9×9 is defined as a small and weak target.

[0032] The dynamic attention module in this method integrates the global perception of self-attention and the sequential scanning capabilities of SSM, significantly improving the model's ability to mine features for multi-band images of different scales. The fusion module processes the feature information extracted by the attention module from different bands. It also uses Fourier transforms to fuse the phase and frequency spectra of the two band features, coupling these fused features with deep features to generate target-enhanced features.

[0033] The method specifically comprises the following steps:

[0034] S1, for small targets, obtain medium-wave infrared image I mand long-wave infrared image I l , the two infrared images are subjected to feature extraction through their respective feature extraction networks, and the medium-wave infrared image I m Extract the medium wave information parameter f m , extract the long-wave information parameter f from the long-wave infrared image l ;

[0035] S2, input the extracted parameters into the dual-band infrared small target detection model (SSD) to obtain the detection results.

[0036] See also Figure 1 The left part is the dual-band infrared small target detection model, which is State Space Model and Self-Attention for Infrared Dual Small and Dim Target Detection, referred to as SSD. The model consists of 4 layers of connected FFUs. Except for the first layer of FFU, the output of each layer of FFU is the input of the next layer of FFU. The number of FFUs is determined according to the extraction accuracy and algorithm complexity. The structure of each layer of FFU is the same. The input of the first layer FFU to the third layer FFU module includes the medium-wave information parameters of the current layer, the long-wave information parameters and the output of the previous layer (including the medium-wave information parameters and the long-wave information parameter f l i+1 ), receives three inputs, and FFU4 receives only two inputs.

[0037] See also Figure 1 The right side of the image is an enlarged view of the FFU. FFU is the abbreviation of Feature Fusion Unit, specifically the feature fusion unit, which is used to process the dual-band image features obtained through the backbone network. Each FFU includes a dynamic attention module (DAM), a fusion module (Fusion Module) and an addition module (Add). The medium-wave information parameter f m and the long-wave information parameter f l After being processed by a dynamic attention module respectively, the respective process features are obtained. After the two process features are fused by the fusion module, the fusion feature is obtained. The fusion feature is combined with the medium wave information parameter output by the previous layer. and the long-wave information parameter f l i+1 After adding them separately ( Figure 1 in Represents two parameters, namely medium wave information parameters and the long-wave information parameter f l i+1), the summation result is passed through the dynamic attention module again to output the medium-wave information parameters of this layer and the long-wave information parameter f l i , which is Figure 1 The two parameters in

[0038] As a preferred solution, a linear projection unit is set before each FFU to increase the parameter scale of the network model and map the features to a high-dimensional space.

[0039] It should be noted that each layer of FFU has its own medium-wave information parameters and long-wave information parameters, that is, the medium-wave information parameters and long-wave information parameters used for each layer of FFU should be extracted from the acquired medium-wave infrared image and long-wave infrared image, which are shown in the figure. and Typically, such parameters are initialized randomly.

[0040] In some embodiments of the present invention, the dynamic attention module is as follows Figure 2 As shown in the figure, in this module, first, the input features are normalized, and the normalized input feature map is evenly split (BN-Split) into X features and Y features. The two features are respectively extracted by the MA (Mamba Attention) feature extraction module and the PTA (Patch Transformer Attention) feature extraction module. The Mamba Attention module obtains the process feature 2, and the Patch Transformer Attention module obtains the process feature 3.

[0041] Considering the significant differences in target and background characteristics between medium- and long-wavelength images, indiscriminately processing feature information from images of different wavelengths can lead to inaccurate target feature extraction and excessive background noise. Therefore, dynamic parameters α and β are introduced to adjust the weights of features processed by the Attention module. Secondly, residual links are used to maximize the preservation of target feature information. Process features 2 and 3, obtained through different feature processing, are merged in the channel dimension (BN-Cat). In this step, process feature 2 is multiplied by α, and process feature 3 is multiplied by β. The two multiplication results are then added to the respective input X and Y values. These added features are then merged in the channel dimension to obtain the merged feature image.

[0042] Finally, a multi-layer perceptron (MLP) is used to map the image after merging features to the preset dimension to obtain the corresponding process feature 1. The MLP consists of convolution, batch normalization, and ReLU activation functions.

[0043] The dynamic attention module dynamically adjusts the feature weights during model training, enabling the model to adaptively learn the differential characteristics of different bands to enhance the model's feature learning ability. The above process is expressed by the following formula (1):

[0044]

[0045] Where, f input is the input feature map, f output is the output feature map, Op split is feature map splitting, Norm is batch normalization, MLP is a multi-layer perceptron consisting of Conv→BN→ReLU, Op concat To merge on the channel, α and β are learnable parameters.

[0046] It should be understood that the dynamic parameters α and β are obtained through training during the learning process of the entire network.

[0047] In the above process, the MA module is a visual module suitable for extracting features of infrared dim targets. As a new network architecture, MA introduces the state space model SSM (State Space Model). This model can achieve sequence-to-sequence mapping by introducing the concept of state variables, which can effectively process long sequence data, thereby enabling MA to capture long-term dependencies in the data. MA's long-range perception capability enables the model to perceive targets in higher dimensions, rather than being limited by the bottleneck problem of increasing the convolution receptive field. Compared with traditional recurrent neural networks, MA has linear time complexity and can process long sequences more efficiently.

[0048] In the above process, see Figure 3During data processing, Patch Transformer Attention first encodes and normalizes the input features through Normalization. The normalized features are then spatially sampled from the input feature map at intervals S, limiting the self-attention calculation to a fixed patch. During the S-sampling process, the data is divided into multiple small blocks according to the step size n. Different linear transformations are used to generate the query vector (Query, Q), key vector (Key, K), and value vector (Value, V). In the "ATT" module, attention scores are calculated using Q, K, and V, and the value vector is weighted and combined to obtain the self-attention result. The concatenated multi-head attention result undergoes another linear transformation, restoring the output tensor to the size of H×W×C. Compared with the Attention method directly based on Windows, interval sampling expands the scope of attention while avoiding the introduction of computational processes such as cyclic window attention, which will greatly increase the computational cost of the model. The sampling result is added to the input feature to obtain the sum feature. The sum result is processed by encoding normalization and then processed by MLP. The feature output by MLP is added to the sum feature to obtain process feature three.

[0049] In some embodiments of the present invention, the fusion module (Fusion Module) process structure diagram is as follows Figure 4 As shown in the figure, the fusion module is a Fourier-based multi-band feature fusion module. First, the process features from different bands are normalized separately. Based on the dynamic attention module, the dual-band features are cross-fused. Each cross-input result is summed with the corresponding input through the summation module. The two added features are merged in the channel dimension. After the merged result is normalized, the image of the merged features is mapped to the preset dimension using a multi-layer perceptron. The MLP consists of convolution, batch normalization, and ReLU activation functions.

[0050] The above dynamic attention module cross-integrates the dual-band features by taking the two normalized results as Figure 2 The X and Y features in Figure 2 process to process.

[0051] In the Path Transformer, the self-attention mechanism is used to learn the interdependencies and interactive information of objects in medium-wave and long-wave infrared images, resulting in a richer and more comprehensive representation of target features and effectively suppressing the impact of noise on model performance. To fully preserve key feature information in the image, the present invention designs a feature fusion module based on Fourier transform. The two summed results are then Fourier fused to obtain the Fourier fusion result.

[0052] In the Fourier fusion process, first, the dual-band image features are Fourier transformed to obtain the amplitude and phase components respectively.

[0053]

[0054] in, Represent the amplitude and phase spectral components of the feature after Fourier transform, respectively.

[0055] Since the medium wave image contains more detailed information, the real and imaginary parts of the high-order fusion features are obtained based on the recovery of the dominant image features of the medium wave image, as follows:

[0056]

[0057] in are the real and imaginary parts of the fused features, respectively.

[0058] Furthermore, the features represented in the frequency domain are converted back to the image space through inverse Fourier transform.

[0059]

[0060] The mapping result, merging result and Fourier fusion result of MLP are added together through the summation module to obtain the output result of the fusion module.

[0061] S3, obtains the output binary image containing only weak targets through the segmentation task head (predict network) and outputs the detection results.

[0062] Some embodiments of the present invention also disclose a method for acquiring a dual-band infrared small target detection model, wherein acquiring the model comprises the following steps:

[0063] Step 1: Obtain infrared image data, pre-process the image, and construct an image dataset Data = {I1, I2, I3, ..., I n}, where I represents the infrared dual-band image pair and N represents the total number of images.

[0064] Step 2: Divide the data set described in step 1 into two groups train and Data test ,Data train As training samples, Data test As a test sample, Data train The data is labeled and the labeled data is obtained label As the input of network training, Mask is a binary image containing only the target, Data label The representation is:

[0065] Data label ={(I,Mask1),...,(I n ,Mask n )}

[0066] Step 3: Build a dual-band infrared small target detection model. You can use a model trained on ImageNet, such as ResNet, Figure 1 In the feature extraction model, the input of the feature extraction model is the original dual-band image in the training set described in step 2, and the output is the dual-band image features of different scales.

[0067] Construct a multi-scale feature fusion unit (FFU). This network is mainly responsible for completing the adaptive fusion of dual-band image features from different levels of the image feature extraction network. This module includes a dynamic attention module and a Fourier fusion module.

[0068] An infrared dual-band small target end-to-end intelligent detection network is constructed. The dual-band features obtained in the above steps are fused using the FFU module, and finally the final intelligent detection network is obtained by combining it with the prediction network.

[0069] In step 4, the training data and labeled data from step 2 are fed into the training object detection model to obtain a trained model for inference.

[0070] In step 5, the model obtained in step 4 is used for inference testing. First, feature extraction is performed on the input long-wave and medium-wave infrared images respectively; second, multi-band features are coupled using a multi-layer cascaded FFU module; finally, the output binary image containing only weak targets is obtained by segmenting the task head.

[0071] The above model was used to test a dataset of weak target images with complex backgrounds. The specific results are shown in Table 1 and Figure 1 The results show that compared with other algorithms, the algorithm proposed in this paper has a greater performance advantage in the dual-band infrared dim small target detection task.

[0072] Table 1 Algorithm performance test results

[0073]

[0074] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A small target end-to-end detection method based on infrared dual-band images, characterized in that: The following steps are involved: S1, obtain medium-wave infrared images and long-wave infrared images of small targets, extract medium-wave information parameters from the medium-wave infrared image, and extract long-wave information parameters from the long-wave infrared image; S2, inputting the medium-wave information parameters and the long-wave information parameters into the dual-band infrared small target detection model; The dual-band infrared dim target detection model includes multiple layers of sequentially connected feature fusion units. Each layer of feature fusion units processes the medium-wave information parameters and the long-wave information parameters respectively through two dynamic attention modules to obtain their respective process features 1. The two process features 1 are processed by the fusion module to obtain fusion features. The fusion features are added to the results of the feature fusion units in the previous layer, and the sum is processed by the dynamic attention module to obtain the output features of the feature fusion unit. The output features of each layer of feature fusion units are the input features of the next layer of feature fusion units; the last layer of feature fusion units outputs the output features of the dual-band infrared dim target detection model; In S2, the dynamic attention module divides the input features evenly to obtain X features and Y features; the X features and Y features are processed by the MA feature extraction module and the PTA feature extraction module respectively to obtain process features 2 and 3; the process features 2 and 3 are multiplied by the corresponding dynamic parameters respectively, and the two products are added to the corresponding X features and Y features respectively, and the two sums are merged in the channel dimension to obtain a merged feature image; the merged feature image is mapped to a preset dimension through a multilayer perceptron to obtain an output feature; A module including SSM is introduced into the MA feature extraction module; The PTA feature extraction module first encodes the input features and normalizes the features, then samples the input feature map with the normalized features at intervals of S in the spatial dimension, adds the sampling results to the input features to obtain the summed features, normalizes the summed results through encoding, passes the processed results through MLP mapping, and adds the MLP mapping results to the summed features to obtain process feature three; S3, after processing the output features of the dual-band infrared small target detection model through the segmentation task head, detects the small target.

2. The method for end-to-end detection of small targets based on infrared dual-band images according to claim 1, characterized in that: The number of medium-wave information parameters and long-wave information parameters obtained in S1 matches the number of feature fusion units in the dual-band infrared small target detection model.

3. The method for end-to-end detection of small targets based on infrared dual-band images according to claim 1, characterized in that: In S2, a linear encoding unit is provided before each of the feature fusion units.

4. The method for end-to-end detection of small targets based on infrared dual-band images according to claim 1, characterized in that: The fusion module normalizes the two process features 1 respectively, cross-fuses the normalized results through the dynamic attention module, and adds each fusion result to the corresponding process feature 1 to obtain the sum of the medium-wave image and the sum of the long-wave image; the two summed results are merged in the channel dimension and subjected to Fourier fusion processing; The merged result is normalized and mapped by MLP in sequence, and then the MLP mapping result is added to the merged result and the Fourier fusion result to obtain the fusion result.

5. The method for end-to-end detection of small targets based on infrared dual-band images according to claim 4 is characterized in that: The process of cross-fusing the normalized processing results through the dynamic attention module is as follows: after processing the two process features 1 respectively through the MA feature extraction module and the PTA feature extraction module, process features 2 and 3 are obtained; after multiplying the process features 2 and 3 by the corresponding dynamic parameters, the two products are added to the corresponding X features and Y features respectively; the two sums are merged in the channel dimension to obtain a merged feature image; the merged feature image is mapped to a preset dimension through a multi-layer perceptron to obtain an output feature.

6. The method for end-to-end detection of small targets based on infrared dual-band images according to claim 5, characterized in that: The Fourier fusion is to first perform Fourier transform on the two summation results to obtain the corresponding amplitude component and phase component; use the amplitude component and phase component of the medium wave image as the dominant image features to obtain the real part and imaginary part of the high-order fusion feature, add the real part and imaginary part of the high-order fusion feature and then perform inverse Fourier transform to obtain the Fourier fusion processing result.

7. The method for end-to-end detection of small targets based on infrared dual-band images according to claim 1 or 4, characterized in that: The MLP mapping consists of convolution, batch normalization and ReLU activation functions.

Images