Weak and small target end-to-end detection method based on infrared dual-band image

Abstract

The invention discloses a weak and small target end-to-end detection method based on an infrared dual-band image, and the method comprises the steps: constructing a dual-band infrared weak and small target detection model; the model comprises a plurality of layers of feature fusion units which are connected in sequence, each feature fusion unit processes medium wave information parameters and long wave information parameters through two dynamic attention modules to obtain respective first process features, the two first process features are processed through a fusion module to obtain fusion features, and the fusion features are used for fusion. After the fusion feature and the result of the feature fusion unit of the upper layer are added, the addition result is processed through a dynamic attention module, and the output feature of the feature fusion unit is obtained; according to the target detection model, the Attention module is reused for multiple times, the whole detection model is simplified, the performance of the algorithm is remarkably improved, and the algorithm has good performance in dual-band infrared weak and small target detection by enhancing the global modeling capability of the model.

Description

Technical Field

[0001] The invention belongs to the field of target detection in digital image processing, and in particular is an end-to-end detection method for small and weak targets based on infrared dual-band images. Background Art

[0002] Infrared small target detection technology is widely used in various infrared search and warning systems, and plays an important role in both civilian and military fields. However, due to the long-distance shooting of the camera and the interference of noise and clutter, infrared targets often appear to have a small pixel space share and dim features, which makes it difficult to detect small infrared targets based on a single band. Compared with single-band infrared images, dual-band infrared detection has the following advantages: ① Improved anti-interference performance against complex backgrounds, infrared long waves have stronger smoke and dust penetration capabilities, so better imaging quality can be obtained; ② Long-wave infrared is more sensitive to changes in ambient temperature, and target detection is stronger in low-temperature environments; ③ Dual-band fusion can obtain richer target detail features and improve the ability to resolve small targets. Therefore, the small target detection technology based on infrared dual-bands has a broader application prospect. However, due to factors such as data acquisition, there are few related studies involving target detection and recognition tasks related to infrared dual-bands.

[0003] At present, the research on fusion detection based on multi-band images is mainly concentrated in the visible light / infrared field. According to the technical route, it can be divided into: fusion first and then detection, and end-to-end detection model. End-to-end detectors provide new possibilities for solving the problem of infrared dual-band weak target detection. However, compared with visible light / infrared detection tasks, it is not only because the target is weak and the texture scale and other features are not obvious that make detection difficult, but the imaging characteristics of the dual band will also aggravate this situation. In particular, when the target appears as a bright feature in one band and a dark feature in another band, the fusion may cause the features to cancel each other out, thereby reducing the detection performance or even causing detection failure. The inaccurate acquisition of target features and the insufficient and unreasonable fusion of multi-band image features are the main reasons for the poor performance of infrared dual-band weak target detection. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide an end-to-end detection method for small targets based on infrared dual-band images, so as to solve the problems of inaccurate acquisition of small target characteristics, insufficient and unreasonable fusion of multi-band image characteristics in the prior art.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A method for detecting small targets end-to-end based on infrared dual-band images comprises the following steps:

[0007] S1, obtaining a medium-wave infrared image and a long-wave infrared image of a small target, extracting medium-wave information parameters from the medium-wave infrared image, and extracting long-wave information parameters from the long-wave infrared image;

[0008] S2, inputting the medium wave information parameters and the long wave information parameters into the dual-band infrared small target detection model;

[0009] The dual-band infrared small target detection model includes multiple layers of sequentially connected feature fusion units. Each layer of feature fusion units processes the medium-wave information parameters and the long-wave information parameters respectively through two dynamic attention modules to obtain respective process features one. The two process features one are processed by the fusion modules respectively to obtain fusion features. The fusion features are added to the results of the feature fusion units of the previous layer, and the sum is processed by the dynamic attention module to obtain the output features of the feature fusion units. The output features of each layer of feature fusion units are the input features of the feature fusion units of the next layer. The feature fusion units of the last layer output the output features of the dual-band infrared small target detection model.

[0010] S3, after processing the output features of the dual-band infrared small target detection model through the segmentation task head, the small target is detected.

[0011] A further improvement of the present invention is:

[0012] Preferably, the number of medium-wave information parameters and the number of long-wave information parameters obtained in S1 match the number of feature fusion units in the dual-band infrared small target detection model.

[0013] Preferably, in S2, a linear encoding unit is provided before each of the feature fusion units.

[0014] Preferably, in S2, the dynamic attention module divides the input features evenly to obtain X features and Y features; the X features and the Y features are processed by the MA feature extraction module and the PTA feature extraction module respectively to obtain process features 2 and 3; the process features 2 and 3 are multiplied by the corresponding dynamic parameters respectively, and the two products are added to the corresponding X features and Y features respectively, and the two sums are merged in the channel dimension to obtain a merged feature image; the merged feature image is mapped to a preset dimension through a multi-layer perceptron to obtain an output feature.

[0015] Preferably, a module including SSM is introduced into the MA feature extraction module.

[0016] Preferably, the PTA feature extraction module first encodes the input features and normalizes the features, then samples the input feature map with an interval of S on the spatial dimension using the normalized features, obtains the summed features by adding the sampling results to the input features, normalizes the summed results through encoding, maps the processed results through MLP, and obtains process feature three by adding the MLP mapping results to the summed features.

[0017] Preferably, the fusion module processes the two process features one respectively through normalization, cross-fuses the normalization results through a dynamic attention module, and adds each fusion result to the corresponding process feature one to obtain the sum result of the medium-wave image and the sum result of the long-wave image; merges the two sum results in the channel dimension and processes them through Fourier fusion; and processes the merged result through normalization and MLP mapping in turn, and then adds the MLP mapping result, the merged result and the Fourier fusion result to obtain a fusion result.

[0018] Preferably, the process of cross-fusion of the normalized processing results through the dynamic attention module is: after processing the two process features 1 respectively through the MA feature extraction module and the PTA feature extraction module, process features 2 and 3 are obtained, after multiplying the process features 2 and 3 with the corresponding dynamic parameters respectively, the two products are added to the corresponding X features and Y features respectively, the two sums are merged in the channel dimension to obtain a merged feature image, and the merged feature image is mapped to a preset dimension through a multi-layer perceptron to obtain an output feature.

[0019] Preferably, the Fourier fusion is to first perform Fourier transform on the two sum results to obtain corresponding amplitude components and phase components; use the amplitude component and phase component of the medium wave image as the dominant image features to obtain the real and imaginary parts of the high-order fusion features, add the real and imaginary parts of the high-order fusion features, and then perform inverse Fourier transform to obtain the Fourier fusion processing result.

[0020] Preferably, the MLP mapping consists of convolution, batch normalization and ReLU activation functions.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] The present invention discloses an end-to-end detection method for small and weak targets based on infrared dual-band images. The method constructs a dual-band infrared small and weak target detection model, which fully utilizes the advantages of dual-band detection in improving target recognizability and enhancing contrast. The model design includes multiple layers of sequentially connected feature fusion units, and adopts multiple reused Attention modules, thereby simplifying the structure of the entire detection model. This design not only improves the performance of the algorithm, but also enhances the model's adaptability to complex backgrounds and environmental conditions. By enhancing the global modeling capability of the model, the algorithm demonstrates excellent performance in dual-band infrared small and weak target detection. This method fully utilizes the fusion advantages of dual-band data in real-time monitoring and dynamic target recognition, ensuring that small and weak targets can be effectively identified under a variety of climatic conditions. Overall, the present invention provides an innovative solution for the application of infrared dual-band detection technology, which can better cope with challenges in actual scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a schematic diagram of the infrared dual-band small target detection algorithm framework;

[0024] Figure 2 Schematic diagram of the dynamic attention module;

[0025] Figure 3 Schematic diagram of the Patch Transformer Attention module;

[0026] Figure 4 is a schematic diagram of the Fourier fusion module;

[0027] Figure 5 A schematic diagram of the test results. DETAILED DESCRIPTION

[0028] In the following, the terms "first", "second", "third", and "fourth" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, a feature defined as "first", "second", "third", and "fourth" may explicitly or implicitly include one or more of the features.

[0029] The co-shooting method provided in the embodiment of the present application can be applied to terminal devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), etc. The embodiment of the present application does not impose any restrictions on the specific type of the terminal device.

[0030] It should be noted that the terms "first", "second", etc. in the specification and drawings of the present invention are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] See also Figure 1 The first aspect of the present invention discloses an end-to-end detection method for small and weak targets based on infrared dual-band images. The main attributes of infrared small and weak targets are "weak" and "small". The so-called "weak" means that in the infrared image, the grayscale value of the target is lower than the surrounding background, that is, the contrast is relatively low. The so-called "small" means that the area occupied by the target on the image is small, which is specifically reflected in the image as the number of pixels occupied by the target is very small. Generally, in an image with a pixel size of 256×256, a target with a signal-to-noise ratio less than 4 and a pixel size less than 9×9 is defined as a small and weak target.

[0032] The dynamic attention module in the method of the present invention integrates the global perception of self-attention and the sequential scanning capability of SSM, which significantly improves the model's feature mining capability for multi-band images of different scales. The fusion module is used to process the feature information extracted by the Attention module from different bands, and at the same time introduces Fourier transform to fuse the phase spectrum and frequency spectrum of the two band features, and couples the fused features with the deep features to obtain the target enhancement features.

[0033] The method specifically comprises the following steps:

[0034] S1, for small targets, obtain medium-wave infrared image I mand long-wave infrared image I l , the two infrared images are feature extracted through their respective feature extraction networks, and the medium-wave infrared image I m Extract the medium wave information parameter f m , extract the long-wave information parameter f from the long-wave infrared image l ;

[0035] S2, input the extracted parameters into the dual-band infrared small target detection model (SSD) to obtain the detection results.

[0036] See also Figure 1 The left part is the dual-band infrared dim target detection model, which is State Space Model and Self-Attention for Infrared Dual Small and Dim Target Detection, referred to as SSD. The model consists of 4 layers of connected FFUs. Except for the first layer of FFU, the output of each layer of FFU is the input of the next layer of FFU. The number of FFUs is determined according to the extraction accuracy and algorithm complexity. The structure of each layer of FFU is the same. The input of the first layer FFU to the third layer FFU module includes the medium-wave information parameters of the current layer, the long-wave information parameters and the output of the previous layer (including the medium-wave information parameters and the long-wave information parameter f l i+1 ), receives three inputs, and FFU4 receives only two inputs.

[0037] See also Figure 1 The right part is an enlarged view of FFU; FFU is the abbreviation of Feature Fusion Unit, specifically the feature fusion unit, which is used to process the dual-band image features obtained through the backbone network. Each FFU includes a dynamic attention module (DAM), a fusion module (Fusion Module) and an addition module (Add). Medium-wave information parameter f m and the long-wave information parameter f l After being processed by a dynamic attention module respectively, the respective process features 1 are obtained. After the two process features 1 are fused by the fusion module, the fusion feature is obtained. The fusion feature is combined with the medium wave information parameter output by the previous layer. and the long-wave information parameter f l i+1 After adding them separately ( Figure 1 In Represents two parameters, namely medium wave information parameters and the long-wave information parameter f l i+1), the summation result is passed through the dynamic attention module again to output the medium-wave information parameters of this layer and the long-wave information parameter f l i , that is Figure 1 The two parameters in

[0038] As a preferred solution, a linear projection unit is provided before each FFU to increase the parameter scale of the network model and map the features to a high-dimensional space.

[0039] It should be noted that each layer of FFU has its own medium-wave information parameters and long-wave information parameters, that is, the medium-wave information parameters and long-wave information parameters used for each layer of FFU should be extracted from the acquired medium-wave infrared image and long-wave infrared image, which are shown in the figure. and Typically, such parameters are initialized randomly.

[0040] In some embodiments of the present invention, the dynamic attention module is as follows Figure 2 As shown in the figure, in this module, first, the input features are standardized, and the input feature map after the standardization is evenly split (BN-Split) into X features and Y features; the two features are respectively extracted by the MA (Mamba Attention) feature extraction module and the PTA (Patch Transformer Attention) feature extraction module, and the Mamba Attenion obtains the process feature 2 and the Patch Transformer Attention obtains the process feature 3.

[0041] Considering the obvious differences between the target and background characteristics in the medium and long wave images, indiscriminately processing the feature information of images in different bands may lead to inaccurate target feature extraction and excessive background noise. Therefore, dynamic parameters α and β are introduced to adjust the weights of the features processed by the Attention module. Secondly, the feature information of the target is retained to the maximum extent through the residual link, and the process features 2 and 3 obtained after different feature processing are merged again in the channel dimension (BN-Cat). In this part, process feature 2 is multiplied by α, and process feature 3 is multiplied by β. After the two multiplication results are added to the respective input X and Y, the added features are merged in the channel dimension to obtain the image after the merged features.

[0042] Finally, a multi-layer perceptron (MLP) is used to map the image after merging features to a preset dimension to obtain the corresponding process feature 1. The MLP consists of convolution, batch normalization, and ReLU activation functions.

[0043] The dynamic attention module dynamically adjusts the feature weights during the model training process, which enables the model to adaptively learn the differential characteristics of different bands to enhance the model's feature learning ability. The above process is expressed by the following formula (1):

[0044]

[0045] In the formula, f input is the input feature map, f output is the output feature map, Op split is feature map splitting, Norm is batch normalization, MLP is a multi-layer perceptron consisting of Conv→BN→ReLU, Op concat To merge on the channel, α and β are learnable parameters.

[0046] It should be understood that the dynamic parameters α and β are obtained through training during the learning process of the entire network.

[0047] In the above process, the MA module is a visual module suitable for infrared dim target feature extraction. As a new network architecture, MA introduces the state model SSM (State Space Model). This model can realize sequence-to-sequence mapping by introducing the concept of state variables, and can effectively process long sequence data, thereby enabling MA to capture long-term dependencies in the data. MA's long-distance perception ability enables the model to perceive targets in higher dimensions, rather than being limited by the bottleneck problem of convolution receptive field improvement. Compared with traditional recursive neural networks, MA has linear time complexity and can process long sequences more efficiently.

[0048] In the above process, see Figure 3In the process of data processing, Patch Transformer Attention first encodes and normalizes the input features through Normalization, and samples the input feature map with interval S in the spatial dimension with the normalized features, limiting the calculation of self-attention within a fixed patch. During the interval S sampling process, the image is divided into multiple small blocks according to the step size n, and the query vector (Query, Q), key vector (Key, K) and value vector (Value, V) are generated through different linear changes. In the "ATT" module, Q, K, and V are used to calculate the attention score, and the value vector is weightedly combined to obtain the self-attention result. The spliced ​​multi-head attention result is once again subjected to a linear transformation, and the size of the output tensor is restored to H×W×C. Compared with the Attention method directly based on Windows, interval sampling expands the scope of attention while avoiding the introduction of cyclic window attention and other calculation processes that will greatly increase the computational cost of the model. The sampling result is added to the input feature to obtain the sum feature. The sum result is processed by encoding normalization, and the processed result is processed by MLP. The feature output by MLP is added to the sum feature to obtain process feature three.

[0049] In some embodiments of the present invention, the fusion module process structure diagram is as follows Figure 4 As shown in the figure, the fusion module is a Fourier-based multi-band feature fusion module. First, the process features from different bands are normalized respectively. Based on the dynamic attention module, the dual-band features are cross-fused, and each cross-input result is added with the corresponding input through the summation module. The two added features are merged in the channel dimension. After the merged result is normalized, the image after the merged features is mapped to the preset dimension using a multi-layer perceptron, where MLP is composed of convolution, batch normalization, and ReLU activation function.

[0050] The above dynamic attention module cross-integrates the dual-band features by taking the two normalized results as Figure 2 The X and Y features in Figure 2 process to process.

[0051] In Path Transformer, the interdependence and interaction information of targets in medium-wave and long-wave infrared images are learned based on the self-attention mechanism to obtain a richer and more comprehensive target feature representation, which effectively suppresses the impact of noise on model performance. In order to fully retain the key feature information in the image, the present invention designs a feature fusion module based on Fourier transform. The above two added results are Fourier fused to obtain the Fourier fusion result.

[0052] In the Fourier fusion process, first, the dual-band image features are Fourier transformed to obtain the amplitude and phase components respectively.

[0053]

[0054] in, They represent the amplitude and phase spectral components of the features after Fourier transform, respectively.

[0055] Since the medium wave image contains more detailed information, the real and imaginary parts of the high-order fusion features are obtained based on the restoration of the dominant image features of the medium wave image, as follows:

[0056]

[0057] in are the real and imaginary parts of the fused features, respectively.

[0058] Furthermore, the features represented in the frequency domain are converted back to the image space through inverse Fourier transform.

[0059]

[0060] The mapping result, merging result and Fourier fusion result of MLP are added together through the summing module to obtain the output result of the fusion module.

[0061] S3, obtains the output binary image containing only weak targets through the segmentation task head (predict network), and outputs the detection result.

[0062] Some embodiments of the present invention also disclose a method for acquiring a dual-band infrared small target detection model, wherein the acquisition of the model comprises the following steps:

[0063] Step 1: Obtain infrared image data, preprocess the image, and construct an image dataset Data = {I1, I2, I3, ..., I n}, where I represents the infrared dual-band image pair and N represents the total number of images.

[0064] Step 2: Divide the data set described in step 1 into two groups train and Data test ,Data train As training samples, Data test As a test sample, Data train The data is labeled to obtain the labeled data label As the input of network training, Mask is a binary image containing only the target, Data label The representation is:

[0065] Data label ={(I,Mask1),...,(I n ,Mask n )}

[0066] Step 3: Build a dual-band infrared small target detection model. You can use a model trained on ImageNet, such as ResNet, Figure 1 In the feature extraction model, the input of the feature extraction model is the original dual-band image in the training set described in step 2, and the output is the dual-band image features of different scales.

[0067] Construct a multi-scale feature fusion module (Feature Fusion Unit, FFU). This network is mainly responsible for completing the adaptive fusion of dual-band image features from different levels of the image feature extraction network. This module includes a dynamic attention module and a Fourier fusion module.

[0068] An infrared dual-band small target end-to-end intelligent detection network is constructed. The dual-band features obtained in the above steps are fused using the FFU module, and finally the final intelligent detection network is obtained by combining the prediction network.

[0069] Step 4: Input the training data and labeled data in step 2 to train the target detection model and obtain the trained model for inference.

[0070] Step 5: Use the model obtained in step 4 to perform inference tests. First, extract features from the input long-wave and medium-wave infrared images respectively. Second, use a multi-layer cascaded FFU module to couple multi-band features. Finally, use the segmentation task head to obtain an output binary image containing only small targets.

[0071] The above model was used to test a dataset of weak target images with complex backgrounds. The specific results are shown in Tables 1 and Figure 1 The results show that compared with other algorithms, the algorithm proposed in this invention has a greater performance advantage in the dual-band infrared dim small target detection task.

[0072] Table 1 Algorithm performance test results

[0073]

[0074] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A small target end-to-end detection method based on infrared dual-band images, characterized in that: The following steps are involved: S1, obtaining a medium-wave infrared image and a long-wave infrared image of a small target, extracting medium-wave information parameters from the medium-wave infrared image, and extracting long-wave information parameters from the long-wave infrared image; S2, inputting the medium wave information parameters and the long wave information parameters into the dual-band infrared small target detection model; The dual-band infrared small target detection model includes multiple layers of sequentially connected feature fusion units, each layer of feature fusion units processes the medium-wave information parameters and the long-wave information parameters respectively through two dynamic attention modules to obtain respective process features 1, and the two process features 1 are processed by the fusion modules respectively to obtain fusion features, and the fusion features are added to the results of the feature fusion units of the previous layer, and the sum is processed by the dynamic attention module to obtain the output features of the feature fusion units; The output features of each layer of feature fusion units are the input features of the next layer of feature fusion units; the last layer of feature fusion units outputs the output features of the dual-band infrared small target detection model; S3, after processing the output features of the dual-band infrared small target detection model through the segmentation task head, the small target is detected.

2. The method for end-to-end detection of small targets based on infrared dual-band images according to claim 1 is characterized in that: The number of medium-wave information parameters and the number of long-wave information parameters obtained in S1 match the number of feature fusion units in the dual-band infrared small target detection model.

3. The end-to-end detection method for small and weak targets based on infrared dual-band images according to claim 1 is characterized in that: In S2, a linear encoding unit is provided before each of the feature fusion units.

4. The end-to-end detection method for small and weak targets based on infrared dual-band images according to claim 1 is characterized in that: In S2, the dynamic attention module divides the input features evenly to obtain X features and Y features; the X features and the Y features are processed by the MA feature extraction module and the PTA feature extraction module respectively to obtain process features 2 and 3; the process features 2 and 3 are multiplied by the corresponding dynamic parameters respectively, and the two products are added to the corresponding X features and Y features respectively, and the two sums are merged in the channel dimension to obtain a merged feature image; the merged feature image is mapped to a preset dimension through a multi-layer perceptron to obtain an output feature.

5. The end-to-end detection method for small and weak targets based on infrared dual-band images according to claim 4 is characterized in that: The MA feature extraction module introduces a module including SSM.

6. The end-to-end detection method for small and weak targets based on infrared dual-band images according to claim 4 is characterized in that: The PTA feature extraction module first encodes the input features and normalizes the features, then samples the input feature map with an interval of S in the spatial dimension with the normalized features, adds the sampling results to the input features to obtain the summed features, normalizes the summed results through encoding, maps the processed results through MLP, and adds the MLP mapping results to the summed features to obtain process feature three.

7. The end-to-end detection method for small and weak targets based on infrared dual-band images according to claim 1 is characterized in that: The fusion module performs normalization processing on the two process features 1 respectively, cross-fusion of the normalization processing results through the dynamic attention module, and adds each fusion result to the corresponding process feature 1 to obtain the sum result of the medium-wave image and the sum result of the long-wave image; the two sum results are combined in the channel dimension and processed by Fourier fusion; The merging result is normalized and mapped by MLP in turn, and then the MLP mapping result is added to the merging result and the Fourier fusion result to obtain the fusion result.

8. The end-to-end detection method for small and weak targets based on infrared dual-band images according to claim 7 is characterized in that: The process of cross-fusion of the normalized processing results through the dynamic attention module is as follows: after processing the two process features 1 respectively through the MA feature extraction module and the PTA feature extraction module, process features 2 and 3 are obtained, after multiplying the process features 2 and 3 with the corresponding dynamic parameters respectively, the two products are added to the corresponding X features and Y features respectively, the two sums are merged in the channel dimension to obtain a merged feature image, and the merged feature image is mapped to a preset dimension through a multi-layer perceptron to obtain an output feature.

9. The method for end-to-end detection of small targets based on infrared dual-band images according to claim 7 is characterized in that: The Fourier fusion is to first perform Fourier transform on the two summed results to obtain the corresponding amplitude component and phase component; use the amplitude component and phase component of the medium wave image as the dominant image features to obtain the real part and imaginary part of the high-order fusion feature, add the real part and imaginary part of the high-order fusion feature and then perform inverse Fourier transform to obtain the Fourier fusion processing result.

10. The method for end-to-end detection of small targets based on infrared dual-band images according to claim 6 or 7, characterized in that: The MLP mapping consists of convolution, batch normalization and ReLU activation functions.

Images