Infrared weak and small target detection method based on multi-stage series full-resolution fusion network

Abstract

The invention discloses an infrared weak and small target detection method based on a multi-stage series full-resolution fusion network, and belongs to the technical field of target tracking. A decoder of a multi-resolution dense interaction U-shaped module receives a feature map subjected to interactive attention fusion as input; the multi-resolution dense interaction U-shaped modules are connected in series to form a cascaded full-resolution fusion U-shaped network; the same-level feature maps of the preorder module are spliced by the execution channels and then are sent to the encoder of the next-level module, and are fused by using interactive attention and then are sent to the decoder of the next-level module; the encoder of each sub-network is normalized to obtain a weak and small target segmentation prediction map, SoftIoU loss is calculated with a true value, and model parameters are optimized; the interactive attention takes a plurality of to-be-fused feature maps as input, respective corresponding attention heat maps are calculated, and the to-be-fused feature maps are multiplied by the corresponding attention heat maps and then added. According to the method, the problem that the weak and small target in the infrared image is relatively weak in gray feature and texture feature and is easy to submerge is solved.

Description

Technical Field

[0001] This invention belongs to the field of target tracking technology, specifically relating to an infrared weak target detection method based on a multi-level serial full-resolution fusion network. Background Technology

[0002] Infrared detection technology utilizes the difference in infrared radiation between a target and its background to sense the object under test. It boasts advantages such as high concealment, all-weather application, and the ability to counter stealth targets to a certain extent, making it widely used in photoelectric tracking, monitoring and early warning, and exploration and surveying. However, when the signal strength of the target object is low and its proportion in the image is small, infrared detection technology tends to focus on bright backgrounds and miss small targets. This is because downsampling in the forward computation of deep neural networks destroys and dilutes the contour details of small targets, resulting in coarse segmentation. On the other hand, not using downsampling and only performing calculations on high-resolution features not only greatly increases the computational load but also leads to slow receptive field growth, a lack of global information, and false alarms. Therefore, fully integrating low-resolution and high-resolution features is of great significance for improving the detection effect of small infrared targets. The technical problem this invention aims to solve is that small targets in infrared images have low intensity and small size, resulting in indistinct features and easy submersion in the background, leading to false detections and missed detections. Summary of the Invention

[0003] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0004] An infrared weak target detection method based on a multi-level cascaded full-resolution fusion network includes:

[0005] Step 1: Construct a multi-resolution dense interactive U-shaped module. The decoder of the multi-resolution dense interactive U-shaped module receives the interactive attention... The fused feature map is used as input;

[0006] Step 2: Use a chaining layer to connect multi-resolution densely interactive U-shaped modules in series to form a cascaded full-resolution fusion U-shaped network; the chaining layer concatenates the execution channels of the same-level feature maps of the preceding modules. The encoder that feeds the code into the next module uses interactive attention on the same-level feature maps of the preceding module. The fused components are then fed into the decoder of the next-stage module; the encoder of each sub-network is... Functional normalization yields a segmentation prediction map of weak targets, and the SoftIoU loss is calculated with the true value to optimize the model parameters;

[0007] Step 3, interactive attention in Steps 1 and 2 Multiple feature maps to be fused are taken as input, and their corresponding attention heatmaps are calculated. Then, the feature maps to be fused are multiplied by their corresponding attention heatmaps and added together.

[0008] The present invention has the following beneficial effects:

[0009] This invention uses a multi-resolution dense interactive U-shaped module for feature extraction and fusion, ensuring that full-level features are applied in each level of decoding calculation, taking into account both high-resolution features rich in detail and low-resolution features that retain global information. The interactive attention module promotes the retention of key information and the suppression of irrelevant information. Moreover, this multi-resolution dense interactive U-shaped module is easy to expand and can be connected in multiple stages to deal with objects that are difficult to identify. It solves the technical problem that the gray-scale features and texture features of small targets in infrared images are weak and easily submerged in the background, and has important application value. Attached Figure Description

[0010] Figure 1 A schematic diagram of the connection path for a multi-resolution, densely interactive U-shaped module;

[0011] Figure 2 A schematic diagram showing the internal computational details of a multi-resolution, densely interactive U-shaped module;

[0012] Figure 3 The diagram shows the structure of a cascaded full-resolution fusion U-shaped network composed of multi-resolution dense interactive U-shaped modules and the internal details of the linking layer. (a) is a schematic diagram of the structure of the cascaded full-resolution fusion U-shaped network composed of multi-resolution dense interactive U-shaped modules, and (b) is a schematic diagram of the internal details of the linking layer.

[0013] Figure 4 A schematic diagram illustrating the principle of interactive attention computation;

[0014] Figure 5 The image shows the prediction results generated when the present invention is applied to the infrared small target detection task; wherein, (a) is a complex infrared small target image, (b) is a target binarized image obtained by segmenting (a) with the present invention applied, (c) is an infrared small target image under strong sunlight interference, (d) is a target binarized image obtained by segmenting (c) with the present invention applied, (e) is an infrared small target image with a background containing clouds and fog, (f) is a target binarized image obtained by segmenting (e) with the present invention applied, (g) is an infrared small target image with a background containing trees, and (h) is a target binarized image obtained by segmenting (g) with the present invention applied. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0016] This invention proposes to improve the model's capabilities by fusing information step by step through a full-resolution feature fusion path during the decoding process, designing an interactive attention mechanism to optimize the transmission of key features, and connecting multiple sub-networks to gradually generate refined predictions.

[0017] This invention provides a method for detecting weak infrared targets based on a multi-level cascaded full-resolution fusion network, comprising the following steps:

[0018] Step 1: Construct a multi-resolution dense interactive U-shaped module. The decoder of the multi-resolution dense interactive U-shaped module receives the interactive attention... The fused feature map is used as input.

[0019] like Figure 1 , Figure 2 As shown, this multi-resolution dense interactive U-shaped module consists of an encoder and a decoder. The encoder contains 6 sub-encoders, and the decoder contains 5 sub-decoders. Each sub-decoder receives the output of the previous sub-encoder as input, as well as the output of a higher-resolution encoder and the output of a lower-resolution decoder. Figure 1 This is a schematic diagram of the connection path of a multi-resolution dense interactive U-shaped module; each node represents a sub-encoder or sub-decoder at a different level. Figure 2 A schematic diagram of the internal computation details of the multi-resolution dense interactive U-shaped module (showing all encoders and first-level decoders); the internal computation details are shown in formulas (1) to (4).

[0020] The encoder employs a feedforward computation method that downsamples and extracts features step by step. The inter-layer computation process can be represented as follows:

[0021] (1)

[0022] (2)

[0023] in, Indicates the input image. Let i be the input feature map of the i-th sub-encoder. Let i be the output feature map of the i-th sub-encoder. For convolution operations, For batch normalization operations, For activation function, This is a bilinear interpolation downsampling with a scale of 0.5 (i.e., the downsampled image size is the same as the original image). The encoder calculates output feature maps at different resolutions. .

[0024] The decoder is designed to integrate multi-resolution features at each stage. This design facilitates the efficient integration of global and local information and suppresses the attenuation of key target features during propagation. The calculation process is as follows:

[0025] (3)

[0026] (4)

[0027] in, Let i be the input feature map of the i-th Conv Block. This is the output feature map of the i-th ConvBlock; It consists of three parts: (1) features upsampled from the previous level decoder; (2) features from the same level encoder; and (3) fusion features obtained by interactive attention of higher resolution encoder features and lower resolution decoder features. Indicates will The tensor sequence within is concatenated along the channel dimension (channel dimension index 1). This means that A is upsampled to the same size as B using bilinear interpolation. This means that A is downsampled to the same size as B using bilinear interpolation. This is the interactive attention layer.

[0028] Step 2: Use a chaining layer (a dedicated network layer for cascading multiple U-shaped modules, which achieves cross-module feature aggregation and transmission through channel splicing and interactive attention mechanisms) to connect the multi-resolution densely interactive U-shaped modules in series, forming a cascaded full-resolution fused U-shaped network, such as... Figure 3 As shown, the chaining layer performs channel concatenation of the sibling feature maps from the preceding modules. The encoder that feeds the code into the next module uses interactive attention on the same-level feature maps of the preceding module. The fused components are then fed into the decoder of the next-level module. The encoder of each sub-network is... Functional normalization yields a segmentation prediction map of weak targets, which is then compared with the true values ​​to calculate the SoftIoU loss (smoothing crossover ratio loss) and optimize the model parameters. Figure 3 This is a schematic diagram illustrating the structure and internal details of a cascaded full-resolution fusion U-shaped network composed of interconnected multi-resolution densely interactive U-shaped modules. Figure 3(a) is a schematic diagram of a cascaded full-resolution fusion U-shaped network composed of multi-resolution dense interactive U-shaped modules connected in series. The modules at each level are connected by a link layer at the same resolution. Figure 3 (b) is a schematic diagram showing the internal details of the bonding layer. The channel splicing is shown in formula (4). Description; interactive attention is described in equations (10) to (12). describe.

[0029] The structure of the first multi-resolution dense interactive U-shaped module is consistent with that described in step 1. The input of the i-th level encoder of the k-th multi-resolution dense interactive U-shaped module is:

[0030]

[0031] in, , For the input and output of the i-th level encoder of the k-th multi-resolution dense interactive U-shaped module, This is the output of the i-th level decoder of the k-th multi-resolution dense interactive U-shaped module; , This is a feature map of the preceding modules in the chain layer aggregation.

[0032] The input of the encoder contains features of the preceding sub-network, which facilitates information flow. The calculation process of the input feature map by each sub-encoder is described in formula (1).

[0033] Input of the i-th level decoder of the k-th multi-resolution dense interactive U-shaped module for:

[0034] (6)

[0035] Compared to step 1, the cascaded encoder increases the connection paths with the pre-sequence encoder and decoder through the chaining layer, and applies interactive attention to the corresponding feature maps. Calculate and use as input.

[0036] for:

[0037] (7).

[0038] The feature map output by the i-th level encoder of the k-th multi-resolution dense interactive U-shaped module is convolved with a size of 1×1 and... Function normalization yields the segmentation prediction map for weak targets. During training, the segmentation prediction map for each level of weak targets is calculated. With real images SoftIoU loss This creates dense relay supervision, suppressing gradient diffusion.

[0039] (8)

[0040] (9)

[0041] in, for and The feature map obtained by element-wise dot product Let d be the scalar obtained by summing every element of the input tensor, and d be the hyperparameter for smoothing the gradient.

[0042] Figure 4 This is a schematic diagram of the interactive attention calculation principle; calculation details are shown in formulas (10) to (12), and channel summation is shown in formula (12). The description and channel splicing are shown in formula (4). The description and channel separation are shown in formula (12). The descriptions of convolution, channel normalization, and activation functions are given in Formula (1).

[0043] Step 3, as follows Figure 4 As shown, the interactive attention mentioned in steps 1 and 2 Multiple feature maps to be fused are taken as input. After calculating the corresponding attention heatmaps for each feature map, the feature maps to be fused are multiplied by their respective attention heatmaps and then summed. The calculation details are as follows:

[0044] (10)

[0045] (11)

[0046] (12)

[0047] In equation (10), This represents n feature maps of the same shape used to compute interactive attention. This represents n attention heatmaps (with 1 channel) multiplied by n feature maps of the same shape. In equation (11), for one, The value ranges from 0 to n; express The fusion feature is that the number of input and output channels of the convolutional layer is n. This indicates that exponential normalization is performed on the channel dimension (channel dimension index 1). This represents taking the sub-feature map of the j-th channel of the feature map. In equation (12), This indicates that the feature maps are added along the channel dimension (channel dimension number 1).

[0048] The embodiments of the present invention are described below in conjunction with the model structure and calculation process.

[0049] The data processed in this example is infrared images. To verify the effectiveness of the infrared small target detection method based on a multi-level cascaded full-resolution fusion network of the present invention, training and testing were carried out on datasets such as the single-frame infrared small target dataset (NUAA-SIRST) released by Nanjing University of Aeronautics and Astronautics.

[0050] Step 1, based on Figures 1 to 3 The network structure was constructed, and the number of output channels for each module was set to 64 in this experiment. This value can be flexibly adjusted according to the complexity of the task and the hardware level. The network parameters were initialized. The training images were converted into single-channel grayscale images, cropped to 256×256, and then fed into the model after data augmentation methods such as flipping and rotation.

[0051] Step 2: According to equations (1)-(5), the first module outputs the predicted image of small target segmentation. Then, the predictions and feature maps of the previous multi-resolution dense interactive U-shaped module are used as inputs to the next multi-resolution dense interactive U-shaped module (see Equations (8)-(12)) until all target segmentation prediction maps are obtained. The loss between the prediction and the true value of each level of all multi-resolution dense interactive U-shaped modules is calculated according to Equations (6)-(7) to optimize the model parameters.

[0052] Step 3: During inference, the pre-trained model parameters are loaded first. The input image is replaced with a single-channel grayscale image, cropped to 256×256, and then fed into the network model. The calculation process in Step 2 is executed, and the prediction from the highest resolution decoder of the last stage of the network is taken, binarized with a threshold of 0.7, and used as the output.

[0053] Figure 5 This invention is used to generate prediction result images for infrared weak target detection tasks; wherein, Figure 5 (a) is a complex infrared small target image. Figure 5 (b) is the application of the present invention to the target binarized image obtained by segmentation in (a). Figure 5 Image (c) is an infrared image of a small target under strong sunlight interference. Figure 5 (d) represents the target binarized image obtained by applying the present invention to the segmentation in (c). Figure 5(e) is an infrared image of a small target against a background of clouds and fog. Figure 5 (f) represents the target binarized image obtained by applying the present invention to the segmentation in (e). Figure 5 (g) is an infrared image of a small target with a background of trees. Figure 5 (h) represents the target binarized image obtained by applying the present invention to (g) segmentation. The above results show that the present invention can effectively identify small-sized, low-signal-intensity targets in various complex scenes.

[0054] The above description is merely an embodiment of the present invention and does not limit the scope of the invention. Any equivalent structural or procedural transformations made based on the description and drawings of this invention, or direct or indirect applications in other related system fields, are similarly included within the protection scope of this invention. Contents not described in detail in this specification are prior art known to those skilled in the art.

Claims

1. A method for detecting weak infrared targets based on a multi-level cascaded full-resolution fusion network, characterized in that, include: Step 1: Construct a multi-resolution dense interactive U-shaped module. The decoder of the multi-resolution dense interactive U-shaped module receives the interactive attention... The fused feature map is used as input; Step 2: Use a chaining layer to connect multi-resolution densely interactive U-shaped modules in series to form a cascaded full-resolution fusion U-shaped network; the chaining layer concatenates the execution channels of the same-level feature maps of the preceding modules. The encoder that feeds the code into the next module uses interactive attention on the same-level feature maps of the preceding module. The fused data is fed into the decoder of the next module; the encoder of each sub-network is normalized to obtain a weak target segmentation prediction map, and the SoftIoU loss is calculated with the true value to optimize the model parameters; Step 3, interactive attention in Steps 1 and 2 Multiple feature maps to be fused are taken as input, and their corresponding attention heatmaps are calculated. Then, the multiple feature maps to be fused are multiplied by their corresponding attention heatmaps and then added together.

2. The infrared weak target detection method based on a multi-level serial full-resolution fusion network according to claim 1, characterized in that, In step 1, the multi-resolution dense interactive U-shaped module consists of two parts: an encoder and a decoder. The encoder contains 6 levels of sub-encoders, and the decoder contains 5 levels of sub-decoders. In addition to receiving the output of the previous sub-encoder as input, each sub-decoder also receives the output of the higher resolution encoder and the output of the lower resolution decoder.

3. The infrared weak target detection method based on a multi-level serial full-resolution fusion network according to claim 2, characterized in that, In step 1, the encoder adopts a feedforward calculation method that downsamples and extracts features step by step. The inter-layer calculation process is as follows: （1） （2） in, For the input image, Let i be the input feature map of the i-th sub-encoder. Let i be the output feature map of the i-th sub-encoder. For convolution operations, For batch normalization operations, For activation function, The model uses bilinear interpolation downsampling with a scale of 0.5; the encoder then calculates output feature maps at different resolutions. .

4. The infrared weak target detection method based on a multi-level serial full-resolution fusion network according to claim 3, characterized in that, Step 1 includes: designing the decoder to integrate multi-resolution features at each stage: （3） （4） in, Let i be the input feature map of the i-th Conv Block. This is the output feature map of the i-th Conv Block; It includes: features upsampled from the previous level decoder, features from the same level encoder, and fused features obtained by interactive attention of higher resolution encoder features and lower resolution decoder features; To be The tensor sequence within is concatenated along the channel dimension, with the channel dimension index being 1; To upsample A to the same size as B using bilinear interpolation, To downsample A to the same size as B using bilinear interpolation, This is the interactive attention layer.

5. The infrared weak target detection method based on a multi-level serial full-resolution fusion network according to claim 4, characterized in that, In step 2, the structure of the first multi-resolution dense interactive U-shaped module is consistent with that described in step 1, and the input of the i-th level encoder of the k-th multi-resolution dense interactive U-shaped module is: in, , For the input and output of the i-th level encoder of the k-th multi-resolution dense interactive U-shaped module, This is the output of the i-th level decoder of the k-th multi-resolution dense interactive U-shaped module; , This is a feature map of the preceding modules in the chain layer aggregation.

6. The infrared weak target detection method based on a multi-level serial full-resolution fusion network according to claim 5, characterized in that, In step 2, the i-th level decoder input of the k-th multi-resolution dense interactive U-shaped module is... for: （6） in, This is the i-th level encoder that is fed into the k-th multi-resolution dense interactive U-shaped module via the linking layer.

7. The infrared weak target detection method based on a multi-level serial full-resolution fusion network according to claim 6, characterized in that, Step 2 includes: the feature map output by the i-th level encoder of the k-th multi-resolution dense interactive U-shaped module, which is then convolved with a 1×1 layer and... Function normalization yields the segmentation prediction map for weak targets. During training, the segmentation prediction map for each level of weak targets is calculated. With real images SoftIoU loss This creates dense relay supervision, suppressing gradient diffusion.

8. The infrared weak target detection method based on a multi-level serial full-resolution fusion network according to claim 1, characterized in that, In step 2, the encoder of each sub-network is... Function normalization yields the segmentation prediction map for weak targets. for: （7）。 9. The infrared weak target detection method based on a multi-level serial full-resolution fusion network according to claim 8, characterized in that, In step 2, the segmentation prediction map for each level of weak targets... With real images SoftIoU loss for: （8） （9） in, for and The feature map obtained by element-wise dot product Let d be the scalar obtained by summing every element of the input tensor, and d be the hyperparameter for smoothing the gradient.

10. The infrared weak target detection method based on a multi-level cascaded full-resolution fusion network according to claim 1, characterized in that, Step 3 includes: multiplying the feature maps to be fused by their corresponding attention heatmaps and then summing them. （10） （11） （12） In equation (10), To compute n feature maps of the same shape for interactive attention, This is to generate n attention heatmaps that are multiplied with n feature maps of the same shape; In equation (11), for one, The value ranges from 0 to n; for The fusion feature is that the number of input and output channels of the convolutional layer is n. To perform exponential normalization along the channel dimension, the channel dimension index is 1. To extract the sub-feature map of the j-th channel from the input feature map; In equation (12), To add the input feature maps along the channel dimension, the channel dimension number is 1.

Images