Infrared dim small target detection method based on space-time feature fusion network

Abstract

The present application relates to the technical field of infrared weak small target detection, and attention guiding mechanism is realized in two dimensions of time and space to fully tap the space-time characteristics in infrared image sequence. The present application is mainly applied to design and manufacturing occasions. The technical scheme adopted by the present application is an infrared weak small target detection method based on a space-time feature fusion network. A space-time feature fusion infrared weak small target detection network model STNet takes adjacent frame infrared images as input, sends them into an STNet backbone network composed of a spatial attention unit SAU to extract visual features, finally sends them into a time sequence guided Transformer structure to complete space-time feature fusion, and finally generates a prediction of the detection result. The STNet backbone network composed of the spatial attention unit SAU refers to in-situ replacement of the convolutional layer of the STNet backbone network by using the SAU. The present application is mainly applied to infrared weak small target detection occasions.

Description

Technical Field

[0001] This invention relates to a deep learning-based target detection method for infrared weak target detection, applicable to various missions such as long-range infrared early warning, anti-drone, and coastal surveillance, and suitable for different platforms including airborne, shipborne, missile-borne, and ground-based systems. Specifically, it relates to an infrared weak target detection method based on a spatiotemporal feature fusion network. Background Technology

[0002] In the application of infrared detection equipment, due to the relatively large distance between the detected target and the equipment, and limited by the aperture of the optical system and atmospheric attenuation in the propagation path, the infrared radiation signal received by the infrared detector from the target is relatively weak. The target typically occupies only a few to a dozen pixels in the infrared detector's image, lacking texture information suitable for identification. Furthermore, unlike visible light cameras that can simultaneously acquire spectral information across the red, green, and blue bands of a target, conventional infrared detectors can only image a specific single infrared band, lacking multispectral characteristics. Therefore, infrared detection equipment needs to identify weak infrared targets with low signal-to-noise ratios, few features, and small areas. In existing single-frame infrared image processing, the limited feature information of the target inevitably leads to false alarms and missed alarms. However, in the application of infrared detection equipment, skilled operators can accurately and effectively identify targets of interest from infrared images. This is because, through long-term experience, operators can distinguish between real and false targets based on the target's motion trajectory, shape changes, and brightness variations. Therefore, efficiently utilizing the spatiotemporal characteristics of the target has become the key to solving the problem of identifying weak infrared targets. With the development of deep neural network technology, especially the emergence of network structures such as LSTM and Transformer, efficient processing of sequential signals has become possible. The Transformer structure originated in the field of natural language processing and was later extended to visual processing tasks. With its introduced self-attention mechanism, the Transformer structure overcomes the limitation of RNN networks in parallel computation, producing more interpretable network structures and becoming a novel approach to solving sequence problems. The self-attention mechanism addresses how, after sequential features are input into the neural network, the network learns and accumulates knowledge to establish a connection between the current vector and subsequent vectors, enabling the model to converge quickly and achieve the desired effect. To this end, the self-attention model introduces a retrieval matrix Q, a key matrix K, and a numerical matrix V, such as... Figure 1As shown, each input feature vector is computed by three independent MLP networks to obtain three vectors: Q, K, and V, and the feature output after attention computation is calculated. The Transformer network follows the encoder-decoder design, stacking multiple self-attention structures in the encoder and decoder, using features as input to the encoder, and finally outputting the prediction result. In existing visual processing tasks, the Transformer network mainly divides the entire frame image into several image blocks, feeding the image blocks and the corresponding encoded information into the encoder-decoder network structure to generate predictions of the target's position and type. This approach treats a single frame image as a sequence, lacking understanding and learning of the temporal sequence of images composed of single frames. This inevitably affects the solution of the infrared weak target recognition problem, which relies on target trajectory, shape changes, and energy changes for identification. Furthermore, existing Transformer network structures mainly use general convolutional neural network structures to extract the visual features of targets, without designing specific model styles for the infrared weak target recognition problem to better suit the target characteristics of infrared weak targets. This also affects the performance of the recognition network. Summary of the Invention

[0003] To overcome the shortcomings of existing technologies and fully exploit the spatiotemporal characteristics of infrared image sequences, this invention aims to propose two network structures: Spatial Attention Units (SAUs) and a temporally guided Transformer. Utilizing a feature extraction backbone network composed of SAUs and a temporally guided Transformer structure, an end-to-end spatiotemporal feature fusion infrared weak target detection network (STNet) is constructed. The technical solution adopted in this invention is an infrared weak target detection method based on a spatiotemporal feature fusion network. The STNet model takes adjacent frames of infrared images as input, feeds them into the STNet backbone network composed of SAUs to extract visual features, and finally feeds them into the temporally guided Transformer structure to complete the spatiotemporal feature fusion, ultimately generating a prediction of the detection result. The STNet backbone network composed of SAUs refers to the in-situ replacement of the STNet backbone network's convolutional layers using SAUs.

[0004] Among them, SAU adds deformable convolutional layers. The deformable convolutional kernels have a variable position offset during the operation, so the receptive field of the convolution operation can have scale transformation and rotation capabilities.

[0005] SAU first uses two dilated convolutions and one deformable convolution to extract multi-scale features, with the kernel size set to 3×3 for all convolutions and the dilation rates set to 2 and 4 respectively. The feature maps after convolution are merged into a unified tensor. To keep the feature dimension unchanged, a 1×1 convolution is used for dimensionality reduction to keep it consistent with the input dimension. At the same time, SAU adds a directly connected path, in which the feature map input to SAU is added to the feature map after the 1×1 convolution to alleviate the gradient vanishing problem caused by the increase in model depth. In order to meet the channel number requirement, when the number of input and output channels of SAU is inconsistent, a 1×1 convolution is added to the directly connected path.

[0006] The ResNet-18 network was selected as the original configuration to form the ResNet-SAU-18 model. The ResNet-SAU-18 model uses a 7×7 convolutional layer and a max pooling layer as the first convolutional group Conv_1. The SAU unit replaces the convolutional layers Conv_2, Conv_3, Conv_4, and Conv_5 in the ResNet-18 in sequence. The output 7×7×512 dimensional features are unfolded and used as 49×512 dimensional features as visual features. The adjacent two frames of infrared images It-1 and It are fed into the ResNet-SAU-18 structure to obtain the corresponding visual features Ft-1 and Ft.

[0007] The Transformer structure uses three MLP networks to obtain the retrieval matrix Q, key matrix K, and content matrix V from the input features Fin, and uses formula (1) to obtain the features after self-attention:

[0008]

[0009] In order to constrain the numerical value of the features, the feature dimension d is introduced. k Divide the product of Q and KT by Utilizing the spatial features f of image sequences s As inputs to the K and V matrices, the time feature f t As input to the Q-matrix, the modified self-attention mechanism is represented as:

[0010]

[0011] The time-guided Transformer encoder and decoder were set to 3 layers. The output features of the decoder were used as the input of the MLP to obtain the prediction of the target type and location. Each MLP network predicted the location box. In STNet, 20 MLP networks were connected after the Transformer encoder. The number of layers of a single MLP was set to 3, and the number of hidden layers in the middle were 512, 256 and 128 respectively.

[0012] The training process of the STNet model is as follows:

[0013] Step 1: Select image frame It and its previous frame It-1 from the infrared image sequence. The two images are fed into the backbone network composed of SAU. The backbone network structure is based on ResNet-18, and some convolutional layers in ResNet-18 are replaced with SAU units to form a ResNet-SAU-18 model. It and It-1 are input into the ResNet-SAU-18 model at a resolution of 224×224. The output 49×512 dimension features are unfolded as the visual features Ft and Ft-1 of the two images.

[0014] Step 2: Calculate the time series feature f t and spatial features f s Where, ft = Ft - Ft-1, f s =F t +F code F code For position encoding;

[0015] Step 3: Put f t and f s The temporally guided Transformer structure is fed in, with spatial features fs serving as input to the encoder's key matrix K and content matrix V, while temporal features f t As input to the retrieval matrix Q in the encoder and decoder, the predicted target detection result Y is obtained. p =[y p1 ,y p2 ,...,y p20 ]T;

[0016] Step 4: Use the Hungarian algorithm to predict the result Y. p and the true value Y gt The matching forms the predicted result Y after pairing. p′ =[y p1 ,y p2 ,...,y pn ]T;

[0017] Step 5: Using Y gt and Y p′ Calculate the loss function Here, is the category loss, and is the cross-entropy loss value between the predicted category and the true category. The L1 norm is calculated from the horizontal center point x, vertical center point y, width w, height h of the predicted box and the parameters of the corresponding box in the true value.

[0018] Step 6: Calculate the value based on the loss function Calculate the gradient and update the model parameters.

[0019] The target prediction process of the STNet model is as follows: Step 1: Select the image frame I' to be detected. t And its previous frame I' t-1 Input STNet, if the current frame is the first frame, then I' t-1 =I' t The prediction results of 20 MLP networks were obtained;

[0020] Step 2: Determine the probability P corresponding to the category among the 20 predicted results. label＝i If the value is greater than the set threshold θ, the predicted bounding box result of the MLP network is output, and the detected bounding box is identified as the location of the i-th type of target.

[0021] The features and beneficial effects of this invention are:

[0022] This invention employs a Spatial Attention Unit (SAU) and a temporally guided Transformer structure to realize a spatiotemporal feature fusion network framework (STNet) for infrared weak target detection. Its advantages are mainly reflected in: (1) Novelty: It fully considers the spatiotemporal characteristics of infrared weak targets and proposes for the first time a Spatial Attention Unit (SAU) structure that can be applied to the extraction of spatial visual features of infrared weak targets. Combined with the temporally guided Transformer structure, a spatiotemporal feature fusion infrared weak target detection network (STNet) is constructed. (2) Effectiveness: Subjective and objective experiments have proven that, compared with some current infrared weak target detection algorithms, the spatiotemporal feature fusion infrared weak target detection network model designed in this invention has relatively improved performance and is therefore more suitable for the infrared weak target detection problem. (3) Practicality: It is simple and feasible. The infrared weak target detection problem solved by this invention has wide applications in infrared early warning, threat perception, coastal surveillance, and anti-drone warfare. Attached image description:

[0023] Figure 1 This is a schematic diagram of the self-attention mechanism.

[0024] Figure 2 This is a schematic diagram of the spatiotemporal feature fusion infrared weak target detection network model.

[0025] Figure 3 This is a comparison of convolution operations of different forms.

[0026] Figure 4 This is a schematic diagram of the spatial attention unit structure.

[0027] Figure 5 This is a schematic diagram of the ResNet-SAU-18 network structure.

[0028] Figure 6 This is a schematic diagram of a timing-guided Transformer structure. Detailed Implementation

[0029] For small infrared targets, the infrared radiation energy is higher than that of the background. How to design a network structure that can focus on local contrast differences is a problem that the spatial attention unit needs to solve. To this end, two types of convolutional layers are introduced: dilated convolution and deformable convolution. Unlike traditional convolution, dilated convolution adds a dilation coefficient during the convolution operation, such as... Figure 3 As shown in (b), the convolution kernel performs convolution operations with the feature map data of the previous layer at certain intervals based on the dilation coefficient. This increases the receptive field of the convolutional layer without increasing computation. Simultaneously, by controlling different dilation coefficients, the contrast difference between the target center point and surrounding pixels can be fully calculated, which better matches the data distribution characteristics of infrared weak targets. Furthermore, to address the possibility of irregular shape changes in infrared weak targets, SAU adds deformable convolutional layers, such as... Figure 3 As shown in (c), the deformable convolution kernel adds a variable positional offset during the operation, thus enabling the receptive field of the convolution operation to have scaling and rotation capabilities, which can better adapt to feature extraction of irregularly shaped objects. Specifically, as Figure 4As shown, SAU first uses two dilated convolutions and one deformable convolution to extract multi-scale features, with the kernel size set to 3×3 for all convolutions and the dilation rates set to 2 and 4 respectively. The feature maps after convolution are merged into a unified tensor. To maintain the feature dimension, a 1×1 convolution is used for dimensionality reduction to match the input dimension. Simultaneously, SAU inherits the "residual" concept from ResNet by adding a directly connected path. In this directly connected path, the feature map input to SAU is added to the feature map after the 1×1 convolution, thus mitigating the gradient vanishing problem caused by increased model depth. To meet the channel count requirement, when the input and output channel counts of SAU are inconsistent, a 1×1 convolution is added to the directly connected path. Regarding the backbone network, SAU is suitable for various mainstream convolutional neural networks such as VGG, GoogleNet, and ResNet. SAU can be used to achieve in-situ replacement of convolutional layers. To balance detection performance and detection speed, this invention selects the ResNet-18 network as the original configuration to form the ResNet-SAU-18 model. Figure 5 As shown, the ResNet-SAU-18 model uses a 7×7 convolutional layer and a max-pooling layer as the first convolutional group Conv_1, where, Figure 5 In this diagram, S represents the convolution stride, P represents the number of padding elements, and C represents the number of channels. The SAU unit sequentially replaces the convolutional layers Conv_2, Conv_3, Conv_4, and Conv_5 in the ResNet-18 architecture, unfolding the output 7×7×512 dimensional features and using 49×512 dimensional features as the visual features. Two adjacent infrared images, It-1 and It, are fed into the ResNet-SAU-18 structure to obtain the corresponding visual features Ft-1 and Ft. After passing through the ResNet-SAU-18 backbone network, the infrared images can fully extract the spatial structure information of weak targets. To explore the temporal correlation of sequential images and improve target detection capabilities, a time-guided Transformer structure is adopted to fuse the spatial structure information of adjacent frames in the sequence, obtaining the final predicted target recognition result. The existing Transformer structure generates attention points for sequence information through a self-attention mechanism. The Transformer structure uses three MLP networks to obtain the retrieval matrix Q, key matrix K and content matrix V from the input feature Fin, and uses formula (1) to obtain the features after self-attention.

[0030]

[0031] In order to constrain the numerical value of the features, the feature dimension d is introduced. k Divide the product of Q and KT by To incorporate the temporal characteristics of target trajectories and energy changes in adjacent frames of an infrared image sequence, the architecture of the Transformer structure, which derives the Q, K, and V matrices from the same feature, was modified. Figure 3 As shown, the spatial features f of the image sequence are utilized. s As inputs to the K and V matrices, the time feature f t As input to the Q matrix, this achieves the effect of improving the temporal features f t Guided by the search, the network pays more attention to spatial features. s The key areas are then represented by the modified self-attention mechanism as follows:

[0032]

[0033] To reduce model complexity, this invention uses a time-guided Transformer encoder and decoder with 3 layers. The decoder's output features are then used as input to an MLP to predict the target type and location. Each MLP network predicts the bounding box. In STNet, the Transformer encoder is followed by 20 MLP networks, meaning a maximum of 20 bounding boxes can be predicted for a single image. Each MLP has 3 layers, with 512, 256, and 128 hidden layers respectively.

[0034] Within the overall detection framework, this invention employs the following... Figure 2 The spatiotemporal feature fusion infrared weak target detection network model (STNet) shown takes adjacent frames of infrared images as input, feeds them into a backbone network composed of spatial attention units (SAU) to extract visual features, and finally feeds them into a temporally guided Transformer structure to complete the spatiotemporal feature fusion and generate a prediction of the detection result.

[0035] The training process of the STNet model is as follows:

[0036] Step 1: Select image frame It and its preceding frame It-1 from the infrared image sequence. These two images are fed into a backbone network composed of SAU units. The backbone network structure is based on ResNet-18, with some convolutional layers replaced by SAU units to form a ResNet-SAU-18 model, as shown below. Figure 5 As shown, It and It-1 are input into the ResNet-SAU-18 model at a resolution of 224×224, and the output 49×512 dimensional features are expanded as the visual features Ft and Ft-1 of the two frames.

[0037] Step 2: Calculate the time series feature f t and spatial features f s Where, ft = Ft - Ft-1, fs =F t +F code F code For position encoding.

[0038] Step 3: Put f t and f s The timing-guided Transformer structure proposed in this invention is fed in, such as... Figure 6 As shown, spatial features fs serve as the input to the encoder's key matrix K and content matrix V, while temporal features f t As input to the retrieval matrix Q in the encoder and decoder, the predicted target detection result Y is obtained. p =[y p1 ,y p2 ,...,y p20 ]T.

[0039] Step 4: Use the Hungarian algorithm to predict the result Y. p and the true value Y gt The matching forms the predicted result Y after pairing. p′ =[y p1 ,y p2 ,...,y pn ]T.

[0040] Step 5: Using Y gt and Y p′ Calculate the loss function Here, is the category loss, and is the cross-entropy loss value between the predicted category and the true category. The loss for the predicted bounding box is calculated by taking the horizontal center point x, vertical center point y, width w, and height h of the predicted bounding box and comparing them with the parameters of the corresponding bounding box in the ground truth. Step 6: Calculate the value based on the loss function. Calculate the gradient and update the model parameters.

[0041] The target prediction process of the STNet model is as follows: Step 1: Select the image frame I' to be detected. t And its previous frame I' t-1 Input STNet, if the current frame is the first frame, then I' t-1 =I' t The prediction results of 20 MLP networks were obtained.

[0042] Step 2: Determine the probability P corresponding to the category among the 20 predicted results. label＝i If the value is greater than a set threshold θ, the predicted bounding box of the MLP network is output, and the detected bounding box is identified as the location of the i-th type of target.

[0043] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting weak infrared targets based on a spatiotemporal feature fusion network, characterized in that, The STNet infrared weak target detection network model, which integrates spatiotemporal features, takes adjacent frames of infrared images as input and feeds them into the STNet backbone network composed of spatial attention units (SAU) to extract visual features. Finally, it feeds them into a temporally guided Transformer structure to complete the spatiotemporal feature fusion and generate a prediction of the detection result. The STNet backbone network composed of spatial attention units (SAU) refers to the use of SAU to achieve in-situ replacement of the convolutional layers of the STNet backbone network. The temporally guided Transformer structure specifically refers to the Transformer structure using three MLP networks to obtain the retrieval matrix Q, the key matrix K, and the content matrix V from the input feature K. The features after self-attention are obtained using formula (1). Attention(Q, K, V) = Softmax( )V (1) In order to constrain the numerical value of features, the feature dimension is introduced. Divide the product of Q and K by Utilizing the spatial features of image sequences 𝑠 As inputs to the K and V matrices, the time feature 𝑓 𝑡 As input to the Q-matrix, the modified self-attention mechanism is represented as: Attention(Q, K, V) = Softmax( ) ; The time-guided Transformer encoder and decoder used are set to have 3 layers. The output features of the decoder are used as the input of the MLP to obtain the prediction of the target type and location. Each MLP network predicts the location box. In STNet, 20 MLP networks are connected after the Transformer encoder. The number of layers of a single MLP is set to 3, and the number of hidden layers in the middle are 512, 256 and 128 respectively.

2. The infrared weak target detection method based on spatiotemporal feature fusion network as described in claim 1, characterized in that, SAU adds deformable convolutional layers, and the deformable convolutional kernels have a variable position offset during the operation, so the receptive field of the convolution operation can have scale transformation and rotation capabilities.

3. The infrared weak target detection method based on spatiotemporal feature fusion network as described in claim 1, characterized in that, SAU first uses two dilated convolutions and one deformable convolution to extract multi-scale features, with the kernel size set to 3×3 for all convolutions and the dilation rates set to 2 and 4 respectively. The feature maps after convolution are merged into a unified tensor. To keep the feature dimension unchanged, a 1×1 convolution is used for dimensionality reduction to keep it consistent with the input dimension. At the same time, SAU adds a directly connected path, in which the feature map input to SAU is added to the feature map after the 1×1 convolution to alleviate the gradient vanishing problem caused by the increase in model depth. To meet the channel number requirement, when the number of input and output channels of SAU is inconsistent, a 1×1 convolution is added to the directly connected path.

4. The infrared weak target detection method based on spatiotemporal feature fusion network as described in claim 1, characterized in that, The ResNet-18 network was selected as the original configuration to form the ResNet-SAU-18 model. The ResNet-SAU-18 model uses a 7×7 convolutional layer and a max pooling layer as the first convolutional group Conv_1. The SAU unit replaces the convolutional layers Conv_2, Conv_3, Conv_4, and Conv_5 in the ResNet-18 in sequence. The output 7×7×512 dimensional features are unfolded and used as 49×512 dimensional features as visual features. The adjacent two frames of infrared images 𝐼𝑡−1 and 𝐼𝑡 are fed into the ResNet-SAU-18 structure to obtain the corresponding visual features 𝐹𝑡−1 and 𝐹𝑡.

5. The infrared weak target detection method based on spatiotemporal feature fusion network as described in claim 1, characterized in that, The training process of the STNet model is as follows: Step 1: Select image frame It and its preceding frame It−1 from the infrared image sequence. The two images are fed into a backbone network composed of SAUs. The backbone network structure is based on ResNet-18, and some convolutional layers in ResNet-18 are replaced with SAU units to form a ResNet-SAU-18 model. It and It−1 are input into the ResNet-SAU-18 model at a resolution of 224×224. The output 49×512 dimensional features are expanded as the visual features It and It−1 of the two images. Step 2: Calculate the time series feature 𝑓 𝑡 and spatial features 𝑠 , where, 𝑓gna = 𝐹gna -𝐹gna−1, 𝑓 𝑠 = 𝐹 𝑡 +𝐹 𝑐𝑜d𝑒 , 𝑐𝑜d𝑒 For position encoding; Step 3: Place the 𝑓 𝑡 and 𝑓 𝑠 The temporally guided Transformer structure is fed in, with spatial features fs serving as input to the encoder's key matrix K and content matrix V, while temporal features f t As input to the retrieval matrix Q in the encoder and decoder, the predicted target detection result Y is obtained. p = [y p1 , y p2 , . . . , y p20 T; Step 4: Use the Hungarian algorithm to predict the result Y. p and the true value Y gt The matching results in a predicted outcome after pairing. 𝑝 ′ = [y p1 , y p2 , . . . , y pn T; Step 5: Using Y gt and 𝑌 𝑝 ′ Calculate the loss function 𝓛= 𝓛 label + α𝓛 box , label Let be the category loss, and let be the cross-entropy loss value between the predicted category and the true category. box The prediction box loss is calculated using the horizontal center point x, vertical center point y, width w, height h of the prediction box, and the parameters of the corresponding box in the true value, along with the 1 norm. Step 6: Calculate the value φ based on the loss function, calculate the gradient, and update the model parameters.

6. The infrared weak target detection method based on spatiotemporal feature fusion network as described in claim 1, characterized in that, The target prediction process of the STNet model is as follows: Step 1: Select the image frame 'F' to be detected. 𝑡 And the previous frame' 𝑡−1 Input STNet, if the current frame is the first frame, then 𝐼' 𝑡−1 = 𝐼' 𝑡 The prediction results of 20 MLP networks were obtained; Step 2: Determine the probability of the corresponding category among the 20 predicted results. l𝑎𝑏𝑒l=𝑖 If the value is greater than the set threshold θ, the predicted bounding box of the MLP network is output, and the detected bounding box is identified as the location of the i-th type of target.

Images