An infrared dim small target detection method based on joint Swin and space-time interaction neural network

Abstract

The application discloses an infrared dim small target detection method based on a joint Swin and space-time interaction neural network, and the method is based on in-depth research on an existing infrared dim small target detection method, converts a to-be-detected image into a time image to obtain time channel features, converts the to-be-detected image into a low-dimensional image, and obtains features containing target detection information after fusing the low-dimensional image and the time channel features. The method captures the spatial information of the image through a Swin-Transformer, and greatly improves the detection capability of the algorithm in a complex scene.

Description

Technical Field

[0001] This invention belongs to the field of target detection in digital image processing, specifically relating to an infrared weak target detection method based on a joint Swing and spatiotemporal interactive neural network. Background Technology

[0002] In modern warfare, airborne, shipborne, and land-based infrared search and track systems play a crucial role in enhancing the autonomous defense of weapon systems and the battlefield survivability of key combat units by detecting and warning of low-altitude aircraft such as UAVs, loitering munitions, and cruise missiles as early as possible. Among these systems, the detection and identification of weak targets in complex environments is a key technology for evaluating system performance. Existing algorithms exhibit reduced detection performance in complex environments, making it difficult to meet operational mission requirements. In particular, the detection of weak targets in low-altitude complex environments has become a major challenge and bottleneck restricting the overall performance of the system.

[0003] Specifically, when infrared search and track systems detect low-altitude aircraft, objects such as ground rocks, farmland, roads, and buildings appear as bright areas under light reflection. In scenarios such as forest gaps or object occlusion, these objects easily create false targets similar to the target. Existing traditional algorithms, such as local contrast analysis, spatial filtering, and sparse matrix factorization—model-driven methods—cannot solve problems related to complex scenes and small targets. While existing deep learning-based infrared weak target detection algorithms still need improvement in some complex scenes, they fail to fully utilize the correlation information between video frames during temporal detection, simply treating the sequence detection task as a superposition of single-frame detections. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide an infrared weak target detection method based on a combined Swing and spatiotemporal interactive neural network, so as to solve the problem that traditional algorithms are difficult to identify weak targets in complex scenes, and to achieve stable identification of weak moving targets in complex scenes.

[0005] To achieve the above objectives, the present invention employs the following technical solution:

[0006] A method for detecting weak infrared targets based on a joint Swing and spatiotemporal interaction neural network, characterized by the following steps:

[0007] S1, convert the image to be detected into a temporal image, convert the temporal image into a one-dimensional signal, and extract the temporal channel features of the image to be detected from the one-dimensional signal;

[0008] S2, the image to be detected is converted into a low-dimensional image, and features are extracted from the low-dimensional image through the TF module. After fusing feature x3 and time channel features, feature x4 is obtained. x4 is magnified and the background is filtered out to obtain features containing target detection information, thereby obtaining infrared weak targets.

[0009] A further improvement of the present invention is that:

[0010] Preferably, in S1, the image to be detected is converted into a time image by the following formula:

[0011] X out =Conv(transpose(X) in ∈R T×C×H×W => X∈R C×T×H×W (2)

[0012] In the formula: Conv — convolution; Transpose — dimensionality transformation, which converts the image spatial channels to temporal channels.

[0013] Preferably, the transformation formula of S2 is:

[0014]

[0015] In the formula: Conv — convolution; Path Embed — relative position encoding; X out This is the output of the Time Transformer module.

[0016] An infrared weak target detection system based on a joint Swing and spatiotemporal interaction neural network includes:

[0017] The Temporal Transformer module is used to convert the image to be detected into a temporal image, and after converting the temporal image into a one-dimensional signal, extract the temporal channel features of the image to be detected from the one-dimensional signal.

[0018] The U-Transformer module is used to convert the image to be detected into a low-dimensional image. The TF module extracts features from the low-dimensional image, and after fusing feature x3 and time channel features, feature x4 is obtained. x4 is then magnified and the background is filtered out to obtain features containing target detection information, thereby obtaining infrared weak targets.

[0019] Preferably, the time Transformer module includes a spatiotemporal interaction module, a relative position encoding module, a W-TF attention extraction module, and a SW-TF attention extraction module.

[0020] Preferably, the spatiotemporal interaction module is used to convert the input spatial image into a temporal image;

[0021] The relative position encoding module is used to expand the input time image into a one-dimensional signal.

[0022] The W-TF attention module is used to extract features from a one-dimensional signal to obtain feature x1;

[0023] The SW-TF attention module is used to extract features from feature x1 to obtain the temporal channel features of the image to be detected.

[0024] Preferably, the U-Transformer module includes several encoders, a TF module, and several decoders.

[0025] Preferably, the encoder is used to convert the input image into a low-dimensional image;

[0026] The TF module is used to obtain feature x3 from the low-dimensional image, and then fuse feature x3 with the temporal channel features to obtain feature x4.

[0027] The decoder is used to amplify the extracted features by x4, filter out the background, obtain features containing target detection information, and thus obtain infrared weak targets.

[0028] Preferably, each encoder consists of a convolutional module, a batch normalization module, and a ReLU activation function; each decoder consists of a dilated convolutional module, a batch normalization module, and a ReLU activation function.

[0029] Preferably, during the training process of the infrared weak target detection system, correction is performed using the following loss function:

[0030] L = L K +L off +λ×L size (4)

[0031] Among them, L K For heatmap loss, L off For offset loss, L size This is for scale loss.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] This invention discloses an infrared weak target detection method based on a joint Swing and spatiotemporal interactive neural network. Building upon in-depth research of existing infrared weak target detection methods, this method adaptively learns inter-frame information using a spatiotemporal interactive network, utilizes a convolutional neural network to learn the spatial characteristics of infrared weak targets, and captures spatial information from images using a Swing-Transformer, significantly improving the algorithm's detection capability in complex scenes. This method proposes a spatiotemporal neural network capable of processing multi-frame image sequences, constructing the temporal information of discontinuously moving small targets as a black-box model, rather than constructing a deterministic mathematical model to describe the motion of small targets. This effectively solves the problem of discontinuous motion of small targets and significantly improves the algorithm's ability to detect targets in complex scenes. Attached Figure Description

[0034] Figure 1 This is a weak target detection algorithm based on spatiotemporal neural networks;

[0035] Figure 2 For the Time Transformer module;

[0036] Figure 3 For spatiotemporal interaction module;

[0037] Figure 4 For the U-Transformer module;

[0038] Figure 5 Flowchart for training and inference of spatiotemporal neural network algorithm;

[0039] Figure 6 These are test results from a real-world scenario.

[0040] Figure 7 These are test results from a simulated scenario.

[0041] Figure 8 For hardware simulation framework;

[0042] Among them, (a) is the video stream input diagram; (b) is the hardware deployment model optimization diagram; and (c) is the output detection information diagram.

[0043] Figure 9 This is a diagram showing the physical hardware connections. Detailed Implementation

[0044] This invention discloses an infrared weak target detection method based on a joint Swing and spatiotemporal interactive neural network. This method effectively integrates the temporal and spatial characteristics of sequential images by utilizing spatiotemporal interactive convolution, enabling the model to effectively achieve spatial and temporal transformation. At the same time, it uses a temporal Transformer to extract temporal channel information.

[0045] The main attributes of infrared weak targets are "weak" and "small". "Weak" means that the target has a lower grayscale value compared to the surrounding background in an infrared image, i.e., lower contrast. "Small" means that the target occupies a small area in the image, specifically reflecting a small number of pixels. Generally, in an image with a pixel size of 256×256, a target with a signal-to-noise ratio of less than 4 and a pixel size of less than 9×9 is defined as a weak target.

[0046] The method specifically includes the following steps:

[0047] S1, input the image to be detected into the time Transformer module.

[0048] This module is primarily responsible for extracting temporal information from images. Since sequential images exhibit continuity along the time axis, and the Transformer structure possesses strong modeling capabilities for long sequences, this invention utilizes the Transformer for feature extraction from sequential images in the time channel, obtaining time channel features, and proposes a temporal Transformer module. The specific structure of this module is as follows: Figure 2 .

[0049] The core of the temporal Transformer module consists of four parts: a spatiotemporal interaction module, a relative position encoding module, a W-TF attention extraction module and a SW-TF attention extraction module in the Swing-Transformer, which ultimately obtain information on the temporal channel of the input image.

[0050] The spatiotemporal interaction module is used to convert the input spatial image into a temporal image and output a temporal image;

[0051] The relative position encoding module is used to expand the input time image into a one-dimensional signal, so that the image processing can be performed in a form that the Transformer can handle.

[0052] The W-TF attention module is used to extract features from a one-dimensional signal to obtain feature x1;

[0053] The SW-TF attention module is used to further extract features from feature one to obtain feature x2, which is the temporal channel feature of the image to be detected.

[0054] This method first uses the spatiotemporal information interaction module proposed in this invention to convert the spatial channel of the image into a temporal channel. Then, it sequentially uses the W-TF attention module and SW-TF attention module in the Swin-Transformer to extract feature information, finally obtaining the information in the temporal channel of the image. The specific transformation formula of the temporal Transformer module is as follows:

[0055]

[0056] In the formula: Conv—convolution; Path Embed—relative position encoding, which processes the image into a form that the Transformer can handle; X out This is the output of the Time Transformer module.

[0057] In some embodiments of the present invention, the structure of the spatiotemporal information interaction module is described below. Figure 3 This module is based on 2D convolution, has low computational cost, and can be directly accelerated using existing AI acceleration libraries (such as TensorRT, NCNN) to transform the feature map X. in ∈R T ×C×H×W Exchanging X∈R in the spatial and temporal channels yields X∈R C×T×H×W Next, feature extraction is performed on the spatiotemporal channel to obtain fused information from different time series; the operation of this module is as follows:

[0058] X out =Conv(transpose(X) in ∈R T×C×H×W => X∈R C×T×H×W (2)

[0059] In the formula: Conv — convolution; Transpose — dimensionality transformation, which converts the image spatial channels to temporal channels.

[0060] S2 connects to the features on the image time channel and the U-Transformer module to obtain features containing target detection information and identify weak infrared targets.

[0061] This module is used for spatial feature extraction of infrared targets, and its specific structure is as follows: Figure 4 It includes an encoder, a decoder, and a TF module. The encoder encodes the features of weak infrared targets into a lower dimension, making the dimensionality of the transformed image lower than that of the original image, thus obtaining a low-dimensional image. This significantly reduces the processing complexity of the TF module. The TF module obtains feature x3 from the low-dimensional image and fuses feature x3 with the features in the temporal channel of the image obtained by the aforementioned temporal Transformer module to obtain feature x4. The decoder is responsible for feature amplification, selection, background filtering, target highlighting, and finally outputting features containing target detection information.

[0062] In some embodiments of the present invention, a number of decoders and encoders are provided. The number of encoders is determined according to the target dimension after the infrared weak target is converted, and the number of decoders is determined according to the feature magnification degree.

[0063] Furthermore, the subsequent encoder performs further low-dimensional transformations on the transformed image from the previous encoder; such as... Figure 4 In this process, encoder 2 further transforms the low-dimensional image output by encoder 1 into an even lower-dimensional image.

[0064] Furthermore, the subsequent decoder processes the features containing target detection information obtained from the previous decoder, further extracting features to obtain features containing target detection information; such as... Figure 4 As shown, decoder 3 extracts from decoder 2, and decoder 2 extracts from decoder 1.

[0065] In some embodiments of the present invention, the image to be processed undergoes two encoding processes in the encoder, each encoding process sequentially passing through convolution, batch normalization, and ReLU activation function; the extracted feature 4 undergoes two decoding processes in the decoding process, each decoding process sequentially passing through dilated convolution, batch normalization, and ReLU activation function; the TF module is the window attention of Swin-Transformer.

[0066] Correspondingly, each encoder includes a convolutional module, a batch normalization module, and a ReLU activation function; each decoder includes a dilated convolutional module, a batch normalization module, and a ReLU activation function.

[0067] Based on this, the initial detection model, which includes the temporal Transformer module and the U-Transformer module, has been completed.

[0068] This invention also discloses an infrared weak target detection system based on a joint Swing and spatiotemporal interaction neural network, comprising:

[0069] The Temporal Transformer module is used to convert the image to be detected into a temporal image, and after converting the temporal image into a one-dimensional signal, extract the temporal channel features of the image to be detected from the one-dimensional signal.

[0070] The U-Transformer module is used to convert the image to be detected into a low-dimensional image. The TF module extracts features from the low-dimensional image, and after fusing feature x3 and time channel features, feature x4 is obtained. x4 is then magnified and the background is filtered out to obtain features containing target detection information, thereby obtaining infrared weak targets.

[0071] In some embodiments of the present invention, the time Transformer module includes a spatiotemporal interaction module, a relative position encoding module, a W-TF attention extraction module, and a SW-TF attention extraction module.

[0072] The spatiotemporal interaction module is used to convert the input spatial image into a temporal image;

[0073] The relative position encoding module is used to expand the input time image into a one-dimensional signal.

[0074] The W-TF attention module is used to extract features from a one-dimensional signal to obtain feature x1;

[0075] The SW-TF attention module is used to extract features from feature x1 to obtain the temporal channel features of the image to be detected.

[0076] In some embodiments of the present invention, the U-Transformer module includes several encoders, a TF module, and several decoders.

[0077] The encoder is used to convert the input image into a low-dimensional image;

[0078] The TF module is used to obtain feature x3 from the low-dimensional image, and then fuse feature x3 with the temporal channel features to obtain feature x4.

[0079] The decoder is used to amplify the extracted features by x4, filter out the background, obtain features containing target detection information, and thus obtain infrared weak targets.

[0080] In some embodiments of the present invention, each encoder consists of a convolution module, a batch normalization module, and a ReLU activation function; each decoder consists of a dilated convolution module, a batch normalization module, and a ReLU activation function.

[0081] This invention also discloses a construction and training process for an infrared weak target detection method system that combines Swin and a spatiotemporal interactive neural network. The process consists of three parts: the first part is the preprocessing of the dataset; the second part is the construction and training of the deep learning model; and the third part is the testing of the infrared weak target detection model using a loss function. The specific content is as follows:

[0082] The first part consists of two steps:

[0083] Step 1: Acquire infrared image data, perform denoising and augmentation preprocessing on the images, and construct an image dataset Data = {I1, I2, I3, ..., I...} n}, where I represents each image in the dataset and N represents the total number of images.

[0084] Step two: Label the dataset from step one with weak targets to obtain labeled data. label The data was divided into two groups. and As training samples, As a test sample.

[0085] Data label ={(I1,x1,y1,width1,hight1),...,(I n ,x n ,y n ,width n ,hight n )} (3)

[0086] Wherein, in the formula I n For the nth image, x n y n x, y coordinates of the center point of the target bounding box, width n The height is the width of the target bounding box. n The height of the target bounding box.

[0087] Part Two consists of two steps:

[0088] Step 3: Construct a detection model. The network model takes the images from the training set described in Step 2 as input and outputs the target's location information. Specifically, as follows... Figure 1 As shown, the detection model involved in this invention includes a temporal feature extraction branch composed of a temporal Transformer module and a spatial feature extraction branch composed of a U-Transformer module. Finally, the model parameters are updated through a loss function to achieve the goal of accurately identifying the target. The temporal Transformer module is implemented based on the spatiotemporal information interaction module proposed in this invention, which can realize the fusion of spatial and temporal features of the image.

[0089] (1) Constructing the Time Transformer module

[0090] This module is primarily responsible for extracting temporal information from images. Since sequential images exhibit continuity along the time axis, and the Transformer structure possesses strong modeling capabilities for long sequences, this invention utilizes the Transformer for feature extraction from sequential images in the time channel, proposing a Temporal Transformer module. The specific structure of this module is as follows: Figure 2 .

[0091] (2) Constructing the U-Transformer module

[0092] This module is used for spatial feature extraction of infrared targets, and its specific structure is as follows: Figure 4It includes an encoder, a decoder, and a TF module. The encoder encodes the features of weak infrared targets into a lower dimension, making the dimensionality of the transformed image lower than that of the original image, thus obtaining a low-dimensional image and significantly reducing the complexity of Transformer processing. The TF module extracts feature x3 from the low-dimensional image and fuses feature x3 with the features in the temporal channel of the image obtained by the aforementioned temporal Transformer module to obtain feature x4. The decoder performs feature amplification and selection, performs background filtering, highlights the target, and finally outputs features containing target detection information.

[0093] Based on this, the initial detection model, which includes the temporal Transformer module and the U-Transformer module, has been completed.

[0094] (3) Correction through loss function

[0095] Obtained through step two The initial detection model is trained, and the loss function during the training process is composed of heatmap loss L. K Offset loss L off Scale loss L size constitute.

[0096] L = L K +L off +λ×L size (4)

[0097] Where λ is the adjustment factor.

[0098] The specific process is as follows: Five consecutive images are received as input from the front-end device, with the first four frames serving as reference frames. Feature extraction is performed through spatiotemporal coupling, and the specific location information of the target in the last image is provided. Model training is as follows: Figure 5 During training, image pairs consisting of 5 frames are constructed based on the temporal relationship of the image frames. The first 4 frames are reference frames, and the last frame is the keyframe, i.e., the image to be detected, thus forming the training dataset. During training, image pairs are randomly selected from the dataset and fed into the detection model. The model is updated based on the difference between the detection result output by the model and the predetermined label value. When the loss value between the model output and the label value is sufficiently small, it is considered that the model has learned the target features well and has the ability to detect weak targets.

[0099] Part Three consists of two steps:

[0100] Step four involves using the detection model obtained in step four for inference. Model testing and inference are offline processes, as the algorithm cannot obtain all image frames beforehand to form image pairs. Therefore, during inference initialization, a data queue consisting of the first frame image is first constructed, with a queue length of 5 (i.e., the first frame data is copied 5 times). The data queue updates data sequentially following the detection video stream (discarding the earliest data that entered the queue when the video stream data is updated), forming a sliding window on the data stream. The detection algorithm then provides the location of the target in the last frame image of the sliding window. Figure 5 .

[0101] Step 5: Build a hardware simulation test framework based on Jetson NX. The framework supports online video streams and local resource input, and tests the algorithm accuracy and execution efficiency of the model on edge devices.

[0102] Example Implementation Results

[0103] The performance of the algorithm in this invention was tested in real-world and simulated scenarios. The experimental results are as follows: Figure 6 and Figure 7 As shown, this invention can effectively solve the infrared weak target detection task in various low signal-to-noise ratio scenarios, and exhibits good performance on both real images and simulation datasets; for example... Figure 8 and Figure 9 The hardware inference framework and real-time inference results of the method of the present invention are described. The method proposed in this invention can run on the edge computing device Jetson NX without loss of accuracy, and the method has certain industrial application capabilities.

[0104] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting weak infrared targets based on a joint Swing and spatiotemporal interaction neural network, characterized in that, Includes the following steps: S1, input the image sequence to be detected, which consists of multiple consecutive infrared images, into the temporal Transformer module. The first few frames in the image sequence to be detected are reference frames, and the last frame is the key frame, i.e., the image to be detected. The spatial image of the image to be detected is converted into a temporal image through the spatiotemporal interaction module. The time Transformer module includes a spatiotemporal interaction module, a relative position encoding module, a W-TF attention module, and a SW-TF attention module; The conversion involves: converting the spatial channels of the image into temporal channels through Transpose dimensionality transformation, then convolving the temporal channels; unfolding the temporal image into a one-dimensional signal through relative position encoding, and extracting features from the one-dimensional signal using a W-TF attention module to obtain the features. ; from the features via the SW-TF attention module Extract features to obtain the temporal channel features of the image to be detected; S2, the image to be detected is converted into a low-dimensional image, and features are extracted from the low-dimensional image using the TF module. , will feature After fusing with time channel features, the features are obtained. ,Will By zooming in and filtering out the background, features containing target detection information are obtained, and then the infrared weak targets in the key frame are obtained.

2. The infrared weak target detection method based on a joint Swing and spatiotemporal interaction neural network according to claim 1, characterized in that, The transformation formula for S1 is: （1） In the formula: Conv --convolution; Path Embed —Relative position encoding; This is the output of the Time Transformer module.

3. An infrared weak target detection system based on a joint Swing and spatiotemporal interaction neural network for implementing the method of claim 1, characterized in that, include: The Temporal Transformer module is used to convert the image to be detected into a temporal image, and after converting the temporal image into a one-dimensional signal, extract the temporal channel features of the image to be detected from the one-dimensional signal. The time Transformer module includes a spatiotemporal interaction module, a relative position encoding module, a W-TF attention module, and a SW-TF attention module; The spatiotemporal interaction module is used to convert the input spatial image into a temporal image; The relative position encoding module is used to expand the input time image into a one-dimensional signal; The W-TF attention module is used to extract features from a one-dimensional signal and obtain the features. ; The SW-TF attention module is used to extract features Extract features to obtain the temporal channel features of the image to be detected; The U-Transformer module is used to convert the image to be detected into a low-dimensional image, and the TF module extracts features from the low-dimensional image. , will feature After fusing with time channel features, the features are obtained. ,Will By magnifying and filtering out the background, features containing target detection information are obtained, thereby identifying weak infrared targets.

4. The infrared weak target detection system based on a joint Swing and spatiotemporal interaction neural network according to claim 3, characterized in that, The U-Transformer module includes several encoders, TF modules, and several decoders.

5. The infrared weak target detection system based on a joint Swing and spatiotemporal interaction neural network according to claim 4, characterized in that, The encoder is used to convert the input image into a low-dimensional image; The TF module is used to obtain features from low-dimensional images. , will feature After fusing with time channel features, the features are obtained. ; The decoder is used to extract features. By magnifying and filtering out the background, features containing target detection information are obtained, thereby identifying weak infrared targets.

6. The infrared weak target detection system based on a joint Swing and spatiotemporal interaction neural network according to claim 3, characterized in that, Each encoder consists of a convolutional module, a batch normalization module, and a ReLU activation function; each decoder consists of a dilated convolutional module, a batch normalization module, and a ReLU activation function.

7. The infrared weak target detection system based on a joint Swing and spatiotemporal interaction neural network according to claim 3, characterized in that, During the training process of the infrared weak target detection system, correction is performed using the following loss function: （4） in, For heatmap loss, For offset loss, For scale loss, This is an adjustment factor.

Images