An infrared dim and small moving target detection method based on space-time hybrid coding

Abstract

The application discloses an infrared weak and small moving target detection method based on space-time hybrid coding, and is applied to the technical field of infrared weak and small moving target detection, and aims at the low detection performance of the prior art in the detection of the infrared weak and small moving target. The method comprises the following steps: firstly, an infrared weak and small moving target detection network based on space-time feature fusion is constructed; a multiscale space-time feature extractor is designed; a space feature branch and a time sequence feature branch are used to perform multiscale feature extraction on the infrared weak and small moving target; a time sequence information self-optimization module is used to perform channel and space information self-optimization on the extracted multiscale time sequence features; a space-time feature hybrid module is used to fuse the corresponding scale space-time information of the multistage space features and the multistage time sequence features, perform cross-scale feature fusion on the multistage space-time features, define a corresponding loss function, perform target positioning prediction, error loss calculation and step-by-step iterative optimization on the output final features, and finally obtain a model with good training parameters, so that the infrared weak and small moving target can be effectively positioned and detected.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to an infrared weak moving target detection technology. Background Art

[0002] The detection of small, infrared moving targets has always been a hot topic and a difficult problem in the field of computer vision. Compared to traditional target detection, the purpose of infrared small target detection is to accurately locate small targets such as aircraft or drones captured by infrared imaging systems. Due to the long imaging distance and inherent imaging mode of infrared imaging, many very large targets in the real world often appear very small against infrared background images or videos, and therefore they often lack distinct color, shape, and texture. On the other hand, infrared targets often have low contrast relative to the background, so such targets are easily submerged in complex background clutter when in motion. In summary, unlike traditional target detection, the difficulty of infrared small moving targets lies primarily in the fact that the targets are too small, have low contrast, and are difficult to capture features, which can easily lead to feature disappearance and detection failure.

[0003] Researchers have made various attempts to address the difficulty of detecting small infrared targets. Current mainstream approaches can be categorized as model-driven and data-driven, depending on whether training data is required. Model-driven approaches typically rely on complex manual feature extraction and model parameter adjustment, calculating the maximum contrast between the image's central pixel and neighboring areas, or exploiting the infrared image's structure to separate the background from the target. However, these methods suffer from poor robustness and perform poorly in detecting small infrared targets in complex clutter and noise environments.

[0004] Among data-driven approaches, learnable models based on deep learning have overcome the shortcomings of traditional models, significantly improving the ability to detect small and dim targets. Within this approach, researchers have proposed numerous single-frame infrared small and dim target segmentation datasets. By learning segmentation models from this large amount of single-frame infrared small and dim target data, these methods have demonstrated greater robustness and accuracy than traditional methods, becoming a mainstream approach to infrared target detection.

[0005] However, this type of single-frame based infrared dim target detection method does not consider the necessary and sufficient spatiotemporal feature information, so it often loses target information when detecting dim infrared moving targets. In addition, they model the infrared target detection task as a more sophisticated segmentation task. For dim infrared target detection, the target is too small and the segmentation is often unreasonable.

[0006] In order to further improve the model's detection performance for infrared weak moving targets, it is necessary to introduce timing information with reference to the human visual system. However, current methods for detecting small moving infrared targets rarely consider spatiotemporal information. The only moving infrared timing information detection models often directly use the inter-frame attention mechanism or LSTM-based models, which will introduce too much interference information and computational load.

[0007] In summary, existing infrared dim moving target detection schemes have three significant flaws. First, model-driven methods lack feature learning capabilities, relying on manual feature and model parameter adjustment, resulting in poor generalization. Second, single-frame data-driven detection methods fail to consider the temporal correlation of moving targets and are not well-suited to modeling them as segmentation tasks. Finally, multi-frame data-driven methods are still in the exploratory stage and have not fully explored spatiotemporal feature extraction and fusion strategies for infrared dim moving targets, which inhibits the detection performance of infrared dim moving targets. Summary of the Invention

[0008] In order to solve the above technical problems, the present invention proposes an infrared dim small target motion detection method based on spatiotemporal feature hybrid coding.

[0009] The technical solution adopted by the present invention is: a method for detecting infrared weak moving targets based on spatiotemporal feature fusion, comprising:

[0010] S1. Build a target detection network; the target detection network includes: a spatiotemporal feature extractor module, a temporal information self-optimization module, a spatiotemporal feature mixing module, and a moving weak target detection module;

[0011] The spatiotemporal feature extractor module includes a spatial feature branch and a temporal feature branch; the spatial feature branch is composed of the backbone network CSPDarknet in YOLOX, which is used to extract the multi-scale spatial features of the sequence images in the current sampling window; the temporal feature branch is composed of the backbone network 3D-ShuffleNetV2, which is used to extract the multi-scale temporal features of the sequence images in the current sampling window;

[0012] The temporal information self-optimization module includes a channel self-attention submodule and a spatial self-attention submodule. The channel self-attention submodule performs weighted adjustment of the channel features on the multi-scale temporal features extracted by the temporal feature branch. The spatial self-attention submodule performs weighted adjustment of the spatial features on the multi-scale temporal features after the weighted adjustment of the channel features, and finally obtains the self-optimized multi-scale temporal features.

[0013] The spatiotemporal feature hybrid module includes a spatiotemporal feature hybrid encoder and a multi-level feature hybrid encoder. The multi-scale spatial features and the self-optimized multi-scale temporal features are fused with the spatiotemporal features of the corresponding scales through the hybrid encoder module. The multi-level feature hybrid encoder performs multi-scale information fusion on the spatiotemporal fusion features of multiple scales output by the hybrid encoder module to obtain the final feature map.

[0014] The moving small target detection module is based on the YOLOX detection head and performs target detection on the feature map output by the spatiotemporal feature mixing module;

[0015] S2, build training data set;

[0016] S3, iteratively optimize the target detection network parameters constructed in step S1 using the training data set based on gradient descent;

[0017] S4. Input the sequence images in the sampling window to be detected into the target detection network obtained after training in step S3, thereby outputting the detection results.

[0018] Beneficial effects of the present invention: The method of the present invention first constructs an infrared weak moving target detection network based on spatiotemporal feature fusion, designs a multi-scale spatiotemporal feature extractor, uses the spatial feature branch and the temporal feature branch therein to extract multi-scale features of infrared weak moving targets respectively, uses the temporal information self-optimization module to self-optimize the channel and spatial information of the extracted multi-scale temporal features, uses the spatiotemporal feature mixing module to fuse the corresponding scale spatiotemporal information of multi-level spatial features and multi-level temporal features, performs cross-scale feature fusion on the multi-level spatiotemporal features, and by defining the corresponding loss function, performs target positioning prediction, error loss calculation, and step-by-step iterative optimization of the target detection and positioning results on the output final features, and finally obtains a model with good training parameters to achieve the purpose of effectively positioning and detecting infrared weak moving targets. The method of the present invention makes the detection and positioning of infrared weak moving targets more accurate and robust, can effectively detect the sequence images of moving weak targets collected by infrared equipment, detect and locate moving weak targets such as drones and trucks in the target area, and can be applied to important application fields such as drone supervision and security. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of an infrared dim moving target detection method based on spatiotemporal feature fusion according to the present invention.

[0020] Figure 2 This is a diagram of the infrared weak moving target detection network structure designed in an embodiment of the present invention.

[0021] Figure 3This is a structural diagram of the temporal feature self-optimization module proposed in an embodiment of the present invention; (a) is the overall structure diagram, (b) is the channel attention detail diagram, and (c) is the spatial attention detail diagram.

[0022] Figure 4 This is a structural diagram of the spatiotemporal feature mixing module proposed in an embodiment of the present invention;

[0023] Among them, (a) is the structure diagram of the spatiotemporal feature hybrid encoder, and (b) is the structure diagram of the multi-level feature hybrid encoder.

[0024] Figure 5 PR curve diagram on the data set in an embodiment of the present invention;

[0025] Among them, (a) is the PR map on IRDST, and (b) is the PR map on ITSDT-15K.

[0026] Figure 6 Graphs showing visualization of detection boxes under two public datasets in an embodiment of the present invention;

[0027] Among them, (a) is the detection visualization result on the IRDST dataset, and (b) is the detection visualization result on the ITSDT-15K dataset. DETAILED DESCRIPTION

[0028] To facilitate those skilled in the art to understand the technical content of the present invention, the present invention is further explained below with reference to the accompanying drawings.

[0029] like Figure 1 As shown, the implementation process of the infrared weak moving target detection method based on spatiotemporal feature fusion of the present invention specifically includes the following steps:

[0030] S1. Construct an infrared weak moving target detection network based on spatiotemporal feature fusion;

[0031] Construct an intelligent detection network specifically for infrared weak moving target detection, that is, an infrared weak moving target detection network based on spatiotemporal feature fusion. The detailed structure of the network is as follows: Figure 2 As shown, it includes: spatiotemporal feature extractor module, temporal information self-optimization module, spatiotemporal feature mixing module, and moving weak target detection module, as follows:

[0032] (1) In the spatiotemporal feature extractor module, in order to enable the model to extract information-rich spatial features and temporal features, such as Figure 2 As shown in the upper part, a multi-scale spatial feature branch and a multi-scale temporal feature branch are designed. In the present invention, the multi-scale spatial feature branch and the multi-scale temporal feature branch will generate three corresponding scale features.

[0033] Firstly, for the spatial feature branch, the last frame in the T sequence frames in the given time domain sampling window is selected as the key frame to be detected, that is, the input T frame picture sequence I s = [I1, I2, …, I T ] key frame is I T .

[0034] Then the key frame I T is sent to the spatial feature for multi-level spatial feature extraction, the difference is that the spatial feature branch in the application uses the pre-trained backbone network in YOLOX as the backbone feature extraction network of the branch, and outputs spatial features with 32 times down-sampling, 16 times down-sampling and 8 times down-sampling respectively The calculation process is as follows:

[0035]

[0036] Among them is the input key frame I T , and the upper layer output in the CSPDarknet network, represents the convolution operation of the dark3 layer in the CSPDarknet.

[0037] Secondly, for the time sequence feature branch, in the given time domain sampling window T sequence frames, the input sequence I s = [I1, I2, …, I T ] is input to the time sequence feature branch for multi-level time sequence feature extraction. The time sequence feature branch uses a 3D-ShuffleNetV2 network as the backbone, and obtains the time sequence feature of the sequence I s = [I1, I2, …, I T ] through the backbone Subsequently is up-sampled to obtain multi-scale time sequence features, and the calculation process is as follows:

[0038]

[0039] Among them Bilinear ×2 and Bilinear ×4 represent 2 times up-sampling and 4 times up-sampling of bilinear interpolation respectively, represents the feature output by the 3D-ShuffleNetV2 backbone network.

[0040] (2) The spatio-temporal feature fusion module includes a channel self-attention module and a spatial self-attention module using the CPCANet idea;

[0041] Among them, the channel self-attention module uses average pooling and maximum pooling as well as two layers of fully connected layer operations to obtain the channel self-attention map for extracting multi-level temporal features, while the spatial self-attention module uses multi-scale depthwise separable convolution modules and aggregates the features they extract to generate the spatial channel self-attention map of multi-level temporal features.

[0042] In the time series information self-optimization module, the multi-scale time series features extracted by the time series feature branch are self-optimized and adjusted to further enhance the time series feature representation, such as Figure 3 As shown in the figure, a temporal information self-optimization module is designed. In this module, the channel attention submodule first outputs the channel self-attention map and generates the temporal features after channel feature adjustment. Then, the spatial attention submodule is used to output the spatial self-attention map and generate the final temporal feature map after channel feature adjustment and spatial feature adjustment.

[0043] The channel attention submodule includes multiple layers of fully connected layers and pooling layers, and the spatial attention submodule includes multiple groups of depth-wise separable convolutions. When performing self-optimization of timing characteristics, the overall calculation formula can be expressed as:

[0044]

[0045] Similarly, input the time series features respectively Self-optimizing timing features Among them, inspired by the CPCANet network, the same as CPCA, f ChannelAttention represents the channel attention operation, represents a depth-wise separable convolution operation with a kernel size of i×j, i.e. represents a depthwise convolution with a kernel size of 5×5. and Represents depthwise convolution with kernel sizes of 7×1 and 1×7, respectively. and Represents depthwise convolution with kernel size of 11×1 and 1×11 respectively, and Represents depthwise convolution with kernel size of 21×1 and 1×21 respectively, Represents a point-by-point convolution with a kernel size of 1×1. Specifically, after obtaining the channel-optimized temporal feature map through a channel attention module consisting of two fully connected layers and a pooling layer, four sets of depthwise separable convolutions generate the corresponding spatial self-attention maps, and 1×1 convolution is used to fuse different spatial feature maps to generate the final spatial feature map and self-optimize the temporal features. Indicates the third-level timing characteristics. In the process of generating multi-level self-optimization timing characteristics, the first, second and third level timing characteristics are input respectively. The timing information self-optimization module generates optimized multi-level timing features

[0046] (3) The spatiotemporal feature mixing module includes: a spatiotemporal feature mixing encoder and a multi-level feature mixing encoder; among them, the spatiotemporal feature mixing encoder is composed of multiple convolutions and channel self-attention operations, and the multi-scale feature mixing encoder is composed of multiple layers of convolution and bilinear interpolation.

[0047] In the spatiotemporal feature mixing module, the extracted multi-level spatial features of infrared weak moving targets and the optimized temporal features are mixed and encoded into information-rich spatiotemporal features to further enhance the feature representation, such as Figure 4 As shown in the figure, a spatiotemporal feature hybrid module consisting of a spatiotemporal feature hybrid encoder and a multi-level feature hybrid encoder is designed. The spatiotemporal feature hybrid encoder is first used to fuse the temporal and spatial features of the corresponding scales to obtain rich multi-level spatiotemporal features. The multi-level feature hybrid encoder is then used to fuse the multi-level spatiotemporal features to obtain the final feature map.

[0048] The spatiotemporal feature hybrid encoder submodule includes multiple convolutional layers and self-attention operations. The specific calculation process of this module can be expressed as:

[0049]

[0050] in They are the self-optimized temporal features and spatial features of the corresponding scales, By the corresponding Generated by timing self-optimization, f self-att represents the self-attention mechanism operation, f conv represents the convolution operation, Represents the corresponding fused spatiotemporal feature output; similarly, and In the same way, as well as The fusion is generated as follows:

[0051]

[0052] The multi-level feature hybrid encoder consists of multiple groups of convolutions, bilinear interpolation upsampling, and N layers of CSPLayer. It is used to receive the multi-level spatiotemporal features fused by the spatiotemporal feature hybrid encoder and fuse the multi-level spatiotemporal features into the final feature map. The specific calculation process of this module can be expressed as:

[0053]

[0054] where f conv represents the convolution operation, f concat represents the feature concatenation operation, fBil stands for twice bilinear interpolation, represents n-layer CSPLayer convolution, It is the final output feature map after the fusion of multi-level spatiotemporal features.

[0055] (4) The moving small target detection module includes: a small target detection head designed based on the YOLOX technology framework; the detection head that has been trained with parameters finally completes the detection and positioning operations of infrared small moving targets.

[0056] S2, extracting primary multi-scale features of the infrared weak moving target based on the temporal feature branch and the spatial feature branch in the feature extractor of the target detection network in step S1;

[0057] In the multi-level spatiotemporal feature extraction stage, the multi-scale spatiotemporal feature extractor uses a spatial feature extractor and a temporal feature extractor to perform multi-scale extraction on the spatial features and temporal features of the infrared weak moving target respectively.

[0058] First, the last frame image I of the sequence image with a sampling time window of T is T As key frame input to the spatial feature extractor for multi-scale spatial feature extraction. T Inputting the 2D backbone of the CSPDarknet pre-trained model, we get the spatial feature encodings of 32x, 16x, and 8x downsampling respectively:

[0059] Secondly, the sequence image I with a sampling time window of T s =[I1,I2,…,I T ] is input into the 3D backbone temporal feature branch of the 3D-ShuffleNetV2 network to obtain a 32x downsampled feature map rich in temporal information. Then the timing characteristic diagram Perform 2x and 4x upsampling respectively to obtain multi-level temporal feature maps

[0060] S3, based on the temporal information self-optimization module of the target detection network in step S1, performing attention adaptive adjustment on the temporal multi-scale feature information extracted by the temporal feature branch in step S2;

[0061] In the multi-level spatiotemporal feature extraction stage, the multi-scale spatiotemporal feature extractor obtains the multi-scale spatial feature X S and multi-scale temporal features X T ;

[0062] In the time series information self-optimization stage, the time series self-optimization module performs self-optimization adjustment on the extracted multi-level time series features, that is, the multi-scale time series features Input the timing self-optimization module, and then adjust the weighted channel features through the channel attention module composed of multiple linear layers and pooling layers. The spatial attention module composed of multiple groups of depth-wise separable convolutions adjusts the weighted spatial features, and finally obtains the self-optimized timing features.

[0063] S4, performing cross-domain and multi-scale information fusion on the multi-scale spatial features extracted in step S2 and the self-optimized multi-scale temporal features extracted in step S3 based on the spatiotemporal feature mixing module of the target detection network in step S1;

[0064] In the multi-scale feature fusion stage, the spatiotemporal feature mixing module first performs spatiotemporal feature fusion of the corresponding scales through the spatiotemporal feature mixing encoder. and are sent to the hybrid encoder module respectively, corresponding to the scale and and and The spatiotemporal features are fused through feature splicing, convolution, and self-attention operations to obtain the corresponding fused spatiotemporal feature map. Then, X is transformed into fused Perform multi-scale information fusion to obtain the final feature map

[0065] S5. Based on the moving small target detection module of the detection network in step S1, by defining a relevant loss function, the output features of step S4 are subjected to target positioning detection, error loss calculation, and iterative optimization of target detection accuracy.

[0066] In the target positioning and detection stage, the infrared weak moving target detection module uses an anchor-free detector from YoloX. It consists of two branches. The input features are respectively passed through two 3×3 convolution upper branches to obtain cls features and two 3×3 convolution lower branches to obtain reg features. The cls features are then passed through a 1×1 convolution to obtain the cls output, and the reg features are passed through a 1×1 convolution to obtain the reg output and obj output. At the same time, the total loss function is calculated for these three output loss items. The calculation rules used are as follows:

[0067] Loss total = λL reg +L cls +L obj (8)

[0068] Among them, cls is the abbreviation of classification, which means classification; reg is the abbreviation of regression, which means bounding box regression; obj is the abbreviation of objectness, which means target information; λ is a hyperparameter used to balance the three loss terms, L reg represents the bounding box regression loss, L cls represents the classification loss, L obj represents the target probability loss; L cls With L obj The ordinary binary cross entropy loss is used for calculation, L reg The ordinary IoU loss is calculated, where IoU represents the intersection-over-union ratio of two overlapping regions, as follows:

[0069] L reg = -log(IoU(B gt ,B pred )) (9)

[0070]

[0071] Among them, B gt represents the true target bounding box, B pred represents the predicted object bounding box, Indicates the predicted category, p i represents the real category, Represents the predicted target information, c i Indicates the real target information.

[0072] Then, based on the calculated loss, the model uses gradient descent to iteratively optimize the parameters. The method of the present invention trains 100 rounds, saves the training model parameters of each round, and then tests the saved model parameters on the test set to select the mAP 50 The model with the best accuracy is selected to obtain the model parameters with higher training performance and the final detection results.

[0073] Those skilled in the art should know that mAP 50 Precision is a very important indicator in object detection, which is used to measure the average precision of the model when the IoU (Intersection over Union) threshold is 0.5.

[0074] The present invention further illustrates the method of the present invention by taking the public infrared image sequence datasets IRDST and ITSDT as examples.

[0075] The IRDST dataset is divided into real datasets and synthetic datasets. The real dataset consists of 85 image sequences and 40,656 images, of which 42 sequences and 20,398 images are used as training sets, and the remaining 43 sequences and 20,258 images are used as test sets; the ITSDT dataset consists of 87 image sequences and 21,750 images, of which 40 sequences and 10,000 images are selected as training sets, and 20 sequences and 5,000 images are selected as test sets to form the new ITSDT-15K dataset.

[0076] (1) Initialization of the experiment:

[0077] Before inputting the infrared image, all images in the training and test sets are converted to 512×512 pixels, the sampling time window T is set to 5, and the image pixel values ​​are normalized before inputting into the model for infrared weak moving target detection and positioning. In this embodiment, SGD is used as the optimizer, the momentum value is set to 0.937, the batch size is set to 4, the initial learning rate is set to 0.01, and the weight decay is 5×10 -4 , the learning rate reduction coefficient is 0.1, and 100 epochs are trained on the dataset. All evaluation experiments of the model of the method of the present invention are carried out on two NVIDIA RTX 3090 graphics cards. In addition, in the infrared weak moving target detection experiment of this embodiment, the evaluation indicators mainly include recall rate, precision, F1 index and mAP 50

[0078] (2) Multi-scale spatiotemporal feature extraction stage:

[0079] After the preprocessed image data is input into the model, the key frames and infrared sequence images in the infrared sequence images are respectively extracted through the spatial feature branch and the temporal feature branch according to the steps of formula (1) and (2) to obtain the multi-scale spatial features X of the infrared dim targets. S and multi-scale temporal features X T .

[0080] (3) Timing information self-optimization stage

[0081] Get multi-scale time series features Afterwards, if Figure 3 As shown, the multi-scale time series feature X T The different scale features of are input into the time series information self-optimization module respectively. The self-time series information optimization module can effectively self-adjust the channel features and spatial features of the multi-scale time series information features according to formula (3), so that more effective multi-scale time series features can be obtained, and more effective spatiotemporal information interaction can be carried out with the multi-scale spatial features.

[0082] (4) Spatiotemporal feature mixing stage:

[0083] Obtain multi-scale spatial features of infrared weak moving targets and self-optimizing timing features After that, the corresponding spatial scale features and temporal scale features are first input into the spatiotemporal feature hybrid encoder. The spatiotemporal feature hybrid encoder can effectively fuse the corresponding spatial scale features and temporal scale features according to formulas (4), (5), and (6) to form a multi-scale spatiotemporal feature. Then the multi-scale spatiotemporal features X fused Input to the multi-level feature hybrid encoder and perform multi-scale spatiotemporal feature fusion according to formula (7) to obtain the final feature

[0084] (5) Detection head and loss function:

[0085] In this embodiment, two branches are used as detection heads to detect small infrared targets. The upper branch predicts the cls output of small infrared targets, while the lower branch predicts the reg output and obj output. Each branch consists of two layers of 3×3 convolution to extract corresponding features. Finally, the loss is calculated using Equation (8), and the backpropagation gradient descent algorithm is used to effectively learn the model, obtaining model parameters with high training performance, thereby achieving the goal of effectively detecting small moving targets in infrared videos.

[0086] In order to verify the effectiveness of the proposed method in infrared moving small target positioning detection, training was performed on the public infrared sequence datasets IRDST and ITSDT-15K. First, the proposed method was compared with some traditional infrared small target detection methods (MAXMEAN, TopHat, RLCM, HBMLCM, PSTNN, WSLCM). In addition, 13 deep learning network methods (ACM, RISTD, ISNet, UIUNet, SANet, AGPCNet, RDIAN, DNANet, ResUnet*DTUM, SIRST5K, MSHNet, RPCANet, SSTNet) were also compared. The results are shown in Table 1:

[0087] Table 1 Comparison of the effects of the method of the present invention and the traditional method

[0088]

[0089] Table 1 compares the mAP50, precision, recall, and F1 metrics of various models on the public infrared sequence datasets IRDST and ITSDT-15K. The best accuracy is marked in bold, and the second best is underlined. Three clear findings can be observed from this comparison.

[0090] 1) Our method consistently outperforms all other existing methods in terms of mAP50. For example, on the IRDST dataset, our method achieves an mAP50 of 77.33%, while SSTNet achieves the second-highest mAP50 of 76.96%. This is a 0.37% improvement over the second-highest mAP50.

[0091] 2) On the IRDST and ITSDT-15K datasets, the proposed method outperforms most other methods in terms of accuracy. For example, in the ITSDT-15K dataset, the Pr index of the proposed method is the highest, with a value of 92.42%. On the IRDST dataset, the Pr index of the proposed method is the second highest, with a value of 87.78%, second only to SSTNet's 88.56%.

[0092] 3) Regarding the F1 metric, our method consistently outperforms all other currently compared methods. For example, on the ITSDT-15K dataset, our method achieves an F1 value of 88.21%, surpassing the second-best SSTNet's 88.07%. On the IRDST dataset, our method achieves an F1 value of 85.96%, surpassing the second-best SSTNet's 85.11%. This demonstrates that while our method's precision and recall are not always superior to those of other methods, it possesses a strong overall detection capability.

[0093] In addition, to further evaluate the efficiency and performance of the present invention, this embodiment selected 13 deep learning-based models for comparison of inference performance. They are ACM, RISTD, ISNet, UIUNet, SANet, AGPCNet, RDIAN, DNANet, SIRST5K, MSHNet, RPCANet, DTUM, and DSFNet. mAP50 and F1 are used to measure model accuracy, Flops and Params are used to measure model complexity, and PCR is the ratio of mAP50 to Flops, which measures the overall efficiency of the model. Table 2 shows the numerical comparison results of this embodiment on the ITSDT-15K dataset.

[0094] Table 2 Numerical comparison results of this embodiment on the ITSDT-15K dataset

[0095]

[0096] Through comparison, we can clearly observe three important findings. First, the model accuracy of our method is higher than that of other methods, and the accuracy of the spatiotemporal feature fusion model is higher than that of other methods. For example, DTUM, SSTNet, and our method all incorporate temporal motion information for detection, and both the mAP50 and F1 metrics are significantly better than the methods relying on spatial features in the upper half of Table 2. Second, among spatiotemporal feature models, our method stands out for its simplicity and lightweightness. For example, among current infrared dim target detection models, the ACM model, which utilizes only spatial features, has the lowest Flops per second (24.66G), while our method ranks second at 41.92G, far lower than the 123.59G of the spatiotemporal feature model SSTNet. Finally, considering the performance-cost ratio (PCR), which is the ratio of mAP50 to Flops per second, our method achieves a PCR value of 1.84, ranking second, achieving more favorable results than the spatiotemporal feature model SSTNet, demonstrating the advantages of our model.

[0097] In order to intuitively compare the overall performance of different detection methods, this example uses precision and recall to create a PR curve, and plots the PR curves on ITSDT-15K and IRDST respectively. Figure 5 As shown in Figure 2, two clear observations can be made. First, the curves for our method are primarily located in the upper right corner of both figures, demonstrating that our method's overall model performance outperforms the other models. Second, another method that uses spatiotemporal features also outperforms other methods that use only spatial features, demonstrating the importance of utilizing temporal information.

[0098] In addition to the numerical performance comparison, in order to intuitively illustrate the detection accuracy of the model of the present invention compared with the current most advanced model method, this embodiment visualizes two sets of small target detection results of eight methods on the IRDST dataset and the ITSDT-15K dataset, respectively. Figure 6 As shown. Through visual comparison, two obvious findings can be observed. First, in the detection results of the IRDST dataset, almost no method produces false detection or missed detection, which may be because the randomly selected samples have a simple background and no occlusion. Despite this, the method of the present invention can still generate a detection frame with the maximum similarity to the actual target and tightly surround the target. On the other hand, in the detection results of the ITSDT-15K dataset, DNANet, AGPCNet and RPCANet failed to detect poor dim targets and occlusions, while the method of the present invention generated a correct and tight detection bounding box, highlighting the superior performance of the method of the present invention in infrared dim moving target detection.

[0099] In conclusion, the method of the present application makes the detection of infrared small moving targets more accurate and robust, can effectively detect the infrared small moving target sequence images collected by the infrared imaging device, detect the infrared small moving targets appearing in the target region, and can be applied in important application fields such as unmanned aerial vehicle supervision and security.

[0100] Those skilled in the art will realize that the embodiments described herein are for the purpose of illustration and should not be construed as limiting the scope of the present application. The present application can be modified and varied in various ways, and it is therefore to be understood that the invention can encompass any such modifications and variations as fall within the scope of the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of the claims of the present application.

Claims

1. A method for detecting infrared small moving targets based on spatiotemporal feature fusion, characterized in that: include: S1. Build target detection network; The target detection network includes: a spatiotemporal feature extractor module, a temporal information self-optimization module, a spatiotemporal feature mixing module, and a motion weak target detection module; The spatiotemporal feature extractor module includes a spatial feature branch and a temporal feature branch; the spatial feature branch is composed of the backbone network CSPDarknet in YOLOX, which is used to extract the multi-scale spatial features of the sequence images in the current sampling window; the temporal feature branch is composed of the backbone network 3D-ShuffleNetV2, which is used to extract the multi-scale temporal features of the sequence images in the current sampling window; Extract the multi-scale spatial features of the sequence image in the current sampling window. Specifically, the last frame of the sequence frames in the current sampling window is used as the key frame to be detected, and the key frame is input into the CSPDarknet network to extract the multi-scale spatial features. Extract multi-scale temporal features of the sequence images in the current sampling window. Specifically, the sequence frames in the current sampling window are input into the 3D-ShuffleNetV2 backbone network to obtain the first-scale temporal features. The first-scale temporal features are upsampled by different multiples to obtain multi-scale temporal features. The temporal information self-optimization module includes a channel self-attention submodule and a spatial self-attention submodule. The channel self-attention submodule performs weighted adjustment of the channel features on the multi-scale temporal features extracted by the temporal feature branch. The spatial self-attention submodule performs weighted adjustment of the spatial features on the multi-scale temporal features after the weighted adjustment of the channel features, and finally obtains the self-optimized multi-scale temporal features. The spatiotemporal feature hybrid module includes a spatiotemporal feature hybrid encoder and a multi-level feature hybrid encoder. The multi-scale spatial features and the self-optimized multi-scale temporal features are fused with the spatiotemporal features of the corresponding scales through the hybrid encoder module. The multi-level feature hybrid encoder performs multi-scale information fusion on the spatiotemporal fusion features of multiple scales output by the hybrid encoder module to obtain the final feature map. The moving small target detection module is based on the YOLOX detection head and performs target detection on the feature map output by the spatiotemporal feature mixing module; S2, dividing the collected infrared images into a training data set and a test data set; S3, iteratively optimize the target detection network parameters constructed in step S1 using the training data set based on gradient descent; and select the optimal target detection network parameters based on the test data set to obtain the optimal target detection network; S4. Input the sequence images in the sampling window to be detected into the optimal target detection network obtained in step S3, thereby outputting the detection results.

2. The infrared weak moving target detection method based on spatiotemporal feature fusion according to claim 1 is characterized in that: The calculation formula corresponding to the timing information self-optimization module is: in, represents the time series feature of the nth scale, f ChannelAttention represents the channel attention operation, is the multiplication operator, represents a depthwise convolution with a kernel size of 5×5. and Respectively represent the depthwise convolution with kernel size of 7×1 and 1×7, and Respectively represent the depthwise convolution with kernel size of 11×1 and 1×11, and Respectively represent the depthwise convolution with kernel size of 21×1 and 1×21, Represents a point-by-point convolution with a kernel size of 1×1.

3. The infrared weak moving target detection method based on spatiotemporal feature fusion according to claim 2 is characterized in that: The moving small target detection module uses the anchor-free detector from YoloX. It consists of two branches, namely the upper branch and the lower branch. Each of the upper and lower branches includes two 3×3 convolutions and one 1×1 convolution. The final feature map obtained by the spatiotemporal feature mixing module is passed through two 3×3 convolutions on the upper branch to obtain classification features; The classification features are output through the 1×1 convolution of the upper branch; The final feature map obtained by the spatiotemporal feature mixing module is passed through two 3×3 convolutions in the lower branch to obtain the bounding box regression feature; The bounding box regression feature is output through the 1×1 convolution of the lower branch to obtain the bounding box regression output and target information output.

4. The infrared weak moving target detection method based on spatiotemporal feature fusion according to claim 3 is characterized in that: The loss function used in step S3 training is: Loss total =λL reg +L cls +L obj Among them, λ represents a hyperparameter, L reg represents the bounding box regression loss, L cls represents the classification loss, L obj represents the target probability loss; L reg = -log(IoU(B gt ,B pred )), B gt represents the true target bounding box, B pred represents the predicted object bounding box, ∩ is the intersection symbol, ∪ is the union symbol, Indicates the predicted category, p i represents the real category, Represents the predicted target information, c i Indicates the real target information.

5. The infrared weak moving target detection method based on spatiotemporal feature fusion according to claim 4 is characterized in that: According to the calculated loss function, the target detection network parameters are iteratively optimized using gradient descent.

6. The infrared weak moving target detection method based on spatiotemporal feature fusion according to claim 5 is characterized in that: The optimal target detection network parameters are selected based on the test data set. Specifically, the target detection network parameters of each round of training are saved, and then the saved target detection network parameters are tested on the test set. The mAP 50 The target detection network parameters with the best accuracy are taken as the optimal target detection network parameters.

Images