Infrared weak and small moving target detection method based on space-time hybrid coding

Abstract

The invention discloses an infrared weak and small moving target detection method based on space-time hybrid coding, which is applied to the technical field of infrared weak and small moving target detection and aims to solve the problem of low detection performance in infrared weak and small moving target detection in the prior art. The method comprises the following steps: firstly, constructing an infrared weak and small moving target detection network based on spatio-temporal feature fusion, designing a multi-scale spatio-temporal feature extractor, and respectively performing multi-scale feature extraction on an infrared weak and small moving target by using a spatial feature branch and a time sequence feature branch in the multi-scale spatio-temporal feature extractor; a time sequence information self-optimization module is used for carrying out channel and space information self-optimization on extracted multi-scale time sequence features, a space-time feature mixing module is used for fusing multi-level space features and corresponding scale space-time information of the multi-level time sequence features, cross-scale feature fusion is carried out on the multi-level space-time features, and a corresponding loss function is defined, so that the multi-scale time sequence features are obtained. Target positioning prediction and error loss calculation are carried out on the output final features, a target detection positioning result is iteratively optimized step by step, a model with good training parameters is finally obtained, and the purpose of effectively positioning and detecting the infrared weak and small moving target is achieved.

Description

Technical Field

[0001] The invention belongs to the field of computer vision, and in particular relates to an infrared weak moving target detection technology. Background Art

[0002] The detection of infrared weak moving targets has always been a hot and difficult problem in the field of computer vision. Compared with traditional target detection, the purpose of infrared weak moving target detection is to accurately locate weak targets such as aircraft or drones captured by infrared imaging systems. Due to the long imaging distance and inherent imaging mode of infrared imaging, many very large targets in the real world usually look very small in infrared background images or videos, so they usually have no obvious color, shape and texture. On the other hand, infrared targets often have low contrast relative to the background, so such targets are easily submerged in complex background clutter when moving. In summary, unlike traditional target detection, the difficulty of infrared weak moving targets mainly lies in the fact that the targets are too small, the contrast is low, and the features are difficult to capture, which easily leads to problems such as feature disappearance and detection failure.

[0003] In order to solve the problem of infrared dim small target detection, researchers have made various attempts. According to whether training data is needed, the current mainstream methods can be divided into model-driven methods and data-driven methods. In model-driven methods, researchers usually rely on complex manual feature extraction and model parameter adjustment to calculate the maximum contrast between the central pixel of the image and the adjacent area or use the structure of the infrared image to separate the background from the target. However, such methods have poor robustness and have poor infrared dim small target detection effects in complex clutter and noise environments.

[0004] In data-driven methods, learnable models based on deep learning have overcome the shortcomings of traditional models and greatly improved the ability to detect small targets. In this type of method, researchers have proposed a large number of infrared small target segmentation datasets based on single frames. By learning segmentation model data of a large number of single-frame infrared small targets, this type of method has stronger robustness and accuracy than traditional methods and has become the mainstream infrared target detection method.

[0005] However, this type of single-frame based infrared dim target detection method does not consider the necessary and sufficient spatiotemporal feature information, so it often loses target information when detecting infrared dim moving targets. In addition, they model the infrared target detection task as a more sophisticated segmentation task. For dim infrared target detection, the target is too small and the segmentation is often unreasonable.

[0006] In order to further improve the detection performance of the model for infrared weak moving targets, it is necessary to introduce timing information with reference to the human visual system. However, the current moving infrared small target detection methods rarely consider spatiotemporal information. The only moving infrared timing information detection models often directly use the inter-frame attention mechanism or LSTM-based models, which will introduce too much interference information and computational load.

[0007] In summary, the existing infrared small moving target detection scheme has three obvious defects. First, the model-driven method has no feature learning ability, relies on manual features and model parameter adjustment, and has poor generalization ability. Second, the data-driven detection method based on a single frame does not consider the temporal correlation of moving targets, and it is not reasonable to model the segmentation task. Finally, the data-driven method based on multiple frames is still in the exploratory stage, and the spatiotemporal feature extraction and fusion strategy of infrared small moving targets have not been fully explored, which inhibits the detection performance of infrared small moving targets. Summary of the invention

[0008] In order to solve the above technical problems, the present invention proposes an infrared dim small target motion detection method based on spatiotemporal feature hybrid coding.

[0009] The technical solution adopted by the present invention is: a method for detecting infrared weak moving targets based on spatiotemporal feature fusion, comprising:

[0010] S1. Build a target detection network; the target detection network includes: a spatiotemporal feature extractor module, a temporal information self-optimization module, a spatiotemporal feature mixing module, and a moving weak target detection module;

[0011] The spatiotemporal feature extractor module includes a spatial feature branch and a temporal feature branch; the spatial feature branch is composed of the backbone network CSPDarknet in YOLOX, which is used to extract the multi-scale spatial features of the sequence images in the current sampling window; the temporal feature branch is composed of the backbone network 3D-ShuffleNetV2, which is used to extract the multi-scale temporal features of the sequence images in the current sampling window;

[0012] The temporal information self-optimization module includes a channel self-attention submodule and a spatial self-attention submodule. The channel self-attention submodule performs weighted adjustment of channel features on the multi-scale temporal features extracted by the temporal feature branch. The spatial self-attention submodule performs weighted adjustment of spatial features on the multi-scale temporal features after weighted adjustment of channel features, and finally obtains the self-optimized multi-scale temporal features.

[0013] The spatiotemporal feature hybrid module includes a spatiotemporal feature hybrid encoder and a multi-level feature hybrid encoder. The multi-scale spatial features and the self-optimized multi-scale temporal features are fused with the spatiotemporal features of the corresponding scales through the hybrid encoder module. The multi-level feature hybrid encoder performs multi-scale information fusion on the spatiotemporal fusion features of multiple scales output by the hybrid encoder module to obtain the final feature map.

[0014] The moving small target detection module is based on the detection head of YOLOX and performs target detection on the feature map output by the spatiotemporal feature mixing module;

[0015] S2, build training data set;

[0016] S3, iteratively optimizing the target detection network parameters constructed in step S1 by using the training data set based on gradient descent;

[0017] S4, inputting the sequence image in the sampling window to be detected into the target detection network obtained after training in step S3, thereby outputting the detection result.

[0018] Beneficial effects of the present invention: The method of the present invention first constructs an infrared weak moving target detection network based on spatiotemporal feature fusion, designs a multi-scale spatiotemporal feature extractor, uses the spatial feature branch and the temporal feature branch to extract multi-scale features of infrared weak moving targets respectively, uses the temporal information self-optimization module to self-optimize the channel and spatial information of the extracted multi-scale temporal features, uses the spatiotemporal feature mixing module to fuse the corresponding scale spatiotemporal information of multi-level spatial features and multi-level temporal features, and performs cross-scale feature fusion on multi-level spatiotemporal features. By defining the corresponding loss function, the output final features are used to predict the location of the target, calculate the error loss, and gradually iterate and optimize the target detection and positioning results, and finally obtain a model with good training parameters to achieve the purpose of effectively positioning and detecting infrared weak moving targets. The method of the present invention makes the detection and positioning of infrared weak moving targets more accurate and robust, can effectively detect the sequence images of moving weak targets collected by infrared equipment, detect and locate the moving weak targets such as drones and trucks in the target area, and can be applied to important application fields such as drone supervision and security. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 The present invention is a flow chart of an infrared weak moving target detection method based on spatiotemporal feature fusion.

[0020] Figure 2 This is a diagram of the infrared weak moving target detection network structure designed in an embodiment of the present invention.

[0021] Figure 32 is a structural diagram of the temporal feature self-optimization module proposed in an embodiment of the present invention; wherein (a) is the overall structural diagram, (b) is the channel attention detail diagram, and (c) is the spatial attention detail diagram.

[0022] Figure 4 It is a structural diagram of the spatiotemporal feature mixing module proposed in an embodiment of the present invention;

[0023] Among them, (a) is the structure diagram of the spatiotemporal feature hybrid encoder, and (b) is the structure diagram of the multi-level feature hybrid encoder.

[0024] Figure 5 It is a PR curve diagram on the data set in an embodiment of the present invention;

[0025] Among them, (a) is the PR graph on IRDST, and (b) is the PR graph on ITSDT-15K.

[0026] Figure 6 Graphs showing visualization results of detection boxes under two public data sets in an embodiment of the present invention;

[0027] Among them, (a) is the detection visualization result on the IRDST dataset, and (b) is the detection visualization result on the ITSDT-15K dataset. DETAILED DESCRIPTION

[0028] To facilitate those skilled in the art to understand the technical content of the present invention, the present invention is further explained below with reference to the accompanying drawings.

[0029] like Figure 1 As shown, the implementation process of the infrared weak moving target detection method based on spatiotemporal feature fusion of the present invention specifically includes the following steps:

[0030] S1. Construct an infrared weak moving target detection network based on spatiotemporal feature fusion;

[0031] Construct an intelligent detection network specifically for infrared weak moving target detection, that is, an infrared weak moving target detection network based on spatiotemporal feature fusion. The detailed structure of the network is as follows: Figure 2 As shown, it includes: a spatiotemporal feature extractor module, a temporal information self-optimization module, a spatiotemporal feature mixing module, and a moving weak target detection module, which are as follows:

[0032] (1) In the spatiotemporal feature extractor module, in order to enable the model to extract information-rich spatial features and temporal features, such as Figure 2 As shown in the upper part, a multi-scale spatial feature branch and a multi-scale temporal feature branch are designed. In the present invention, the multi-scale spatial feature branch and the multi-scale temporal feature branch will generate three corresponding scale features.

[0033] First, for the spatial feature branch, the last frame is selected from the T sequence frames in the given time domain sampling window as the key frame to be detected, that is, the image sequence I of the input T frames is s =[I1,I2,…,I T ]Key frame is I T .

[0034] Then key frame I T The spatial feature branch in the present invention uses the pre-trained backbone network in YOLOX as the backbone feature extraction network of this branch, and outputs 32 times downsampling, 16 times downsampling, and 8 times downsampling spatial features respectively. The calculation process is as follows:

[0035]

[0036] in Is the input keyframe I T After that, the upper layer output in the CSPDarknet network, Represents the convolution operation of the dark3 layer in CSPDarknet.

[0037] Secondly, for the temporal feature branch, in a given time domain sampling window of T sequence frames, the input sequence I s =[I1,I2,…,I T ] to the time series feature branch for multi-level time series feature extraction. The time series feature branch uses the 3D-ShuffleNetV2 network as the backbone, through which the sequence I is obtained. s =[I1,I2,…,I T The timing characteristics of Then will Upsampling is performed to obtain multi-scale time series features, and the calculation process is as follows:

[0038]

[0039] Among them, Bilinear ×2 and Bilinear ×4 They represent 2x upsampling and 4x upsampling of bilinear interpolation respectively. Represents the features output by the 3D-ShuffleNetV2 backbone network.

[0040] (2) The spatiotemporal feature fusion module includes: a channel self-attention module and a spatial self-attention module based on CPCANet;

[0041] Among them, the channel self-attention module uses average pooling and maximum pooling as well as two layers of fully connected layer operations to obtain the channel self-attention map for extracting multi-level temporal features, while the spatial self-attention module uses multi-scale depthwise separable convolution modules and aggregates the features they extract to generate spatial channel self-attention maps of multi-level temporal features.

[0042] In the time series information self-optimization module, the multi-scale time series features extracted by the time series feature branch are self-optimized and adjusted to further enhance the time series feature representation, such as Figure 3 As shown in the figure, a temporal information self-optimization module is designed, in which the channel attention submodule first outputs the channel self-attention map and generates the temporal features after the channel features are adjusted, and then the spatial attention submodule is used to output the spatial self-attention map and generate the final channel feature adjustment and spatial feature adjustment temporal feature map;

[0043] The channel attention submodule includes multiple layers of fully connected layers and pooling layers, and the spatial attention submodule includes multiple groups of depth-wise separable convolutions. When performing self-optimization of timing characteristics, the overall calculation formula can be expressed as:

[0044]

[0045] Similarly, input the time series features Self-optimizing timing features Among them, inspired by the CPCANet network, the same as CPCA, f ChannelAttention represents the channel attention operation, represents a depth-wise separable convolution operation with a kernel size of i×j, i.e. represents a depthwise convolution with a kernel size of 5×5. and They represent depth-wise convolutions with kernel sizes of 7×1 and 1×7, respectively. and They represent depth-wise convolutions with kernel sizes of 11×1 and 1×11, respectively. and They represent depth-wise convolutions with kernel sizes of 21×1 and 1×21, respectively. Represents a point-by-point convolution with a kernel size of 1×1. Specifically, after obtaining the temporal feature map after channel feature optimization through a channel attention module consisting of two layers of full connection and pooling layers, four sets of depthwise separable convolutions generate the corresponding spatial self-attention map, and 1×1 convolution is used to fuse different spatial feature maps to generate the final spatial feature map and self-optimize the temporal features; Indicates the third-level timing characteristics. In the process of generating multi-level self-optimization timing characteristics, the first, second and third level timing characteristics are input respectively. The timing information self-optimization module generates optimized multi-level timing features

[0046] (3) The spatiotemporal feature mixing module includes: a spatiotemporal feature mixing encoder and a multi-level feature mixing encoder; wherein the spatiotemporal feature mixing encoder is composed of multiple convolutions and channel self-attention operations, and the multi-scale feature mixing encoder is composed of multiple layers of convolution and bilinear interpolation.

[0047] In the spatiotemporal feature mixing module, the extracted multi-level spatial features of infrared weak moving targets and the optimized temporal features are mixed and encoded into information-rich spatiotemporal features to further enhance the feature representation, such as Figure 4 As shown in the figure, a spatiotemporal feature hybrid module consisting of a spatiotemporal feature hybrid encoder and a multi-level feature hybrid encoder is designed. The spatiotemporal feature hybrid encoder is first used to fuse the temporal and spatial features of the corresponding scale to obtain rich multi-level spatiotemporal features, and then the multi-level spatiotemporal features are fused using the multi-level feature hybrid encoder to obtain the final feature map.

[0048] The spatiotemporal feature hybrid encoder submodule includes multiple convolutional layers and self-attention operations. The specific calculation process of this module can be expressed as:

[0049]

[0050] in They are the self-optimized temporal and spatial features of the corresponding scales, By the corresponding Generated by timing self-optimization, f self-att represents the self-attention mechanism operation, f conv represents the convolution operation, Represents the corresponding fused spatiotemporal feature output; similarly, and In the same way, as well as The fusion is produced as follows:

[0051]

[0052] The multi-level feature hybrid encoder consists of multiple groups of convolutions, bilinear interpolation upsampling and N layers of CSPLayer, which is used to receive the multi-level spatiotemporal features fused by the spatiotemporal feature hybrid encoder and fuse the multi-level spatiotemporal features into the final feature map. The specific calculation process of this module can be expressed as:

[0053]

[0054] where f conv represents the convolution operation, f concat represents the feature concatenation operation, fBil stands for twice bilinear interpolation, represents n layers of CSPLayer convolution, It is the final output feature map after the fusion of multi-level spatiotemporal features.

[0055] (4) The moving small target detection module includes: a small target detection head designed based on the YOLOX technology framework; the detection head that has been trained with parameters finally completes the detection and positioning operations of infrared small moving targets.

[0056] S2, extracting primary multi-scale features of infrared weak moving targets respectively based on the temporal feature branch and the spatial feature branch in the feature extractor of the target detection network in step S1;

[0057] In the multi-level spatiotemporal feature extraction stage, the multi-scale spatiotemporal feature extractor uses a spatial feature extractor and a temporal feature extractor to perform multi-scale extraction on the spatial features and temporal features of the infrared weak moving target respectively.

[0058] First, the last frame image I of the sequence image with a sampling time window of T is T As key frame input to the spatial feature extractor for multi-scale spatial feature extraction. T Input into the 2D backbone of the CSPDarknet pre-trained model, and get the spatial feature encoding of 32 times, 16 times and 8 times downsampling respectively:

[0059] Secondly, the sequence image I with a sampling time window of T s =[I1,I2,…,I T ] is input into the 3D backbone temporal feature branch with 3D-ShuffleNetV2 as the backbone network, and a 32-fold downsampled feature map containing rich temporal information is obtained. Then the time series characteristic diagram Perform 2x and 4x upsampling respectively to obtain multi-level time series feature maps

[0060] S3, based on the temporal information self-optimization module of the target detection network in step S1, performing attention adaptive adjustment on the temporal multi-scale feature information extracted by the temporal feature branch in step S2;

[0061] In the multi-level spatiotemporal feature extraction stage, the multi-scale spatiotemporal feature extractor obtains the multi-scale spatial feature X S and multi-scale temporal features X T ;

[0062] In the time series information self-optimization stage, the time series self-optimization module performs self-optimization adjustment on the extracted multi-level time series features, that is, the multi-scale time series features Input the timing self-optimization module, and then use the channel attention module composed of multiple linear layers and pooling layers to perform weighted adjustment of channel features, and the spatial attention module composed of multiple groups of deep separable convolutions to perform weighted adjustment of spatial features, and finally obtain the self-optimized timing features.

[0063] S4, based on the spatiotemporal feature mixing module of the target detection network in step S1, performing cross-domain and multi-scale information fusion on the multi-scale spatial features extracted in step S2 and the self-optimized multi-scale temporal features extracted in step S3;

[0064] In the multi-scale feature fusion stage, the spatiotemporal feature mixing module first performs spatiotemporal feature fusion of corresponding scales through the spatiotemporal feature mixing encoder. and are sent to the hybrid encoder module respectively, corresponding to the scale and and and The spatiotemporal features are fused through feature concatenation, convolution, and self-attention operations to obtain the corresponding fused spatiotemporal feature map. Then, X is transformed into fused Perform multi-scale information fusion to obtain the final feature map

[0065] S5. Based on the moving small target detection module of the detection network described in step S1, by defining a relevant loss function, the output features of step S4 are used to perform target positioning detection, error loss calculation, and iterative optimization of target detection accuracy.

[0066] In the target positioning and detection stage, the infrared weak moving target detection module uses an anchor-free detector from YoloX; it includes two branches, the input features are respectively passed through two 3×3 convolution upper branches to obtain cls features and two 3×3 convolution lower branches to obtain reg features, and then the cls features are passed through a 1×1 convolution to obtain the cls output, and the reg features are passed through a 1×1 convolution to obtain the reg output and obj output, and the total loss function is calculated for these three output loss items at the same time, and the calculation rules used are as follows:

[0067] Loss total = λL reg +L cls +L obj (8)

[0068] Among them, cls is the abbreviation of classification, which means classification; reg is the abbreviation of regression, which means bounding box regression; obj is the abbreviation of objectness, which means target information; λ is a hyperparameter used to balance the three loss terms, L reg represents the bounding box regression loss, L cls represents the classification loss, L obj represents the target probability loss; L cls With L obj The ordinary binary cross entropy loss is used for calculation, L reg The common IoU loss is used for calculation. IoU represents the intersection-over-union ratio of two overlapping areas, as follows:

[0069] L reg = -log(IoU(B gt ,B pred )) (9)

[0070]

[0071] Among them, B gt represents the true target bounding box, B pred represents the predicted object bounding box, represents the predicted category, p i represents the real category, represents the predicted target information, c i Indicates the real target information.

[0072] Then, according to the calculated loss, the model uses gradient descent to iteratively optimize the parameters. The method of the present invention trains 100 rounds, saves the training model parameters of each round, and then tests the saved model parameters on the test set to select the mAP 50 The model with the best accuracy is selected to obtain the model parameters with higher training performance and the final detection results.

[0073] Those skilled in the art will know that mAP 50 Precision is a very important indicator in object detection, which is used to measure the average accuracy of the model when the IoU (intersection over union) threshold is 0.5.

[0074] The present invention also takes the public infrared image sequence data sets IRDST and ITSDT as examples to further illustrate the method of the present invention.

[0075] The IRDST dataset is divided into a real dataset and a synthetic dataset. The real dataset consists of 85 image sequences and 40,656 images, of which 42 sequences with 20,398 images are used as training sets, and the remaining 43 sequences with 20,258 images are used as test sets; the ITSDT dataset consists of 87 image sequences and 21,750 images, of which 40 sequences with 10,000 images are selected as training sets, and 20 sequences with 5,000 images are selected as test sets to form the new ITSDT-15K dataset.

[0076] (1) Initialization of the experiment:

[0077] Before inputting the infrared image, all images in the training set and the test set are converted to 512×512 pixels, the sampling time window T is set to 5, and then the image pixel values ​​are normalized and input into the model for infrared weak moving target detection and positioning. In this embodiment, SGD is used as the optimizer, the momentum value is set to 0.937, the batch size is set to 4, the initial learning rate is set to 0.01, and the weight decay is 5×10 -4 , the learning rate reduction coefficient is 0.1, and 100 epochs are trained on the dataset. All evaluation experiments of the model of the method of the present invention are carried out on two NVIDIA RTX 3090 graphics cards. In addition, in the infrared weak moving target detection experiment of this embodiment, the evaluation indicators mainly include recall rate, precision, F1 index and mAP 50

[0078] (2) Multi-scale spatiotemporal feature extraction stage:

[0079] After the preprocessed image data is input into the model, the key frames and infrared sequence images in the infrared sequence images are respectively extracted through the spatial feature branch and the temporal feature branch according to the steps of formula (1) and (2) to obtain the multi-scale spatial features X of the infrared dim target. S and multi-scale temporal features X T .

[0080] (3) Timing information self-optimization stage

[0081] Get multi-scale time series features Afterwards, if Figure 3 As shown, the multi-scale time series feature X T The different scale features of are input into the time series information self-optimization module respectively. The self-time series information optimization module can effectively self-adjust the channel features and spatial features of the multi-scale time series information features according to formula (3), so that more effective multi-scale time series features can be obtained, and more effective spatiotemporal information interaction can be carried out with multi-scale spatial features.

[0082] (4) Spatiotemporal feature mixing stage:

[0083] Get the multi-scale spatial characteristics of infrared weak moving targets and self-optimizing timing features After that, the corresponding spatial scale features and temporal scale features are first input into the spatiotemporal feature hybrid encoder. The spatiotemporal feature hybrid encoder can effectively fuse the corresponding spatial scale features and temporal scale features according to equations (4), (5), and (6) to form a multi-scale spatiotemporal feature. Then the multi-scale spatiotemporal feature X fused Input to the multi-level feature hybrid encoder to fuse multi-scale spatiotemporal features according to formula (7) to obtain the final feature

[0084] (5) Detection head and loss function:

[0085] In this embodiment, two branches are used as detection heads to detect infrared weak targets. The upper branch predicts the cls output of infrared weak targets, and the lower branch predicts the reg output and obj output. Each branch consists of two layers of 3×3 convolution to extract corresponding features. Finally, the loss is calculated by formula (8), and the back propagation gradient descent algorithm is used to effectively learn the model to obtain model parameters with higher training performance, thereby achieving the purpose of effectively detecting weak moving targets in infrared videos.

[0086] In order to verify the effectiveness of the proposed method in infrared moving small target positioning detection, training was performed on the public infrared sequence data sets IRDST and ITSDT-15K respectively. First, the proposed method is compared with some traditional infrared small target detection methods (MAXMEAN, TopHat, RLCM, HBMLCM, PSTNN, WSLCM. In addition, 13 deep learning network methods (ACM, RISTD, ISNet, UIUNet, SANet, AGPCNet, RDIAN, DNANet, ResUnet*DTUM, SIRST5K, MSHNet, RPCANet, SSTNet) are also compared, and the results are shown in Table 1:

[0087] Table 1 Comparison of the effects of the method of the present invention and the traditional method

[0088]

[0089] Table 1 shows the comparison of mAP50, precision, recall, and F1 indicators of each model on the public infrared sequence datasets IRDST and ITSDT-15K, where the best precision is marked in bold and the second is marked by underline. Through the comparison, three obvious findings can be observed.

[0090] 1) The mAP50 index of the proposed method is always significantly better than all other current methods. For example, on the IRDST dataset, the mAP50 index of the proposed method is 77.33%, while the second highest index of SSTNet is 76.96%. The mAP50 index of the proposed method is 0.37% higher than that of the second highest method.

[0091] 2) On the IRDST and ITSDT-15K datasets, the proposed method outperforms most other methods in terms of accuracy. For example, in the ITSDT-15K dataset, the Pr index of the proposed method is the highest, with a value of 92.42%. On the IRDST dataset, the Pr index of the proposed method is the second highest, with a value of 87.78%, second only to 88.56% of SSTNet.

[0092] 3) For the F1 index, the method of the present invention is always better than all other currently compared methods. For example, on the ITSDT-15K dataset, the F1 value of the method of the present invention is 88.21%, which is better than the second highest SSTNet's 88.07%. On the IRDST dataset, the F1 value of the method of the present invention is 85.96%, which is better than the second highest SSTNet's 85.11%. This shows that although the precision and recall rate of the method of the present invention are not always better than other methods, the comprehensive detection ability of the present invention is stronger.

[0093] In addition, in order to further evaluate the efficiency performance of the present invention, this embodiment selects 13 deep learning-based models for comparison of reasoning performance. They are ACM, RISTD, ISNet, UIUNet, SANet, AGPCNet, RDIAN, DNANet, SIRST5K, MSHNet, RPCANet, DTUM and DSFNet. mAP50 and F1 are used to measure the accuracy of the model, Flops and Params are used to measure the complexity of the model, and PCR is the ratio of mAP50 and Flops defined to measure the overall efficiency of the model. Table 2 shows the numerical comparison results of this embodiment on the ITSDT-15K dataset.

[0094] Table 2 Numerical comparison results of this embodiment on the ITSDT-15K dataset

[0095]

[0096] By comparison, it is clear that we can observe three important findings. First, the model accuracy of the proposed method is higher than that of other methods, and the accuracy of the spatiotemporal feature fusion model is higher than that of other methods. For example, DTUM, SSTNet and the proposed method all combine temporal motion information for detection, and the mAP50 index and F1 index are significantly better than the methods that rely on spatial features in the upper half of Table 2. Secondly, in the spatiotemporal feature model, the proposed method stands out for its simplicity and lightness. For example, among the current infrared weak target detection models, the ACM model that only uses spatial features has the lowest Flops of 24.66G, while the proposed method ranks second at 41.92G, which is much lower than the 123.59G of the spatiotemporal feature model SSTNet. Finally, considering the performance cost ratio PCR, which is the ratio of mAP50 to Flops, the proposed method has a PCR value of 1.84, ranking second, and obtains more favorable results than the spatiotemporal feature model SSTNet, indicating that our model has advantages.

[0097] In order to intuitively compare the overall performance of different detection methods, this example creates a PR curve based on precision and recall, and plots the PR curves on ITSDT-15K and IRDST respectively. Figure 5 As shown. Two obvious things can be found. First, the curve of the method of the present invention is mainly located in the upper right of the two figures, which shows that the comprehensive performance of the model of the method of the present invention is better than that of other models. Second, another method using spatiotemporal features is also higher than other methods that only use spatial features, showing the importance of using temporal information.

[0098] In addition to the numerical performance comparison, in order to intuitively illustrate the detection accuracy of the model of the method of the present invention compared with the current most advanced model method, this embodiment visualizes two sets of small target detection results of eight methods on the IRDST dataset and the ITSDT-15K dataset, respectively. Figure 6 As shown. Through visual comparison, two obvious findings can be observed. First, in the detection results of the IRDST dataset, almost no method produces false detection or missed detection, which may be because the randomly selected samples have a simple background and no occlusion. Despite this, the method of the present invention can still generate a detection box with the maximum similarity to the actual target and tightly surround the target. On the other hand, in the detection results of the ITSDT-15K dataset, DNANet, AGPCNet and RPCANet failed to detect poor dim targets and occlusions, while the method of the present invention generated a correct and tight detection bounding box, highlighting the superior performance of the method of the present invention in infrared dim moving target detection.

[0099] In summary, the method of the present invention makes the detection of infrared weak moving targets more accurate and robust, can effectively detect the infrared moving weak target sequence images collected by the infrared imaging equipment, and detect the infrared weak moving targets appearing in the target area, and can be applied to important application fields such as drone supervision and security.

[0100] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. For those skilled in the art, the present invention may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A method for detecting infrared weak moving targets based on spatiotemporal feature fusion, characterized in that: include: S1, build target detection network; The target detection network includes: a spatiotemporal feature extractor module, a temporal information self-optimization module, a spatiotemporal feature mixing module, and a moving weak target detection module; The spatiotemporal feature extractor module includes a spatial feature branch and a temporal feature branch; the spatial feature branch is composed of the backbone network CSPDarknet in YOLOX, which is used to extract the multi-scale spatial features of the sequence images in the current sampling window; the temporal feature branch is composed of the backbone network 3D-ShuffleNetV2, which is used to extract the multi-scale temporal features of the sequence images in the current sampling window; The temporal information self-optimization module includes a channel self-attention submodule and a spatial self-attention submodule. The channel self-attention submodule performs weighted adjustment of channel features on the multi-scale temporal features extracted by the temporal feature branch. The spatial self-attention submodule performs weighted adjustment of spatial features on the multi-scale temporal features after weighted adjustment of channel features, and finally obtains the self-optimized multi-scale temporal features. The spatiotemporal feature hybrid module includes a spatiotemporal feature hybrid encoder and a multi-level feature hybrid encoder. The multi-scale spatial features and the self-optimized multi-scale temporal features are fused with the spatiotemporal features of the corresponding scales through the hybrid encoder module. The multi-level feature hybrid encoder performs multi-scale information fusion on the spatiotemporal fusion features of multiple scales output by the hybrid encoder module to obtain the final feature map. The moving small target detection module is based on the detection head of YOLOX and performs target detection on the feature map output by the spatiotemporal feature mixing module; S2, dividing the collected infrared images into a training data set and a test data set; S3, iteratively optimize the target detection network parameters constructed in step S1 by using the training data set based on the gradient descent method; and select the optimal target detection network parameters based on the test data set to obtain the optimal target detection network; S4, inputting the sequence image in the sampling window to be detected into the optimal target detection network obtained in step S3, thereby outputting the detection result.

2. The infrared weak moving target detection method based on spatiotemporal feature fusion according to claim 1 is characterized in that: The multi-scale spatial features of the sequence images in the current sampling window are extracted. Specifically, the last frame of the sequence frames in the current sampling window is used as the key frame to be detected, and the key frame is input into the CSPDarknet network to extract the multi-scale spatial features.

3. The infrared weak moving target detection method based on spatiotemporal feature fusion according to claim 2 is characterized in that: The multi-scale temporal features of the sequence images in the current sampling window are extracted. Specifically, the sequence frames in the current sampling window are input into the 3D-ShuffleNetV2 backbone network to obtain the first-scale temporal features, and the first-scale temporal features are upsampled by different multiples to obtain multi-scale temporal features.

4. The infrared weak moving target detection method based on spatiotemporal feature fusion according to claim 3 is characterized in that: The calculation formula corresponding to the timing information self-optimization module is: in, represents the time series features of the nth scale, is the multiplication operator, represents a depthwise convolution with a kernel size of 5×5. and They represent depth-wise convolution with kernel sizes of 7×1 and 1×7 respectively. and They represent depth-wise convolution with kernel sizes of 11×1 and 1×11 respectively. and They represent depth-wise convolution with kernel sizes of 21×1 and 1×21 respectively. Represents a point-wise convolution with a kernel size of 1×1.

5. The infrared weak moving target detection method based on spatiotemporal feature fusion according to claim 4 is characterized in that: The moving small target detection module uses the anchor-free detector from YoloX; it includes two branches, namely the upper branch and the lower branch; the upper and lower branches each include two 3×3 convolutions and one 1×1 convolution; The final feature map obtained by the spatiotemporal feature mixing module is passed through two 3×3 convolutions in the upper branch to obtain classification features; The classification features are output through the 1×1 convolution of the upper branch; The final feature map obtained by the spatiotemporal feature mixing module is passed through two 3×3 convolutions in the lower branch to obtain the bounding box regression feature; The bounding box regression feature is output through the 1×1 convolution of the lower branch to obtain the bounding box regression output and target information output.

6. The infrared weak moving target detection method based on spatiotemporal feature fusion according to claim 5 is characterized in that: The loss function used in step S3 training is: Loss total =λL reg +L cls +L obj Among them, λ represents a hyperparameter, L reg represents the bounding box regression loss, L cls represents the classification loss, L obj represents the target probability loss; L reg = -log(IoU(B gt , B pred )), B gt represents the true target bounding box, B pred represents the predicted object bounding box, ∩ is the intersection symbol, ∪ is the union symbol, represents the predicted category, p i represents the real category, represents the predicted target information, c i Indicates the real target information.

7. The infrared weak moving target detection method based on spatiotemporal feature fusion according to claim 6 is characterized in that: According to the calculated loss function, the target detection network parameters are iteratively optimized using gradient descent.

8. The infrared weak moving target detection method based on spatiotemporal feature fusion according to claim 7 is characterized in that: The optimal target detection network parameters are selected based on the test data set. Specifically, the target detection network parameters of each round of training are saved, and then the saved target detection network parameters are tested on the test set. The mAP 50 The target detection network parameters with the best accuracy are taken as the optimal target detection network parameters.

Images