Infrared weak and small target detection tracking method based on time-space domain trajectory fitting

Abstract

The invention relates to an infrared weak and small target detection and tracking method based on time-space domain trajectory fitting, and belongs to the technical field of infrared weak and small target detection and tracking. An infrared weak and small target track fitting data set Data-match is constructed; building an infrared weak and small target detection network model Model-Det; the Model-Det is trained and verified by using the Data-match repeatedly, and the Model-Det is verified; obtaining an infrared weak and small target detection candidate box Bbox; the method comprises the following steps of: inputting a target trajectory A of each sequence into a Kalman filtering unit, and generating a target trajectory Trajectory-A of each sequence; building an infrared weak and small target track segmentation network model Model-Seg; the Model-Seg is trained and verified by repeatedly using the Data-match, and the Model-Seg is verified; a target track Trajectory-B of each sequence is obtained; and taking an intersection of the Trajectory-A and the Trajectory-B to obtain a final infrared weak and small target detection tracking result and target trajectory output. According to the method, the data dependence is reduced, and the infrared weak and small target can be stably and accurately detected in different scenes.

Description

Technical Field

[0001] This invention belongs to the field of infrared weak target detection and tracking technology, and relates to an infrared weak target detection and tracking method based on spatiotemporal trajectory fitting. Background Technology

[0002] In the field of target detection, the detection of small infrared targets has always been an important research hotspot. Small infrared targets have limited pixels, indistinct geometric features, and weak intensity, making them susceptible to interference from strong clutter and noise. During movement, they may exhibit phenomena such as flickering and discontinuity, making accurate detection and tracking difficult.

[0003] To meet the requirements of infrared weak target detection and tracking, it is necessary to combine current advanced technologies and optimize the technology according to the characteristics of infrared weak targets. In recent years, deep learning-based target detection models have been successfully applied to infrared target detection. Compared with traditional methods, deep learning models do not require manual feature design and can autonomously learn deep target features, significantly improving algorithm development efficiency and detection accuracy. However, deep learning-based infrared weak target detection methods also have two main problems: ① This method only utilizes the spatial domain information of a single frame of infrared image and does not utilize the temporal domain information of target motion in previous and subsequent frames; ② This method is highly dependent on data, performs well in detecting learned infrared weak targets and scene-specific targets, but has poor generalization ability to different scenes and different targets. Summary of the Invention

[0004] The technical problem solved by this invention is to overcome the shortcomings of the prior art and propose an infrared weak target detection and tracking method based on spatiotemporal trajectory fitting. This method fully explores the temporal and spatial information in infrared image sequences, while reducing data dependence, and can achieve stable and accurate detection of infrared weak targets in different scenarios.

[0005] The solution of the present invention is:

[0006] An infrared weak target detection and tracking method based on spatiotemporal trajectory fitting includes:

[0007] Step 1: Construct the infrared weak target trajectory fitting dataset Data-match; the dataset Data-match includes a continuous frame infrared image sequence, target location information of a single frame image in the sequence, and target trajectory information of the entire sequence;

[0008] Step 2: Construct the infrared weak target detection network model Model-Det; use 3D residual convolution as the input unit at the detection input end; the feature extraction network adopts a multi-attention bottleneck residual module; to achieve targeted feature extraction for infrared weak targets; the input of the infrared weak target detection network model Model-Det is 10 consecutive frames of infrared image sequence, and the output is target candidate box information;

[0009] Step 3: Repeat the infrared weak target trajectory fitting dataset from Step 1 (Data-match) to train and validate the infrared weak target detection network model (Model-Det) from Step 2 until an infrared weak target detection model with a detection rate exceeding 95% is obtained; the resulting infrared weak target detection candidate boxes (Bboxes) are obtained.

[0010] Step 4: Input the infrared weak target detection candidate box (Bbox) from Step 3 into the Kalman filter unit to achieve preliminary target tracking; generate the target trajectory Trajectory-A for each sequence;

[0011] Step 5: Construct the infrared weak target trajectory segmentation network model Model-Seg, with the main architecture being U-Net; the feature extraction network adopts a multi-attention bottleneck residual module; the first layer of conventional convolution in each feature downsampling is optimized into dilated convolution; thus improving trajectory extraction performance; the input of this model is the entire infrared sequence image, and the output is the overall trajectory information of all targets in the sequence;

[0012] Step 6: Repeat the Data-match training of the infrared weak target trajectory fitting dataset in Step 1 and the infrared weak target trajectory segmentation network model Model-Seg in Step 5 until a trajectory segmentation model with an overall accuracy of over 90% is obtained, resulting in the target trajectory Trajectory-B for each sequence.

[0013] Step 7: Fit and compare the target trajectory Trajectory-A from Step 4 with the target trajectory Trajectory-B from Step 6, and take the intersection of the two to obtain the final infrared weak target detection and tracking result and target trajectory output.

[0014] In the above-mentioned infrared weak target detection and tracking method based on spatiotemporal trajectory fitting, in step one, the continuous frame infrared image sequence is in 8-bit grayscale PNG format; the image is identified by the file name to indicate the frame number;

[0015] The target location information of a single frame image in the sequence is in txt format. The file name corresponds one-to-one with the continuous frame sequence images. The location content includes 5 columns of attribute data, namely Class, Xi, Yi, Width, and Height. Among them, Class is the category of the labeled object; Xi and Yi are the center points of the labeled object; Width and Height are the width and height of the labeled box; all values ​​are percentages of the actual image.

[0016] The target trajectory information for the entire sequence is in JSON format or converted to PNG image format; the PNG format visualizes the trajectory of all targets in the entire infrared image sequence, using 8-bit storage, with a background pixel value of 0 and a target trajectory pixel value of 255.

[0017] In the aforementioned infrared weak target detection and tracking method based on spatiotemporal trajectory fitting, the data-match dataset is constructed as follows:

[0018] First, obtain the base image of the real infrared sequence data: Data = {I1, I2, ... I n}; where I i This represents a single frame image in an infrared sequence base map; based on the characteristics of infrared aerial targets, target simulation is added to the base map Data to obtain an infrared weak target trajectory fitting dataset Data-match;

[0019] The image size is 1024×1024, and the target is one to several pixels. During simulation, the target size is set to 1×1, 1×2, 1×3, 2×2, 2×3, and 3×3. The target energy is set to 8 to 10 times the temporal signal-to-noise ratio of the infrared base image sequence. The target energy is the pixel value of the target. The energy diffuses and decays from the center, and the decay function is a Gaussian function. The target's initial position is set to random. The target trajectory is set to a linear function, a quadratic function, a cubic function, or a Bézier curve. The target velocity is set to move 1 to 3 pixels between adjacent frames. During the simulation, the target is sequentially superimposed onto the real infrared sequence image base image I. i The simulation is completed, and the following data are obtained: target sequence images, target location information in each frame, and target trajectory information for the entire image sequence. The simulation dataset is divided into a training set (train), a validation set (val), and a test set (test). The ratio of training set to validation set to test set is 3:1:1. The training set contains 18,000 images, the validation set contains 3,600 images, and the test set contains 3,600 images. Each sequence contains 360 images. The infrared weak target trajectory fitting dataset (Data-match) is obtained.

[0020] In the aforementioned infrared weak target detection and tracking method based on spatiotemporal trajectory fitting, the 3D residual convolution input unit in step two is specifically:

[0021] When an image is input to the network input terminal, a sliding window input strategy is used; the current frame image is I. n At that time, taking the current frame as the intermediate moment, 10 adjacent frames are superimposed, specifically {I n-4 ,I n-3 ,I n-2 ,I n-1 ,I n ,I n+1 ,I n+2,I n+3 ,I n+4 ,I n+5 The parameter is adjusted according to the image background and target characteristics; the 3D residual convolution input unit fuses the input 10 frames of original images with the output feature map through 3D convolution, and inputs it into the subsequent feature extraction network to realize the extraction of spatiotemporal information.

[0022] The multi-attention bottleneck residual module is:

[0023] By leveraging spatial attention and channel attention, combined with depthwise separable convolution and residual structures, we can improve the deep feature extraction capability of the backbone network while maintaining model lightweightness, reducing hardware computing power requirements, and increasing detection speed. Channel attention calculates the weight of each channel based on input features. Different channels typically represent different objects. In the infrared small target detection problem, channel attention is used to distinguish between targets and target-like noise. Spatial attention calculates the weight of each pixel based on input features. In the infrared small target detection problem, spatial attention is used to enhance pixels at the target location while suppressing non-target pixels.

[0024] In the aforementioned infrared weak target detection and tracking method based on spatiotemporal trajectory fitting, the parameter count and computational cost of depth-separable convolution and standard convolution during convolution operations are compared as follows:

[0025] The formula for reducing the parameter quantity is:

[0026]

[0027] The formula for reducing computational load is:

[0028]

[0029] In the formula, D K The kernel size;

[0030] D w The width of the feature map;

[0031] D H The height of the feature map;

[0032] M and N are the number of input and output feature map channels, respectively. When building a convolutional neural network, the absolute kernel size is 3×3. Substituting into the formula, we can see that the number of parameters and computational cost of a single depthwise separable convolution is about 1 / 9 of that of a standard convolution. This effectively reduces the number of model parameters and improves feature extraction efficiency.

[0033] In the above-mentioned infrared weak target detection and tracking method based on spatiotemporal trajectory fitting, in step three, the infrared weak target detection network model Model-Det is trained and validated using the training set and validation set in the infrared weak target trajectory fitting dataset Data-match; the training set contains 18,000 images and the validation set contains 3,600 images.

[0034] Model building and training were implemented using Python 3.8, with PyTorch 1.10 as the deep learning framework, and NVIDIA GPUs were used for training with CUDA version 11.6.

[0035] Hyperparameter settings: 100 epochs, 64 images per batch, 1024×1024 image input size, 0.001 initial learning rate, Adam optimizer, and mosaic data augmentation enabled.

[0036] During training, the model validation effect is monitored in real time. If the conditions are not met, the hyperparameters are adjusted on the basis of the model and training continues. The learning rate is reduced by 0.5 times, and the model training and validation process is repeated.

[0037] The formula for the target detection rate is:

[0038]

[0039] After training, a pth format infrared small target detection network model Model-Det-Trained with weighted parameters is obtained. Each sequence from the test set in the infrared small target trajectory fitting dataset Data-match is input into the infrared small target detection network model Model-Det-Trained for target detection, resulting in infrared small target detection candidate boxes (Bboxes). The test set contains 3600 images, with each sequence consisting of 360 images, for a total of 10 sequences.

[0040] In the aforementioned infrared weak target detection and tracking method based on spatiotemporal trajectory fitting, in step four, the target tracking algorithm based on Kalman filtering uses a motion model for data association, matches the candidate box information output by the target detection network, and employs 8-dimensional spatial variables. This characterizes the physical state of the trajectory at a certain moment; where (u,v) represents the coordinates of the center point of the candidate box; γ represents the ratio of the height to the width of the candidate box; and h represents the height of the candidate box. The motion information of the target candidate box is represented; the motion trajectory is predicted and updated using a Kalman filter, employing a uniform velocity model and a linear observation model, with the observation variables being (u, vγ, h);

[0041] The detection and tracking process is as follows:

[0042] After a sequence from the test set is input into the target detection network, the network extracts deep feature information from the image, locates and identifies weak infrared targets, and obtains preliminary candidate boxes. The candidate box information is first subjected to non-maximum suppression (NMS) to remove overlapping candidate boxes. It is then input into the Kalman filter for prediction and the matching degree is measured. The Hungarian algorithm is used for cascaded matching using intersection-over-union (IoU) information, and finally the target detection and tracking results of the current frame are output. The target detection network model and Kalman filter tracking are combined to achieve the fusion of temporal and spatial information of the infrared image. The target detection network model uses the spatial information of the image to gradually extract deep features of the target through convolution and downsampling operations to achieve target detection. The Kalman filter uses the temporal information of the infrared image sequence to filter the detected targets according to the motion model and eliminate false alarms caused by targets that do not meet the motion rules. Each frame result is saved as a txt file, where each line contains the motion information of a target, including the target number, target x-coordinate, target y-coordinate, target box height, and target box width.

[0043] After tracking all candidate bounding boxes (Bboxes) for infrared weak target detection in each sequence of the test set of the infrared weak target trajectory fitting dataset Data-match, read all the txt files corresponding to each sequence and generate the target trajectory Trajectory-A for each sequence.

[0044] In the aforementioned infrared weak target detection and tracking method based on spatiotemporal trajectory fitting, step five involves a U-Net encoder-decoder architecture for the target trajectory semantic segmentation network model. The left side of the network is an encoder that extracts deep features of the weak target trajectory through convolution and pooling, and progressive downsampling enables feature extraction at different levels, extracting features of the target trajectory at various scales. The right side of the network is a decoder that decodes the features of the weak target trajectory through convolution and upsampling. During decoding, the encoder on the left and the decoder on the right interact through a concatenation operation, achieving feature fusion at the same level. The encoder module also employs a multi-attention bottleneck residual module, utilizing spatial attention and channel attention, combined with depthwise separable convolution and residual structures, to improve the deep feature extraction capability of the backbone network while maintaining model lightweightness. Simultaneously, dilated convolution is used to expand the receptive field of the convolution without increasing the number of network parameters, fusing longer trajectory features and improving the accuracy of infrared weak target trajectory segmentation.

[0045] The dilated convolution module achieves a larger receptive field than ordinary convolution without changing the number of parameters; when using dilated convolution, the formula for calculating the equivalent convolution kernel size is:

[0046] k′=k+(k-1)×(r-1)

[0047] In the formula, k′ is the equivalent convolution kernel size;

[0048] k is the kernel size;

[0049] r is the expansion rate;

[0050] RF i+1 =RF i +(k′-1)×S i

[0051] In the formula, RF i+1 The size of the receptive field of the current layer;

[0052] RF i The size of the receptive field of the next higher layer;

[0053] S i It is the product of the step sizes of all previous layers.

[0054] In the aforementioned infrared weak target detection and tracking method based on spatiotemporal trajectory fitting, step six involves training and validating the infrared weak target trajectory segmentation network model Model-Seg using the training and validation sets in the mid-infrared weak target trajectory fitting dataset Data-match. Model building and training validation are implemented using Python 3.8, with PyTorch 1.10 as the deep learning framework, and training is performed using an NVIDIA GPU with CUDA version 11.6. Each sequence is treated as a whole trajectory for training and validation. Hyperparameter settings include: 50 epochs, 16 images per batch, 1024×1024 image input size, an initial learning rate of 0.001, and Adam as the optimizer.

[0055] During training, the model's validation effect is monitored in real time. If the conditions are not met, the hyperparameters are adjusted based on the model and training continues. The learning rate is reduced by 0.5 times, and the training and validation process is repeated until a trajectory segmentation model with an overall accuracy of over 90% is obtained.

[0056] In the aforementioned infrared weak target detection and tracking method based on spatiotemporal trajectory fitting, the target detection rate formula is:

[0057]

[0058] After training, the infrared weak target trajectory segmentation network model Model-Seg-Trained with weight parameters in pth format is obtained. Each sequence of the test set in the infrared weak target trajectory fitting dataset Data-match is input into the infrared weak target trajectory segmentation network model Model-Seg-Trained for trajectory segmentation, and the target trajectory Trajectory-B of each sequence is obtained.

[0059] The advantages of this invention compared to the prior art are:

[0060] (1) The dual-trajectory fitting method proposed in this invention takes into account the relationship between the local and the overall situation, effectively reducing the false alarm rate of infrared weak target detection and tracking. This method has higher reliability and robustness.

[0061] (2) This invention solves the problem of insufficient utilization of spatiotemporal information in existing infrared weak target detection networks. First, it uses 3D residual convolution units to extract information from 10 consecutive frames of images. Second, it uses Kalman filtering to fit the temporal motion state of the target. Third, it uses trajectory segmentation to compress the temporal and spatial features of the target in the entire sequence into a single trajectory image, thereby achieving full utilization of the temporal and spatial information of the infrared image sequence.

[0062] (3) The multi-attention bottleneck residual module proposed in this invention can effectively improve the feature extraction effect of infrared weak targets, while reducing the number of model parameters and computation. Attached Figure Description

[0063] Figure 1 This is a block diagram of the overall scheme of the infrared weak target detection and tracking method based on spatiotemporal trajectory fitting of the present invention;

[0064] Figure 2 This is a schematic diagram of the 3D residual convolution input unit of the present invention;

[0065] Figure 3 This is a schematic diagram of the multi-attention bottleneck residual module structure of the present invention;

[0066] Figure 4 This is a schematic diagram illustrating the principle of dilated convolution in this invention;

[0067] Figure 5 This is a schematic diagram of the U-Net network structure of the present invention;

[0068] Figure 6 This is a flowchart of the detection and tracking process based on the target detection network and Kalman filtering of this invention. Detailed Implementation

[0069] The present invention will be further described below with reference to the embodiments.

[0070] To address the problems of insufficient utilization of spatiotemporal information and excessive reliance on data in existing deep learning-based infrared weak target detection methods, this invention proposes an infrared weak target detection and tracking method based on spatiotemporal trajectory fitting. This method fully mines the temporal and spatial information in infrared image sequences while reducing data dependence, and can achieve stable and accurate detection of infrared weak targets in different scenarios.

[0071] Infrared weak target detection and tracking methods based on spatiotemporal trajectory fitting, such as Figure 1 As shown, the specific steps of this method are as follows:

[0072] (1) Construct an infrared weak target trajectory fitting dataset, Data-match. The dataset consists of three parts: a continuous frame infrared image sequence, target location information for individual frames within the sequence, and target trajectory information for the entire sequence. The specific formats for these three parts are as follows: the continuous frame infrared image sequence is in 8-bit grayscale PNG format, with frame numbers identified by filenames. The target location information for individual frames within the sequence is in TXT format, with filenames corresponding one-to-one with the continuous frame image sequence. The location information includes:

[0073] The data consists of five columns: (Class, Xi, Yi, Width, Height). Class represents the category of the labeled object, (Xi, Yi) represents the center point of the labeled object, and (Width, Height) represents the width and height of the labeled box. All values ​​are percentages of the actual image. The target trajectory information for the entire sequence is in JSON format, which can also be converted to PNG image format. PNG format allows visualization of the trajectories of all targets in the entire infrared image sequence, using 8-bit storage with a background pixel value of 0 and a target trajectory pixel value of 255.

[0074] First, obtain the base image of the real infrared sequence data: Data = {I1, I2, ... I n}, where I i This represents a single frame image from an infrared sequence base map. Based on the characteristics of infrared aerial targets, a target simulation synthesis dataset, Data-match, is obtained by adding targets to the base map Data. The image size is 1024×1024, with the target occupying only one to a few pixels. During simulation, target sizes are set to 1×1, 1×2, 1×3, 2×2, 2×3, and 3×3. The target energy (pixel value) is set to 8 to 10 times the temporal signal-to-noise ratio of the infrared base map sequence, with energy diffused and attenuated from the center using a Gaussian function. The target's starting position is set to random, and the target trajectory is set to linear, quadratic, cubic, or Bézier curve functions, etc. The target velocity is set to move 1 to 3 pixels between adjacent frames. During the simulation, the target is sequentially superimposed onto the real infrared sequence image base map I. iAfter the simulation, three parts are obtained: the sequence of images containing the target, the target location information in each frame, and the target trajectory information for the entire image sequence. The simulation dataset is divided into a training set (train), a validation set (val), and a test set (test), with a training set:validation set:test set ratio of 3:1:1. The training set contains 18,000 images, the validation set contains 3,600 images, and the test set contains 3,600 images. Each sequence contains 360 images. This results in the infrared weak target trajectory fitting dataset, Data-match.

[0075] (2) An infrared weak target detection network model, Model-Det, was constructed. The main architecture is YOLOv8. At the detection input end, the conventional two-dimensional convolution was optimized into a 3D residual convolution input unit designed in this invention. The feature extraction network adopts the multi-attention bottleneck residual module designed in this invention. These two optimizations can perform targeted feature extraction for infrared weak targets. The input of this model is 10 consecutive frames of infrared image sequence, and the output is target candidate box information.

[0076] The 3D residual convolution input unit is as follows Figure 2 As shown, when an image is input to the network input terminal, a sliding window input strategy is adopted, and the current frame image is I. n At that time, taking the current frame as the intermediate moment, 10 adjacent frames are superimposed, specifically {I n-4 ,I n-3 ,I n-2 ,I n-1 ,I n ,I n+1 ,I n+2 ,I n+3 ,I n+4, I n+5 This parameter is adjustable and can be adjusted according to the image background and target characteristics. The 3D residual convolution input unit fuses the 10 original frames of input images with the output feature map after 3D convolution, and inputs it into the subsequent feature extraction network, which can effectively realize the extraction of spatiotemporal information.

[0077] The multi-attention bottleneck residual module, such as Figure 3 As shown, by utilizing spatial attention and channel attention, combined with depthwise separable convolution and residual structures, the deep feature extraction capability of the backbone network can be improved while maintaining model lightweightness, reducing hardware computing power requirements, and increasing detection speed. Channel attention calculates the weight of each channel based on input features. Different channels typically represent different objects. In the infrared small target detection problem, channel attention is used to distinguish between targets and target-like noise. Spatial attention calculates the weight of each pixel based on input features. In the infrared small target detection problem, spatial attention is used to enhance pixels at the target location while suppressing non-target pixels.

[0078] The following is a comparison of the number of parameters and computational cost between depthwise separable convolution and standard convolution when performing convolution operations:

[0079] The formula for reducing the parameter quantity is:

[0080]

[0081] The formula for reducing computational load is:

[0082]

[0083] D w The width of the feature map;

[0084] In the formula, D K The kernel size;

[0085] D H The height of the feature map;

[0086] M and N are the number of input and output feature map channels, respectively;

[0087] When building convolutional neural networks, the kernel size is usually 3×3. Substituting this into the formula, we can see that the number of parameters and computational cost of a depthwise separable convolution is about 1 / 9 of that of a standard convolution. This can effectively reduce the number of model parameters and improve feature extraction efficiency.

[0088] (3) Use the training set and validation set (18,000 images in the training set and 3,600 images in the validation set, with 360 images for each sequence) in the infrared weak target trajectory fitting dataset Data-match from step (1) to train and validate the infrared weak target detection network model Model-Det built in step (2);

[0089] Model building and training were implemented using Python 3.8, with PyTorch 1.10 as the deep learning framework. Training was performed using an NVIDIA GPU with CUDA version 11.6. Hyperparameter settings included: 100 epochs, 64 images per batch, 1024×1024 image input size, an initial learning rate of 0.001, Adam optimizer, and enabled mosaic data augmentation. Model validation performance was monitored in real-time during training. If the model did not meet the validation requirements, hyperparameters were adjusted, and training continued, reducing the learning rate by 0.5 times. This training and validation process was repeated until an infrared weak target detection model with a detection rate exceeding 95% was obtained. The target detection rate formula is:

[0090]

[0091] After training, a pth format infrared small target detection network model Model-Det-Trained with weight parameters is obtained. Each sequence from the test set (3600 images in total, 360 images per sequence, 10 sequences in total) in the infrared small target trajectory fitting dataset Data-match is input into the infrared small target detection network model Model-Det-Trained for target detection, and infrared small target detection candidate boxes (Bbox) are obtained.

[0092] (4) Input the infrared weak target detection candidate box (Bbox) information obtained in step (3) into the Kalman filter unit to achieve preliminary target tracking. The target tracking algorithm based on Kalman filter uses a motion model to perform data association, matches the candidate box information output by the target detection network, and uses 8-dimensional spatial variables. The physical state of the trajectory at a certain moment is represented by (u, v), where (u, v) represents the coordinates of the center point of the candidate box; γ represents the ratio of the height to the width of the candidate box; and h represents the height of the candidate box. The motion information of the target candidate box is represented, and then the motion trajectory is predicted and updated using a Kalman filter. A uniform velocity model and a linear observation model are adopted, with the observation variables being (u, v, γ, h). Due to its deep correlation features, it can better handle situations where the target is occluded for a long time, and can improve the tracking effect of weak infrared targets.

[0093] Its detection and tracking process is as follows: Figure 4 As shown, after inputting a sequence from the test set into the target detection network, the network extracts deep feature information from the image, locates and identifies weak infrared targets, and obtains preliminary candidate boxes. These candidate boxes are first processed by Non-Maximum Suppression (NMS) to remove overlapping boxes, then input into a Kalman filter for prediction and matching degree measurement. Subsequently, a Hungarian algorithm cascade matching is performed using intersection-over-union (IoU) information, and finally, the target detection and tracking result for the current frame is output. The combination of the target detection network model and Kalman filter tracking achieves a certain degree of fusion of temporal and spatial information from infrared images. The target detection network model mainly utilizes the spatial information of the image, gradually extracting deep target features through convolution and downsampling operations to achieve target detection. The Kalman filter mainly utilizes the temporal information of the infrared image sequence, filtering detected targets according to a motion model and eliminating false alarms caused by targets that do not meet the motion rules. Each frame's result is saved as a txt file, where each line contains motion information for one target, including target number, target x-coordinate, target y-coordinate, target box height, and target box width.

[0094] After tracking all the candidate bounding boxes (Bbox) for infrared weak target detection in each sequence of the test set of the infrared weak target trajectory fitting dataset Data-match (3600 images in total, 360 images per sequence, 10 sequences in total), read all the txt files corresponding to each sequence and generate the target trajectory Trajectory-A for each sequence.

[0095] (5) Construct an infrared weak target trajectory segmentation network model Model-Seg. The main architecture is U-Net. The feature extraction network adopts the multi-attention bottleneck residual module designed in this invention. The first layer of conventional convolution in each feature downsampling is optimized into dilated convolution. Both optimizations can improve the trajectory extraction performance. The input of this model is the entire infrared sequence image, and the output is the overall trajectory information of all targets in the sequence.

[0096] The target trajectory semantic segmentation network model adopts the U-Net encoder-decoder architecture, such as... Figure 5 As shown, the left side of the network is an encoder, whose main function is to extract deep features of weak target trajectories through convolution and pooling. Stepwise downsampling enables feature extraction at different levels, extracting features of the target trajectory at various scales. The right side of the network is a decoder, whose main function is to decode the features of weak target trajectories through convolution and upsampling. During decoding, the encoder on the left and the decoder on the right interact through a concatenation operation, achieving feature fusion at the same level. In the encoder module, this scheme also employs a multi-attention bottleneck residual module, utilizing spatial attention and channel attention, combined with depthwise separable convolution and residual structures, to improve the deep feature extraction capability of the backbone network while maintaining model lightweightness. Furthermore, this scheme uses dilated convolution to expand the receptive field of the convolution without increasing the number of network parameters, fusing longer trajectory features and improving the accuracy of infrared weak target trajectory segmentation.

[0097] The dilated convolution module, as shown below Figure 6 As shown, Figure 4 This represents the receptive field obtained by dilated convolution. Figure 6 -a represents the receptive field obtained after a standard 3×3 convolution; Figure 6 -b represents the receptive field obtained by a 3×3 dilated convolution with a dilation rate of 1; Figure 6 -c represents the receptive field obtained after a 3×3 dilated convolution with a dilation rate of 2. It can be seen that dilated convolution can achieve a larger receptive field than ordinary convolution without changing the number of parameters.

[0098] When using dilated convolution, the formula for calculating the equivalent convolution kernel size is:

[0099] k′=k+(k-1)×(r-1)

[0100] In the formula, k′ is the equivalent convolution kernel size, k is the convolution kernel size, and r is the dilation rate.

[0101] RF i+1 =RF i +(k′-1)×S i

[0102] In the formula, RF i+1 RF is the receptive field size of the current layer. i S is the size of the receptive field of the previous layer. i It is the product of the step sizes of all previous layers.

[0103] In the target trajectory semantic segmentation task, the trajectory appears continuously and has a large range. This scheme uses dilated convolution to expand the receptive field without increasing the amount of computation, thereby improving the semantic feature extraction capability.

[0104] (6) The infrared weak target trajectory fitting dataset Data-match (18,000 images in the training set and 3,600 images in the validation set, with 360 images per sequence) was used in step (1) to train and validate the infrared weak target trajectory segmentation network model Model-Seg built in step (5). The model building, training and validation were implemented using Python language, version 3.8, and the deep learning framework PyTorch 1.10 was used. NVIDIA GPU was used for training, and CUDA version 11.6 was used. Each sequence was used as a whole trajectory for training and validation. The hyperparameter settings were: 50 epochs, 16 images per batch, 1024×1024 image input size, initial learning rate of 0.001, and Adam optimizer. The model validation effect was monitored in real time during training. If the conditions were not met, the hyperparameters were adjusted on the basis of the model and training continued. The learning rate was reduced by 0.5 times, and the training and validation process was repeated until a trajectory segmentation model with an overall accuracy of over 90% was obtained.

[0105] The formula for the target detection rate is:

[0106]

[0107] After training, the infrared weak target trajectory segmentation network model Model-Seg-Trained with weight parameters in pth format is obtained. Each sequence from the test set (3600 images in total, 360 images per sequence, 10 sequences in total) in the infrared weak target trajectory fitting dataset Data-match is input into the infrared weak target trajectory segmentation network model Model-Seg-Trained for trajectory segmentation, and the target trajectory Trajectory-B of each sequence is obtained.

[0108] (7) Fit and compare the trajectory Trajectory-A obtained in step (4) with the trajectory Trajectory-B obtained in step (6), take the intersection of the two, and obtain the final infrared weak target detection and tracking result and target trajectory output.

[0109] The method of this invention, through steps such as target detection, Kalman filtering, and trajectory segmentation, makes full use of the temporal and spatial information of the infrared image sequence. By combining the single-frame target detection results of each frame with the overall trajectory results of the entire sequence, it can effectively improve the detection rate of weak infrared targets, reduce the false alarm rate, and improve robustness.

[0110] The dual-trajectory fitting method based on target detection tracking trajectory and semantic segmentation trajectory proposed in this invention is intended to be protected.

[0111] This invention addresses the problem of insufficient utilization of spatiotemporal information in existing infrared weak target detection networks by: first, extracting information from 10 consecutive frames of images using 3D residual convolutional units; second, fitting the target's temporal motion state using Kalman filtering; and third, compressing the temporal and spatial features of the target throughout the sequence into a single trajectory image using trajectory segmentation, thus achieving full utilization of the temporal and spatial information of the infrared image sequence. This spatiotemporal information fusion method is proposed for protection.

[0112] The multi-attention bottleneck residual module proposed in this invention can effectively improve the feature extraction effect of infrared weak targets, while reducing the number of model parameters and computational load. The model structure is intended to be protected.

[0113] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications to the technical solutions of the present invention by utilizing the methods and techniques disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall fall within the protection scope of the technical solutions of the present invention.

Claims

1. A method for detecting and tracking weak infrared targets based on spatiotemporal trajectory fitting, characterized in that: include: Step 1: Construct the infrared weak target trajectory fitting dataset Data-match; The dataset Data-match includes a continuous frame infrared image sequence, target location information of a single frame image in the sequence, and target trajectory information for the entire sequence. Step 2: Construct the infrared weak target detection network model Model-Det; use 3D residual convolution as the input unit at the detection input end; the feature extraction network adopts a multi-attention bottleneck residual module; This system enables targeted feature extraction for weak infrared targets. The Model-Det infrared target detection network takes 10 consecutive frames of infrared images as input and outputs target candidate bounding box information. Step 3: Repeat the infrared weak target trajectory fitting dataset from Step 1 (Data-match) to train and validate the infrared weak target detection network model (Model-Det) from Step 2 until an infrared weak target detection model with a detection rate exceeding 95% is obtained; the resulting infrared weak target detection candidate boxes (Bboxes) are obtained. Step 4: Input the infrared weak target detection candidate box (Bbox) from Step 3 into the Kalman filter unit to achieve preliminary target tracking; generate the target trajectory Trajectory-A for each sequence; Step 5: Construct the infrared weak target trajectory segmentation network model Model-Seg, with the main architecture being U-Net; the feature extraction network adopts a multi-attention bottleneck residual module; the first layer of regular convolution in each feature downsampling is optimized into dilated convolution; thus improving trajectory extraction performance; the input of this model is the entire infrared sequence image, and the output is the overall trajectory information of all targets in the sequence; Step 6: Repeat the Data-match training of the infrared weak target trajectory fitting dataset in Step 1 and the infrared weak target trajectory segmentation network model Model-Seg in Step 5 until the trajectory segmentation model with an overall accuracy of over 90% is obtained, and the target trajectory Trajectory-B of each sequence is obtained. Step 7: Fit and compare the target trajectory Trajectory-A from Step 4 with the target trajectory Trajectory-B from Step 6, and take the intersection of the two to obtain the final infrared weak target detection and tracking result and target trajectory output.

2. The infrared weak target detection and tracking method based on spatiotemporal trajectory fitting according to claim 1, characterized in that: In step one, the continuous frame infrared image sequence is in 8-bit grayscale PNG format; the image is identified by the frame number through the file name. The target location information of a single frame image in the sequence is in txt format. The file name corresponds one-to-one with the continuous frame sequence images. The location content includes 5 columns of attribute data, namely Class, Xi, Yi, Width, and Height. Among them, Class is the category of the labeled object; Xi and Yi are the center points of the labeled object; Width and Height are the width and height of the labeled box; all values ​​are percentages of the actual image. The target trajectory information for the entire sequence is in JSON format or converted to PNG image format; the PNG format visualizes the trajectory of all targets in the entire infrared image sequence, using 8-bit storage, with a background pixel value of 0 and a target trajectory pixel value of 255.

3. The infrared weak target detection and tracking method based on spatiotemporal trajectory fitting according to claim 2, characterized in that: The Data-match dataset is constructed as follows: Firstly, the real infrared sequence image base Data = {I1, I2,... I n} is acquired; wherein I i represents a single frame image in the infrared sequence base; according to the characteristics of the infrared aerial target, the target simulation is added on the base Data to obtain the infrared weak small target trajectory fitting data set Data-match; The image size is 1024×1024, and the target is one to several pixels. During simulation, target sizes are set to 1×1, 1×2, 1×3, 2×2, 2×3, and 3×3. The target energy is set to 8 to 10 times the temporal signal-to-noise ratio of the infrared base image sequence. The target energy is the pixel value of the target. The energy diffuses and decays from the center, and the decay function is a Gaussian function. The target's initial position is set to random. The target trajectory is set to a linear function, a quadratic function, a cubic function, and a Bézier curve. The target velocity is set to move 1 to 3 pixels between adjacent frames. During the simulation, the target is sequentially superimposed onto the real infrared sequence image base image Ii. After the simulation, the following are obtained: a sequence of images containing the target, target position information in each frame, and target trajectory information for the entire image sequence. The simulation dataset is divided into a training set (train), a validation set (val), and a test set (test). The ratio of training set:validation set:test set is 3:1:1, with 18,000 images in the training set, 3,600 in the validation set, and 3,600 in the test set. Each sequence consists of 360 images. The infrared weak target trajectory fitting dataset Data-match was obtained.

4. The infrared weak target detection and tracking method based on spatiotemporal trajectory fitting according to claim 3, characterized in that: In step two, the 3D residual convolution input unit is specifically: When an image is input to the network input terminal, a sliding window input strategy is adopted; when the current frame image is In, the images of the next 10 adjacent frames are superimposed with the current frame as the intermediate time, specifically {I}. n-4 ,I n-3 ,I n-2 ,I n-1 ,I n ,I n+1 ,I n+2 ,I n+3 ,I n+4 ,I n+5 The parameter is adjusted according to the image background and target characteristics; the 3D residual convolution input unit fuses the input 10 frames of original images with the output feature map through 3D convolution, and inputs it into the subsequent feature extraction network to realize the extraction of spatiotemporal information. The multi-attention bottleneck residual module is: By leveraging spatial attention and channel attention, combined with depthwise separable convolution and residual structures, we can improve the deep feature extraction capability of the backbone network while maintaining model lightweightness, reducing hardware computing power requirements, and increasing detection speed. Channel attention calculates the weight of each channel based on input features. Different channels typically represent different objects. In the infrared small target detection problem, channel attention is used to distinguish between targets and target-like noise. Spatial attention calculates the weight of each pixel based on input features. In the infrared small target detection problem, spatial attention is used to enhance pixels at the target location while suppressing non-target pixels.

5. The infrared weak target detection and tracking method based on spatiotemporal trajectory fitting according to claim 4, characterized in that: The following is a comparison of the number of parameters and computational cost between depthwise separable convolution and standard convolution when performing convolution operations: The formula for reducing the parameter quantity is: The formula for reducing computational load is: In the formula, D K The kernel size; D w The width of the feature map; D H The feature map height; M and N are the number of input and output feature map channels, respectively. When building a convolutional neural network, the absolute kernel size is 3×3. Substituting into the formula, we can see that the number of parameters and computational cost of a single depthwise separable convolution is about 1 / 9 of that of a standard convolution. This effectively reduces the number of model parameters and improves feature extraction efficiency.

6. The infrared weak target detection and tracking method based on spatiotemporal trajectory fitting according to claim 5, characterized in that: In step three, the infrared weak target detection network model Model-Det is trained and validated using the training and validation sets in the infrared weak target trajectory fitting dataset Data-match; the training set contains 18,000 images and the validation set contains 3,600 images. Model building and training were implemented using Python 3.8, with PyTorch 1.10 as the deep learning framework, and NVIDIA GPUs were used for training with CUDA version 11.

6. Hyperparameter settings: 100 epochs, 64 images per batch, 1024×1024 image input size, 0.001 initial learning rate, Adam optimizer, and mosaic data augmentation enabled. During training, the model validation effect is monitored in real time. If the conditions are not met, the hyperparameters are adjusted on the basis of the model and training continues. The learning rate is reduced by 0.5 times, and the model training and validation process is repeated. The formula for the target detection rate is: After training, a pth format infrared weak target detection network model Model-Det-Trained with weighted parameters is obtained. Each sequence from the test set in the infrared weak target trajectory fitting dataset Data-match is input into the infrared weak target detection network model Model-Det-Trained for target detection, resulting in infrared weak target detection candidate boxes (Bboxes). The test set contains 3600 images, with each sequence consisting of 360 images, for a total of 10 sequences.

7. The infrared weak target detection and tracking method based on spatiotemporal trajectory fitting according to claim 6, characterized in that: In step four, the Kalman filter-based target tracking algorithm uses a motion model for data association, matches the candidate box information output by the target detection network, and employs an 8-dimensional spatial variable. This characterizes the physical state of the trajectory at a certain moment; where (u,v) represents the coordinates of the center point of the candidate box; γ represents the ratio of the height to the width of the candidate box; and h represents the height of the candidate box. The motion information of the target candidate box is represented; the motion trajectory is predicted and updated using a Kalman filter, employing a uniform velocity model and a linear observation model, with the observation variables being (u,v,γ,h). The detection and tracking process is as follows: After a sequence from the test set is input into the target detection network, the target detection network extracts deep feature information from the image, locates and identifies weak infrared targets, and obtains preliminary target candidate boxes. The candidate box information is first subjected to non-maximum suppression (NMS) to remove overlapping candidate boxes. The information is then input into the Kalman filter for prediction and the matching degree is measured. The Hungarian algorithm is used for cascaded matching using intersection-union information, and the target detection and tracking results of the current frame are finally output. The target detection network model was combined with Kalman filter tracking to achieve the fusion of temporal and spatial information of infrared images; The target detection network model utilizes the spatial information of the image to gradually extract deep features of the target through convolution and downsampling operations to achieve target detection; Kalman filtering utilizes the temporal information of the infrared image sequence to filter the detected targets according to the motion model and eliminate false alarms caused by targets that do not meet the motion rules. Each frame result is saved as a txt file, where each line contains motion information for a target, including target number, target x-coordinate, target y-coordinate, target bounding box height, and target bounding box width; After tracking all candidate bounding boxes (Bboxes) for infrared weak target detection in each sequence of the test set of the infrared weak target trajectory fitting dataset Data-match, read all the txt files corresponding to each sequence and generate the target trajectory Trajectory-A for each sequence.

8. The infrared weak target detection and tracking method based on spatiotemporal trajectory fitting according to claim 6, characterized in that: In step five, the target trajectory semantic segmentation network model adopts a U-Net encoder-decoder architecture. The left side of the network is an encoder, which extracts deep features of weak target trajectories through convolution and pooling. Stepwise downsampling can realize feature extraction at different levels, extracting features of the target trajectory at various scales. The right side of the network is a decoder, which decodes the features of weak target trajectories through convolution and upsampling. During the decoding process, the encoder on the left and the decoder on the right complete data interaction through concatenation operations to achieve feature fusion at the same level. In the encoder module, a multi-attention bottleneck residual module is also adopted. By utilizing spatial attention and channel attention, combined with depthwise separable convolution and residual structure, the model can be kept lightweight while improving the deep feature extraction capability of the backbone network. At the same time, dilated convolution is used to expand the receptive field of convolution without increasing the number of network parameters, and to fuse longer trajectory features to improve the accuracy of infrared weak target trajectory segmentation. The dilated convolution module achieves a larger receptive field than ordinary convolution without changing the number of parameters; when using dilated convolution, the formula for calculating the equivalent convolution kernel size is: k' = k + (k-1) × (r-1) In the formula, k′ is the equivalent convolution kernel size; k is the kernel size; r is the expansion rate; RF i+1 =RF i +(k′-1)×S i In the formula, RF i+1 The size of the receptive field of the current layer; RF i The size of the receptive field of the next higher layer; S i It is the product of the step sizes of all previous layers.

9. The infrared weak target detection and tracking method based on spatiotemporal trajectory fitting according to claim 6, characterized in that: In step six, the infrared weak target trajectory fitting dataset Data-match is used to train and validate the infrared weak target trajectory segmentation network model Model-Seg. The model building and training validation are implemented using Python 3.8, the deep learning framework is PyTorch 1.10, and training is performed using NVIDIA GPU with CUDA version 11.

6. Each sequence is treated as a whole trajectory for training and validation. The hyperparameter settings are as follows: 50 epochs, 16 images per batch, 1024×1024 image input size, initial learning rate of 0.001, and Adam optimizer. During training, the model's validation effect is monitored in real time. If the conditions are not met, the hyperparameters are adjusted based on the model and training continues. The learning rate is reduced by 0.5 times, and the training and validation process is repeated until a trajectory segmentation model with an overall accuracy of over 90% is obtained.

10. The infrared weak target detection and tracking method based on spatiotemporal trajectory fitting according to claim 9, characterized in that: The formula for the target detection rate is: After training, the infrared weak target trajectory segmentation network model Model-Seg-Trained with weight parameters in pth format is obtained; Each sequence from the test set of the infrared weak target trajectory fitting dataset Data-match is input into the infrared weak target trajectory segmentation network model Model-Seg-Trained for trajectory segmentation, resulting in the target trajectory Trajectory-B for each sequence.

Images