A spatio-temporal adaptive thermal infrared tracking method based on coordinate information

Abstract

The present application belongs to the technical field of image processing, and particularly relates to a space-time adaptive thermal infrared tracking method based on coordinate information. The space-time adaptive thermal infrared tracking method based on coordinate information provided by the present application uses a Transformer structure with bidirectional information flow, unifies feature extraction and feature fusion, simplifies the tracking process and can extract more rich discriminative features; in order to cope with the low signal-to-noise ratio of infrared images and the lack of color features, the present application uses coordinate information as prompt information of infrared features to predict the target bounding box autoregressively, and captures the appearance change of the target through a dynamic template, thereby enhancing the stability of thermal infrared tracking. The present application is tested on two large public datasets, and compared with other advanced infrared trackers, the present application achieves the optimal performance in each index on the two datasets.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a spatiotemporal adaptive thermal infrared tracking method based on coordinate information. Background Technology

[0002] Thermal infrared target tracking is a fundamental and crucial task in the field of artificial intelligence. Unlike visible light imaging mechanisms, thermal infrared imaging captures the thermal information of a target, making thermal infrared trackers effective even in harsh environments such as rain, fog, and low light. Therefore, thermal infrared tracking has wide applications in intelligent surveillance, maritime rescue, and unmanned aerial vehicle (UAV) platforms. However, infrared images suffer from low signal-to-noise ratios and lack rich color and texture information, making the extraction of discriminative features and the use of coordinate and temporal information as supplementary visual features particularly important.

[0003] Initial thermal infrared trackers were designed based on traditional machine learning methods. For single-channel thermal infrared images lacking color information, Venkataraman et al. estimated the target's appearance representation using local standard deviation and pixel intensity histograms, validating the model's ability to improve tracking accuracy across multiple sequences. However, traditional machine learning-based methods are limited by the lack of adaptability due to handcrafted features. In recent years, convolutional neural networks and Transformers have demonstrated powerful representational capabilities in computer vision tasks and have been gradually applied to thermal infrared tracking. MLSSNet can simultaneously learn global semantic information and local detail information, enhancing the network's discriminative ability. Inspired by visual tracking methods, Siamese network-based algorithms have been gradually introduced into infrared target tracking. To learn diverse fine-grained features, Yang et al. further improved upon the TransT network, designing a fine-grained, diversified loss function and a fine-grained network. While these models exhibit excellent performance, most employ a two-stream architecture that separates feature extraction and feature fusion. Such a design is not only complex but also only utilizes the infrared image features of the initial frame. For infrared images lacking detailed information, this method struggles to extract fine-grained discriminative features and cannot update the target's appearance features in a timely manner, thus affecting the robustness of tracking. Summary of the Invention

[0004] The purpose of this invention is to provide a spatiotemporally adaptive thermal infrared tracking method based on coordinate information. Firstly, a Transformer structure with bidirectional information flow is used to unify feature extraction and feature fusion, thereby extracting richer discriminative features. To address the low signal-to-noise ratio and lack of color features in infrared images, this invention uses coordinate information as a cue for infrared features to autoregressively predict the target bounding box, and captures changes in the target's appearance through dynamic templates, thus enhancing the stability of thermal infrared tracking.

[0005] The technical solution adopted in this invention is as follows:

[0006] A spatiotemporal adaptive thermal infrared tracking method based on coordinate information includes the following steps:

[0007] Step 1: Preprocess the infrared sequence dataset, specifically including:

[0008] Step 1.1: First, randomly sample a video sequence from the infrared sequence dataset. Then, randomly sample three different infrared images from the video sequence. Sort the three sampled infrared images according to the number of video frames. The smallest frame is the template image, followed by the dynamic template image, and the largest frame is the search area image. The maximum sampling interval is set to 50 frames.

[0009] Step 1.2: Cropped images from the three sampled infrared images, centered on the target bounding box, are simultaneously enhanced with predetermined probabilities, including image flipping and grayscale value changes. The resulting preprocessed template image T, dynamic template image T_O, and search region image S form the input image set for the encoder.

[0010] Step 2: Construct a spatiotemporal adaptive thermal infrared tracking network based on coordinate information, mainly including a Transformer-based encoder, a Transformer decoder with masking, a multilayer perceptron, and a dynamic template update branch; the tracking network processes image groups as follows:

[0011] Step 2.1: First, pass each image in the input image group through a convolutional layer with a stride of 16 and a kernel size of 16 to convert the images into patches. Then, flatten the patches and concatenate the patches of the three images into a feature vector.

[0012] Step 2.2: Embed sin-con positional encoding into the feature vector. The feature vector with positional encoding information is used to enhance the network's ability to perceive positional information.

[0013] Step 2.3: Construct a Transformer-based encoder for feature extraction and feature fusion of the template and search region. The feature vector with positional encoding information is used as the input to the encoder. The attention mechanism in the encoder is as follows:

[0014] K M =Concat(K Z ,K X ),V M =Concat(V Z V X ),

[0015]

[0016]

[0017] Attn = Concat(Attn) Z Attn X )

[0018] Q Z ,K Z V Z Represented as queries, keys, and values ​​in template features, Q X ,K X V X Represented as queries, keys, and values ​​for the search region features, where d represents the dimension of the key, and Attn Z Attn represents the self-attention result of template features. X This represents the cross-attention result of the search region features. The output of the i-th layer of the encoder can be represented as:

[0019] f i =MLP(Norm(Attn(Norm(f) i-1 ))+f i-1 ))+f i-1 i = 1, 2, ..., L

[0020] Where MLP stands for Multilayer Perceptron, Norm stands for LayerNorm, and f i f represents the output of the i-th layer. 0 =[Z P ,X P Attn is formula (1);

[0021] Step 2.4: Extract the features of the search region from the feature vector output by the encoder and use them as input to the masked Transformer decoder;

[0022] Step 2.5: Discretize the coordinate information into a coordinate sequence as another input to the Transformer decoder with a mask;

[0023] Step 2.6: Construct a masked Transformer decoder, which mainly includes masked self-attention and cross-attention. The masked self-attention is used to enhance the feature representation of the coordinate sequence, and then the enhanced coordinate features are used as a cue for the features of the search region. Cross-attention is used to further enhance the features of the search region.

[0024] Step 2.7: The multilayer perceptron mainly consists of a softmax layer and three linear layers, used to regress the probability distribution of the coordinate sequence, thereby obtaining the physical coordinates of the target;

[0025] Step 2.8: The dynamic template update branch is used to evaluate the score of the dynamic template to determine whether the template should be updated over time. The reliability of the current template can be determined simply by calculating the average of the softmax scores of the probability distributions of the four coordinate sequences. The reliability score of the dynamic template can be expressed as:

[0026] Score=Mean(SoftMax(x),SoftMax(y),SoftMax(w),SoftMax(h))

[0027] Mean represents the mean, SoftMax represents the softmax layer, and x, y, w, and h represent the top-left corner coordinates (x, y), width, and height of the target bounding box, respectively.

[0028] Step 3: Construct the loss function for the spatiotemporal adaptive thermal infrared tracking network based on coordinate information. Only a simple cross-entropy loss function is needed:

[0029]

[0030] Z t X represents the template image at the current time t. t Y represents the image of the search region corresponding to the current time t. 0 :t-1 This represents the previously predicted target sequence at time t, and is related to the template image and the search region image. P represents the softmax probability. t represents the time step of the token, and T represents the length of the target sequence. The input coordinate sequence is [cmd x yw h], and the corresponding target sequence is [xywh end].

[0031] Step 4: Training strategy for single segments;

[0032] The tracking framework in step 2 is implemented using a Python interpreter and the PyTorch framework. The model is trained using the preprocessed data from step 1 and the loss function constructed in step 3, with the encoder's learning rate set to 1e. -5 The learning rate of the non-encoder module is 1e -4 The weight decay is set to 1e. -4 The training process lasted 120 epochs, with 30,000 image groups sampled per epoch.

[0033] Step 5: Inference process: Using the model trained in step 4.1, the template and dynamic template are initialized using the first frame image in the sequence. The dynamic template is updated based on the reliability score. When the coordinate sequence is first input, an identifier is set as a prompt for the infrared features. Each generated coordinate sequence is used as the input for the next step, gradually prompting the infrared features to predict the bounding box coordinates.

[0034] The beneficial effects of this invention are as follows:

[0035] Most existing thermal infrared tracking methods employ a Siamese network architecture. This invention provides a coordinate-information-based spatiotemporally adaptive thermal infrared tracking method that leverages the Transformer's ability to simultaneously model global and local features. A single-stream Transformer encoder can simultaneously achieve feature extraction and feature fusion without requiring an additional feature fusion module. To address the lack of texture information in thermal infrared features, this invention utilizes the target's coordinate information as autoregressive predicted bounding box coordinates for infrared feature cues, thereby improving the robustness of thermal infrared tracking. To utilize temporal information to simultaneously address changes in target appearance, thermal infrared tracking typically employs a template update strategy. However, most template update strategies require constructing an additional score prediction branch and undergoing two-stage training. This invention only needs to calculate the average softmax score of the probability distributions of four coordinate sequences to determine the reliability of the current template. Experiments on large-scale public datasets show that this invention outperforms other advanced thermal infrared tracking methods across multiple evaluation metrics. Attached Figure Description

[0036] Figure 1 This is a flowchart illustrating the overall process of the method of the present invention.

[0037] Figure 2 This is a schematic diagram of the Transformer decoder with a mask in this invention;

[0038] Figure 3 A visualization of a tracker when a target is occluded in a video sequence, where (a) is frame 50, (b) is frame 140, and (c) is frame 238.

[0039] Figure 4 A visual representation of the tracker as the appearance of a target changes in a video sequence, where (a) is frame 21, (b) is frame 40, and (c) is frame 230.

[0040] Figure 5 The image shows a visualization of a tracker when there are multiple similar targets in a video sequence, where (a) is frame 28, (b) is frame 110, and (c) is frame 208. Detailed Implementation

[0041] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0042] This embodiment provides a spatiotemporal adaptive thermal infrared tracking method based on coordinate information, including the following steps:

[0043] Step 1: Preprocess the infrared sequence dataset, specifically including:

[0044] Step 1.1: First, randomly sample a video sequence from the infrared sequence dataset. Then, randomly sample three different infrared images from the video sequence. Sort the three sampled infrared images according to the number of video frames. The smallest frame is the template image, the next is the dynamic template image, and the largest frame is the search area image. The maximum sampling interval is set to 50 frames.

[0045] Step 1.2: Cropping the three sampled infrared images centered on the target bounding box, and simultaneously performing image enhancement on the cropped images with specific probabilities, including image flipping and grayscale value changes. The resulting preprocessed template image T, dynamic template image T_O, and search region image S form the input image group, which serves as the encoder input.

[0046] Step 2: Construct a spatiotemporal adaptive thermal infrared tracking network based on coordinate information, such as... Figure 1 As shown, it mainly includes a Transformer-based encoder, a Transformer decoder with a mask, a multilayer perceptron, and a dynamic template update branch;

[0047] Step 2.1: The images in the input image group are first passed through a convolutional layer with a stride of 16 and a kernel of 16 to convert each image into patches. Then the patches are flattened and the patches of the three images are concatenated into a feature vector.

[0048] Step 2.2: Embed sin-con positional encoding into the feature vector to enhance the network's ability to perceive positional information;

[0049] Step 2.3: Construct a Transformer-based encoder for feature extraction and feature fusion of the template and search region. The feature vector with positional encoding information is used as the input to the encoder. The attention mechanism in the encoder is as follows:

[0050] K M =Concat(K Z ,K X ),V M =Concat(V Z V X ),

[0051]

[0052]

[0053] Attn = Concat(Attn) Z Attn X )

[0054] Q Z ,K Z V Z Queries, keys, and values, Q X ,K X V X Represented as queries, keys, and values ​​for the search region features, where d represents the dimension of the key, and Attn Z Attn represents the self-attention result of template features. X This represents the cross-attention result of the search region features. The output of the i-th layer of the encoder can be represented as:

[0055] f i =MLP(Norm(Attn(Norm(f) i-1 ))+f i-1 ))+f i-1 i = 1, 2, ..., L

[0056] Where MLP stands for Multilayer Perceptron, Norm stands for LayerNorm, and f i f represents the output of the i-th layer. 0 =[Z P ,X P Attn is formula (1);

[0057] Step 2.4: Extract the features of the search region from the feature vector output by the encoder and use them as input to the masked Transformer decoder;

[0058] Step 2.5: Discretize the coordinate information into a coordinate sequence as another input to the Transformer decoder with a mask;

[0059] Step 2.6: Construct a Transformer decoder with a mask, such as... Figure 2 As shown, it mainly includes self-attention with mask and cross attention. Self-attention with mask is used to enhance the feature representation of the coordinate sequence. Then, the enhanced coordinate features are used as a cue for the features of the search region. Cross attention is used to further enhance the features of the search region.

[0060] Step 2.7: The multilayer perceptron mainly consists of a softmax layer and three linear layers, used to regress the probability distribution of the coordinate sequence, thereby obtaining the physical coordinates of the target;

[0061] Step 2.8: The dynamic template update branch is used to evaluate the score of the dynamic template to determine whether the template should be updated over time. The reliability of the current template can be determined simply by calculating the average of the softmax scores of the probability distributions of the four coordinate sequences. The reliability score of the dynamic template can be expressed as:

[0062] Score=Mean(SoftMax(x),SoftMax(y),SoftMax(w),SoftMax(h))

[0063] Mean represents the mean, SoftMax represents the softmax layer, and x, y, w, and h represent the top-left corner coordinates (x, y), width, and height of the target bounding box, respectively.

[0064] Step 3: Construct the loss function for the spatiotemporal adaptive thermal infrared tracking network based on coordinate information. Only a simple cross-entropy loss function is needed:

[0065]

[0066] Z t X represents the template image at the current time t. t Y represents the image of the search region corresponding to the current time t. 0 :t-1 This represents the previously predicted target sequence at time t, and is related to the template image and the search region image. P represents the softmax probability. t represents the time step of the token, and T represents the length of the target sequence. The input coordinate sequence is [cmd x yw h], and the corresponding target sequence is [xywh end].

[0067] Step 4: Single-segment training strategy: Implement the tracking framework from Step 2 using a Python interpreter and the PyTorch framework. Train the model using the preprocessed data from Step 1 and the loss function constructed in Step 3, setting the encoder's learning rate to 1e. -5 The learning rate of the non-encoder module is 1e -4 The weight decay is set to 1e. -4 The training process lasted 120 epochs, with 30,000 image groups sampled per epoch.

[0068] Step 5: Inference process: Using the model trained in step 4.1, the template and dynamic template are initialized using the first frame image in the sequence. The dynamic template is updated based on the reliability score. The coordinate sequence starts with "cmdtoken" as a prompt for infrared features. Each generated coordinate sequence is used as the input for the next step, gradually prompting the infrared features to predict the bounding box coordinates.

[0069] According to the above implementation scheme, the spatiotemporal adaptive thermal infrared tracking effect based on coordinate information proposed in this invention is compared and analyzed. Comparative experiments are conducted with other advanced thermal infrared trackers on the LSOTB-TIR and PTB-TIR public datasets. Success, Precision, and Norm Precision are used as evaluation indicators for evaluation, as shown in Table 1 and Table 2.

[0070] Table 1 compares the tracker of this invention with 10 other trackers on the PTB-TIR benchmark. Bold text indicates the best.

[0071]

[0072] Table 1 presents the experimental results of this invention and other excellent infrared target tracking methods on the PTB-TIR dataset. The tracking algorithms compared include Siamese-FC, CFNet, MCFTS, ECO-deep, HSSNet, MLSSNet, ECO-MM, MMNet, ECO-tir, and SiamMSS. The table shows that the algorithm of this invention achieves the best performance on the PTB-TIR dataset, with a success rate of 0.683 and an accuracy of 0.856. Compared to the Siamese-FC tracker, the algorithm of this invention improves the success rate by 20.3% and the accuracy by 23.3%. Compared to the SiamMss tracker, the success rate is improved by 4% and the accuracy by 3.3%. This is mainly due to the fact that this invention fully utilizes temporal and coordinate information.

[0073] Table 2 compares the tracker of this invention with nine other trackers on the LSOTB-TIR benchmark. Bold text indicates the best.

[0074]

[0075]

[0076] To further verify the effectiveness of the infrared target tracking algorithm of this invention, comparative experiments were conducted on the largest infrared tracking dataset currently available. Table 2 shows the experimental results of this invention and other excellent infrared target tracking methods on the LSOTB-TIR dataset. The tracking algorithms compared include Siamese-FC, CFNet, MCFTS, ECO-deep, HSSNet, MLSSNet, ECO-stir, ECO-tir, and SiamMSS. As can be seen from Table 2, this invention achieved first place in all metrics on LSOTB-TIR, with a success rate of 0.696, an accuracy of 0.838, and a normalized accuracy of 0.752, thus demonstrating the superiority and generalization ability of the thermal infrared target tracking algorithm in this invention.

[0077] like Figure 3 , Figure 4 and Figure 5 The image shows a visualization of the tracker of the present invention, where the red box represents the target bounding box predicted by the tracker. Figure 3 This verifies that the tracker can still accurately track the target even when it is partially occluded; Figure 4 It demonstrates that the tracker can capture changes in the target's appearance in real time and update dynamic templates; Figure 5 This demonstrates that the tracker can still accurately track objects even when there is interference from similar objects. These results fully demonstrate the robustness and accuracy of the tracker of this invention.

Claims

1. A spatiotemporal adaptive thermal infrared tracking method based on coordinate information, comprising the following steps: Step 1: Preprocess the infrared sequence dataset, specifically including: Step 1.1: Randomly sample a video sequence from the infrared sequence dataset, then randomly sample three different infrared images from the video sequence, and sort the three sampled infrared images according to the number of video frames. The smallest frame is the template image, followed by the dynamic template image, and the largest frame is the search area image. The maximum sampling interval is set to 50 frames. Step 1.2: Crop the three sampled infrared images with the target bounding box as the center, and simultaneously perform image enhancement on the cropped images with a set probability, including image flipping and image grayscale value changes; finally, the preprocessed template image T, dynamic template image T_O, and search region image S are combined into an input image group as the input of the encoder. Step 2: Construct a spatiotemporal adaptive thermal infrared tracking network based on coordinate information, including a Transformer-based encoder, a Transformer decoder with masking, a multilayer perceptron, and a dynamic template update branch; the tracking network processes image groups as follows: Step 2.1: First, pass each image in the input image group through a convolutional layer with a stride of 16 and a kernel size of 16 to convert the images into patches. Then, flatten the patches and concatenate the patches of the three images into a feature vector. Step 2.2: Embed sin-con positional encoding into the feature vector to obtain a feature vector with positional encoding information; Step 2.3: The Transformer-based encoder is used for feature extraction and feature fusion of the template and search region. The feature vector with positional encoding information is used as the input to the encoder. The attention mechanism in the encoder is as follows: K M =Concat(K Z ,K X ),V M =Concat(V Z ,V X ), Attn=Concat(Attn Z ,Attn X ) Q Z ,K Z V Z Represented as queries, keys, and values ​​in template features, Q X ,K X V X Represented as queries, keys, and values ​​for the search region features, where d represents the dimension of the key, and Attn Z Attn represents the self-attention result of template features. X The cross-attention result represents the search region features; the output of the i-th layer of the encoder is represented as: f i =MLP(Norm(Attn(Norm(f i-1 ))+f i-1 ))+f i-1 i=1,2,……L Where MLP stands for Multilayer Perceptron, Norm stands for LayerNorm, and f i f represents the output of the i-th layer. 0 =[Z P ,X P ]; Step 2.4: Extract the features of the search region from the feature vector output by the encoder and use them as input to the masked Transformer decoder; Step 2.5: Discretize the coordinate information into a coordinate sequence as another input to the Transformer decoder with a mask; Step 2.6: The masked Transformer decoder includes masked self-attention and cross-attention. The masked self-attention is used to enhance the feature representation of the coordinate sequence. The enhanced coordinate features are used as cues for the features of the search region. After further enhancing the features of the search region by cross-attention fusion, the output is sent to the multilayer perceptron. Step 2.7: The probability distribution of the regression coordinate sequence is obtained by using a multilayer perceptron, thereby obtaining the target's physical coordinates. The input is then fed into the dynamic template update branch. The multilayer perceptron consists of one softmax layer and three linear layers. Step 2.8: Evaluate the score of the dynamic template through the dynamic template update branch to determine whether the template should be updated over time. This requires calculating the average of the softmax scores of the probability distributions of the four coordinate sequences to determine the reliability of the current template. The reliability score of the dynamic template can be expressed as: Score=Mean(SoftMax(x),SoftMax(y),SoftMax(w),SoftMax(h)) Mean represents the mean, SoftMax represents the softmax layer, and x, y, w, and h represent the top-left corner coordinates (x, y), width, and height of the target bounding box, respectively. Step 3: Construct the loss function for the spatiotemporal adaptive thermal infrared tracking network based on coordinate information, specifically using the cross-entropy loss function: Z t X represents the template image at the current time t. t Y represents the image of the search region corresponding to the current time t. 0:t-1 This represents the previously predicted target sequence at the current time t, and is related to the template image and the search region image. P represents the softmax probability, t represents the time step of the token, and T represents the length of the target sequence. The input coordinate sequence is defined as [cmd xywh], and the corresponding target sequence is [xywh end]. Step 4: Train the tracking network model using the preprocessed data from Step 1 and the loss function constructed in Step 3, setting the encoder's learning rate to 1e. -5 The learning rate of the non-encoder module is 1e -4 The weight decay is set to 1e. -4 The training process lasted 120 epochs, with 30,000 image groups sampled per epoch. Step 5: Using the trained tracking network model, initialize the template and dynamic template using the first frame image in the sequence. The dynamic template is updated based on the reliability score. When the coordinate sequence is first input, set an identifier as a prompt for the infrared features. Each generated coordinate sequence is used as the input for the next step, gradually prompting the infrared features to predict the bounding box coordinates.

Images