Thermal infrared target tracking Siamese network tracker

Abstract

The invention discloses a thermal infrared target tracking Siamese network tracker, which relates to the technical field of computer vision and comprises a fine-grained feature parallel learning convolution block, a multi-layer fine-grained feature fusion module, a Siamese residual error refining module and a significance loss function. Wherein the fine-grained feature parallel learning convolution block captures key global features from a shallow layer, the feature diversity is enhanced, and fine-grained information loss in residual connection is reduced; the multi-layer fine-grained feature fusion module effectively integrates depth and shallow features through bilinear matrix multiplication; the Siamese residual error refining module uses a residual error learning technology to correct a saliency map prediction error, and combines depth supervision to gradually optimize a prediction result; the saliency loss function restrains saliency prediction and guides the network to pay attention to highly distinguished fine-grained features. Experiments show that the tracker is excellent in performance on a plurality of benchmark tests.

Description

Technical Field

[0001] This invention relates to the field of computer vision, specifically to a thermal infrared target tracking Siamese network tracker. Background Technology

[0002] Thermal infrared (TIR) ​​target tracking is an important research direction in the field of computer vision. With the continuous advancement of uncooled TIR imaging technology, TIR sensors have become more compact and can capture higher-resolution images, laying a solid foundation for developing more efficient TIR tracking algorithms. Compared with visible light tracking, TIR tracking has significant advantages, such as all-weather operation and strong resistance to environmental interference, making it particularly valuable in applications such as autonomous driving, video surveillance, and maritime rescue.

[0003] However, TIR targets typically exhibit low signal-to-noise ratios, poor contrast, limited texture detail, and blurred edges. These issues severely impact the performance of traditional feature extraction models, making it difficult to generate clear and discriminative target representations, thus increasing the complexity of the tracking process. Traditional methods rely on hand-designed features, which are limited in effectiveness with TIR images. Although deep learning has made some progress, it still faces challenges in tracking visually or semantically similar targets, as pre-trained models struggle to extract highly discriminative features.

[0004] To address these issues, recent research has focused on fine-grained feature learning, but problems still exist, such as the difficulty in extracting distinguishing features when the target and background are highly similar. Summary of the Invention

[0005] The purpose of this invention is to provide a thermal infrared target tracking Siamese network tracker to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a thermal infrared target tracking Siamese network tracker, comprising fine-grained feature parallel learning convolutional blocks (FGPCBs), employing a two-stream architecture, using convolutional kernels of different sizes, achieving downsampling by adjusting the convolutional kernel size and setting a stride of 2, preserving fine-grained information in the residual connections, and employing the LeakyReLU activation function, defined as: f(x)=max(αx,x), where 0<α<1, x represents the feature matrix element values ​​input to the activation function after convolution operations and batch normalization;

[0007] The multi-layer fine-grained feature fusion module is used to fuse deep and shallow feature matrices through matrix outer product and normalization operations, projecting feature information from low-dimensional space to high-dimensional space. The outputs of the last three layers are fused through matrix multiplication, and then processed by a fully connected layer and a softmax function to retain the region with the highest response value. The output feature matrix is ​​defined as follows: This allows for more accurate capture and differentiation of regions, thereby improving the quality of feature representation. Specifically, feature matrices X, Y, and Z are obtained through Conv4_1, Conv4_2, and Conv4_3 layers, respectively, with dimensions B, C, and H×W. Here, B represents the batch size, C represents the number of channels, and H×W represents the spatial size of each feature map. The matrix products of feature matrices X and Y, X and Z, and Y and Z are calculated, normalized, and then added to form the input of the fully connected layer, resulting in a multi-layer fine-grained feature fusion model that effectively fuses fine-grained features from different layers.

[0008] The Siam residual refinement module corrects prediction errors in the initial saliency map. This module alternately uses low-level or high-level ensemble feature streams and the saliency map from the previous iteration as input, calculates residuals, and adds them to the previous saliency map to generate a more accurate saliency map. The residual calculation is: Residual = Φi × (cat(si-1, F)). The saliency map is updated as follows: Residue i, where si-1 is the predicted saliency map generated in the previous step, F is the feature map, and Φ is the calculation function;

[0009] The saliency loss function is used to constrain saliency prediction, guiding the network to focus on highly discriminative fine-grained features. The total loss function is LFGSGT = λ. cls L cls +λ reg L reg +λ sal L sal , where λ cls , λ reg and λ sal These represent the weights of the classification loss, regression loss, and significance prediction loss, respectively. The classification loss is defined using the cross-entropy loss, the regression loss uses the L1 loss function, and the significance prediction loss is the sum of the losses of all the prediction significance maps.

[0010] Preferably, the fine-grained feature parallel learning convolutional block adopts a two-stream architecture, which concatenates different feature representation matrices through connection operations, increases the channel dimension to enhance feature diversity, uses 1×3 and 3×1 convolutional kernels for feature extraction, retains conv1×1 convolution as the key element in residual connections, removes pooling layers, and directly replaces connection operations with addition in the fourth layer.

[0011] Preferably, the multi-layer fine-grained feature fusion model is obtained through the following steps:

[0012] Let X ∈ ? Hx, wx, Cx and Y ∈ ? Hy, wy, Cy and Z ∈ ? Hz, where ? denotes the bilinear operator between layers, and wz and Cz denote the multi-layer feature matrices extracted from parallel convolutional blocks, defined as follows:

[0013] X = M(conv) 3×1 (conv 1×3 (concat(conv 1×1 (F),conv 3×3 (F)))))+conv 3×3 (F)

[0014] Y = conv 3×1 (conv 1×3 (conv 1×1 (X)+conv 3×3 (X)))+X

[0015] Z = conv 3×1 (conv 1×3 (conv 1×1 (Y)+conv 3×3 (Y)))+Y

[0016] Where conv contains convolution, batch normalization, and activation layers; F represents the feature map of the previous layer; conv 1×3 conv 3×1 conv 1×1 and conv 3×3 These represent convolutional kernels of sizes 1×3, 3×1, 1×1, and 3×3, respectively. Downsampling is achieved through max pooling to reduce the size of the feature map.

[0017] The multi-layer fine-grained feature fusion model is: o bp =σ(N(β(X,Y))+N(β(X,Z))+N(β(Y,Z))), where σ represents the softmax function, N represents the normalization operation, and β is the bilinear operator in each feature matrix.

[0018] Preferably, the specific implementation logic for forming the fully connected layer in the multi-layer fine-grained feature fusion module is as follows: [The following text appears to be incomplete and requires further context: "to..."] Adjust from (B, C, C) dimensions to (B, C × C), where X is a bilinear operator. T Y T Z T The transposes of X, Y, and Z are respectively; then the process is repeated for the three dimensions X, Y, and Z, and the results of these three dimensions are added together to form the input of the fully connected layer.

[0019] Preferably, the Siam residual refinement module forms a residual refinement saliency prediction network. This network adopts a deep supervision mechanism. During the training process, each recursive step applies a supervision signal to the saliency map. The saliency prediction is gradually optimized through the Siam residual refinement module. The low-level ensemble feature stream is generated by upsampling the feature maps of Conv1, Conv2, and Conv3 and then performing connection and convolution operations. The high-level ensemble feature stream is generated by connecting and convolution operations on the feature maps of Conv4 and Conv5.

[0020] Preferably, the formula for generating the low-level ensemble feature flow is Flow = fconv(cat(F1, F2, F3)), where F1, F2, and F3 are the upsampled feature maps of Conv1, Conv2, and Conv3 layers, respectively; cat represents the concatenation of feature maps; and fconv refers to the feature fusion network, which consists of three convolutional layers and a PReLU activation function. The formula for generating the high-level ensemble feature flow is Fhigh = fconv(cat(F4, F5)), where F4 and F5 are the feature maps of Conv4 and Conv5 layers, respectively.

[0021] Preferably, the classification loss L cls The cross-entropy loss is defined as follows:

[0022]

[0023] Where u j Represents the probability predicted by the network. Let be the true value of the j-th sample;

[0024] The regression loss L reg The L1 loss function is used, and its definition is as follows:

[0025]

[0026] Where, N pos Indicates the number of positive samples. Indicates when When the value is greater than 0, the function equals 1, L Iou For IoU loss; when the value is equal to 0, L1 represents the L1 norm loss, λ G λ1 and p are regularization parameters. j Represents the predicted bounding box of the j-th sample; Indicates the actual bounding box label;

[0027] The significance prediction loss L sal Defined as the sum of the losses of all predicted saliency maps, it is defined as:

[0028]

[0029] Where w0 and y0 represent the weights and loss of the initial significance prediction, respectively, and w i and y i Then, represents the weight and loss of the i-th recursive step, and N is the number of recursive steps in the refining process of the Siam residual refining module.

[0030] Preferably, the fine-grained feature parallel learning convolutional block architecture uses ResNet-50 as the backbone network, adjusts the stride in Conv4 and Conv5 blocks to 8 pixels, expands the receptive field by using dilated convolutions, adds a 1×1 convolutional layer after each residual block, reduces the number of output channels to 256, and processes the feature maps of Conv3, Conv4 and Conv5 by three independent FGSGT_RPN modules to generate feature maps with uniform spatial resolution. After weighted summation, they are aggregated through a weighted fusion layer.

[0031] Preferably, in the FGSGT_RPN module, the extracted multi-level features are fused by the encoder module to form a unified representation. A deep cross-correlation operation is introduced to independently calculate the correlation of each input channel. Subsequently, a classification CLS score and a bounding box regression Reg value are generated through a cross-correlation layer and a fully convolutional layer. Finally, the outputs of multiple RPN modules are aggregated through weighted fusion to obtain the final tracking result. Among them, the classification score CLS is generated to determine the probability that the current region belongs to the target and to distinguish the target from the background. The bounding box regression value Reg is generated to predict the bounding box coordinates of the target, so as to achieve accurate locking of the target position and reduce tracking drift.

[0032] Compared with existing technologies, the beneficial effects of this invention are as follows: The Siamese network tracker for thermal infrared target tracking proposed in this invention extracts global context information from shallow layers through parallel learning of convolutional blocks with fine-grained features, enhancing spatial details and reducing the loss of fine-grained information in residual connections; it effectively integrates deep and shallow features through a multi-layer fine-grained feature fusion module; it corrects prediction errors in the saliency map through a Siamese residual refinement module combining residual learning and deep supervision, iteratively refining prediction quality; and it guides the network to focus on highly discriminative fine-grained features by constraining saliency prediction through a saliency loss function.

[0033] Extensive experimental results show that this tracker outperforms existing trackers in terms of accuracy, success rate, and robustness on benchmark tests such as PTB-TIR, LSOTB-TIR, and VOT-TIR. It can effectively solve the tracking interference and drift problems caused by the difficulty of feature extraction in thermal infrared target tracking, and has high practical value. Attached Figure Description

[0034] Figure 1The diagram shows the FGSGT architecture of the present invention, where (a) is an overall preview of FGSGT, (b) is the FGSGT_RPN module architecture, and (c) is the key fine-grained information saliency guidance network architecture.

[0035] Figure 2 This is a schematic diagram of the fine-grained feature parallel learning convolutional block (FGPCB) structure of the present invention;

[0036] Figure 3 This is a schematic diagram of the multi-layer fine-grained feature fusion model structure of the present invention;

[0037] Figure 4 This is a schematic diagram showing the performance comparison results of the PTB-TIR benchmark.

[0038] Figure 5 The results are for the LSOTB-TIR benchmark performance comparison.

[0039] Figure 6 A visual representation of the tracking results in the PTB-TIR benchmark test;

[0040] Figure 7 This is a visualization of the tracking results in the LSOTB-TIR benchmark test. Detailed Implementation

[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] Please see Figure 1 This invention provides a technical solution: a thermal infrared target tracking Siamese network tracker, including fine-grained feature parallel learning convolutional blocks (FGPCBs), adopting a two-stream architecture, using convolutional kernels of different sizes, achieving downsampling by adjusting the convolutional kernel size and setting a stride of 2, preserving fine-grained information in the residual connections, and employing the Leaky ReLU activation function, defined as: f(x)=max(αx,x), where 0<α<1, x represents the feature matrix element values ​​input to the activation function after convolution operations and batch normalization;

[0043] Fine-grained feature parallel learning convolutional blocks employ a two-stream architecture, such as... Figure 2As shown, different feature representation matrices are concatenated through connection operations to increase channel dimension and enhance feature diversity. 1×3 and 3×1 convolutional kernels are used for feature extraction, while conv1×1 convolution is retained as the key element in residual connection. Pooling layers are removed, and addition is used directly to replace connection operations in the fourth layer.

[0044] The fine-grained feature parallel learning convolutional block architecture uses ResNet-50 as the backbone network, adjusts the stride in Conv4 and Conv5 blocks to 8 pixels, expands the receptive field by using dilated convolutions, adds a 1×1 convolutional layer after each residual block, and reduces the number of output channels to 256. The feature maps of Conv3, Conv4 and Conv5 are processed by three independent FGSGT_RPN modules to generate feature maps with uniform spatial resolution, which are then aggregated by weighted summation and weighted fusion layer.

[0045] The multi-layer fine-grained feature fusion module is used to fuse deep and shallow feature matrices through matrix outer product and normalization operations, projecting feature information from low-dimensional space to high-dimensional space. The outputs of the last three layers are fused through matrix multiplication, and then processed by a fully connected layer and a softmax function to retain the region with the highest response value. The output feature matrix is ​​defined as follows: This allows for more accurate capture and differentiation of regions, thereby improving the quality of feature representation. Specifically, feature matrices X, Y, and Z are obtained through Conv4_1, Conv4_2, and Conv4_3 layers, respectively, with dimensions B, C, and H×W. Here, B represents the batch size, C represents the number of channels, and H×W represents the spatial size of each feature map. The matrix products of feature matrices X and Y, X and Z, and Y and Z are calculated, normalized, and then added to form the input of the fully connected layer, resulting in a multi-layer fine-grained feature fusion model that effectively fuses fine-grained features from different layers.

[0046] The multi-layer fine-grained feature fusion model is obtained through the following steps:

[0047] Let X ∈ ? Hx, wx, Cx and Y ∈ ? Hy, wy, Cy and Z ∈ ? Hz, where ? denotes the bilinear operator between layers, and wz and Cz denote the multi-layer feature matrices extracted from parallel convolutional blocks, defined as follows: X = M(conv 3×1 (conv 1×3 (concat(conv 1×1 (F),conv 3×3 (F)))))+conv 3×3 (F)

[0048] Y = conv 3×1 (conv 1×3 (conv1×1 (X)+conv 3×3 (X)))+X

[0049] Z = conv 3×1 (conv 1×3 (conv 1×1 (Y)+conv 3×3 (Y)))+Y

[0050] Where conv contains convolution, batch normalization, and activation layers; F represents the feature map of the previous layer; conv 1×3 conv 3×1 conv 1×1 and conv 3×3 These represent convolutional kernels of sizes 1×3, 3×1, 1×1, and 3×3, respectively. Downsampling is achieved through max pooling to reduce the size of the feature map.

[0051] The multi-layer fine-grained feature fusion model is: o bp =σ(N(β(X,Y))+N(β(X,Z))+N(β(Y,Z))), where σ represents the softmax function, N represents the normalization operation, and β is the bilinear operator in each feature matrix. The model architecture is as follows: Figure 3 As shown.

[0052] The specific implementation logic for forming the fully connected layer in the multi-layer fine-grained feature fusion module is as follows: The input to the fully connected layer is... Adjust from (B, C, C) dimensions to (B, C × C), where X is a bilinear operator. T Y T Z T The transposes of X, Y, and Z are respectively; then the process is repeated for the three dimensions X, Y, and Z, and the results of these three dimensions are added together to form the input of the fully connected layer.

[0053] Siam Residual Refining Module (SRRB), such as Figure 1 As shown in (c), to correct the prediction error in the initial saliency map, this module alternately uses low-level ensemble feature streams or high-level ensemble features and the saliency map from the previous iteration as input, calculates the residuals, and adds them to the previous saliency map to generate a more accurate saliency map. The refinement operation performed by SRRB can be represented as follows: The residual is calculated as: Residual = Φi × (cat(si-1, F)), and the saliency map is updated to... Residue i, where in the i-th recursive step, the predicted saliency map si-1 generated in the previous step is concatenated with the feature map F. Then, the function Φ is applied to calculate the residual residuei; finally, the residual residuei is added element by element to si-1, and the updated saliency map si-1 is calculated.

[0054] A residual refinement saliency prediction network is used. To effectively learn salient regions during the refinement process, a series of salient region enhancers (SRRBs) are used to progressively improve the accuracy of saliency prediction. A fine-grained feature extraction module generates a series of multi-scale feature maps. These deep feature maps capture high-level semantic information of salient objects, while shallow feature maps preserve the fine structural details of salient regions. The output feature maps of the convolutional layers are divided into two categories: low-level feature maps, including Conv1, Conv2, and Conv3, which are generated by upsampling and convolution operations; and high-level feature maps, including Conv4 and Conv5, which are also generated by convolution operations.

[0055] The Siam residual refinement module forms the residual refinement saliency prediction network. This network adopts a deep supervision mechanism. During the training process, each recursive step applies a supervision signal to the saliency map, and the saliency prediction is gradually optimized through the Siam residual refinement module.

[0056] The formula for generating the low-level ensemble feature flow is Flow = fconv(cat(F1, F2, F3)), where F1, F2, and F3 are the upsampled feature maps of Conv1, Conv2, and Conv3 layers, respectively; cat represents the concatenation of feature maps; and fconv refers to the feature fusion network, which consists of three convolutional layers and a PReLU activation function. The formula for generating the high-level ensemble feature flow is Fhigh = fconv(cat(F4, F5)), where F4 and F5 are the feature maps of Conv4 and Conv5 layers, respectively.

[0057] First, the network predicts a saliency map s0 from the high-level ensemble feature Fhigh. This map captures the location of salient objects but lacks fine-grained saliency details. Then, based on s0, the saliency prediction is progressively optimized through a series of SRRBs. Since the low-level ensemble feature Flow provides finer saliency details, the first SRRB is constructed by setting the feature map F to Flow (as shown in the equation residual = Φi × (cat(si-1, F))), which refines s0 and generates a more detailed saliency map s1. However, Flow also contains a significant amount of non-saliency information, which can lead to irrelevant regions in S1. To address this, the second SRRB is constructed by replacing Flow with Fhigh in the equation residual = Φi × (cat(si-1, F)), which helps eliminate the non-saliency regions introduced by Flow. Since Fhigh captures the semantic features of salient objects, this operation effectively removes non-saliency details outside the semantic regions.

[0058] The algorithm steps are as follows:

[0059] Algorithm: Use SRRB for saliency refinement

[0060] Input: Initial saliency graph s0; Low-level feature flow graph Flow;

[0061] At each iteration i = 1 to N, s0 is initialized from the high-level feature map Fhigh.

[0062] If i is odd, then;

[0063] If Input ← Flow; otherwise;

[0064] Set the F input to ←F high;

[0065] End time;

[0066] The residual is calculated based on the equation: residual = Φi × (cat(si―1, F)).

[0067] According to the formula The significance map of residue i is updated to si, which is supervised during training;

[0068] Finish;

[0069] Output: Final refined significance graph sfinal←sN.

[0070] To further improve the accuracy of saliency prediction, we employ a series of SRRBs (Saliency Region Enhancers), iteratively optimizing predictions by alternately combining high-level features Fhigh and low-level features Flow. Furthermore, we introduce a deep supervision mechanism, applying a supervisory signal to the saliency map at each recursive step during training. Figure 1 As shown in (c), by introducing auxiliary supervision in the intermediate stage, each SRRB can directly learn the residual with the true value, thus simplifying the network optimization process. Finally, the saliency map sfinal generated in the last recursive step is used as the final output of the network;

[0071] The saliency loss function is used to constrain saliency prediction, guiding the network to focus on highly discriminative fine-grained features. The total loss function is LFGSGT = λ. cls L cls +λ reg L reg +λ sal L sal , where λ cls , λ reg and λ sal These represent the weights of the classification loss, regression loss, and significance prediction loss, respectively. The classification loss is defined using the cross-entropy loss function, the regression loss uses the L1 loss function, and the significance prediction loss is the sum of the losses from all predicted significance maps. Classification loss Lcls The cross-entropy loss is defined as follows:

[0072]

[0073] Where u j Represents the probability predicted by the network. Let be the true value of the j-th sample;

[0074] Regression loss L reg The L1 loss function is used, and its definition is as follows:

[0075]

[0076] Where, N pos Indicates the number of positive samples. Indicates when When the value is greater than 0, the function equals 1, L Iou For IoU loss; when the value is equal to 0, L1 represents the L1 norm loss, λ G λ1 and p are regularization parameters. j Represents the predicted bounding box of the j-th sample; Indicates the actual bounding box label;

[0077] Significance prediction loss L sal Defined as the sum of the losses of all predicted saliency maps, it is defined as:

[0078]

[0079] Where w0 and y0 represent the weights and loss of the initial significance prediction, respectively, and w i and y i Then, represents the weight and loss of the i-th recursive step, and N is the number of recursive steps in the refining process of the Siam residual refining module.

[0080] In the FGSGT_RPN module, extracted multi-level features are fused into a unified representation through the encoder module. A deep cross-correlation operation is introduced to independently calculate the correlation of each input channel. Subsequently, a classification CLS score and bounding box regression Reg value are generated through cross-correlation layers and fully convolutional layers. Finally, the outputs of multiple RPN modules are aggregated through weighted fusion to obtain the final tracking result. Specifically, the classification score (CLS) determines the probability that the current region belongs to the target, distinguishing the target from the background; the bounding box regression value (Reg) predicts the bounding box coordinates of the target, achieving precise target location locking and reducing tracking drift. The architecture of the FGSGT_RPN module is as follows: Figure 1 As shown in (b).

[0081] The following description is based on specific embodiments:

[0082] The thermal infrared target tracking Siamese network tracker of this invention is an FGSGT tracker, and its overall architecture is as follows: Figure 1 As shown in (a), ResNet-50 is used as the backbone network. The stride in Conv4 and Conv5 blocks is adjusted to 8 pixels. The receptive field is expanded by using dilated convolutions. A 1×1 convolutional layer is added after each residual block to reduce the number of output channels to 256. The feature maps of Conv3, Conv4, and Conv5 are processed by three independent FGSGT_RPN modules to generate feature maps with uniform spatial resolution. After weighted summation, they are aggregated through a weighted fusion layer. The specific implementation logic is as follows:

[0083] Datasets and Evaluation Criteria. The PTB-TIR benchmark is a TIR pedestrian dataset designed specifically for short-term tracking tasks, containing 60 manually annotated TIR pedestrian sequences with a total of over 30,000 frames. The LSOTB-TIR benchmark;

[0084] This is currently the largest TIR dataset, containing 47 object types and 1400 TIR video sequences, with a total of over 600,000 frames. Furthermore, the dataset provides annotations for 730,000 bounding boxes. Following the methods described in the PTB-TIR and LSOTB-TIR benchmarks, we use center position error (accuracy) and overlap score (success rate) as evaluation metrics for the PTB-TIR benchmark. Additionally, normalized accuracy (NP) is introduced as an evaluation metric for the LSOTB-TIR benchmark. VOT-TIR2015 and VOT-TIR2017 are short-term TIR object tracking datasets released by the Visual Object Tracking (VOT) Committee. These benchmarks contain 45 test sequences covering eight object categories, with an average sequence length of 651 frames. As described in VOT-TIR2015 and VOT-TIR2017, we use accuracy, robustness, and expected average overlap (EAO) as evaluation metrics for both benchmarks.

[0085] Experimental Platform. The proposed FGSGT tracker was implemented in Python 3.9 and trained on the LSOTB-TIR TIR dataset and the visible light datasets VOT 2017 and VOT 2019. Training was accelerated using CUDA 11.8 and cuDNN 7.6. The image patch size for the search region was 255×255, while the image patch size for the template region was 127×127. All experiments were performed on a 24GB NVIDIA GeForce RTX 3090 GPU.

[0086] Ablation experiment:

[0087] Ablation experiments were performed on the PTB-TIR and LSOTB-TIR benchmarks to evaluate the effectiveness of the proposed Fine-Grained Feature Parallel Learning Convolutional Block (FGPCB), Residual Refined Saliency Prediction Network (RRSPN), and FGSGT loss function. Experimental results are shown in Table 1, with SiamRPN++ used as the baseline tracker.

[0088] Table 1. Ablation results on PTB-TIR and LSOTB-TIR baselines

[0089]

[0090] As shown in Table 1, the Siam_FGPCB model outperforms the baseline tracker SiamRPN++ in overall performance. On the PTB-TIR dataset, it improves accuracy by 3.5% and success rate by 2.7%. On the LSOTB-TIR dataset, it improves accuracy by 2.7%, normalized accuracy by 1.7%, and success rate by 3.1%. These results demonstrate that integrating fine-grained features and learning convolutional blocks in parallel can significantly improve tracking performance. Furthermore, the Siam_RRSPN model improves accuracy by 3.0% and 2.0%, respectively, and success rate by 4.0% and 4%, respectively. On the PTB-TIR and LSOTB-TIR datasets, the success rate reaches 4%, validating the effectiveness of the residual refinement saliency prediction network. Moreover, the Siam_T model, built upon Siam_RRSPN, further enhances performance, improving accuracy and success rate by 2.5% and 1.0%, respectively, on the PTB-TIR dataset. On the LSOTB-TIR dataset, the Siam_T model improved accuracy by 6.3%, normalized accuracy by 2.5%, and success rate by 1.1%.

[0091] Performance comparison with the latest technology:

[0092] To evaluate the proposed FGSGT, we compared it with several state-of-the-art trackers and presented experimental results on the LSOTB-TIR, PTB-TIR, VOT-TIR2015, and VOT-TIR2017 benchmarks. See details in [link to relevant documentation]. Figure 4 , Figure 5 and Table 2;

[0093] Results of PTB-TIR and LSOTB-TIR. For example... Figure 4 As shown, the proposed FGSGT;

[0094] On PTB-TIR, the tracker achieved the best performance across all evaluation metrics.

[0095] In benchmark tests, FGSGT achieved an accuracy score of 0.830 and a success rate of 0.625. Compared to the baseline tracker SiamRPN++, FGSGT improved accuracy by 9.0% and success rate by 6.0%. Furthermore, compared to Transformer-based trackers such as TransT and DFG, FGSGT achieved performance improvements of 2.1% and 1.5%, respectively.

[0096] Compared to correlation filter-based trackers (such as ECOstir and MCCT), FGSGT improves success rate by 2.6% and 8.5%, respectively.

[0097] Regarding the test results for LSOTB-TIR. For example... Figure 5 As shown, FGSGT achieved an accuracy of 0.845, surpassing AMFT, DFG, and TransT. Specifically, FGSGT was 1.1% more accurate and 0.9% more successful than DFG. Furthermore, FGSGT achieved the highest success rate.

[0098] Compared with deep learning-based trackers DiMP and VITAL, the proposed tracker improves the success rate by 5.0% and 21.4%, respectively.

[0099] Results from VOT-TIR 2015 and VOT-TIR 2017. As shown in Table 2, the proposed FGSGT tracker achieved an accuracy score of 0.78 in the VOT-TIR 2015 benchmark, tying for first place with DFG. In the VOT-TIR 2017 benchmark, its accuracy score reached 0.75, surpassing methods such as SiamSAV, Ocean, and TransT. Furthermore, FGSGT achieved an EAO score of 0. In the 2015 benchmark, FGSGT achieved an accuracy of 375, exceeding SiamRPN++ and ranking only behind UDCT. Compared to the baseline tracker SiamRPN++, FGSGT improved accuracy by 4.0% and 6.0% in the 2015 and 2017 benchmarks, respectively, and improved EAO by 6.2% and 6.4%, respectively. Furthermore, FGSGT achieved robustness scores of 1.59 and 1.90 in benchmarks in 2015 and 2017, respectively, surpassing Siamese network-based trackers such as MLSSNet and MMNet, as well as Transformer-based tracker DFG.

[0100] Table 2: Tracking Challenge Results in LSOTB-TIR Benchmark Tests

[0101]

[0102]

[0103] Visualization of tracking results:

[0104] To more intuitively demonstrate the effectiveness of the proposed FGSGT tracker, Figure 6 and Figure 7 The results show visual tracking of six challenging sequences using FGSGT, AMFT, TransT, and DeepSTRCF on the LSOTB-TIR and PTB-TIR benchmarks. In both figures, the green lines represent the ground truth bounding boxes of the targets.

[0105] like Figure 6 and Figure 7 As shown, FGSGT achieves more accurate tracking in most sequences. Specifically, in Figure 6 (a) and Figure 6 (b) In the "Birds" and "Classroom 2" sequences shown, the presence of interfering objects caused AMFT and TransT to experience tracking failures or drift in frames 65 and 224, respectively. In contrast, FGSGT achieves accurate localization by learning highly discriminative, fine-grained features of the target to distinguish between interfering objects and the actual target.

[0106] Similarly, in Figure 7 In the “person_D_004” sequence shown in (a), AMFT and TransT experienced tracking drift in frame 606 due to water background interference and partial occlusion, while DeepSTRCF lost the target in frame 657. However, FGSGT was still able to accurately locate the target. Furthermore, Figure 7 (b) and Figure 7 (c) Shows the sequences “person_H_002” and “person_S_018”, characterized by low resolution and poor signal-to-noise ratio, resulting in weak target feature representation and the presence of multiple similar objects. These challenges caused DeepSTRCF, TransT, and AMFT to lose targets in frames 39 and 41, respectively. Due to its superior target representation capabilities, FGSGT demonstrated more accurate localization in these two sequences.

[0107] In TIR images, traditional trackers are susceptible to interference from visually similar objects and experience tracking drift due to sparse feature details. This invention introduces several key innovations in its tracker, including a dual-stream architecture and parallel learning of convolutional blocks with fine-grained features of varying kernel sizes. This not only enhances feature diversity but also preserves fine-grained details. The invention also proposes a multi-layer feature fusion module that effectively integrates deep and shallow features through bilinear matrix multiplication. A Siamese residual refinement module refines saliency map predictions through residual learning and deep supervision, thereby improving accuracy. Finally, a dedicated saliency loss function guides the network to focus on highly discriminative fine-grained features. Extensive experimental results demonstrate that FGSGT outperforms existing trackers in accuracy, success rate, and robustness on PTB-TIR, LSOTBTIR, and VOT-TIR benchmarks.

[0108] This invention discloses a Siamese network tracker for thermal infrared target tracking, relating to the field of computer vision technology. It includes a fine-grained feature parallel learning convolutional block, a multi-layer fine-grained feature fusion module, a Siamese residual refinement module, and a saliency loss function. Specifically, the fine-grained feature parallel learning convolutional block captures key global features from shallow layers, enhancing feature diversity and reducing the loss of fine-grained information in residual connections; the multi-layer fine-grained feature fusion module effectively integrates deep and shallow features through bilinear matrix multiplication; the Siamese residual refinement module uses residual learning techniques to correct saliency map prediction errors and gradually optimizes prediction results combined with deep supervision; and the saliency loss function constrains saliency prediction, guiding the network to focus on highly discriminative fine-grained features. Experiments show that this tracker performs excellently on multiple benchmark tests.

[0109] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A thermal infrared target tracking Siamese network tracker, characterized in that, include: Fine-grained feature parallel learning convolutional blocks are implemented using a two-stream architecture with convolutional kernels of different sizes. Downsampling is achieved by adjusting the kernel size and setting a stride of 2. Fine-grained information is preserved in the residual connections, and the Leaky ReLU activation function is used, which is defined as: f(x)=max(αx,x), where 0<α<1, and x represents the feature matrix element values ​​input to the activation function after convolution and batch normalization. The multi-layer fine-grained feature fusion module is used to fuse deep and shallow feature matrices through matrix outer product and normalization operations, projecting feature information from low-dimensional space to high-dimensional space. The outputs of the last three layers are fused through matrix multiplication, and then processed by a fully connected layer and a softmax function to retain the region with the highest response value. The output feature matrix is ​​defined as follows: This allows for more accurate capture and differentiation of regions, thereby improving the quality of feature representation. Specifically, feature matrices X, Y, and Z are obtained through Conv4_1, Conv4_2, and Conv4_3 layers, respectively, with dimensions B, C, and H×W. Here, B represents the batch size, C represents the number of channels, and H×W represents the spatial size of each feature map. The matrix products of feature matrices X and Y, X and Z, and Y and Z are calculated, normalized, and then added to form the input of the fully connected layer, resulting in a multi-layer fine-grained feature fusion model that effectively fuses fine-grained features from different layers. The Siam residual refinement module is used to correct prediction errors in the initial saliency map. This module alternately uses low-level ensemble feature streams or high-level ensemble features and the saliency map from the previous iteration as input, calculates residuals and adds them to the previous saliency map to generate a more accurate saliency map. The residual is calculated as: residual = Φi × (cat(si-1, F)). The saliency map is updated as si = si-1 ⊕ residue i, where si-1 is the predicted saliency map generated in the previous step, F is the feature map, and Φ is the calculation function. The saliency loss function is used to constrain saliency prediction, guiding the network to focus on highly discriminative fine-grained features. The total loss function is LFGSGT = λ. cls L cls +λ reg L reg +λ sal L sal , where λ cls , λ reg and λ sal These represent the weights of the classification loss, regression loss, and significance prediction loss, respectively. The classification loss is defined using the cross-entropy loss, the regression loss uses the L1 loss function, and the significance prediction loss is the sum of the losses of all the prediction significance maps.

2. The thermal infrared target tracking Siamese network tracker according to claim 1, characterized in that: The fine-grained feature parallel learning convolutional block adopts a two-stream architecture, which concatenates different feature representation matrices through connection operations, increases the channel dimension to enhance feature diversity, uses 1×3 and 3×1 convolutional kernels for feature extraction, retains conv1×1 convolution as the key element in residual connections, removes pooling layers, and directly replaces connection operations with addition in the fourth layer.

3. The thermal infrared target tracking Siamese network tracker according to claim 1, characterized in that: The multi-layer fine-grained feature fusion model is obtained through the following steps: Let X ∈ ? Hx, wx, Cx and Y ∈ ? Hy, wy, Cy and Z ∈ ? Hz, where ? denotes the bilinear operator between layers, and wz and Cz denote the multi-layer feature matrices extracted from parallel convolutional blocks, defined as follows: X=M(conv 3×1 (conv 1×3 (concat(conv 1×1 (F),conv 3×3 (F)))))+conv 3×3 (F) Y=conv 3×1 (conv 1×3 (conv 1×1 (X)+conv 3×3 (X)))+X Z=conv 3×1 (conv 1×3 (conv 1×1 (Y)+conv 3×3 (Y)))+Y Where conv contains convolution, batch normalization, and activation layers; F represents the feature map of the previous layer; conv 1×3 conv 3×1 conv 1×1 and conv 3×3 These represent convolutional kernels of sizes 1×3, 3×1, 1×1, and 3×3, respectively. Downsampling is achieved through max pooling to reduce the size of the feature map. The multi-layer fine-grained feature fusion model is: o bp =σ(N(β(X,Y))+N(β(X,Z))+N(β(Y,Z))), where σ represents the softmax function, N represents the normalization operation, and β is the bilinear operator in each feature matrix.

4. A thermal infrared target tracking Siamese network tracker according to claim 1, characterized in that: The specific implementation logic for forming the fully connected layer in the multi-layer fine-grained feature fusion module is as follows: The input to the fully connected layer is... Adjust from (B, C, C) dimensions to (B, C × C), where X is a bilinear operator. T Y T Z T The transposes of X, Y, and Z are respectively; then the process is repeated for the three dimensions X, Y, and Z, and the results of these three dimensions are added together to form the input of the fully connected layer.

5. A thermal infrared target tracking Siamese network tracker according to claim 1, characterized in that: The Siam residual refinement module forms a residual refinement saliency prediction network. This network adopts a deep supervision mechanism. During the training process, each recursive step applies a supervision signal to the saliency map. The saliency prediction is gradually optimized through the Siam residual refinement module. The low-level ensemble feature stream is generated by upsampling the feature maps of Conv1, Conv2, and Conv3 and then performing connection and convolution operations. The high-level ensemble feature stream is generated by connecting and convolution operations on the feature maps of Conv4 and Conv5.

6. A thermal infrared target tracking Siamese network tracker according to claim 5, characterized in that: The formula for generating the low-level ensemble feature flow is Flow = fconv(cat(F1, F2, F3)), where F1, F2, and F3 are the upsampled feature maps of Conv1, Conv2, and Conv3 layers, respectively; cat represents the concatenation of feature maps; and fconv refers to the feature fusion network, which consists of three convolutional layers and a PReLU activation function. The formula for generating the high-level ensemble feature flow is Fhigh = fconv(cat(F4, F5)), where F4 and F5 are the feature maps of Conv4 and Conv5 layers, respectively.

7. A thermal infrared target tracking Siamese network tracker according to claim 1, characterized in that: The classification loss L cls The cross-entropy loss is defined as follows: Where u j Represents the probability predicted by the network. Let be the true value of the j-th sample; The regression loss L reg The L1 loss function is used, and its definition is as follows: Where, N pos Indicates the number of positive samples. Indicates when When the value is greater than 0, the function equals 1, L Iou For IoU loss; when the value is equal to 0, L1 represents the L1 norm loss, λ G λ1 and p are regularization parameters. j Represents the predicted bounding box of the j-th sample; Indicates the actual bounding box label; The significance prediction loss L sal Defined as the sum of the losses of all predicted saliency maps, it is defined as: Where w0 and y0 represent the weights and loss of the initial significance prediction, respectively, and w i and y i Then, represents the weight and loss of the i-th recursive step, and N is the number of recursive steps in the refining process of the Siam residual refining module.

8. The one according to claim 1, characterized in that: The fine-grained feature parallel learning convolutional block architecture uses ResNet-50 as the backbone network, adjusts the stride in Conv4 and Conv5 blocks to 8 pixels, expands the receptive field by using dilated convolutions, adds a 1×1 convolutional layer after each residual block, and reduces the number of output channels to 256. The feature maps of Conv3, Conv4 and Conv5 are processed by three independent FGSGT_RPN modules to generate feature maps with uniform spatial resolution, which are then aggregated by weighted summation and weighted fusion layer.

9. A thermal infrared target tracking Siamese network tracker according to claim 8, characterized in that: In the FGSGT_RPN module, the extracted multi-level features are fused by the encoder module to form a unified representation. A deep cross-correlation operation is introduced to independently calculate the correlation of each input channel. Subsequently, the classification CLS score and bounding box regression Reg value are generated through the cross-correlation layer and the fully convolutional layer. Finally, the output of multiple RPN modules is aggregated by weighted fusion to obtain the final tracking result. Among them, the classification score CLS is generated to determine the probability that the current region belongs to the target and to distinguish the target from the background. Generate bounding box regression value Reg: Predict the bounding box coordinates of the target to achieve precise target location locking and reduce tracking drift.

Images