Visible-thermal infrared target tracking method based on three-flow feature aggregation twin network

Abstract

The application discloses a visible light-thermal infrared target tracking method based on a three-flow feature aggregation twin network, which utilizes a three-flow backbone network to realize high-performance and high-efficiency visible light-thermal infrared tracking. The three-flow backbone network can effectively aggregate multi-modal features, wherein two feature extraction flows respectively adopt pre-trained Swin Transformers to extract visible light features and thermal infrared features, and a third feature fusion flow is specially designed to bridge the two feature extraction flows and realize feature fusion at each stage, which provides more sufficient multi-modal feature aggregation at all scales and improves the accuracy of target detection. The application designs a joint-complementary feature aggregation module, and further proposes a depth separable shared attention structure, which reduces the computational complexity of the attention module by sharing the first convolution layer features and using a depth separable convolution, thereby speeding up the tracking speed of the model.

Description

Technical Field

[0001] This invention relates to the field of image detection technology, and in particular to a visible light-thermal infrared target tracking method based on a three-stream feature aggregation twin network. Background Technology

[0002] Target tracking is a task where a target of any type is selected in the first frame of a video, and in each subsequent frame, the tracker needs to determine the target's position and size. Target tracking has wide applications in autonomous driving, drones, and video surveillance. However, target tracking methods relying solely on visible light imaging perform poorly in challenging scenarios such as inclement weather, extreme lighting, cluttered backgrounds, and motion blur. Therefore, researchers have attempted to introduce data from other modalities besides visible light to assist in tracking tasks; this is known as multimodal target tracking. Among these, the most common multimodal target tracking is visible light-thermal infrared tracking. It uses thermal infrared images to assist the model in making judgments. For targets that can generate heat, such as people, animals, and cars, the thermal infrared modality often provides target information that is highly distinct from the background. Therefore, visible light-thermal infrared target tracking has higher robustness and accuracy.

[0003] Currently, most visible-thermal-infrared trackers are built within the framework of visible-light trackers, and can be broadly categorized into MDNet-based and Siamese network-based methods. The MDNet model employs two important strategies: offline learning and online tracking. The offline learning strategy enables the shared backbone network to be trained offline to extract general features. The online tracking strategy allows the model to be trained in a specific domain to classify candidate boxes during tracking. Typically, MDNet-based visible-thermal-infrared trackers incorporate modules suitable for multimodal feature fusion, and by applying online tracking and offline learning strategies, they exhibit excellent performance. Lu et al. introduced hierarchical divergence loss and a dynamic fusion module, and adjusted some network structures. They utilized VGG-M as a feature extractor and designed a dual cross-conditional auxiliary module, leveraging information from one modality to guide feature extraction from another.

[0004] Siamese networks are an important paradigm in target tracking. Visible-light-thermal-infrared tracking methods based on Siamese networks, including the SiamFT and SiamCDA models, achieve fast tracking speeds due to the high-speed characteristics of the Siamese network paradigm. The SiamFT model is an early model applying the Siamese network paradigm to visible-light-thermal-infrared tracking; it is an improvement on the SiamFC model, fusing multimodal features through modal weight calculation. The SiamCDA model is an improvement on the SiamRPN++ model, proposing a complementary sensing multimodal feature fusion module to improve the discriminability of the fused features and an interference sensing region proposal selection module to improve the model's robustness.

[0005] Despite the existence of many high-performance visible-thermal infrared (V-IR) tracking methods, the introduction of thermal infrared modes leads to complex feature extraction and fusion, which degrades model efficiency and hinders the practical application of V-IR tracking. It is difficult for proposed V-IR tracking methods to simultaneously achieve high performance in both accuracy and efficiency. For example, MDNet-based V-IR tracking methods generate a large number of candidate boxes during tracking and require calculating positive scores for each candidate box, significantly reducing tracking speed. Siamese network-based V-IR tracking methods also face challenges in achieving high-precision target tracking due to the difficulty of multimodal feature fusion.

[0006] Therefore, there is an urgent need to design a visible light-thermal infrared target tracking method based on a three-flow feature aggregation twin network to achieve high-precision and fast target tracking. Summary of the Invention

[0007] (a) Technical problems to be solved

[0008] To address the aforementioned issues, this invention provides a visible-thermal infrared target tracking method based on a three-stream feature aggregation twin network. Experiments have shown that it can achieve high-precision and fast target tracking.

[0009] (II) Technical Solution

[0010] To address the aforementioned technical issues, this invention provides a visible light-thermal infrared target tracking method based on a three-stream feature aggregation twin network. The three-stream feature aggregation twin network includes a backbone network, correlation operations, and a prediction head. The original image is extracted and fused through the backbone network to obtain search features and template features. Then, the search features and template features are correlated pixel by pixel to generate a similarity map. The similarity map is input into the prediction head to generate tracking results. The backbone network contains two parallel feature extraction streams and one feature fusion stream. The two feature extraction streams are parallel Swin-Transformers, and the feature fusion stream is composed of four joint-complementary feature aggregation modules connected in series.

[0011] Will Let i and y represent the visible light feature extraction stream, thermal infrared feature extraction stream, and feature fusion stream output from stage i, respectively, where i ∈ {1, 2, 3, 4}. The network at stage i is represented as follows:

[0012]

[0013]

[0014]

[0015] in, and These represent the network modules of the i-th stage corresponding to the three flows; and This represents the initial input to the three streams.

[0016] In the joint-complementary feature aggregation module, the feature fusion flow in the i-th stage is described as follows:

[0017]

[0018] Where, ψ m (·), ψ s (·) represent the modal feature fusion process and the stage feature fusion process, respectively; the channel and spatial attention weights are...

[0019]

[0020] in This represents element-wise addition, where M∈{com,rgb,tir} and D∈{c,s}.

[0021] The modal feature fusion process is represented as:

[0022]

[0023] in This represents element-wise multiplication.

[0024] The phase feature fusion process is represented as follows:

[0025]

[0026]

[0027] Where AP(·), BN(·), and Conv(·) represent average pooling, batch normalization, and convolutional layer operations, respectively. and These represent the channel dimension weights corresponding to the fusion features of the current stage and the fusion features of the previous stage, respectively.

[0028] Furthermore, and The i-th stage network module representing the Swing Transformer consists of block merging and Swing Transformer blocks; block merging implements downsampling and is replaced by block partitioning and linear mapping operations in the first stage; Swing Transformer blocks apply local window self-attention in the window multi-head self-attention module and shifted window self-attention in the moving window multi-head self-attention module.

[0029] Furthermore, Among them, I rgb and I tir represents the visible light image and the thermal infrared image, respectively, and || represents the connection along the channel dimension.

[0030] Furthermore, the concatenated multimodal features are fed into a convolutional module to obtain intermediate shared features F. c The description is as follows:

[0031]

[0032] Among them, GAP(·), Conv DW (·) and δ(·) represent the global average pooling layer, the depthwise separable convolutional layer, and the ReLU activation function, respectively; subsequently, based on the shared feature F c Three channel attention weights are generated respectively.

[0033]

[0034] Where σ represents the Sigmoid activation function, Conv M This represents the corresponding 1×1 convolutional layer; in the spatial dimension, the three spatial attention weights are expressed by the following formula:

[0035]

[0036] V s M =σ(Conv M (F s )),M∈{rgb,tir,com}

[0037] Furthermore, during the stage feature fusion process, depthwise separable shared attention is also used to generate channel-dimensional weights W corresponding to different stages. c cur and Right now:

[0038]

[0039]

[0040] Furthermore, the correlation operation employs a pixel-by-pixel correlation method. After obtaining the search features and template features, they are represented as follows: and For pixel-wise correlation operations, the template feature Z is first decomposed into H0×W0 1×1 convolutional kernels. Then, the search feature X output by the backbone network is convolved with each convolution kernel, and the results of each calculation are stacked along the channel dimension to obtain the correlation graph. This process is represented as:

[0041]

[0042] Where * represents convolution operation.

[0043] Furthermore, the prediction head is a center point prediction head. The correlation map M is input into the three FCN branches in the prediction head to obtain the classification score map, offset map and size map. The center of the predicted target is determined by the coordinates of the maximum value on the classification score map, and the bounding box is obtained using the corresponding offset map and size map.

[0044] Furthermore, the loss function consists of three parts: L cls L giou and L1; L cls It is a classification loss, which is based on the predicted classification score map P. xy Gaussian plot generated from the true values Calculate the Focal loss; for each frame, the center coordinates of the target box are represented as (p x ,p y The Gaussian map is generated by the following formula:

[0045]

[0046] Where σ is the adaptive standard deviation of the object size;

[0047] The classification loss is expressed as:

[0048]

[0049] L giou L1 and L2 are regression losses obtained by calculating the generalized IoU loss and mean absolute error loss for the predicted bounding box and the ground truth bounding box, respectively; the total loss function is expressed as:

[0050] L = L cls +λ1L giou +λ2L1

[0051] Where λ1 and λ2 are regularization loss parameters.

[0052] Furthermore, the AdamW optimizer was used to optimize the network. The initial learning rate was set to 0.0001, and the learning rate was reduced to 0.2 times every 40 epochs. The network was trained for a total of 120 epochs, and the batch size was set to 8.

[0053] The present invention also discloses a visible light-thermal infrared target tracking system based on a three-stream feature aggregation twin network, comprising: at least one processor; and at least one memory communicatively connected to the processor, wherein: the memory stores program instructions executable by the processor, and the processor can execute the visible light-thermal infrared target tracking method based on the three-stream feature aggregation twin network by calling the program instructions.

[0054] (III) Beneficial Effects

[0055] The above-described technical solution of the present invention has the following advantages:

[0056] (1) This invention proposes a visible light-thermal infrared target tracking method based on a three-stream feature aggregation Siamese network. This method utilizes a three-stream backbone network to achieve high-performance and high-efficiency visible light-thermal infrared tracking. The three-stream backbone network can effectively aggregate multimodal features, where two feature extraction streams use pre-trained SwinTransformers to extract visible light and thermal infrared features, respectively. A third feature fusion stream bridges the two feature extraction streams and performs feature fusion at each stage, which provides more comprehensive multimodal feature aggregation at all scales and improves the accuracy of target detection.

[0057] (2) This invention designs a joint-complementary feature aggregation module specifically adapted for three-stream feature aggregation. It is a basic component of the feature fusion stream, which utilizes joint-complementary attention to guide the aggregation of multimodal features. Visible-thermal infrared tracking tasks must fully utilize information from both modalities while distinguishing potentially invalid information in specific modalities. By integrating features from both modalities, joint attention can focus on the target's spatial location information and key semantic information, while complementary attention is used to distinguish between useful and invalid information in each modality, with useful information being enhanced and invalid information being suppressed.

[0058] (3) Furthermore, this invention proposes a lightweight attention structure called depthwise separable shared attention. Since joint-complementary attention is widely used in this model, the calculation of attention weights will greatly affect the tracking efficiency. To further improve the tracking speed, this invention proposes a depthwise separable shared attention structure, which reduces the computational complexity of the attention module by sharing the features of the first convolutional layer and using depthwise separable convolution, thereby accelerating the model's tracking speed. Attached Figure Description

[0059] The features and advantages of the invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the invention in any way. In the drawings:

[0060] Figure 1 This is a diagram showing the overall structure of the three-flow feature aggregation twin network in this invention.

[0061] Figure 2 This is the specific structure of the joint-complementary feature aggregation module proposed in this invention.

[0062] Figure 3 The diagrams show the structures of three types of attention, where (a) represents ordinary attention, (b) represents the improved shared attention, and (c) represents the depth-separable shared attention proposed in this invention.

[0063] Figure 4 The proposed method is compared with other methods on the RGBT210 dataset in terms of PR and SR metrics.

[0064] Figure 5 The proposed method is compared with other methods on the RGBT234 dataset in terms of PR and SR metrics.

[0065] Figure 6 The proposed method is compared with other methods on the LasHeR dataset in terms of PR and SR metrics.

[0066] Figure 7The proposed method is compared with other methods on the VTUAV dataset in terms of PR and SR metrics.

[0067] Figure 8 Visualization of the joint-complementary attention weight map for three scenarios. (a) shows the target exhibiting weak discriminative power in two modes. (b) shows the target being affected by high illumination in the visible light mode. (c) shows the target's temperature being similar to the background. The red box represents the cropped area, and the green box represents the target area.

[0068] Figure 9 This is a qualitative comparison of the tracking method proposed in this invention with six state-of-the-art tracking methods across three video sequences. (a) The target is partially occluded in RGB mode. (b) The appearance and shape of the target change. (c) The target is affected by high illumination in the visible light mode. For each sequence, the left image is a frame in the visible light mode, and the right image is a frame in the thermal infrared mode. Detailed Implementation

[0069] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0070] To address the problem that existing visible light-thermal infrared tracking methods cannot simultaneously achieve high precision and high speed, this invention proposes a visible light-thermal infrared target tracking method based on a three-stream feature aggregation twin network.

[0071] First, addressing the challenge of achieving high accuracy, this invention employs an improved multimodal feature fusion method to obtain high-precision tracking. Specifically, this invention proposes a three-stream backbone network to realize multimodal feature extraction and fusion. This backbone network includes two parallel Swing Transformer feature extraction streams and a feature fusion stream composed of a joint-complementary feature aggregation module, thereby achieving three-stream feature aggregation. The core joint-complementary feature aggregation module utilizes joint-complementary attention to guide the aggregation of multimodal features. Joint attention can focus the features of both modalities on the spatial location information and key semantic information of the target, while complementary attention is used to enhance useful information from one modality and suppress invalid information from another, thereby improving high-precision target recognition and tracking.

[0072] Secondly, addressing the challenge of achieving high-speed tracking, the proposed method employs Siamese networks as its fundamental paradigm. The shared backbone network and lightweight correlation operations of the Siamese network paradigm lay the foundation for the method's basic speed performance. Furthermore, since joint-complementary attention is widely used in the model of this invention, the calculation of attention weights significantly impacts tracking efficiency. To further improve tracking speed, this invention proposes a lightweight attention structure called depthwise separable shared attention. It reduces the computational complexity of the attention module by sharing features from the first convolutional layer and using depthwise convolutions.

[0073] like Figure 1 The diagram shown illustrates the overall structure of the proposed three-stream feature aggregation Siamese network. Following the basic architecture of the Siamese network paradigm, this network comprises three key components: a backbone network, correlation operations, and a prediction head. Initially, the original image is extracted and fused through the backbone network to obtain search features X and template features Z. Then, pixel-wise correlation is performed on the search features X and template features Z to generate a similarity map. Finally, this similarity map M is input into the prediction head to produce tracking results. The main innovation of this invention is the backbone network. The backbone network contains two parallel feature extraction streams and one feature fusion stream. The two feature extraction streams are parallel Swin-Transformers, and the feature fusion stream consists of four joint-complementary feature aggregation modules cascaded together. Therefore, this invention proposes a joint-complementary feature aggregation module to effectively aggregate multimodal features, thereby improving tracking accuracy. Furthermore, an innovative lightweight, deep separable shared attention structure is used in the joint-complementary feature aggregation module to achieve high-speed tracking.

[0074] The overall framework of the present invention is as follows: Figure 1 As shown, Figure 1 The structure of the joint-complementary feature aggregation module (hereinafter referred to as the complementary feature aggregation module) in the backbone network is as follows: Figure 2 As shown, the deep separable shared attention structure used in the joint-complementary feature aggregation module structure is as follows: Figure 3 As shown, the following will be based on the above. Figure 1-3 The improved network structure is described below:

[0075] 1. Feature extraction and fusion

[0076] (1) Constructing a three-stream feature aggregation network

[0077] To effectively extract and fuse multimodal features, this invention designs a three-stream feature aggregation network as the backbone, including a visible light feature extraction stream, a thermal infrared feature extraction stream, and a feature fusion stream, such as... Figure 1As shown, the visible light and thermal infrared feature extraction streams extract semantic features of their respective modalities. They have the same structure but different weight parameters. This invention uses the Swin Transformer as the feature extractor because of its powerful global modeling capabilities and high efficiency. The Swin Transformer backbone network is divided into four stages, and the visible light and thermal infrared features extracted in each stage are input into the corresponding feature fusion stream. The feature fusion stream is designed to effectively aggregate visible light and thermal infrared features, fusing corresponding multimodal features at each stage and outputting the fused features to the next stage. It consists of four sequentially connected complementary feature aggregation modules.

[0078] To clearly describe the processing flow of the backbone network proposed in this invention, Let i and n represent the visible light feature extraction stream, thermal infrared feature extraction stream, and feature fusion stream output from stage i, respectively, where i ∈ {1, 2, 3, 4}. The network at stage i can be represented as follows:

[0079]

[0080]

[0081]

[0082] in and These represent the network modules of the i-th stage corresponding to the three flows; and This represents the initial input to the three streams, i.e., Among them, I rgb and I tir represents the visible light image and the thermal infrared image, respectively, and || represents the connection along the channel dimension.

[0083] like Figure 1 As shown, and The i-th stage network module representing the Swing Transformer consists of block merging and Swing Transformer blocks. Block merging implements downsampling and is replaced by block partitioning and linear mapping operations in the first stage. The Swing Transformer applies local window self-attention in the window multi-head self-attention module and shifted window self-attention in the moving window multi-head self-attention module. This approach enhances global modeling capabilities and reduces computational complexity by facilitating the interaction of features within local windows.

[0084] (2) Complementary Feature Aggregation Module

[0085] This invention proposes a complementary feature aggregation module. Currently, attention mechanisms are widely used in various visual tasks. These mechanisms amplify key information relevant to the target while reducing the impact of distracting data. In visible-thermal infrared tracking tasks, it is essential to fully utilize information from both modalities while distinguishing potentially invalid information within a specific modality. Therefore, this invention proposes a complementary feature aggregation module (i.e., a joint-complementary feature aggregation module) that utilizes a joint-complementary attention mechanism to extract joint and complementary information.

[0086] The structure of the complementary feature aggregation module is as follows: Figure 2 As shown, the joint-complementary feature aggregation module can be divided into two stages. The initial stage involves aggregating features from the visible light and thermal infrared modes to obtain the multimodal fusion features of the current stage, while the subsequent stage involves aggregating the fusion features of the current stage and the previous stage. Therefore, the feature fusion flow of the i-th stage can be described as follows:

[0087]

[0088] Where, ψ m (·), ψ s (·) represent the modal feature fusion process and the stage feature fusion process, respectively.

[0089] Specifically, taking the channel dimension as an example, the depth-separable shared channel attention module will generate three types of channel attention weights. in We focus on common information from both modalities, which will be applied to features in both modalities simultaneously. After network learning, we extract joint features. and Focusing on complementary information from the visible light and thermal infrared modes, this information is applied to each mode separately. After network learning, this enhances the features of the dominant mode and suppresses features of low-quality modes. Similarly, in the spatial dimension, the three attention weights are represented as follows: Furthermore, to avoid over-suppression of features, residual connections are applied to each attention weight:

[0090]

[0091] in This represents element-wise addition, where M∈{com,rgb,tir} and D∈{c,s}. Specifically, this can be calculated and extracted using the depth-separable shared channel attention method, which will be introduced later.

[0092] The modal feature fusion process is represented as:

[0093]

[0094] in This indicates element-wise multiplication.

[0095] In the stage feature fusion process, the fused features from the previous stage are input into a network including convolutional layers, batch normalization layers, and average pooling layers. These features are remapped to align with the current stage features. Next, both sets of stage features are fused using channel attention. Finally, the two sets of features are added together. The stage feature fusion process can be represented as:

[0096]

[0097]

[0098] Where AP(·), BN(·), and Conv(·) represent average pooling, batch normalization, and convolutional layer operations, respectively. and These represent the channel dimension weights corresponding to the fusion features of the current stage and the fusion features of the previous stage, respectively.

[0099] (3) Depth-separable shared attention

[0100] In the joint-complementary feature aggregation module, joint-complementary attention is used multiple times. To reduce computational complexity, this invention proposes a lightweight structure for generating attention weights, called depth-separable shared attention, which includes depth-separable shared spatial attention and depth-separable shared channel attention. Figure 3 Figures (a) to (c) illustrate a comparison between the structure of ordinary attention and the depth-separable shared space / channel attention structure proposed in this invention. Typical joint-complementary attention requires generating three attention weights. One approach is to use three ordinary attention modules, such as... Figure 3 As shown in (a). This invention proposes that attention weights should be able to share most of the information after the first convolutional layer, therefore only one shared feature is generated in the first convolutional layer. Subsequently, it is possible to base... Figure 3 The shared features shown in (b) generate corresponding attention weights in the following 1×1 convolutional layers, which reduces a significant amount of redundant computation. To further minimize computation, this invention also improves the convolutional layers to depthwise separable convolutions. The structure of the depthwise separable shared attention is as follows: Figure 3 As shown in (c). Subsequent experiments will demonstrate that this structure improves the tracker's speed without sacrificing accuracy.

[0101] First, we introduce the details of depthwise separable shared attention along the channel dimension. The concatenated multimodal features are fed into convolutional modules to obtain intermediate shared features F.c This process can be described as follows:

[0102]

[0103] Among them, GAP(·), Conv DW (·) and δ(·) represent the global average pooling layer, the depthwise separable convolutional layer, and the ReLU activation function, respectively. Subsequently, based on the shared feature F... c Three attention weights are generated respectively.

[0104]

[0105] Where σ represents the Sigmoid activation function, Conv M This represents the corresponding 1×1 convolutional layer. Similarly, in the spatial dimension, it can be represented by the following formula:

[0106]

[0107] W s M =σ(Conv M (F s ),M∈{rgb,tir,com}

[0108] Furthermore, during the stage feature fusion process, depthwise separable shared attention is also used to generate channel-dimensional weights W corresponding to different stages. c cur and Right now:

[0109]

[0110]

[0111] 2. Related Operations

[0112] This invention employs a pixel-by-pixel correlation method. Compared to naive correlation methods, this method can more accurately represent the target's position and scale features. From the previous feature extraction and fusion steps, search frame features and template frame features can be obtained, which are represented as follows: and For pixel-wise correlation operations, the template frame feature Z is first decomposed into H0×W0 1×1 convolutional kernels. Then, the search feature X output by the backbone network is convolved with each convolution kernel, and the results of each calculation are stacked along the channel dimension to obtain the correlation graph. This process can be represented as:

[0113]

[0114] Where * represents convolution operation.

[0115] 3. Predicting the head

[0116] This invention utilizes a center point prediction head. The correlation map M is input into three FCN (Fully Convolutional Networks) branches within the prediction head to obtain a classification score map, an offset map, and a size map. The center of the predicted target is determined by the coordinates of the maximum value on the classification score map, and the bounding box is obtained using the corresponding offset and size maps.

[0117] 4. Loss Function

[0118] The loss function consists of three parts: L cls L giou And L1. L cls It is a classification loss, which is based on the predicted classification score map P. xy Gaussian plot generated from the true values Calculate the Focal loss. For each frame, the center coordinates of the bounding box are represented as (p... x ,p y The Gaussian map is generated by the following formula:

[0119]

[0120] Where σ is the adaptive standard deviation of the object size.

[0121] The classification loss can be expressed as:

[0122]

[0123] L giou L1 and L2 are regression losses obtained by calculating the generalized IoU loss and mean absolute error loss for the predicted and ground truth bounding boxes, respectively. The total loss function can be expressed as:

[0124] L = L cls +λ1L giou +λ2L1,

[0125] Where λ1 and λ2 are regularization loss parameters.

[0126] 5. Scheduler and optimizer settings

[0127] This invention uses the AdamW optimizer to optimize the network, with an initial learning rate of 0.0001. The learning rate decreases to 0.2 times every 40 epochs, and the network is trained for a total of 120 epochs with a batch size of 8. Appropriate data augmentation is performed during training, but not during testing.

[0128] 6. Environment Configuration

[0129] This invention is implemented based on the PyTorch 1.12.1 framework. All source code is written in Python 3.8. The operating system is Ubuntu 22.04.2, the CPU is an AMD 1950X, the memory is 32GB, and the GPU is an NVIDIA GeForce RTX 3090ti.

[0130] In addition, to verify that the improved target tracking method described above has excellent high accuracy and fast processing performance, the present invention conducted the following performance comparison test.

[0131] This invention evaluates its proposed method on four popular visible-thermal infrared tracking benchmark datasets: RGBT210, RGBT234, LasHeR, and VTUAV. In the evaluation experiments, two common target tracking evaluation metrics were used: Precision Rate (PR) and Success Rate (SR). PR is defined as the percentage of all frames where the Euclidean distance between the predicted target center and the ground reality center is less than a given threshold. SR is defined as the percentage of all frames where the Intersection over Union (IoU) ratio between the predicted bounding box and the ground reality bounding box is greater than a given threshold. PR and SR plots are used to illustrate the performance of the proposed method against other tracking methods at different thresholds for these two metrics.

[0132] (1) Evaluation of the RGBT210 dataset

[0133] The RGBT210 dataset is an early proposed large-scale visible-thermal infrared tracking dataset, containing 210 video sequences and over 104,700 frame pairs. The tracking method proposed in this invention was compared with 15 advanced tracking methods (mfDiMP, CAT, TFNet, MDNet, ACFN, C-COT, ECO, MEEM, BACF, SRDCF, Staple, KCF+RGBT, SGT, DSST+RGBT, and SOWP+RGBT). The results are as follows: Figure 4 As shown, the method proposed in this invention achieved the best PR and SR scores of 79.7% and 56.3%, respectively. In terms of PR / SR scores, it outperformed the second-best tracker (CAT / mfDiMP) by 0.5% and 0.8%, respectively.

[0134] (2) Evaluation of the RGBT234 dataset

[0135] RGBT234 is currently the most popular visible-thermal-infrared tracking dataset, and many visible-thermal-infrared tracking methods have been evaluated on it. It contains 234 video sequences, some of which are shared with the RGBT210 dataset. RGBT234 includes 12 annotation attributes, including background clutter (BC), camera moving (CM), deformation (DEF), fast motion (FM), heavy occlusion (HO), low illumination (LI), low resolution (LR), motion blur (MB), no occlusion (NO), partial occlusion (PO), scale variation (SV), and thermal crossover (TC). The tracking method proposed in this invention was compared with state-of-the-art tracking methods on the RGBT234 dataset, including TFNet, DAFNet, MANet++, JMMAC, FANet, MANet, DAPNet, SiamCDA, DAT+RGBT, MDNet+RGBT, SGT, C-COT, ECO, SOWP+RGBT, and SRDCF. The results are as follows: Figure 5 As shown, this indicates that the tracking method of the present invention achieved optimal performance, with PR and SR scores of 82.2% and 59.2%, respectively.

[0136] Furthermore, this invention selected seven high-performance tracking methods from the comparisons above to supplement the proposed method with comparative experiments on tracking speed and attribute-based PR / SR scores on the RGBT234 dataset. The experimental results are shown in Table 1. The results demonstrate that the proposed tracking method exhibits the fastest tracking speed at 37 frames per second and ranks among the top three in most attributes. This indicates that the proposed method achieves superior performance in both accuracy and speed.

[0137] Table 1 compares the tracking speed and attribute-based PR / SR (%) scores of seven visible-thermal infrared tracking methods on the RGBT234 dataset. The best, second best, and third best results are represented in red, green, and blue, respectively.

[0138]

[0139] (3) Evaluation of the LasHeR dataset

[0140] LasHeR is currently the largest visible-thermal infrared tracking dataset in terms of video sequences, containing 1224 sequences and over 730,000 image pairs. LasHeR provides more complex and challenging scene sequences. It is divided into training and testing sets, and this tracking method was evaluated on the testing set. Figure 6 As shown, compared with 13 other tracking methods, the tracking method proposed in this invention (APFNet, DMCNet, MaCNet, MANet++, MANet, CAT, DAFNet, mfDiMP, FANet, DAPNet, SGT++, CMR, and SGT) exhibits extremely superior performance. Experimental results show that the PR and SR scores of the proposed method reach 62.5% and 48.1%, respectively. Compared with the second-ranked tracker APFNet, this method improves the PR and SR scores by 12.5% ​​and 11.9%, respectively. Experimental results indicate that the tracker of this invention has greater potential on more challenging datasets.

[0141] (4) Evaluation of VTUAV dataset

[0142] VTUAV is a visible-thermal infrared (VTI) tracking dataset from a UAV perspective, containing 500 video sequences and 1.7 million image pairs. It boasts the highest 1080p resolution among all VTI datasets. VTUAV is divided into four parts: a short-term training set, a short-term test set, a long-term training set, and a long-term test set. Our tracking method was evaluated on the short-term test set. This invention compares our tracking method with five state-of-the-art tracking methods (HMFT, mfDiMP, FSRPN, ADRNet, and DAFNet), with the following results: Figure 7 As shown, the tracking method proposed in this invention achieves PR / SR scores of 82.1% and 67.9%, respectively, which are 6.3% and 5.2% higher than the second-best tracking method, HMFT. Experimental results demonstrate that the proposed tracking method performs well even under challenging UAV perspectives.

[0143] (5) Visualization of joint-complementary attention

[0144] This invention visualizes the weight graph of joint-complementary attention in a spatial dimension to analyze the focus of joint-complementary attention. This invention selects three example scenarios from the RGBT234 dataset, such as... Figure 8 As shown. In Figure 8 In the scenario shown in (a), the target is affected by background clutter in the visible light mode and contains only partial useful information in the thermal infrared mode. Joint attention combines information from both modes and accurately focuses on the target's spatial location. Figure 8(b) shows a scene with strong illumination in the visible light mode, where the complementary attention mechanism successfully suppresses interference from the visible light mode and enhances the feature of the target in the thermal infrared mode. Figure 8 Figure (c) illustrates a case where the target's temperature is similar to the background, causing the target to lose its discriminative features in the visible light mode. The weight map shows that complementary attention suppresses features in the target's thermal infrared mode while enhancing features in the visible light mode. These visualizations demonstrate that the joint-complementary attention proposed in this invention can effectively guide the aggregation of multimodal features.

[0145] (6) Visualization of tracking results

[0146] This invention visualizes some tracking results of the proposed tracking method and five other advanced tracking methods (APFNet, MaCNet, DMCNet, mfDiMP, and MANet++) on the LasHeR dataset, such as... Figure 9 As shown. In Figure 9 In (a), all tracking methods were initially able to track the target in a simple scene, but when the target was partially occluded in the visible light mode, only this tracking method used information from the thermal infrared mode to accurately track the target. Figure 9 Figure (b) describes a scenario in which the target undergoes significant changes in appearance and aspect ratio, but the tracking method is still able to successfully track the target. Figure 9 Image (c) presents a scenario where the target loses its discriminative properties in the visible light mode due to strong illumination. In this challenging scenario, only this tracker achieved perfect tracking.

[0147] (7) Ablation test

[0148] 1) Backbone Network Architecture: To verify the effectiveness of the proposed backbone network architecture, experiments were conducted on the RGBT234 dataset to explore the contributions of different numbers of flows and feature extractors. The results are shown in Table 2. The first row shows the results of the proposed tracking method after replacing the feature extraction network from Swin Transformer to ResNet-50. The second row shows the results of the proposed tracking method after removing the feature fusion flow. The last row shows the results of the proposed tracking method. In summary, the experimental results show that using a three-flow network structure and Swin Transformer as the feature extractor significantly improves the accuracy of the tracking method, with PR / SR scores improved by 3.7% / 3.0% and 1.5% / 1.5% respectively compared to the other two architectures. The experimental results demonstrate the effectiveness of the backbone network architecture of the proposed tracking method.

[0149] 1) Backbone Network Architecture: To verify the effectiveness of the proposed backbone network architecture, experiments were conducted on the RGBT234 dataset to explore the contributions of different numbers of flows and feature extractors. The results are shown in Table 2. The first row shows the results of the proposed tracking method after replacing the feature extraction network from Swin Transformer to ResNet-50. The second row shows the results of the proposed tracking method after removing the feature fusion flow. The last row shows the results of the proposed tracking method. In summary, the experimental results show that using a three-flow network structure and Swin Transformer as the feature extractor significantly improves the accuracy of the tracking method, with PR / SR scores improved by 3.7% / 3.0% and 1.5% / 1.5% respectively compared to the other two architectures. The experimental results demonstrate the effectiveness of the backbone network architecture of the proposed tracking method.

[0150] Table 2 compares the PR / SR (%) scores of different backbone network architectures on the RGBT234 dataset.

[0151]

[0152] 2) Joint-Complementary Attention: To verify the effectiveness of joint-complementary attention, three variants were obtained by removing the joint and complementary attention modules from this model. The performance of this tracking method was compared with the three variants on the RGBT234 dataset, and the results are shown in Table 3. Tracking methods without two attention modules were used as the baseline. Experimental data show that combining joint or complementary attention modules can improve the tracker's accuracy. Combining the joint and complementary attention modules results in even greater improvements, with PR / SR scores increasing by 1.9% and 1.8%, respectively. This indicates that the proposed joint-complementary attention module makes a significant contribution to improving tracker performance.

[0153] Table 3 compares the PR / SR (%) scores of models using different attention modules. √ indicates that this attention module was used.

[0154]

[0155] 3) Depth-Separable Shared Attention: To evaluate the performance impact of the proposed depth-separable shared attention method, this invention employs... Figure 3 The three structures shown were evaluated for accuracy and speed on the RGBT234 dataset. The results are shown in Table 4. Experimental results demonstrate that the depthwise separable shared attention proposed in this invention improves FPS from 31 to 37 without sacrificing tracker accuracy.

[0156] Table 4 compares PR / SR (%) scores and tracking speeds on the RGBT234 dataset for different types of attention structures.

[0157]

[0158] It should be noted that the image processing method of the present invention can be converted into software program instructions. It can be implemented using a detection system including a processor and memory, or it can be implemented using computer instructions stored in a non-transitory computer-readable storage medium. The integrated unit implemented as a software functional unit described above can be stored in a computer-readable storage medium. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0159] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit it; although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A visible light-thermal infrared target tracking method based on a three-flow feature aggregation twin network, characterized in that, The three-stream feature aggregation twin network includes a backbone network, correlation operations, and a prediction head. The search image and template image are extracted and fused through the backbone network to obtain search features and template features. Both the search image and template image include visible light images and thermal infrared images. Then, the search features and template features are correlated pixel by pixel to generate a similarity map. The similarity map is input into the prediction head to generate tracking results. The backbone network contains two parallel feature extraction streams and one feature fusion stream. The two feature extraction streams are parallel Swin Transformers, and the feature fusion stream is composed of four joint-complementary feature aggregation modules connected in series. Will , , Represented as the first The stage output includes a visible light feature extraction stream, a thermal infrared feature extraction stream, and a feature fusion stream, among which... , No. The network representation of the stage is as follows in, , and These represent the first corresponding to the three streams. The network module of the stage; , and This represents the initial input to the three streams. and Representing the first of the Swing Transformer The staged network module consists of block merging and a Swing Transformer block; block merging implements downsampling and is replaced by block partitioning and linear mapping operations in the first stage; the Swing Transformer block applies local window self-attention in the window multi-head self-attention module and shifted window self-attention in the moving window multi-head self-attention module. , , ;in, and These represent visible light images and thermal infrared images, respectively. Indicates a connection along the channel dimension; In the joint-complementary feature aggregation module, the first The stage feature fusion flow is described as follows: in, , These represent the modal feature fusion process and the stage feature fusion process, respectively; the channel and spatial attention weights are... in This represents element-wise addition. , ; The modal feature fusion process is represented as: in This represents element-wise multiplication. The phase feature fusion process is represented as follows: in , , These represent average pooling, batch normalization, and convolutional layer operations, respectively. and These represent the channel dimension weights corresponding to the fusion features of the current stage and the fusion features of the previous stage, respectively. The concatenated multimodal features are fed into a convolutional module to obtain intermediate shared features F. c The description is as follows: Among them, GAP , , These represent the global average pooling layer, the depthwise separable convolutional layer, and the ReLU activation function, respectively; subsequently, based on the shared feature F... c Three channel attention weights are generated based on depth-separable shared attention. , , : in This represents the Sigmoid activation function. This represents the corresponding 1×1 convolutional layer; in the spatial dimension, the three spatial attention weights are expressed by the following formula: Furthermore, during the stage feature fusion process, depthwise separable shared attention is also used to generate channel-dimensional weights corresponding to different stages. and ,Right now: In this context, the different weights generated by the depthwise separable shared attention layer share the same preceding depthwise separable convolutional layer.

2. The visible light-thermal infrared target tracking method based on a three-stream feature aggregation twin network according to claim 1, characterized in that, The correlation operation employs a pixel-by-pixel correlation method. After obtaining the search features and template features, they are represented as follows: and For pixel-wise related operations, firstly, the template features are... Decomposed into 1×1 convolution kernel Then, for the search features output by the backbone network Perform convolution operations with each convolution kernel separately, and then stack the results of each calculation along the channel dimension to obtain the correlation graph. This process is represented as: ； in, This represents the convolution operation.

3. The visible light-thermal infrared target tracking method based on a three-stream feature aggregation twin network according to claim 2, characterized in that, The prediction head is a center point prediction head, which will correlate with the graph. The data is fed into the three FCN branches in the prediction head to obtain the classification score map, offset map, and size map. The center of the predicted target is determined by the coordinates of the maximum value on the classification score map, and the bounding box is obtained using the corresponding offset map and size map.

4. The visible light-thermal infrared target tracking method based on a three-stream feature aggregation twin network according to claim 3, characterized in that, The loss function consists of three parts: , and ; It is a classification loss, which is based on the predicted classification score map. Gaussian plot generated from the true values Calculate the Focal loss; for each frame, the center coordinates of the bounding box are represented as ( The Gaussian map is generated by the following formula: in It is the standard deviation of object size adaptation; The classification loss is expressed as: and The regression loss is obtained by calculating the generalized IoU loss and mean absolute error loss for the predicted bounding box and the ground truth bounding box, respectively; the total loss function is expressed as: in and It is the regularization loss parameter.

5. The visible light-thermal infrared target tracking method based on a three-stream feature aggregation twin network according to claim 4, characterized in that, The AdamW optimizer was used to optimize the network. The initial learning rate was set to 0.0001, and the learning rate was reduced to 0.2 times every 40 epochs. The network was trained for a total of 120 epochs, and the batch size was set to 8.

6. A visible light-thermal infrared target tracking system based on a three-flow feature-converged twin network, characterized in that, include: At least one processor; and at least one memory communicatively connected to the processor, wherein: The memory stores program instructions that can be executed by the processor, and the processor can execute the visible light-thermal infrared target tracking method based on a three-stream feature aggregation twin network as described in any one of claims 1 to 5 by calling the program instructions.

Images