RGBT tracking method based on diversified semantic aggregation

Abstract

The invention discloses an RGBT tracking method based on diversified semantic aggregation. According to the method, on the basis of an RGBT tracking framework of a single-branch Transform, local semantics and global semantics of 1-12 layers of features are aggregated through a designed diversified semantic aggregation method to be used for target positioning so as to improve tracking accuracy. Specifically, firstly, a shallow random feature is obtained from a plurality of shallow features by adopting an adaptive random selection method, and then the shallow random feature containing local semantics and a plurality of deep features containing global semantics are subjected to semantic fusion in a cascade mode. Wherein semantic fusion takes a semantic convergence template as a medium to learn semantic information from significance information of a previous layer semantic fusion feature, and the semantic convergence template and a current layer feature are fused based on a double-branch cross attention enhancement method to obtain a previous layer semantic fusion feature and a current layer semantic fusion feature. Diversified semantic aggregation results not only provide rich semantics for target positioning, but also can promote tracking accuracy.

Description

Technical Field

[0001] This invention relates to RGBT target tracking methods, and particularly to an RGBT tracking method based on diverse semantic aggregation. Background Technology

[0002] In the field of computer vision, target tracking involves continuously predicting the spatial position and size changes of a target in a sequence of subsequent frames, given its initial position. Compared to traditional single-modal target tracking, RGBT (visible light and thermal infrared) tracking has greater application value due to the nighttime light sensitivity and penetration capability of the thermal infrared mode (TIR). RGBT tracking has broad application prospects in many fields such as video surveillance, robotics, and autonomous driving.

[0003] Vision Transformer (ViT) networks have become a commonly used backbone network in RGB-T tracking algorithms. One approach is to extract features from the two modalities separately using ViT, and then fuse the features during or after feature extraction (e.g., BAT algorithm: Cao, B., Guo, J., Zhu, P., et al. Bi-directional adapter for multimodal tracking. In AAAI, 2024.; TBSI algorithm: Hui, T., Xun, Z., Peng, F., et al. Bridging search region interaction with template for RGB-T tracking. In CVPR, 2023.). Another approach is to uniformly extract and fuse features from both modalities using a single-branch ViT algorithm (e.g., USTrack algorithm: Xia J, Shi DX, Song K, et al. Unified Single-Stage Transformer Network for Efficient RGB-T Tracking. In IJCAI, 2024.). Both approaches share a basic framework comprising three parts: inputs—two modal target templates and a search region image; a single-branch or dual-branch backbone network for feature extraction; and a prediction head for target localization. The backbone network, ViT, typically contains 12 encoder layers. However, existing methods focus on inter-modal information fusion to effectively utilize the complementary information of the two modalities, using the simplified features output from the last layer of the backbone network as the basis for target localization. This approach fails to effectively utilize the tracking-friendly information from all layers of the backbone network, inevitably impacting tracking accuracy.

[0004] This invention employs a single-branch Transformer RGBT tracking framework, comprising three parts: input, backbone network, and prediction head. The input consists of template images of two modalities and a search region image. The backbone network, similar to the USTrack algorithm, uses a single-branch ViT network to achieve unified processing of feature extraction and fusion interaction. The prediction head uses a processing method similar to the TBSI algorithm.

[0005] Unlike TBSI, this invention uses a single-branch ViT network as its backbone, eliminating the need for an additional complex intermodal information fusion network. Furthermore, unlike the USTrack algorithm's backbone network, which uses features from layer 12 as the basis for target localization, this invention uses the diverse semantic fusion results from layers 1-12 as the basis for target localization. This diverse semantic fusion result provides richer and more varied semantics for target localization, thereby effectively improving tracking accuracy. Summary of the Invention

[0006] To address the problems existing in the prior art, the present invention aims to provide an RGBT tracking method based on diversified semantic aggregation. It introduces a diversified semantic aggregation method on the basis of the single-branch Transformer RGBT tracking framework, aggregating local or global semantics from different levels of the backbone network for target localization to improve tracking accuracy.

[0007] To achieve the above objectives, the technical solution of the present invention is as follows:

[0008] An RGBT tracking method based on diversified semantic aggregation is proposed. This method utilizes a single-branch Transformer RGBT tracking framework and employs a diversified semantic aggregation method to combine local and global semantics from layers 1-12 for target localization, thereby improving tracking accuracy. Specifically, an adaptive random selection method is first used to obtain shallow random features from multiple shallow features. Then, the shallow random features containing local semantics are semantically fused with multiple deep features containing global semantics in a cascaded manner. The semantic fusion learns semantic information from the saliency information of the semantic fusion features of the previous layer using a semantic convergence template as a medium. Finally, a two-branch cross-attention enhancement method is used to fuse the semantic convergence template with the current layer features, resulting in a semantic fusion feature of the previous and current layers. The diversified semantic aggregation results not only provide rich semantics for target localization but also improve tracking accuracy.

[0009] Furthermore, the method of the present invention comprises the following ten steps:

[0010] Step 1: Input the first frame of the video;

[0011] A video sequence contains several frames, each frame having two modes: RGB (visible light) and TIR (thermal infrared), and containing at least one target. The target's position in the first frame is known, while the target's position in each of the remaining frames is unknown. The number of video frames is a positive integer greater than 1. The top-left corner of each frame in the video frame sequence is the origin. The width and height are respectively and Input the first frame image and automatically or manually mark the rectangular area of ​​the tracked target. That is, the selected tracking target. Among them, This represents the coordinates of the top-left corner of the rectangular region. Let represent the width and height of the rectangular region, respectively. Indicates the current frame number, the first frame number The target selected in the first frame is also called the tracking result of the current frame. = .

[0012] Step 2: Generate the target template token sequence;

[0013] Zoom in on the rectangular area of ​​the selected target in the first frame. 2 The magnified area corresponds to a change in image size. , as the target template image for both modalities and Where the height and width are equal, that is , The number of channels in an image, symbol Indicates a template.

[0014] right and The image is divided into blocks, each with a pixel resolution of [missing value]. It can form a sequence of template image blocks with two modalities. and ,in This indicates the number of target template image patches for a modality. This indicates the token embedding dimension. and After linear projection and add position embedding and and modal embedding and It can generate two modal target template token sequences. and ,Right now:

[0015] ;

[0016] 。;

[0017] Step 3: Input the next frame and generate the search area token sequence;

[0018] Step 4: Shallow Feature Extraction;

[0019] Step 5: Shallow random feature selection;

[0020] Step 6: Deep Feature Extraction;

[0021] Deep feature extraction calculation Output features of layer encoder The calculation process is as follows:

[0022] ;

[0023] ;

[0024] in , and These represent conventional multi-head self-attention operation, feedforward neural network, and layer normalization operation, respectively. The attention matrix after activation during the operation is denoted as: .when Time Layer output features As the input feature of the current layer, when Use the previous layer's fusion features As input features for the current layer.

[0025] Step 7: Semantic Fusion Module;

[0026] The semantic fusion stage will fuse the features from the previous layer. Compared with the output features of the current layer The fusion process is performed modally, and the fusion result of the two modalities is... and By connecting the links, the final fused features of the current layer can be obtained. The two-modal fusion process can be represented as follows:

[0027] ;

[0028] ;

[0029] in, , These represent the semantic fusion operation and the saliency semantic extraction operation, respectively. Since the two modality fusion methods are the same, the following uses the RGB modality as an example to illustrate the implementation process of the saliency semantic extraction and semantic fusion operations; the TIR modality is similar.

[0030] Step 7.1 Saliency semantic extraction operation;

[0031] Saliency semantic extraction operation RGB modal data with fused features from the previous layer can be used. Extract its saliency semantic information , can be represented as:

[0032] ;

[0033] From the current layer attention matrix Obtain the correlation submatrix between the RGB modal template and the search region:

[0034] ;

[0035] Mean pooling operation right Taking the average of each column yields a relevance vector, where each mean represents the relevance between the corresponding search token and the target. Binarization is then applied. The top elements with higher correlation in the correlation vector Set the ratio to 1 and the rest to 0 to generate an RGB modal search token mask vector. This process can be represented as:

[0036] ;

[0037] in This indicates the transpose operation. Search token area With mask vector Dot product yields the semantic saliency extraction result of the search region, which can then be further compared with... Template token area The saliency semantic information can be obtained by merging. This process can be represented as:

[0038] ;

[0039] Step 7.2 Semantic fusion operation;

[0040] The implementation process of semantic fusion operation includes two steps: learning the semantic information of the previous layer and fusing it with the semantic information of the current layer.

[0041] (1) Learning semantic information from the previous layer;

[0042] Learning semantic information from the previous layer involves using template tokens from the current layer as a medium to learn useful semantics from the salient semantics of the previous layer, forming a semantic convergence template. Current layer features. This includes template features and search features for both RGB and TIR modalities, where the features for the RGB modality are: The saliency semantics of the upper-level RGB modality is: .

[0043] First, the RGB modal template features of the current layer are... Compared with the previous layer RGB modal saliency semantic template features Perform a connection and linear transformation to form a aggregation template. :

[0044] ;

[0045] in and These represent the join operation and the linear transformation operation, respectively.

[0046] Next, aggregate the template. As query information, it is related to the semantic saliency of the RGB modality in the previous layer. Perform multi-head cross attention This process can be expressed as a formula:

[0047] ;

[0048] in, For the Softmax function, , and For the learnable projection parameters corresponding to the query key and value, The dimension is the key. Multi-head cross-attention results and aggregation template. Residual integration and layer normalization are performed. Enhanced aggregation template. This can be expressed as a formula:

[0049] ;

[0050] in, Presentation layer normalization operation. Enhanced aggregation template. After feedforward neural network Subsequently, it is integrated with the enhanced aggregated template residual and subjected to layer normalization. This process enables the convergence of semantic information from the previous layer into the template features of the current frame to form a semantic converged template. :

[0051] ;

[0052] (2) Integration with semantic information of the current layer;

[0053] The purpose of fusing semantic information with the current layer is to create a semantic convergence template that incorporates semantic information from the previous layer. By fully interacting with the feature information of the current layer, the semantic fusion result of the previous layer and the current layer is obtained. This fusion process is divided into two branches, one of which obtains the template features for RGB modality semantic fusion. Another branch yields search features based on RGB modal semantic fusion. .

[0054] For template features of semantic fusion First, the semantic convergence template Compared with the current layer RGB modal features Enhanced interaction. As a query As keys and values, their multi-head cross-attention can be expressed as a formula:

[0055] ;

[0056] in, , and For the learnable projection parameters corresponding to the query key and value, The dimension is the key. Multi-head cross-attention results and semantic convergence template. Residual integration and layer normalization are performed. Enhanced semantic convergence template. This can be expressed as a formula:

[0057] ;

[0058] Enhanced semantic convergence template After feedforward neural network Afterwards, with Residual integration and layer normalization are performed. Template features for RGB modal semantic fusion. It combines semantic information from the previous layer and semantic information from the current layer, and can be represented by the formula:

[0059] ;

[0060] Search features for semantic fusion This is the reverse of the template feature acquisition process in semantic fusion. First, the RGB modal features of the current layer are... As a query, it integrates semantic convergence templates that incorporate semantic information from the previous layer. As keys and values, multi-head cross-attention can be expressed as the formula:

[0061] ;

[0062] in, , and For the learnable projection parameters corresponding to the query key and value, The dimension is the key. Multi-head cross-attention results are compared with the current layer features. Residual integration followed by layer normalization yields enhanced search features. It can be expressed as a formula:

[0063] ;

[0064] Enhanced search features After feedforward neural network Afterwards, with Residual integration and layer normalization can be represented as intermediate variables. :

[0065] ;

[0066] in, It consists of two parts: template token and search token. The search token part Search features as semantic fusion It combines semantic information from the previous layer and semantic information from the current layer, and can be expressed as a formula:

[0067] ;

[0068] The RGB modality semantic fusion features can be obtained by merging the semantically fused template features with the search features. It can be expressed as the following formula:

[0069] ;

[0070] Step 8: Determine if it is the last encoder layer.

[0071] If the current layer number Proceed to step six.

[0072] Step Nine: Determine the Tracking Results

[0073] The semantic fusion features of layer 12 The search features of the two modalities are added together. This serves as the basis for target localization. The localization process is a standard procedure in the target localization stage of single-stream tracking, primarily involving reinterpreting the search region feature sequence into a two-dimensional feature map, passing it through a multi-layered fully convolutional network with a Conv-BN-ReLU structure, and determining the target location based on information such as the classification score map and local offsets. The total training loss of the tracking model is:

[0074] ;

[0075] Among them, classification loss Weighed Focal Loss was used for classification, and L1 loss was used for regression. and Generalized IoU loss Perform bounding box regression. The target localization result is used as the tracking result for the current frame. .

[0076] Step 10: Determine the last frame;

[0077] If the current frame is the last frame, the tracking ends; otherwise, proceed to step three.

[0078] Furthermore, step three specifically involves:

[0079] Input a new video frame as the current frame, and increment the current frame number by 1. The tracking results from the previous frame. Enlarge the rectangular region b 2 The magnified area corresponds to a change in image size. The search region image serves as both modalities. and ,in and These represent the height and width of the search region image, respectively. , The number of channels in an image, symbol Indicates the search area.

[0080] right and The image is divided into blocks, each with a pixel resolution of [missing value]. This can form a sequence of search region image patches in two modalities. and ,in This indicates the number of image patches in a modal search region. This indicates the token embedding dimension, which is the same as the template image block embedding dimension. and After linear projection and add position embedding and and modal embedding and It can generate two modal search region token sequences. and ,Right now:

[0081] ;

[0082] ;

[0083] The target template token sequence of the two modalities and and the search region token sequence of the two modalities. and Connect into a token sequence and will The features are fed into the backbone network as input features for the first layer encoder. Represented as:

[0084] .

[0085] Furthermore, step four specifically involves:

[0086] This invention uses ViT as the backbone network, comprising 12 encoder layers. Each encoder layer mainly includes two modules: a multi-head self-attention network and a feedforward network. In the shallow feature extraction stage, only the front-end network is computed. Layer features ,in Indicates the floor number.

[0087] Each encoder layer is based on input features Calculate output features The calculation process is as follows:

[0088] ;

[0089] ;

[0090] in , and These represent conventional multi-head self-attention operation, feedforward neural network, and layer normalization operation, respectively. It also serves as the input feature for the next layer of encoder.

[0091] Furthermore, step five specifically involves:

[0092] The shallow features of the ViT network are rich in target details or texture information, which can provide valuable local semantic information for the tracking task. The number of tokens in the first m shallow features is several times that of tokens in a single layer. To effectively utilize the shallow local semantics and minimize interference, a shallow random feature extraction operation is employed. From the past Layer features Obtain shallow random features It can also be called in the past. The first layer features obtained Layer fusion features can be represented as:

[0093] .

[0094] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0095] (1) Tracking performance across multiple tracking datasets;

[0096] On the LasHeR dataset, the method of this invention outperforms BAT by 0.55% and 0.43% in PR and SR, respectively. On the GTOT dataset, the method of this invention achieves PR and SR of 93.5 and 78.3, respectively, significantly outperforming other methods and exceeding VLCTrack by 3.4% and 1.5%, respectively. On the RGB-T210 dataset, the method of this invention achieves PR and SR of 84.4 and 60.7, respectively, showing a significant improvement over other methods and exceeding MKFTracker by 3.7% and 2.5%, respectively. On the RGB-T234 dataset, the method of this invention achieves MPR and MSR of 86.6 and 63.6, respectively, also outperforming other methods and exceeding CMDTrack by 0.7% and 1.8%, respectively. This indicates that the method of this invention has better tracking performance and generalization ability on the large-scale dataset LasHeR, the small-object dataset GTOT, and the small-scale datasets RGBT234 and RGBT210.

[0097] (2) Tracking performance under various complex environments;

[0098] In the LasHeR dataset, the accuracy and success rate of the model across multiple attribute dimensions were systematically evaluated in nineteen typical challenge scenarios. Experimental results demonstrate significant advantages; compared to current state-of-the-art methods such as TBSI, SDSTrack, MANet++, CAT, and mfDiMP, the proposed solution outperforms these methods in almost all attribute evaluations. Particularly in low-light (LI) environments, it effectively overcomes the feature loss inherent in traditional methods due to insufficient utilization of TIR modalities. In motion-blurred (MB) scenarios, it successfully addresses the target localization deviation problem in dynamically blurred images. Furthermore, in the target deformation (DEF) challenge, it exhibits excellent adaptability to changes in the shape of non-rigid targets. These performance improvements are attributed to the innovative construction of a diverse semantic aggregation architecture, achieving deep fusion of infrared and visible light information. While retaining the advantages of each modality, it significantly improves information complementarity efficiency. Simultaneously, the invention dynamically fuses detailed texture information from shallow networks with deep semantic features, constructing a multi-layered feature structure. This design effectively alleviates feature degradation in extreme scenarios, providing a reliable guarantee for high-precision visual target tracking. Attached Figure Description

[0099] Figure 1 Schematic diagram of diverse semantic aggregation

[0100] Figure 2 : Schematic diagram of the tracking method of the present invention;

[0101] Figure 3 Radar charts of accuracy and success rate for 19 challenge scenarios on the LasHer dataset;

[0102] Figure 4 : A schematic diagram of the network model of this invention;

[0103] Figure 5 : A schematic diagram of the semantic fusion operation of this invention;

[0104] Figure 6 The selection of the tracking region in the first frame of the ab_bikeoccluded sequence;

[0105] Figure 7 Partial tracking results; Detailed Implementation

[0106] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0107] like Figure 1-7 As shown,

[0108] According to the technical solution of this invention, the video sequence ab_bikeoccluded in the LasHeR dataset is selected for tracking, which has two challenging attributes: target disappearance over a long period of time and scale change.

[0109] Step 1: Input the first frame of the video;

[0110] The ab_bikeoccluded video sequence contains 294 frames; the top left corner of each frame is the origin. The width and height are respectively and Input the first frame image and automatically or manually mark the rectangular region (550, 341, 50, 118) of the tracked target; this is the selected tracked target. Here, (550, 341) represents the coordinates of the top-left corner of the rectangular region, and 50, 118 represent the width and height of the rectangular region, respectively. The selected target in the first frame is also called the tracking result of the current frame. = (550, 341, 50, 118), where the subscript 1 indicates the current frame number.

[0111] Step 2: Generate the target template token sequence;

[0112] The rectangular regions of the selected targets in the first frame of both modalities are enlarged respectively, and the image size corresponding to the enlarged regions is distorted to 128×128, which is used as the target template image. and Where the height and width are equal, that is 3 indicates the number of channels in the image, symbol Indicates a template.

[0113] right and The image is divided into blocks, each with a pixel resolution of [missing value]. It can form a sequence of image patches. and ,in Indicates the number of target template image blocks. This indicates the token embedding dimension. and After linear projection And add position embeddings respectively , and add position embedding and and modal embedding and It can generate two modal target template token sequences. and ,Right now and .

[0114] Step 3: Input the next frame and generate the search area image;

[0115] Input the second video frame as the current frame image. Enlarge the rectangular region (550, 341, 50, 118) of the tracking result of the previous frame (the first frame). The image size of the enlarged region is distorted to 256×256, and this region is used as the search area image. , The height and width 3 indicates the number of channels in the image, symbol Indicates the search area.

[0116] right and The image is divided into blocks, each with a pixel resolution of [missing value]. This can form a sequence of search region image patches in two modalities. and ,in This indicates the number of image patches in a modal search region. This indicates the token embedding dimension, which is the same as the template image block embedding dimension. and After linear projection and add position embedding and and modal embedding and It can generate two modal search region token sequences. and ,Right now and .

[0117] The target template token sequence of the two modalities and and the search region token sequence of the two modalities. and Connect into a token sequence and will The features are fed into the backbone network as input features for the first layer encoder. Represented as:

[0118] .

[0119] Step 4: Shallow Feature Extraction;

[0120] This invention uses ViT as the backbone network, comprising 12 encoder layers. Each encoder layer mainly includes two modules: a multi-head self-attention network and a feedforward network. In the shallow feature extraction stage, only the features from the first 9 layers are calculated. ,in Indicates the floor number.

[0121] Each encoder layer is based on input features Calculate output features The calculation process is as follows:

[0122] ;

[0123] ;

[0124] in , and These represent conventional multi-head self-attention operation, feedforward neural network, and layer normalization operation, respectively. It also serves as the input feature for the next layer of encoder.

[0125] Step 5: Shallow random feature selection;

[0126] The shallow features of the ViT network are rich in target details or texture information, which can provide valuable local semantic information for tracking tasks. The number of tokens in the first nine shallow features is several times that of tokens in a single layer. To effectively utilize shallow local semantics and minimize interference, a shallow random feature extraction operation is employed. From the first 9 layers of features Obtain shallow random features This can also be called the 9th layer fusion feature obtained from the first 9 layers of features, and can be represented as:

[0127] ;

[0128] Step 6: Deep Feature Extraction;

[0129] Deep feature extraction calculation Output features of layer encoder The calculation process is as follows:

[0130] ;

[0131] ;

[0132] in , and These represent conventional multi-head self-attention operation, feedforward neural network, and layer normalization operation, respectively. The attention matrix after activation during the operation is denoted as: .when Time Layer output features As the input feature of the current layer, when Use the previous layer's fusion features As input features for the current layer.

[0133] Step 7: Semantic Fusion Module;

[0134] The semantic fusion stage will fuse the features from the previous layer. Compared with the output features of the current layer The fusion process is performed modally, and the fusion result of the two modalities is... and By connecting the links, the final fused features of the current layer can be obtained. The two-modal fusion process can be represented as follows:

[0135] ;

[0136] ;

[0137] in, , These represent the semantic fusion operation and the saliency semantic extraction operation, respectively. Since the two modality fusion methods are the same, the following uses the RGB modality as an example to illustrate the implementation process of the saliency semantic extraction and semantic fusion operations; the TIR modality is similar.

[0138] 7.1 Saliency semantic extraction operation;

[0139] Saliency semantic extraction operation RGB modal data with fused features from the previous layer can be used. Extract its saliency semantic information , can be represented as:

[0140] ;

[0141] From the current layer attention matrix Obtain the correlation submatrix between the RGB modal template and the search region:

[0142] ;

[0143] Mean pooling operation right Taking the average of each column yields a relevance vector, where each mean represents the relevance between the corresponding search token and the target. Binarization is then applied. The top elements with higher correlation in the correlation vector Set the ratio to 1 and the rest to 0 to generate an RGB modal search token mask vector. This process can be represented as:

[0144] ;

[0145] in This indicates the transpose operation. Search token area With mask vector Dot product yields the semantic saliency extraction result of the search region, which can then be further compared with... Template token area The saliency semantic information can be obtained by merging. This process can be represented as:

[0146] ;

[0147] 7.2 Semantic fusion operation;

[0148] The implementation process of semantic fusion operation includes two steps: learning the semantic information of the previous layer and fusing it with the semantic information of the current layer.

[0149] (1) Learning semantic information from the previous layer;

[0150] Learning semantic information from the previous layer involves using template tokens from the current layer as a medium to learn useful semantics from the salient semantics of the previous layer, forming a semantic convergence template. Current layer features. This includes template features and search features for both RGB and TIR modalities, where the features for the RGB modality are: The semantic significance of the RGB modality at the next higher level is:

[0151] ;

[0152] First, the RGB modal template features of the current layer are... Compared with the previous layer RGB modal saliency semantic template features Perform a connection and linear transformation to form a aggregation template. :

[0153] ;

[0154] in and These represent the join operation and the linear transformation operation, respectively.

[0155] Next, aggregate the template. As query information, it is related to the semantic saliency of the RGB modality in the previous layer. Perform multi-head cross attention This process can be expressed as a formula:

[0156] ;

[0157] in, For the Softmax function, , and For the learnable projection parameters corresponding to the query key and value, The dimension is the key. Multi-head cross-attention results and aggregation template. Residual integration and layer normalization are performed. Enhanced aggregation template. This can be expressed as a formula:

[0158] ;

[0159] in, Presentation layer normalization operation. Enhanced aggregation template. After feedforward neural network Subsequently, it is integrated with the enhanced aggregated template residual and subjected to layer normalization. This process enables the convergence of semantic information from the previous layer into the template features of the current frame to form a semantic converged template. :

[0160] ;

[0161] (2) Integration with semantic information of the current layer;

[0162] The purpose of fusing semantic information with the current layer is to create a semantic convergence template that incorporates semantic information from the previous layer. By fully interacting with the feature information of the current layer, the semantic fusion result of the previous layer and the current layer is obtained. This fusion process is divided into two branches, one of which obtains the template features for RGB modality semantic fusion. Another branch yields search features based on RGB modal semantic fusion. .

[0163] For template features of semantic fusion First, the semantic convergence template Compared with the current layer RGB modal features Enhanced interaction. As a query As keys and values, their multi-head cross-attention can be expressed as a formula:

[0164] ;

[0165] in, , and For the learnable projection parameters corresponding to the query key and value, The dimension is the key. Multi-head cross-attention results and semantic convergence template. Residual integration and layer normalization are performed. Enhanced semantic convergence template. This can be expressed as a formula:

[0166] ;

[0167] Enhanced semantic convergence template After feedforward neural network Afterwards, with Residual integration and layer normalization are performed. Template features for RGB modal semantic fusion. It combines semantic information from the previous layer and semantic information from the current layer, and can be represented by the formula:

[0168] ;

[0169] Search features for semantic fusion This is the reverse of the template feature acquisition process in semantic fusion. First, the RGB modal features of the current layer are... As a query, it integrates semantic convergence templates that incorporate semantic information from the previous layer. As keys and values, multi-head cross-attention can be expressed as the formula:

[0170] ;

[0171] in, , and For the learnable projection parameters corresponding to the query key and value, The dimension is the key. Multi-head cross-attention results are compared with the current layer features. Residual integration followed by layer normalization yields enhanced search features. It can be expressed as a formula:

[0172] ;

[0173] Enhanced search features After feedforward neural network Afterwards, with Residual integration and layer normalization can be represented as intermediate variables. :

[0174] ;

[0175] in, It consists of two parts: template token and search token. The search token part Search features as semantic fusion It combines semantic information from the previous layer and semantic information from the current layer, and can be expressed as a formula:

[0176] ;

[0177] The RGB modality semantic fusion features can be obtained by merging the semantically fused template features with the search features. It can be expressed as the following formula:

[0178] ;

[0179] Step 8: Determine if it is the last encoder layer;

[0180] If the current layer number Proceed to step six.

[0181] Step Nine: Determine the tracking results;

[0182] The semantic fusion features of layer 12 The search features of the two modalities are added together. This serves as the basis for target localization. The localization process is a standard procedure in the target localization stage of single-stream tracking, primarily involving reinterpreting the search region feature sequence into a two-dimensional feature map, passing it through a multi-layered fully convolutional network with a Conv-BN-ReLU structure, and determining the target location based on information such as the classification score map and local offsets. The total training loss of the tracking model is:

[0183] ;

[0184] Among them, classification loss Weighed Focal Loss was used for classification, and L1 loss was used for regression. and Generalized IoU loss Perform bounding box regression. The target localization result is used as the tracking result for the current second frame. .

[0185] Step 10: Determine the last frame;

[0186] The current frame is the second frame, so proceed to step three, and continue tracking until the last frame.

[0187] Finally, the examples of this invention were trained and tracked on an NVIDIA RTX 4090 GPU, and some tracking results are shown below. Figure 7 As shown in the figure. In the figure, the yellow label in the upper left corner is the video frame number, the green box indicates the actual position of the video marker, the red box indicates the tracking result of this invention, and other colored boxes indicate other tracking models. See the figure caption for details.

[0188] The present invention has the following advantages: improved tracking performance across multiple tracking datasets;

[0189] On the LasHeR dataset, the method of this invention outperforms BAT by 0.55% and 0.43% in PR and SR, respectively. On the GTOT dataset, the method of this invention achieves PR and SR of 93.5 and 78.3, respectively, significantly outperforming other methods and exceeding VLCTrack by 3.4% and 1.5%, respectively. On the RGB-T210 dataset, the method of this invention achieves PR and SR of 84.4 and 60.7, respectively, showing a significant improvement over other methods and exceeding MKFTracker by 3.7% and 2.5%, respectively. On the RGB-T234 dataset, the method of this invention achieves MPR and MSR of 86.6 and 63.6, respectively, also outperforming other methods and exceeding CMDTrack by 0.7% and 1.8%, respectively. This indicates that the method of this invention has better tracking performance and generalization ability on the large-scale dataset LasHeR, the small-object dataset GTOT, and the small-scale datasets RGBT234 and RGBT210.

[0190]

[0191] Table 1 shows the comparison results between the current state-of-the-art trackers and the LasHer and GTOT datasets.

[0192]

[0193] Table 2 shows the comparison results with the current state-of-the-art trackers on the RGBT210 and RGBT234 datasets.

[0194] Tracking performance in various complex environments;

[0195] In the LasHeR dataset, the accuracy and success rate of the model across multiple attribute dimensions were systematically evaluated in nineteen typical challenge scenarios. Experimental results demonstrate significant advantages; compared to current state-of-the-art methods such as TBSI, SDSTrack, MANet++, CAT, and mfDiMP, the proposed solution outperforms these methods in almost all attribute evaluations. Particularly in low-light (LI) environments, it effectively overcomes the feature loss inherent in traditional methods due to insufficient utilization of TIR modalities. In motion-blurred (MB) scenarios, it successfully addresses the target localization deviation problem in dynamically blurred images. Furthermore, in the target deformation (DEF) challenge, it exhibits excellent adaptability to changes in the shape of non-rigid targets. These performance improvements are attributed to the innovative construction of a diverse semantic aggregation architecture, achieving deep fusion of infrared and visible light information. While retaining the advantages of each modality, it significantly improves information complementarity efficiency. Simultaneously, the invention dynamically fuses detailed texture information from shallow networks with deep semantic features, constructing a multi-layered feature structure. This design effectively alleviates feature degradation in extreme scenarios, providing a reliable guarantee for high-precision visual target tracking.

[0196] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions conceived without inventive effort should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.

Claims

1. An RGBT tracking method based on diversified semantic aggregation, characterized in that: This method is based on the RGBT tracking framework of a single-branch Transformer. It uses a variety of designed semantic aggregation methods to aggregate the local and global semantics of features from layers 1-12 for target localization to improve tracking accuracy. Specifically, firstly, an adaptive random selection method is used to obtain shallow random features from multiple shallow features. Then, the shallow random features containing local semantics are semantically fused with multiple deep features containing global semantics in a cascade manner. The semantic fusion learns semantic information from the saliency information of the semantic fusion features of the previous layer by using a semantic convergence template as a medium. The semantic convergence template is then fused with the current layer features based on a two-branch cross-attention enhancement method to obtain the semantic fusion features of the previous and current layers. Diverse semantic aggregation results not only provide rich semantics for target localization but also improve tracking accuracy.

2. The method according to claim 1, characterized in that: This method consists of the following ten steps: Step 1: Input the first frame of the video; A video sequence contains several frames of images. Each frame has two modes: RGB (visible light) and TIR (thermal infrared). Each frame contains at least one target. The target position is known in the first frame, while the target position is unknown in the remaining frames. The number of video frames is a positive integer greater than 1; the top-left corner of each frame in the video frame sequence is the origin. The width and height are respectively and Input the first frame image and automatically or manually mark the rectangular area of ​​the tracked target. That is, the selected tracking target; among which, This represents the coordinates of the top-left corner of the rectangular region. Let the width and height of the rectangular region be respectively; Indicates the current frame number, the first frame number The target selected in the first frame is also called the tracking result of the current frame. = ; Step 2: Generate the target template token sequence; Zoom in on the rectangular area of ​​the selected target in the first frame. 2 The magnified area corresponds to a change in image size. , as the target template image for both modalities and Where the height and width are equal, that is , The number of channels in an image, symbol Indicates a template; right and The image is divided into blocks, each with a pixel resolution of [missing value]. It can form a sequence of template image blocks with two modalities. and ,in This indicates the number of target template image patches for a modality. Indicates the token embedding dimension; and After linear projection and add position embedding and and modal embedding and It can generate two modal target template token sequences. and ,Right now: ； ；； Step 3: Input the next frame and generate the search area token sequence; Step 4: Shallow Feature Extraction; Step 5: Shallow random feature selection; Step 6: Deep Feature Extraction; Deep feature extraction calculation Output features of layer encoder The calculation process is as follows: ； ； in , and These represent conventional multi-head self-attention operations, feedforward neural network operations, and layer normalization operations, respectively. The attention matrix after activation during the operation is denoted as: ;when Time Layer output features As the input feature of the current layer, when Use the previous layer's fusion features As input features for the current layer; Step 7: Semantic Fusion Module; The semantic fusion stage will fuse the features from the previous layer. Compared with the output features of the current layer The fusion process is performed modally, and the fusion result of the two modalities is... and By connecting the links, the final fused features of the current layer can be obtained. The two modality fusion processes can be represented as follows: ； ； in, , These represent the semantic fusion operation and the saliency semantic extraction operation, respectively. Since the two modality fusion methods are the same, the following uses the RGB modality as an example to illustrate the implementation process of the saliency semantic extraction operation and the semantic fusion operation. The same applies to the TIR modality. Step 7.1 Saliency semantic extraction operation; Saliency semantic extraction operation RGB modal data with fused features from the previous layer can be used. Extract its saliency semantic information , can be represented as: ； From the current layer attention matrix Obtain the correlation submatrix between the RGB modal template and the search region: ； Mean pooling operation right Taking the mean of each column yields a relevance vector, where each mean represents the relevance between the corresponding search token and the target; binarization is then applied. The top elements with higher correlation in the correlation vector Set the ratio to 1 and the rest to 0 to generate an RGB modal search token mask vector. This process can be represented as: ； in Indicates the transpose operation; Search token area With mask vector Dot product yields the semantic saliency extraction result of the search region, which can then be further compared with... Template token area The saliency semantic information can be obtained by merging. This process can be represented as: ； Step 7.2 Semantic Fusion Operation The implementation process of semantic fusion operation includes two steps: learning the semantic information of the previous layer and fusing it with the semantic information of the current layer. (1) Learning semantic information from the previous layer; Learning semantic information from the previous layer will use the template tokens of the current layer as a medium to learn useful semantics from the salient semantics of the previous layer, forming a semantic convergence template; current layer features This includes template features and search features for both RGB and TIR modalities, where the features for the RGB modality are: The saliency semantics of the upper-level RGB modality are: ; First, the RGB modal template features of the current layer are... Compared with the previous layer RGB modal saliency semantic template features Perform a connection and linear transformation to form a aggregation template. : ； in and These represent the join operation and the linear transformation operation, respectively. Next, aggregate the template. As query information, it is related to the semantic saliency of the RGB modality in the previous layer. Perform multi-head cross attention This process can be expressed as a formula: ； in, For the Softmax function, , and For the learnable projection parameters corresponding to the query key and value, The dimension of the key; multi-head cross-attention results and aggregation template Residual integration and layer normalization; enhanced aggregation template This can be expressed as a formula: ； in, Presentation layer normalization operation; enhanced aggregation template After feedforward neural network Subsequently, it is integrated with the enhanced aggregated template residual and subjected to layer normalization; this process enables the convergence of semantic information from the previous layer into the template features of the current frame to form a semantic converged template. : ； (2) Fusion with semantic information of the current layer The purpose of fusing semantic information with the current layer is to create a semantic convergence template that incorporates semantic information from the previous layer. By fully interacting with the feature information of the current layer, the semantic fusion result of the previous layer and the current layer is obtained. This fusion process is divided into two branches, one of which obtains the template features for RGB modality semantic fusion. Another branch yields search features based on RGB modal semantic fusion. ; For template features of semantic fusion First, the semantic convergence template Compared with the current layer RGB modal features Enhanced interaction; As a query As keys and values, their multi-head cross-attention can be expressed as a formula: ； in, , and For the learnable projection parameters corresponding to the query key and value, The dimension of the key; multi-head cross-attention results and semantic convergence template Residual integration and layer normalization; enhanced semantic convergence template This can be expressed as a formula: ； Enhanced semantic convergence template After feedforward neural network Afterwards, with Residual integration and layer normalization; template features for RGB modal semantic fusion It combines semantic information from the previous layer and semantic information from the current layer, and can be represented by the formula: ； Search features for semantic fusion In contrast to the template feature acquisition process of semantic fusion, firstly, the RGB modal features of the current layer are... As a query, it integrates semantic convergence templates that incorporate semantic information from the previous layer. As keys and values, multi-head cross-attention can be expressed as the formula: ； in, , and For the learnable projection parameters corresponding to the query key and value, The dimension of the key; multi-head cross-attention results and current layer features. Residual integration followed by layer normalization yields enhanced search features. It can be expressed as a formula: ； Enhanced search features After feedforward neural network Afterwards, with Residual integration and layer normalization can be represented as intermediate variables. : ； in, It consists of two parts: template token and search token. ; Search token part Search features as semantic fusion It combines semantic information from the previous layer and semantic information from the current layer, and can be expressed as a formula: ； The RGB modality semantic fusion features can be obtained by merging the semantically fused template features with the search features. It can be expressed as the following formula: ； Step 8: Determine if it is the last encoder layer; If the current layer number Proceed to step six; Step Nine: Determine the tracking results; The semantic fusion features of layer 12 The search features of the two modalities are added together. This serves as the basis for target localization; the localization process is a standard method for target localization in single-stream tracking, mainly including reinterpreting the feature sequence of the search region into a two-dimensional feature map, passing it through a multi-layered fully convolutional network with a Conv-BN-ReLU structure, and determining the target position based on information such as the classification score map and local offset; the total training loss of the tracking model is: ； Among them, classification loss Weighed Focal Loss was used for classification, and L1 loss was used for regression. and Generalized IoU loss Perform bounding box regression; the target localization result is used as the tracking result for the current frame. ; Step 10: Determine the last frame; If the current frame is the last frame, the tracking ends; otherwise, proceed to step three.

3. The method according to claim 2, characterized in that: Step three specifically involves: Input a new video frame as the current frame, and increment the current frame number by 1. ; Tracking results from the previous frame Enlarge the rectangular region b 2 The magnified area corresponds to a change in image size. The search region image serves as both modalities. and ,in and These represent the height and width of the search region image, respectively. , The number of channels in an image, symbol Indicates the search area; right and The image is divided into blocks, each with a pixel resolution of [missing value]. This can form a sequence of search region image patches in two modalities. and ,in This indicates the number of image patches in a modal search region. This indicates the token embedding dimension, which is the same as the template image patch embedding dimension. and After linear projection and add position embedding and and modal embedding and It can generate two modal search region token sequences. and ,Right now: ； ； The target template token sequence of the two modalities and and the search region token sequence of the two modalities. and Connect into a token sequence and will The features are fed into the backbone network as input features for the first layer encoder. Represented as: 。 4. The method according to claim 3, characterized in that: Step four is as follows: This invention uses ViT as the backbone network, which contains 12 encoder layers. Each encoder layer mainly consists of two modules: a multi-head self-attention network and a feedforward network. In the shallow feature extraction stage, only the front-end network is computed. Layer features ,in Indicates the floor number; Each encoder layer is based on input features Calculate output features The calculation process is as follows: ； ； in , and These represent conventional multi-head self-attention operation, feedforward neural network, and layer normalization operation, respectively. It also serves as the input feature for the next layer of encoder.

5. The method according to claim 4, characterized in that: Step five is as follows: The shallow features of the ViT network are rich in target details or texture information, which can provide valuable local semantic information for the tracking task. The number of tokens in the first m shallow features is several times that of tokens in a single layer. To effectively utilize shallow local semantics and minimize interference, a shallow random feature extraction operation is employed. From the past Layer features Obtain shallow random features It can also be called in the past. The first layer features obtained Layer fusion features can be represented as: 。

Images