An RGB-T target tracking method and device based on a Transformer architecture

Abstract

The present application belongs to the technical field of RGB-T target tracking, and particularly relates to a kind of RGB-T target tracking method and device based on Transformer architecture. Visible light image and infrared image sequence to be tracked are acquired, and corresponding template frame and search frame are extracted;The extraction result is input into the trained target tracking model to obtain the position of the target in each search frame;Wherein, the target tracking model includes an improved double-branch Vision Transformer encoder with shared weights, and a prediction head;The improvement includes inserting a bidirectional frequency domain guiding mechanism in the middle deep and deep layers of the encoder, which is used to: based on the input two modal features, extract the high-frequency component in the RGB modal feature and the low-frequency component in the TIR modal feature, respectively, and then based on the high-frequency component, calibrate the TIR feature in space, and based on the low-frequency component, anchor the RGB high-frequency details through channel modulation. The present application significantly improves the tracking robustness and positioning accuracy in complex scenes such as severe light changes, target deformation and background clutter.

Description

Technical Field

[0001] This invention belongs to the field of RGB-T target tracking technology, specifically relating to an RGB-T target tracking method and apparatus based on the Transformer architecture. Background Technology

[0002] RGB-T target tracking aims to fuse the complementary characteristics of visible light and thermal infrared images to achieve all-weather, highly robust visual target localization, and is widely used in scenarios such as autonomous driving, security monitoring, drone inspection, intelligent transportation, and border control. However, visible light and thermal infrared sensors have inherent differences in their imaging mechanisms, and their information distribution and modal reliability for the same scene differ significantly, making it difficult for existing methods to fully explore the potential of cross-modal collaboration. Especially in complex scenarios with drastic changes in illumination, target occlusion, or cluttered backgrounds, the degradation of single-modal features and the misalignment of cross-modal features become prominent, seriously affecting the continuity and accuracy of tracking.

[0003] Chinese invention patent application CN121482105A, published on February 6, 2026, discloses an RGB-T target tracking method based on spatiotemporal state evolution. This method is a typical example of RGB-T target tracking, where the RGB and TIR branches share ViT encoder weights and employ an iterative processing framework. In each iteration, RGB and TIR modal features are extracted separately, and then cross-modal temporal context modeling is performed via a modality-aware temporal Mamba module. This module achieves long-term, target-aware representation learning through a cross-modal coupled state transition mechanism and a cue-guided strategy. Finally, a cross-modal Mamba aggregation module fuses intra-modal and inter-modal features in the spatial dimension, and the target position is output via a prediction head. This patent achieves persistent cross-modal state evolution with linear complexity, effectively fusing spatiotemporal features of visible light and infrared light.

[0004] Existing RGB-T tracking methods mainly suffer from improper frequency-spatial domain coupling. Most methods directly perform cross-modal feature fusion in the spatial domain, neglecting the pre-filtering and preprocessing of frequency domain information. This leads to invalid features such as illumination interference and thermal crossover in the spatial domain being mixed with the core target features. This lack of cross-domain processing order makes it difficult for the model to distinguish between artifacts caused by environmental interference and apparent changes in the target itself. Under strong backlight or low illumination conditions, it easily causes feature distortion and tracking drift, severely restricting the stable expression and effective fusion of bimodal features in extreme scenarios.

[0005] Existing methods suffer from coarse-grained processing in modal representation enhancement. A few methods attempt to decompose and enhance modal features in the frequency domain, but these largely rely on fixed frequency band divisions and static component weighting strategies, failing to dynamically capture the structural changes in the frequency domain distribution of RGB (visible light) and TIR (thermal infrared) modes under different imaging conditions. When the target undergoes non-rigid deformation, local occlusion, or scale changes, the high- and low-frequency component distributions of the two modes shift significantly. Static decomposition strategies lead to cross-modal feature misalignment, disrupting a consistent representation of the target's appearance and affecting tracking accuracy.

[0006] Existing RGB-T tracking methods generally lack spatial inductive bias in feature modeling. Mainstream global attention mechanisms perform indiscriminate correlation calculations on all spatial locations within the feature map, ignoring the inherent spatial local consistency prior of the visual target. This fully connected modeling approach not only introduces a large amount of background noise and invalid long-range associations unrelated to the target but also disrupts the local structural integrity of the target. In challenging scenarios such as dense interference, small targets, or rapid movement, the lack of spatial constraints in attention diffusion is particularly severe, leading to multi-peak and diffuse phenomena in the response map, significantly reducing the tracking accuracy and robustness. Therefore, there is an urgent need to design a fusion architecture that can adaptively enhance modal complementary features and suppress redundant interference to achieve robust target tracking in complex scenarios. Summary of the Invention

[0007] The purpose of this invention is to provide an RGB-T target tracking method and apparatus based on the Transformer architecture, in order to solve the problem of coarse processing in modal representation enhancement of existing RGB-T tracking methods.

[0008] To address the aforementioned technical problems, the first aspect of this invention provides an RGB-T target tracking method based on the Transformer architecture, the method comprising:

[0009] Acquire the visible light and infrared image sequences to be tracked, and extract the corresponding template frames and search frames for each; input the extraction results into the trained target tracking model to obtain the position of the target in each search frame;

[0010] The target tracking model includes an improved shared-weight dual-branch Vision Transformer encoder for independent feature extraction of RGB and TIR modes, and a prediction head for predicting results based on input features. The improvement includes inserting a bidirectional frequency domain guidance mechanism in the mid-deep and deep layers of the encoder. The bidirectional frequency domain guidance mechanism is used to: extract high-frequency components from the RGB mode features and low-frequency components from the TIR mode features based on the features of the two input modes, respectively; then, spatially calibrate the TIR features based on the extracted high-frequency components to achieve spatial guidance enhancement; and anchor the RGB high-frequency details through channel modulation based on the extracted low-frequency components to achieve channel guidance enhancement, thereby obtaining the enhanced dual-modal search region features.

[0011] In one possible implementation, the bidirectional frequency domain guidance mechanism extracts the high-frequency components from the RGB modal features and the low-frequency components from the TIR modal features in the following way:

[0012] Channel compression is performed on the features of the two input modalities, and the compressed features of the two modalities are fused to obtain the fused guided features;

[0013] Based on the fusion-guided features, high-pass filter kernels and low-pass filter kernels are dynamically generated at each spatial location;

[0014] For the input RGB modal features, a content-aware feature smoothing upsampling operator with a high-frequency filter kernel as weight is used to extract pure high-frequency detail components, which are then added to the input RGB modal features to obtain the high-frequency components in the RGB modal features. For the input TIR modal features, a content-aware feature upsampling operator with a low-pass filter kernel as weight is used to extract the low-frequency components in the stable TIR modal features.

[0015] In one possible implementation, spatial guided enhancement is performed as follows: using the extracted high-frequency components as guides, firstly max pooling and average pooling are performed separately. The max pooling result is input into two parallel convolutional branches with different dilation rates, and the average pooling result is input into a third convolutional branch with a dilation rate to extract multi-scale features. Then, the multi-scale features are fused to generate a spatial attention map. The spatial attention map is used to modulate the extracted low-frequency components element-wise to achieve spatial guided enhancement.

[0016] In one possible implementation, channel-guided enhancement is performed as follows: the extracted low-frequency components are used as guides and processed through two parallel branches. The first branch has a global average pooling and a multilayer perceptron connected in sequence, and the second branch has a global max pooling and a multilayer perceptron connected in sequence. The results of the two branches are added together and then passed through an activation function to generate a channel attention vector. The channel attention vector is used to recalibrate the extracted high-frequency components to achieve channel-guided enhancement.

[0017] In one possible implementation, the improvement further includes adding an Euclidean constraint mechanism between the encoder and the prediction head, which is used to generate a spatial attenuation mask based on spatial Euclidean distance to enhance the feature correlation of the target local region and effectively suppress invalid interference from distant background regions.

[0018] In one possible implementation, the specific processing steps of the Euclidean constraint mechanism include:

[0019] The enhanced dual-modal search region features are spliced ​​along the channel dimension to obtain the fused features. The spatial coordinate tensor of the fused features is constructed using meshgrid. After flattening, the squared Euclidean distance between any two spatial locations is calculated and normalized.

[0020] Based on the normalized distance matrix, spatial decay masks corresponding to the two modes are constructed so that positions with closer spatial distances receive higher attention weights, while the weights of distant positions decay exponentially.

[0021] Self-attention calculation is performed on the fused features based on the spatial attenuation mask to obtain the spatially modulated attention outputs from RGB to TIR and from TIR to RGB; the final optimized search features are obtained based on the two outputs.

[0022] In one possible implementation, the spatial attenuation mask is:

[0023]

[0024] In the formula, For spatial attenuation mask, It is the ReLU activation function. For the normalized distance matrix, For learnable decay rate parameters, This is the bias parameter.

[0025] In one possible implementation, the method for calculating self-attention for the fused features based on a spatial decay mask is as follows:

[0026] Spatial attenuation masks corresponding to the two modes respectively and It applies to the cross-modal self-attention score matrix, specifically in the following manner:

[0027]

[0028] In the formula, , These are the attention score matrices after modulation from RGB to TIR and from TIR to RGB, respectively. , These are the original attention score matrices from RGB to TIR and from TIR to RGB, respectively, derived from the query vector. and key vector dot product The result after scaling; It represents the Hadamardi (or Hadama) stack;

[0029] Using two modulated attention score matrices , Aggregate value vector :

[0030]

[0031] In the formula, , This is the attention output after two-way spatial modulation.

[0032] In one possible implementation, the final optimized search features are obtained from the two outputs by concatenating the two spatially modulated attention outputs, and then generating gating parameters through average pooling and a multilayer perceptron network. The gating parameters are then used to fuse the two spatially modulated attention outputs to obtain the final optimized search features.

[0033] To address the aforementioned technical problems, a second aspect of the present invention provides an RGB-T target tracking device based on a Transformer architecture, comprising a processor for executing a computer program to implement the RGB-T target tracking method based on a Transformer architecture of the present invention, the method comprising:

[0034] Acquire the visible light and infrared image sequences to be tracked, and extract the corresponding template frames and search frames for each; input the extraction results into the trained target tracking model to obtain the position of the target in each search frame;

[0035] The target tracking model includes an improved shared-weight dual-branch Vision Transformer encoder for independent feature extraction of RGB and TIR modes, and a prediction head for predicting results based on input features. The improvement includes inserting a bidirectional frequency domain guidance mechanism in the mid-deep and deep layers of the encoder. The bidirectional frequency domain guidance mechanism is used to: extract high-frequency components from the RGB mode features and low-frequency components from the TIR mode features based on the features of the two input modes, respectively; then, spatially calibrate the TIR features based on the extracted high-frequency components to achieve spatial guidance enhancement; and anchor the RGB high-frequency details through channel modulation based on the extracted low-frequency components to achieve channel guidance enhancement, thereby obtaining the enhanced dual-modal search region features.

[0036] In one possible implementation, the bidirectional frequency domain guidance mechanism extracts the high-frequency components from the RGB modal features and the low-frequency components from the TIR modal features in the following way:

[0037] Channel compression is performed on the features of the two input modalities, and the compressed features of the two modalities are fused to obtain the fused guided features;

[0038] Based on the fusion-guided features, high-pass filter kernels and low-pass filter kernels are dynamically generated at each spatial location;

[0039] For the input RGB modal features, a content-aware feature smoothing upsampling operator with a high-frequency filter kernel as weight is used to extract pure high-frequency detail components, which are then added to the input RGB modal features to obtain the high-frequency components in the RGB modal features. For the input TIR modal features, a content-aware feature upsampling operator with a low-pass filter kernel as weight is used to extract the low-frequency components in the stable TIR modal features.

[0040] In one possible implementation, spatial guided enhancement is performed as follows: using the extracted high-frequency components as guides, firstly max pooling and average pooling are performed separately. The max pooling result is input into two parallel convolutional branches with different dilation rates, and the average pooling result is input into a third convolutional branch with a dilation rate to extract multi-scale features. Then, the multi-scale features are fused to generate a spatial attention map. The spatial attention map is used to modulate the extracted low-frequency components element-wise to achieve spatial guided enhancement.

[0041] In one possible implementation, channel-guided enhancement is performed as follows: the extracted low-frequency components are used as guides and processed through two parallel branches. The first branch has a global average pooling and a multilayer perceptron connected in sequence, and the second branch has a global max pooling and a multilayer perceptron connected in sequence. The results of the two branches are added together and then passed through an activation function to generate a channel attention vector. The channel attention vector is used to recalibrate the extracted high-frequency components to achieve channel-guided enhancement.

[0042] In one possible implementation, the improvement further includes adding an Euclidean constraint mechanism between the encoder and the prediction head, which is used to generate a spatial attenuation mask based on spatial Euclidean distance to enhance the feature correlation of the target local region and effectively suppress invalid interference from distant background regions.

[0043] In one possible implementation, the specific processing steps of the Euclidean constraint mechanism include:

[0044] The enhanced dual-modal search region features are spliced ​​along the channel dimension to obtain the fused features. The spatial coordinate tensor of the fused features is constructed using meshgrid. After flattening, the squared Euclidean distance between any two spatial locations is calculated and normalized.

[0045] Based on the normalized distance matrix, spatial decay masks corresponding to the two modes are constructed so that positions with closer spatial distances receive higher attention weights, while the weights of distant positions decay exponentially.

[0046] Self-attention calculation is performed on the fused features based on the spatial attenuation mask to obtain the spatially modulated attention outputs from RGB to TIR and from TIR to RGB; the final optimized search features are obtained based on the two outputs.

[0047] In one possible implementation, the spatial attenuation mask is:

[0048]

[0049] In the formula, For spatial attenuation mask, It is the ReLU activation function. For the normalized distance matrix, For learnable decay rate parameters, This is the bias parameter.

[0050] In one possible implementation, the method for calculating self-attention for the fused features based on a spatial decay mask is as follows:

[0051] Spatial attenuation masks corresponding to the two modes respectively and It applies to the cross-modal self-attention score matrix, specifically in the following manner:

[0052]

[0053] In the formula, , These are the attention score matrices after modulation from RGB to TIR and from TIR to RGB, respectively. , These are the original attention score matrices from RGB to TIR and from TIR to RGB, respectively, derived from the query vector. and key vector dot product The result after scaling; It represents the Hadamardi (or Hadama) stack;

[0054] Using two modulated attention score matrices , Aggregate value vector :

[0055]

[0056] In the formula, , This is the attention output after two-way spatial modulation.

[0057] In one possible implementation, the final optimized search features are obtained from the two outputs by concatenating the two spatially modulated attention outputs, and then generating gating parameters through average pooling and a multilayer perceptron network. The gating parameters are then used to fuse the two spatially modulated attention outputs to obtain the final optimized search features.

[0058] The beneficial effects of this invention are as follows: This invention constructs a target tracking model for RGB-T target tracking based on a shared-weight dual-branch Vision Transformer encoder. A bidirectional frequency domain guidance mechanism is inserted into the mid-to-deep network of the encoder, enabling complementary purification of the features of the two modalities before spatial modeling. Specifically, high-frequency enhanced RGB features and TIR low-frequency features (i.e., high-frequency components in the RGB modal features and low-frequency components in the TIR modal features, respectively) are extracted based on the features of the two modalities. This achieves precise decoupling of RGB high-frequency details and TIR low-frequency structures. Then, channel-guided enhancement and spatial-guided enhancement are performed based on the high-frequency enhanced RGB features and TIR low-frequency features (i.e., TIR features are spatially calibrated based on RGB high-frequency details, and RGB high-frequency details are anchored through channel modulation based on the TIR low-frequency structure). This cross-modal complementary purification from two directions significantly improves the consistency and discriminative power of the dual-modal features under conditions such as drastic changes in illumination and occlusion. This invention significantly improves tracking robustness and positioning accuracy in complex scenarios such as drastic changes in illumination, target deformation, and cluttered backgrounds. Attached Figure Description

[0059] Figure 1 This is a flowchart of the RGB-T target tracking method based on the Transformer architecture of the present invention;

[0060] Figure 2 This is the overall framework structure of the RGB-T target tracking method based on the Transformer architecture of the present invention;

[0061] Figure 3 This invention provides a bidirectional frequency domain guidance mechanism for the RGB-T target tracking method based on the Transformer architecture.

[0062] Figure 4 This invention relates to the Euclidean constraint mechanism for the RGB-T target tracking method based on the Transformer architecture.

[0063] Figure 5 This is a structural diagram of the RGB-T target tracking device based on the Transformer architecture of the present invention. Detailed Implementation

[0064] The core of this invention includes: First, designing a bidirectional frequency domain guidance mechanism to extract high-frequency detail components of the RGB mode and low-frequency structural components of the TIR mode, respectively. This is achieved by spatially calibrating the low-frequency components with high-frequency data and anchoring the channel weights of the high-frequency components with low-frequency data, thus realizing structured complementary enhancement of the dual-modal features. Second, proposing an Euclidean constraint mechanism to construct a spatial Euclidean distance matrix and generate an adaptive attenuation mask, strengthening the feature correlation of the target's local region while suppressing invalid interference from distant background regions. This effectively improves the discriminative power and robustness of target features in complex scenes, enhances the model's adaptability to drastic changes in illumination, target deformation, and background interference, and significantly improves the accuracy and stability of RGB-T target tracking. To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings.

[0065] An implementation method for RGB-T target tracking based on Transformer architecture:

[0066] This invention proposes an RGB-T target tracking method based on the Transformer architecture. This method achieves synergistic enhancement of cross-modal features and robust tracking by constructing a cascaded architecture of frequency domain pre-filtering and spatial post-propagation. Key aspects include: First, introducing a bidirectional frequency domain guidance mechanism (BFDG) in the encoder to extract high-frequency detail components of the RGB mode and low-frequency structural components of the TIR mode. This achieves structured complementary enhancement of dual-modal features by calibrating the spatial localization of low frequencies and anchoring the channel weights of high frequencies with low frequencies. Second, proposing an Euclidean constraint mechanism (EC) to construct a spatial Euclidean distance matrix and generate an adaptive attenuation mask, strengthening the feature correlation of local target regions while suppressing invalid interference from distant background regions. Finally, the frequency-enhanced reliable features are sent to the spatial propagation stage, where the Euclidean constraint mechanism performs precise spatial modulation before outputting the tracking result.

[0067] The implementation process of the RGB-T target tracking method based on the Transformer architecture of the present invention is as follows: Visible light and thermal infrared image sequences to be tracked are acquired; template frames and search frames are extracted from the visible light and thermal infrared image sequences, respectively; the extracted template frames and search frames are input into a trained target tracking model; the bidirectional frequency domain guided (BFDG) module in the target tracking model is used for complementary feature enhancement; and the Euclidean constraint (EC) module in the target tracking model is used for precise spatial modulation of features. Finally, the optimized search features are output through the prediction head in the target tracking model to determine the target classification confidence and bounding box regression results, thereby obtaining the target's position in each search frame and achieving target tracking.

[0068] like Figure 2The diagram shows the overall framework of the target tracking model of this invention. Its basic framework uses ViT-B as the backbone network, specifically including an encoder module, an Euclidean constraint module, and a prediction head. The encoder mainly adopts a shared-weight dual-branch Transformer structure, consisting of a PatchEmbed module and 12 Transformer blocks. The step size of PatchEmbed is set to 16. Each Transformer block contains a multi-head self-attention layer (MHSA) and a feedforward network layer (MLP). This invention inserts bidirectional frequency domain guidance (BFDG) modules in the middle and deep layers of the encoder to perform frequency domain decoupling and cross-modal complementary enhancement of the search region features. In this embodiment, to preserve modality-specific features and enhance complementary information, specifically, bidirectional frequency domain guidance modules are inserted between layers 7 and 8 (ViT6 and ViT7) and between layers 10 and 11 (ViT9 and ViT10) of the 12-layer Transformer block, performing cross-modal frequency domain guidance and recalibration only on the search region tokens. The Euclidean constraint module is deployed between the encoder and the prediction head, primarily used to enhance the feature relevance of local target regions while suppressing invalid interference from distant background regions. The prediction head mainly employs an anchor-box-independent dense prediction paradigm, including classification and regression branches, to project the optimized features output by the Euclidean constraint module onto the original dimensions and output the target classification confidence and bounding box regression results to obtain the target tracking results.

[0069] The target tracking model will be described in detail below.

[0070] I. Encoder Module.

[0071] The encoder module employs a weighted, dual-branch Transformer encoder to independently extract features from RGB (visible light) and TIR (thermal infrared) images. The images are input into the PatchEmbed module of the encoder, converting them into sequential data that the Transformer block can process. Feature extraction and enhancement are then performed at each layer of the Transformer block. The enhanced features output from layer 7 (ViT6) are input to the first Bidirectional Frequency-Guided (BFDG) module for RGB and TIR mode feature fusion enhancement. The fused enhanced features are then input to layer 8 (ViT7) of the Transformer block for further processing. The enhanced features output from layer 10 (ViT9) are then input to the second BFDG module for feature fusion enhancement. The fused enhanced features are then input to layer 11 (ViT10) of the Transformer block for further processing. Finally, layer 12 (ViT11) of the Transformer block outputs the final fused and enhanced dual-modal features. Through a weight-sharing mechanism, the same encoder parameters process the inputs of both modalities simultaneously, reducing the number of parameters while maintaining modality-specific characterization capabilities.

[0072] 1. PatchEmbed module.

[0073] 1) Divide the input RGB image and infrared image (including the search image and template image respectively) into non-overlapping image blocks of size P×P, and map them into token sequences. and :

[0074]

[0075] In the formula, D represents the channel dimension. and These represent the number of image blocks in the template frame and the search frame, respectively.

[0076] 2) For the same modality (RGB or TIR), the template token and search token are concatenated along the token dimension to form the encoder input sequence:

[0077]

[0078] In this embodiment, the step size of PatchEmbed is set to 16, and the backbone network parameters are initialized using a weight file pre-trained based on the ODTrack framework at a search region scale of 256×256.

[0079] 2. Bidirectional Frequency Domain Guidance (BFDG) module.

[0080] To preserve modal-specific features and enhance complementary information, bidirectional frequency-domain guided (BFDG) modules are inserted between layers 7 and 8 (ViT6 and ViT7) and between layers 10 and 11 (ViT9 and ViT10) of the encoder. Cross-modal frequency-domain guided and recalibrated features are performed only on tokens in the search region. This bidirectional frequency-domain guided mechanism is used to: extract high-frequency components from the RGB modal features and low-frequency components from the TIR modal features based on the features of the two input modalities; then, spatially calibrate the TIR features based on the extracted high-frequency components to achieve spatial guided enhancement; and anchor RGB high-frequency details through channel modulation based on the extracted low-frequency components to achieve channel guided enhancement. Let the RGB features of a certain layer of the input encoder be... TIR characteristics are All sizes The RGB and TIR features of the first layer (ViT0) of the input encoder are the encoder input sequence formed by the PatchEmbed module. The input features of layers 2-7 (ViT1-ViT6), 9-10 (ViT8-ViT9), and 12 (ViT11) are the output features of the upper layers. The input features of layers 8 (ViT7) and 11 (ViT10) are the fused features of the BFDG output. The detailed steps of the Bidirectional Frequency Domain Guided (BFDG) module for frequency domain decoupling and cross-modal complementary enhancement of the search region features are as follows... Figure 3 As shown, the following is based on Figure 3 A detailed introduction to the bidirectional frequency domain guidance mechanism:

[0081] 1) Features of the input BFDG module and The grid feature dimensions are transformed using Reshap to 2D Grid to obtain the transformed 2D grid features. Specifically, the search region tokens are converted into 2D grid features through dimensional rearrangement. To accommodate subsequent convolution operations, where B is the batch size and C is the channel dimension. , The height and width of the feature map of the search region.

[0082] 2) Perform channel compression. The converted 2D mesh features are processed using a 1×1 convolution (Conv) to compress the channels of the two modal features, reducing the number of channels from... Down to In this embodiment, the number of channels is set to a default value. =256, therefore the number of channels is reduced from =256 dropped to =64:

[0083]

[0084] In the formula, and They are respectively and Features after channel compression.

[0085] 3) The compressed two-modal features are fused to obtain the fused guided features. :

[0086]

[0087] In the formula, For scale alignment operations, it degenerates into an identity mapping when the resolutions of the two branches are the same.

[0088] 4) Based on fusion guidance features Low-pass and high-pass filter kernels are dynamically generated at each spatial location using two parallel 3×3 convolutional predictors:

[0089]

[0090] In the formula, and It is a 3×3 convolutional predictor; , These refer to the sizes of the low-pass core and the high-pass core, respectively. In this embodiment, the low-pass core size is set to [size]. Qualcomm core size .

[0091] 5) Apply kernel softmax normalization and a window function to the kernel vector (flattened kernel vector) output at each spatial location. Regularization is applied, and layer normalization is performed in Normalize to suppress reconstruction artifacts and improve the smoothness of kernel weights.

[0092]

[0093] In the formula, The original dynamic kernel output by the convolution predictor can be taken as... or , This is the intermediate kernel after softmax and window function regularization. For the final normalized executable filter kernel, It represents the Hadamah accumulation. To ensure numerical stability, a small constant is used. When the original dynamic kernel is... At that time, the final output Low-pass core When the original dynamic kernel is At that time, the final output Qualcomm core .

[0094] 6) Obtain the executable filter kernel Then, the frequency components of the two modes (TIR low-frequency component and RGB high-frequency component) are extracted respectively.

[0095] ① For TIR modalities, the content-aware feature upsampling (CAFU) operator combined with a normalized low-pass kernel is used. Extracting stable low-frequency structural components (i.e., TIR low-frequency characteristics):

[0096]

[0097] In the formula, Indicated by The content-aware feature upsampling operator for weights performs feature upsampling synchronously when the scale factor is greater than 1; The TIR features of this layer are input to the encoder.

[0098] The specific calculation formula for Content-Aware Feature Upsampling (CAFU) is as follows:

[0099]

[0100] In the formula, Indicates the feature channel index. Indicates the current spatial location to be reorganized; Indicates the position of participation The spatial location of the neighborhood of feature recombination; This represents the low-frequency characteristics of the TIR obtained by CAFU; Indicates position Surrounding size is The local neighborhood, Indicates the location The corresponding neighborhood position in the dynamically generated recombination kernel The weight.

[0101] ② For the RGB modality, the Content-Aware Feature Smoothing Upsampling (CAFSU) operator is used in conjunction with a normalized high-pass kernel. Pure high-frequency detail components are extracted using a residual method and then added back to the original features to obtain high-frequency enhanced RGB features. :

[0102]

[0103] In the formula, Indicated by Content-aware feature smoothing upsampling operator for weights; The RGB features of this layer are used as input to the encoder; For pure high-frequency detail components; It is a high-frequency enhanced RGB feature.

[0104] The specific formula for calculating Content-Aware Feature Smoothing Upsampling (CAFSU) is as follows:

[0105]

[0106] In the formula, This represents the RGB smoothing feature obtained from CAFSU; Indicates position Surrounding size is The local neighborhood; Indicates the location The corresponding neighborhood position in the dynamically generated sufficient kernel The weight.

[0107] CAFU uses a dynamic low-pass reconstruction kernel to directly weight and reconstruct TIR features, and its output itself is a low-frequency structural feature. CAFSU, on the other hand, uses a dynamic local reconstruction kernel to first estimate the smoothed component of RGB features, and then obtains high-frequency detail features in the form of residuals by subtracting the original RGB features from the smoothed components. Therefore, CAFU is a direct low-frequency extraction, while CAFSU is an indirect high-frequency extraction based on content-aware smoothing and residual subtraction.

[0108] 7) Complete frequency decomposition to obtain TIR low-frequency characteristics. and high-frequency enhanced RGB features Then, bidirectional cross-modal guidance enhancement is performed.

[0109] ① During the Spatial Guided Enhancement (SGE) stage, RGB features are enhanced with high frequency. As a guide, we first perform max pooling and average pooling along the channel dimension:

[0110]

[0111] Specifically, multi-scale spatial cues using max pooling are extracted through two parallel convolutional branches with different dilation rates, and multi-scale spatial cues using average pooling are extracted through another convolutional branch with a different dilation rate. The multi-scale spatial cues from the three branches are then fused with 1×1 convolutions via channel concatenation (Concat). Batch normalization (BN) is then applied to normalize the fused features, and finally, a spatial attention map is generated using 7×7 convolutions and sigmoid activation. And the low-frequency characteristics of TIR Perform element-wise modulation:

[0112]

[0113] In the formula, , , These are three parallel convolutional branches with different dilation rates; This is a high-frequency enhanced RGB feature; Use the Sigmoid activation function; It exhibits low-frequency characteristics of TIR; For spatially enhanced TIR features.

[0114] ② During the Channel Guided Enhancement (CGE) phase, using the low-frequency characteristics of TIR... As a guide, channel descriptors are generated in parallel using global average pooling (Adaptive AvgPool2d(1)) and global max pooling (Adaptive MaxPool2d(1)). The two generated channel descriptors are then element-wise summed using an MLP network (Conv→ReLU→Conv), and a sigmoid activation is used to generate the channel attention vector. RGB features enhanced by high frequency Perform channel recalibration to suppress noise, texture, and background interference:

[0115]

[0116] In the formula, Use the Sigmoid activation function; , These represent global average pooling and global max pooling, respectively. Enhanced RGB features for the channel.

[0117] 8) The final enhanced bimodal feature stream is reshaped into a 1D token sequence and written back to subsequent Transformer layers for further processing.

[0118] II. Euclidean Constraints (EC) Module.

[0119] The enhanced bimodal search region features output from encoder layer 12 (ViT11) are concatenated along the channel dimension and then fed into a Euclidean constraint mechanism for spatial modulation to suppress long-distance background interference; such as Figure 2 As shown, the Euclidean constraint mechanism is applied to the fusion search region features after the encoder output. The specific process of the Euclidean constraint mechanism is as follows: Figure 4 As shown.

[0120] 1) The enhanced bimodal search region features are stitched together along the channel dimension to obtain a size of... Fusion characteristics The meshgrid expands a one-dimensional coordinate array to generate a two-dimensional network coordinate matrix, thus constructing the spatial coordinate tensor of the feature map. After flattening, calculate the squared Euclidean distance between any two spatial locations:

[0121]

[0122] In the formula, express and The square of the Euclidean distance between them.

[0123] 2) To unify the distance scale of feature maps of different sizes, the distance matrix is ​​normalized to its maximum value:

[0124]

[0125] In the formula, It is the largest single distance value in the entire matrix. The normalized values ​​have diagonal elements always set to 0 and far-end elements to 1, accurately quantifying the spatial proximity of all pixel pairs in the feature map.

[0126] 3) Based on the normalized distance matrix, a learnable attenuation rate parameter is introduced. and bias parameters Construct a spatial attenuation mask:

[0127]

[0128] In the formula, ReLU activation ensures that the decay rate is non-negative, so that locations that are closer in space receive higher attention weights, while the weights of distant locations decay exponentially. It is a natural exponential function. The near-diagonal area of ​​the mask is highlighted, and the far-diagonal area gradually darkens.

[0129] 4) Apply the spatial decay mask to the cross-modal self-attention score matrix, and perform element-wise multiplication before Softmax normalization:

[0130]

[0131] In the formula, and It is the original attention score matrix without spatial mask modulation and Softmax normalization; and These are the attention score matrices (also known as attention weight matrices) modulated by spatial attenuation masks and normalized to Softmax, representing RGB to TIR and TIR to RGB respectively, derived from the query vector. and key vector dot product The result after scaling; and These are the modulated attention score matrices; This indicates element-wise multiplication; and This is the spatial attenuation mask corresponding to the two modes.

[0132] Among them, query vector Key vector and weighted aggregate value vector It was obtained using the following method:

[0133] First, the enhanced RGB features and TIR features are concatenated to obtain the fused feature vector. :

[0134]

[0135] Then, a three-parameter independent linear mapping layer is used to fuse the feature vectors. Perform mapping to obtain query vectors respectively. Key vector Sum value vector :

[0136]

[0137] In the formula, and These are trainable weight matrices for three linear mapping layers, used to map the fused feature vector to the query feature space, key feature space, and value feature space; and These are the trainable bias vectors for the three linear mapping layers; each weight matrix and bias vector is learned during model training.

[0138] 5) Modulate the attention score matrix and Each is compared with the aggregated value vector of the modulated attention weights. Perform matrix multiplication:

[0139]

[0140] In the formula, This represents matrix multiplication.

[0141] 6) The two spatially modulated attention outputs are concatenated, and the concatenated attention matrix is ​​then processed by average pooling (AvgPool) and an MLP network to generate learnable gating parameters. The contributions of dynamic balancing of original and modulation features:

[0142]

[0143] In the formula, Indicates channel-dimensional splicing; These are the final optimized search features.

[0144] III. Predicting the Head.

[0145] The modulated fused features are projected onto the original dimension through a lightweight convolutional layer, and the input prediction head outputs the target classification confidence and bounding box regression results.

[0146] The prediction head employs an anchor-frame-independent dense prediction paradigm. The optimization features output by the Euclidean constraint mechanism are used. Lightweight convolutional layers reduce the channel dimension from Projection Back The data is fed into the classification and regression branches of the prediction head. The classification branch outputs the target confidence score for each search location, and the regression branch outputs the bounding box offset for the corresponding location.

[0147] After the target tracking model is built, it needs to be trained so that the model can mark the target in the input image to be tracked.

[0148] 1. Model training.

[0149] During the training phase, the training set of the LasHeR dataset (a large-scale, highly diverse RGB-T tracking benchmark dataset) was used as the data source. It contains 1224 strictly spatiotemporally aligned bimodal video sequences, approximately 1.47 million frames, covering 19 challenging attributes such as drastic changes in lighting, target deformation, and cluttered backgrounds. Samples were taken from the synchronous bimodal video sequences of the RGB-T tracking dataset to obtain template frames of size 128×128 and search frames of size 256×256.

[0150] Total loss function during training phase Designed as follows:

[0151]

[0152] In the formula, Weighted focus loss is used to constrain classification confidence levels; use Loss-constrained bounding box coordinate regression; The GIoU loss function is used to constrain the shape overlap quality between the predicted bounding box and the ground truth bounding box. , , represents the weight hyperparameters for each loss term. During the testing phase, the location with the highest classification confidence within the search area is selected as the target center estimate, and the final target bounding box is generated by combining it with the offset output from the regression branch.

[0153] 2. Model testing.

[0154] During the model testing phase, according to... Figure 1 The process shown is used to test the trained model: First, the acquired visible light and thermal infrared images to be tracked are converted into sequence data by the PatchEmbed module in the model, and template frames and search frames are extracted respectively. Then, the dual-branch Transformer encoder with shared weights in the model is used to extract features from the two modal images independently, and the bidirectional frequency domain guided (BFDG) module inserted in the encoder performs frequency domain decoupling and cross-modal complementary enhancement of the search region features. After that, the enhanced dual-modal search region features are concatenated along the channel dimension and fed into the Euclidean constraint (EC) module in the model for spatial modulation to suppress long-distance background interference. Finally, the modulated fused features are projected onto the original dimension through lightweight convolution, and the features projected onto the original dimension are input into the prediction head in the model. The prediction head outputs the target classification confidence and bounding box regression results, thereby obtaining the target tracking results.

[0155] In this embodiment, the testing phase evaluates the cross-dataset generalization performance on four publicly available benchmark datasets: LasHeR test set, RGBT210, RGBT234, and VTUAV. The RGB-T target tracking method based on the Transformer architecture provided by this invention, using ViT-B as the backbone network, achieves the following results: on the LasHeR dataset test set, the method achieves 72.1% accuracy, 68.3% normalized accuracy, and 57.4% success rate; on the RGBT234 dataset, it achieves 89.5% accuracy and 66.4% success rate; on the RGBT210 dataset, it achieves 89.3% accuracy and 64.5% success rate; and on the VTUAV dataset, it achieves 86.4% accuracy and 73.5% success rate, all reaching leading levels in the field.

[0156] An implementation of an RGB-T target tracking device based on the Transformer architecture:

[0157] The present invention provides an RGB-T target tracking device based on the Transformer architecture, such as... Figure 5 As shown, the system includes a memory, a processor, a system bus, and a computer program stored in the memory. The processor and memory communicate and exchange data with each other via the system bus. The processor executes the computer program to implement the RGB-T target tracking method based on the Transformer architecture of this invention.

[0158] The processor can be a general-purpose ARM Cortex A series (such as RK3588) to implement low-power embedded vision processing, or a DSP (such as TMS320C6748) to complete high-speed Fourier transform and convolution acceleration of frequency domain features. Alternatively, a GPU (such as Jetson Orin) can be used to accelerate the self-attention calculation of the Transformer encoder and the dynamic kernel prediction of the bidirectional frequency domain guidance mechanism. The memory can use non-volatile MRAM or FRAM to store key model parameters and template frame feature prototypes to ensure power-loss safety, while a large-capacity NAND Flash is used to store modal feature cache in the historical search frame sequence.

[0159] This device can be deployed in multiple real-world locations. The lightest deployment is on the local end of an in-vehicle embedded computing platform (e.g., based on an ARM Cortex-A55 processor, equipped with LPDDR4 memory and embedded eMMC). The processor runs a lightweight FSTrack model in real time, while the memory stores recent visible light and thermal infrared template frame features, as well as intermediate features enhanced in the frequency domain. Its advantages include fast response, low bandwidth requirements, and stable tracking via thermal infrared modality in low-light scenarios such as nighttime or tunnels, without relying on the cloud. Alternatively, it can be deployed on the autonomous driving domain controller. The in-vehicle camera and thermal infrared sensor transmit raw image pairs via in-vehicle Ethernet or GMSL, and the domain controller executes a complete bidirectional frequency domain guidance and Euclidean constraint inference process. Its advantages include strong computing power, large storage capacity, utilization of the in-vehicle GPU to accelerate Transformer encoding and cross-modal attention computation, and the ability to share features with the perception fusion module. Furthermore, it can be deployed on roadside edge computing nodes (e.g., Jetson AGXOrin, equipped with a large-capacity SSD), where data from multiple roadside RGB-T cameras converge to the node for multi-target joint tracking. The advantage is that it can fuse multi-view, multi-modal data and maintain identity consistency through trajectory correlation when the target switches between cameras or is severely occluded. In practical applications, the choice is usually based on the requirements for real-time performance, power consumption, computing power, and the deployment platform (vehicle-mounted, roadside, or drone).

[0160] In summary, this invention first constructs a structured information flow paradigm of frequency domain pre-filtering and spatial post-constraint. Specifically, the bidirectional frequency domain guidance mechanism achieves precise decoupling of RGB high-frequency details and TIR low-frequency structure through a content-adaptive dynamic kernel, and performs cross-modal complementary purification through spatial guidance enhancement and channel guidance enhancement, significantly improving the consistency and discriminative power of bimodal features under conditions of drastic illumination changes and occlusion. Next, the Euclidean constraint mechanism, by constructing a spatial distance attenuation mask, injects explicit geometric distance priors into the Transformer's self-attention calculation, effectively suppressing interference from distant background noise, enabling the model to generate more compact and accurate target responses in complex backgrounds. The two mechanisms work together to significantly improve tracking robustness and positioning accuracy in complex scenarios such as drastic illumination changes, target deformation, and cluttered backgrounds.

[0161] It should be noted that, in this document, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0162] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An RGB-T target tracking method based on Transformer architecture, characterized in that, The method includes: Acquire the visible light and infrared image sequences to be tracked, and extract the corresponding template frames and search frames for each; input the extraction results into the trained target tracking model to obtain the position of the target in each search frame; The target tracking model includes an improved shared-weight dual-branch Vision Transformer encoder for independent feature extraction of RGB and TIR modes, and a prediction head for predicting results based on input features. The improvement includes inserting a bidirectional frequency domain guidance mechanism in the mid-deep and deep layers of the encoder. The bidirectional frequency domain guidance mechanism is used to: extract high-frequency components from the RGB mode features and low-frequency components from the TIR mode features based on the features of the two input modes, respectively; then, spatially calibrate the TIR features based on the extracted high-frequency components to achieve spatial guidance enhancement; and anchor the RGB high-frequency details through channel modulation based on the extracted low-frequency components to achieve channel guidance enhancement, thereby obtaining the enhanced dual-modal search region features.

2. The RGB-T target tracking method based on Transformer architecture according to claim 1, characterized in that, The bidirectional frequency domain guidance mechanism extracts the high-frequency components from the RGB modal features and the low-frequency components from the TIR modal features in the following way: Channel compression is performed on the features of the two input modalities, and the compressed features of the two modalities are fused to obtain the fused guided features; Based on the fusion-guided features, high-pass filter kernels and low-pass filter kernels are dynamically generated at each spatial location; For the input RGB modal features, a content-aware feature smoothing upsampling operator with a high-frequency filter kernel as weight is used to extract pure high-frequency detail components, which are then added to the input RGB modal features to obtain the high-frequency components in the RGB modal features. For the input TIR modal features, a content-aware feature upsampling operator with a low-pass filter kernel as weight is used to extract the low-frequency components in the stable TIR modal features.

3. The RGB-T target tracking method based on Transformer architecture according to claim 1, characterized in that, The spatial guided enhancement method is as follows: using the extracted high-frequency components as guides, first perform max pooling and average pooling respectively. The max pooling result is input into two parallel convolution branches with different dilation rates, and the average pooling result is input into a third convolution branch with a dilation rate to extract multi-scale features. Then, the multi-scale features are fused to generate a spatial attention map. The spatial attention map is used to modulate the extracted low-frequency components element by element to achieve spatial guided enhancement.

4. The RGB-T target tracking method based on Transformer architecture according to claim 1, characterized in that, The channel-guided enhancement method is as follows: the extracted low-frequency components are used as guides and processed through two parallel branches. The first branch has a global average pooling and a multilayer perceptron connected in sequence, and the second branch has a global max pooling and a multilayer perceptron connected in sequence. The results of the two branches are added together and then passed through an activation function to generate a channel attention vector. The channel attention vector is used to recalibrate the extracted high-frequency components to achieve channel-guided enhancement.

5. The RGB-T target tracking method based on Transformer architecture according to any one of claims 1 to 4, characterized in that, The improvement also includes adding an Euclidean constraint mechanism between the encoder and the prediction head, which is used to generate a spatial attenuation mask based on spatial Euclidean distance to enhance the feature correlation of the target local region and effectively suppress invalid interference from distant background regions.

6. The RGB-T target tracking method based on Transformer architecture according to claim 5, characterized in that, The specific processing steps of the Euclidean constraint mechanism include: The enhanced dual-modal search region features are spliced ​​along the channel dimension to obtain the fused features. The spatial coordinate tensor of the fused features is constructed using meshgrid. After flattening, the squared Euclidean distance between any two spatial locations is calculated and normalized. Based on the normalized distance matrix, spatial decay masks corresponding to the two modes are constructed so that positions with closer spatial distances receive higher attention weights, while the weights of distant positions decay exponentially. Self-attention calculation is performed on the fused features based on the spatial attenuation mask to obtain the spatially modulated attention outputs from RGB to TIR and from TIR to RGB; the final optimized search features are obtained based on the two outputs.

7. The RGB-T target tracking method based on Transformer architecture according to claim 6, characterized in that, The spatial attenuation mask is: In the formula, For spatial attenuation mask, It is the ReLU activation function. For the normalized distance matrix, For learnable decay rate parameters, This is the bias parameter.

8. The RGB-T target tracking method based on Transformer architecture according to claim 6, characterized in that, The method for calculating self-attention for fused features based on a spatial attenuation mask is as follows: Spatial attenuation masks corresponding to the two modes respectively and It applies to the cross-modal self-attention score matrix, specifically in the following manner: In the formula, , These are the attention score matrices after modulation from RGB to TIR and from TIR to RGB, respectively. , These are the original attention score matrices from RGB to TIR and from TIR to RGB, respectively, derived from the query vector. and key vector dot product The result after scaling; It represents the Hadamardi (or Hadama) stack; Using two modulated attention score matrices , Aggregate value vector : In the formula, , This is the attention output after two-way spatial modulation.

9. The RGB-T target tracking method based on Transformer architecture according to claim 6, characterized in that, The method for obtaining the final optimized search features based on the two outputs is as follows: the two spatially modulated attention outputs are concatenated, and the concatenated features are then subjected to average pooling and a multilayer perceptron network to generate gating parameters. By fusing the two spatially modulated attention outputs using gating parameters, the final optimized search features are obtained.

10. An RGB-T target tracking device based on Transformer architecture, comprising a processor, characterized in that, The processor executes a computer program to implement the steps of the method according to any one of claims 1 to 9.

Images