RGB-T tracking method and system based on adaptive space-time prompt

Abstract

The invention provides an RGB-T tracking method and system based on adaptive space-time prompt, and the method comprises the steps: obtaining an image sequence containing RGB and TIR modes, and carrying out the preprocessing of the image sequence; performing basic feature extraction based on the preprocessed image sequence; features of different semantic levels are extracted based on a modal complementation prompter, and channel complementarity is learned; time node marks are generated based on a time state observer, and the tracking process is perfected by using time sequence features; a self-adaptive dynamic space-time module based on a self-adaptive weight mechanism keeps optimal balance of space-time information; and inputting the features of the final layer into an autoregressive decoder, and predicting a final target position in an autoregressive manner. According to the method, the effective space-time information is generated, the continuity of the time information and the complexity of the space-time information are fully mined, the optimal balance of the space-time information is realized in combination with the space-time module, and the complementary information of the RGB and TIR modes is effectively fused through the autoregression encoder, so that the target features can be quickly captured with relatively high precision.

Description

Technical Field

[0001] This invention relates to the field of target tracking technology, and in particular to an RGB-T tracking method and system based on adaptive spatiotemporal cues. Background Technology

[0002] With the rapid development of computer vision technology, target tracking, as one of its core tasks, has a wide range of applications in fields such as intelligent monitoring, autonomous driving, and robot navigation. Target tracking methods based on visible light (RGB) images have made significant progress in normal environments due to their rich texture and color information. However, in extremely complex scenes, RGB images are easily affected by changes in lighting, background interference, or loss of target information, leading to a significant decrease in tracking accuracy and insufficient robustness.

[0003] To address this issue, existing methods employ RGB-T tracking based on thermal infrared (TIR) ​​images. TIR images capture the target's thermal radiation information, are unaffected by lighting conditions, and complement RGB images, providing more stable target contour information even in extreme scenarios. Most mainstream RGB-T tracking methods are based on deep learning frameworks, primarily considering the combination of spatial and temporal information. Spatial information focuses on changes in the tracked object's appearance, relying on a fixed initial target appearance as reference information throughout the tracking process. However, over time, the target may deform or become occluded, making it impossible for the initial template frame to accurately capture its current state. Therefore, some methods consider incorporating temporal information, allowing updates at specific time points, but these only depend on sparse temporal or spatiotemporal relationships (i.e., updates limited to specific conditions). These methods either fail to consider temporal information during the tracking process or only consider simple temporal or spatiotemporal relationships under specific conditions, neglecting the continuity of temporal information and the complexity of spatiotemporal relationships. In video tracking tasks, the changes and movements of targets often exhibit certain trends. If the temporal information between frames and the complex spatiotemporal relationships are ignored, it is difficult to capture and express these trends, thus limiting the performance of the tracker. Currently, there is a need for an RGB-T tracking method and system based on adaptive spatiotemporal cues. Summary of the Invention

[0004] To address the problem of neglecting the modeling of inter-frame temporal information and complex spatiotemporal relationships in traditional target tracking, this invention provides an RGB-T tracking method and system based on adaptive spatiotemporal cues.

[0005] In a first aspect, the present invention provides an RGB-T tracking method based on adaptive spatiotemporal cues, which adopts the following technical solution:

[0006] The specific steps include the following:

[0007] S1, acquire image sequences containing both RGB and TIR modes, and preprocess the acquired image sequences;

[0008] S2, based on the preprocessed image sequence, performs basic feature extraction, including constructing a backbone network composed of multiple cascaded encoder blocks based on the Vision Transformer framework;

[0009] S3 generates effective prompt words based on the prompt learning method and extracts semantic features at different levels, including parallel embedding of modal complementary prompters into the encoder blocks of the backbone network;

[0010] S4, Construct a time state observer based on the time information of historical frames, including generating the current time node marker, using time series features to improve the tracking process, and promoting the generation of subsequent time node markers;

[0011] S5 constructs an adaptive dynamic spatiotemporal module based on the spatiotemporal features of a parallel modal complementary cue and a time state observer, cleverly integrating the dual advantages of temporal and spatial features; including the introduction of an adaptive weighting mechanism to effectively balance the features of spatiotemporal branches;

[0012] S6 inputs the final fused features into the autoregressive decoder, converts the bounding box into discrete labels, and autoregressively predicts the final target location to complete target tracking.

[0013] Furthermore, the preprocessing of the acquired image sequence includes spatial alignment of the RGB and TIR image sequences, extraction of template and search images from the image sequences, segmenting the processed RGB template image, RGB search image, TIR template image, and TIR search image into the same number of patches, encoding the patches through linear projection, converting the two-dimensional patches into one-dimensional feature vectors, and generating a visible light modal feature sequence.

[0014] Further, the basic feature extraction based on the preprocessed image sequence includes inputting the RGB dominant modality patch sequence obtained by the patch embedding layer into the encoder block of the backbone network, firstly performing multi-head self-attention calculation on the input features, then performing layer normalization on the output of the multi-head self-attention, and finally performing residual connection between the layer-normalized features and the input features to obtain the first intermediate features. The first intermediate features are then input into a feedforward convolutional network for nonlinear transformation, and the output of the feedforward convolutional network is then residually connected with the first intermediate features to obtain the output features of the encoder block. The output relationship between the l-th encoder block and the (l-1)-th encoder block can be expressed as:

[0015] ,

[0016] ,

[0017] in, For the first The output characteristics of each encoder block For layer normalization, For the sake of the bulls' self-attention, It is a convolutional feedforward network. These are intermediate features after multi-head self-attention and residual connections.

[0018] Furthermore, the cue-based learning method generates effective cue words and extracts semantic features at different levels. This includes ensuring that all parameters in the encoder block of the fixed backbone network do not participate in training gradient updates, and embedding modal complementarity cueing devices in parallel into the encoder block of the backbone network. The modal complementarity cueing device performs channel deformation on the output features of the previous layer encoder and cueing device. Then, it spatially focuses the deformed RGB features and fuses them with TIR features. Simultaneously, it performs channel deformation again to restore the original feature shape and generate channel cueing, which is then input into the encoder's original input. The process expression is:

[0019] ,

[0020] ,

[0021] in, This is represented as the output of MCP. These are the outputs from the previous encoder and the MCP, respectively. This indicates the characteristics of the current encoder input prompt and the dominant RGB mode.

[0022] Furthermore, the construction of the time-state observer based on the time information of historical frames includes selecting a state-space model such that the model's parameters depend on the input data, from the two modalities of the input. =[ ]and =[ Extract features from [the previous n frames] to generate temporal information learned from the previous n frames. and Guide the target state at the current time point, and insert an empty token. This is used to store the target information at the current time point. Then... and The time information is discretized, and the input is subjected to one-dimensional convolution and linear projection to generate a feature matrix. The current time token is generated by exchanging matrix C and combining supplementary information from another modality. This current time token is then added to the total time set T and input into the next frame. The discretization processing expression is:

[0023]

[0024]

[0025]

[0026]

[0027] in, These are predefined time-scale parameters, continuous parameters. and The corresponding discrete parameters mapped to the discrete space and , ,and It is the length of the visual marker. The discrete process is the same as described above. and The final output is obtained by adding them together.

[0028] Furthermore, the adaptive dynamic spatiotemporal module constructed based on the spatiotemporal features of the parallel modal complementary cue and the temporal state observer includes the introduction of an adaptive weighting mechanism, cleverly integrating the dual advantages of temporal and spatial features. The adaptive weighting method defines the two-modal input feature tokens as a set x = [ After inputting the linearization layer and generating the average value for each channel, the three branches are input in parallel. Then, the aforementioned features are input into a sequence module consisting of two cascaded linear layers and a ReLU activation function. Adaptive weights are dynamically calculated and assigned to the MCP and TSO branches. Simultaneously, the adaptive weights y are subjected to softmax processing to normalize them to weight values ​​z between 0 and 1. Finally, the weights a and b of the two branches are dynamically extracted through the adaptive weight mechanism, added together, and output to the next encoder layer. The expression for calculating and outputting the branch weights is as follows:

[0029]

[0030]

[0031]

[0032]

[0033] In this context, the output y of the adaptive weights is a 2D vector representing the weight ratio, z represents the normalized weight value, and i, j, k, l represent the indices of each dimension of the input x.

[0034] Furthermore, the final fused features are input into an autoregressive decoder, replacing the traditional tracking head, to autoregressively predict the final target position, thus completing target tracking. This includes converting the bounding box into a discrete token sequence. It introduces special tokens "start" and "end" to represent the beginning and end of the sequence, respectively. Simultaneously, it transforms continuous coordinate values ​​on the search image into discrete vocabulary units and superimposes learnable positional codes to inject sequence order information. The decoder then indexes from these positional codes and sequentially generates and processes the target location. The values ​​are sequentially input into a masked multi-head self-attention mechanism, a multi-head self-attention mechanism, a feedforward network, and a multilayer perceptron. After processing by the feedforward network, a hidden layer representation is output. Then, through a linear projection layer and a softmax function, the most likely token is predicted from the preset positional encodings until the four coordinate tokens representing the bounding box are fully generated. It terminates with the special token "end".

[0035] Secondly, an RGB-T tracking system based on adaptive spatiotemporal cues includes:

[0036] The data acquisition module is configured to acquire image sequences containing both RGB and TIR modes, and to preprocess the acquired image sequences.

[0037] The feature extraction module is configured to perform basic feature extraction based on the preprocessed image sequence, including constructing a backbone network consisting of multiple cascaded encoder blocks based on the Vision Transformer framework;

[0038] The prompting learning module is configured to: generate effective prompt words based on the prompting learning method, extract semantic features at different levels, including parallel embedding of modal complementary prompters into the encoder blocks of the backbone network;

[0039] The time feature module is configured to: construct a time state observer based on the time information of historical frames, including generating the current time node marker, using time series features to improve the tracking process, and promoting the generation of subsequent time node markers;

[0040] The adaptive balancing module is configured to: construct an adaptive dynamic spatiotemporal module based on the spatiotemporal features of a parallel modal complementarity cue and a temporal state observer, cleverly integrating the dual advantages of temporal and spatial features. This includes introducing an adaptive weighting mechanism to effectively balance the features of the spatiotemporal branches;

[0041] The output module is configured to input the final fused features into the autoregressive decoder, convert the bounding box into discrete labels, and autoregressively predict the final target location to complete target tracking.

[0042] In summary, the present invention has the following beneficial technical effects:

[0043] 1. This invention proposes a time-state observation module that combines cue learning with historical time information and uses an autoregressive mechanism to generate new time markers. This process not only provides rich temporal features to guide the tracking process but also acts as a bridge for information transmission, effectively supporting target tracking at subsequent time nodes.

[0044] 2. This invention proposes an adaptive dynamic spatiotemporal module that cleverly integrates the dual advantages of temporal and spatial features, fully explores the continuity of time information and the complexity of spatiotemporal information, and uses an adaptive weighting mechanism to enable the model to more accurately capture and understand the continuous change patterns and complex characteristics of input features in the temporal and spatiotemporal domains, ensuring that the spatiotemporal branches achieve optimal balance, so as to achieve more accurate and stable tracking.

[0045] 3. This invention proposes an autoregressive decoder to replace the tracking head in receiving output features to predict the final target position. It converts bounding boxes into discrete marker sequences and combines RGB visual features with TIR features using a self-attention mechanism to predict the target position in an autoregressive manner, thereby improving target tracking accuracy and efficiency. Attached Figure Description

[0046] Figure 1 This is an overall structural diagram of an RGB-T tracking method based on adaptive spatiotemporal cues in an embodiment of the present invention.

[0047] Figure 2 This is a structural diagram of a modal complementarity cue model for an RGB-T tracking method based on adaptive spatiotemporal cues, as described in an embodiment of the present invention.

[0048] Figure 3 This is a structural diagram of the time state observation model of an RGB-T tracking method based on adaptive spatiotemporal cues in an embodiment of the present invention.

[0049] Figure 4 This is an adaptive dynamic spatiotemporal model structure diagram of an RGB-T tracking method based on adaptive spatiotemporal cues in an embodiment of the present invention.

[0050] Figure 5 This is a model structure diagram of an autoregressive decoder based on an adaptive spatiotemporal cue-based RGB-T tracking method in an embodiment of the present invention.

[0051] Figure 6 This is an evaluation comparison curve of an RGB-T tracking method based on adaptive spatiotemporal cues in an embodiment of the present invention on the LasHeR dataset with other open-source methods.

[0052] Figure 7 This is an evaluation comparison curve of an RGB-T tracking method based on adaptive spatiotemporal cues in an embodiment of the present invention on the RGBT234 dataset with other open-source methods.

[0053] Figure 8 This is an evaluation comparison curve of an RGB-T tracking method based on adaptive spatiotemporal cues in an embodiment of the present invention on the GTOT dataset with other open-source methods.

[0054] Figure 9 This is a visualization of the results of the present invention and current mainstream networks in this embodiment of the invention. Detailed Implementation

[0055] The present invention will be further described in detail below with reference to the accompanying drawings.

[0056] Example 1

[0057] Reference Figure 1 The RGB-T tracking method based on cross-modal fusion and adapter fine-tuning in this embodiment includes the following specific steps:

[0058] Acquire image sequences containing both RGB and TIR modes, and preprocess the acquired image sequences;

[0059] Basic feature extraction is performed based on the preprocessed image sequence, including the construction of a backbone network consisting of multiple cascaded encoder blocks based on the Vision Transformer framework;

[0060] Effective prompt words are generated based on prompt learning methods, and semantic features at different levels are extracted, including parallel embedding of modal complementary prompters into the encoder blocks of the backbone network.

[0061] A time state observer is constructed based on the time information of historical frames, including generating the current time node marker, using temporal features to improve the tracking process, and promoting the generation of subsequent time node markers.

[0062] An adaptive dynamic spatiotemporal module is constructed based on the spatiotemporal features of a parallel modal complementarity cue and a temporal state observer, cleverly integrating the dual advantages of temporal and spatial features. This includes introducing an adaptive weighting mechanism to effectively balance the features of the spatiotemporal branches.

[0063] The final fused features are input into the autoregressive decoder, which converts the bounding box into discrete labels and autoregressively predicts the final target location, thus completing target tracking.

[0064] Specifically, an RGB-T tracking method based on cross-modal fusion and adapter fine-tuning includes the following steps:

[0065] S1. Obtain an image sequence containing both RGB and TIR modes, and preprocess the obtained image sequence;

[0066] First, acquire bimodal image sequence datasets covering extremely complex scenes (such as strong light, occlusion, darkness, low resolution, fast motion, etc.), such as LasHeR, RGBT234, GTOT, RGBT210, etc. These datasets need to contain time-synchronized RGB and TIR image frames to ensure that the two modalities correspond to the same scene and target at the same time, providing basic data for subsequent cross-modal fusion.

[0067] Next, spatial alignment processing is performed on the acquired RGB and TIR image sequences. The spatial transformation matrix between the RGB and TIR images is calculated using the camera calibration parameters provided in the dataset or an image registration algorithm based on feature matching. The TIR image is then geometrically corrected based on this matrix to ensure that the spatial positions of the targets in the two modal images are completely matched, thus avoiding the problem of feature mismatch during subsequent feature fusion due to modal misalignment.

[0068] Then, template images and search images are extracted from the aligned image sequence: the template image is cropped from the initial frame of the image sequence or the target initialization frame specified by the user to initialize the target features; the search image is cropped from subsequent frames of the template frame with the predicted position of the target in the previous frame as the center, and contains the target candidate region to track the target position change in real time. At the same time, the template image and search image are adjusted to a uniform size to adapt to the subsequent network input.

[0069] Next, the processed RGB template image, RGB search image, TIR template image, and TIR search image are patched, each divided into the same number of non-overlapping patches of a fixed size to ensure consistent patch counts for the same type of image across different modalities. Then, each patch is linearly projectively encoded using a patch embedding layer (PatchEmbed) to convert the two-dimensional patch into a one-dimensional feature vector, generating a feature sequence. The image patch sequence for the visible light modality is as follows: =[ ]∈ The image patch sequence in infrared mode is =[ ]∈ ,in, The encoding sequence for the RGB template patch. The encoded sequence for the RGB search image patch. The encoded sequence of the TIR template patch. The encoded sequence for the TIR search image patch. and These are the number of tokens for the template and the search image, respectively. For the token dimension.

[0070] S2. Based on the preprocessed image sequence, perform basic feature extraction, including constructing a backbone network composed of multiple cascaded encoder blocks based on the Vision Transformer framework;

[0071] Based on the preprocessed image sequence, basic features are extracted, and OSTrack, with Vision Transformer as its backbone, is selected. OSTrack is a single-stage target tracking algorithm based on the Transformer framework, supporting both single-modal and multi-modal target tracking. The RGB dominant modality patch sequence obtained from the patch embedding layer in step S1 is input into the encoder of the backbone network. The encoder of the backbone network consists of multiple cascaded encoder blocks, each containing multi-head self-attention (MSA), a normalization layer, and a feedforward convolutional network (FFN). For the RGB modality, the output relationship between the l-th encoder block and the (l-1)-th encoder block is expressed by the following formula:

[0072] ,

[0073] in, For the first The output features of each encoder block are first subjected to layer normalization (LN) processing, and then input into a multi-head self-attention (MSA) module for computation to capture long-range dependencies between features. The processing results are then compared with... Perform residual connections to obtain intermediate features after multi-head self-attention and residual connections. Then through the formula FFN is a feedforward convolutional network that first processes intermediate features. Perform layer normalization, then input FFN and compare the results with... Perform residual connections to obtain the output features of the l-th encoder block. The encoder block processing process for TIR mode is completely consistent with that for RGB mode. Through multiple serial encoder blocks, the deep features of different modes are finally extracted.

[0074] S3. Generate effective prompt words based on prompt learning methods, extract semantic features at different levels, including parallel embedding of modal complementary prompters into the encoder blocks of the backbone network;

[0075] When extracting multi-level semantic features, the first step is to ensure that all parameters of the encoder blocks in the backbone network are fixed, meaning they are not updated with gradients during training, thus maintaining the stability and effectiveness of the backbone network. Then, a modal complementarity cue module is added to this fixed encoder framework, such as... Figure 2 As shown, these prompts are embedded in parallel into each encoder block.

[0076] The modal complementarity cueing mechanism works by reshaping (or deforming) the output features of the previous encoder and the features generated by the cueing mechanism along the channel dimension. This deformation aims to better integrate cross-modal information. Next, the deformed RGB features undergo spatial focusing to enhance the feature representation of important regions through an attention mechanism, ensuring that important information is not lost. Following this, the focused RGB features are fused with thermal infrared (TIR) ​​features, combining complementary information from different modalities.

[0077] The fused features undergo another channel-wise deformation to restore their original shape, enabling more efficient transmission through the encoder. Finally, the generated channel cue signals are fed into the encoder's original input, serving as new input signals to aid in more accurate feature extraction and representation. Through this series of operations, the model can better fuse multimodal information, improving its overall performance.

[0078] ,

[0079] ,

[0080] In this process This represents the output features generated by the Modal Complementary Cue (MCP). Specifically, the output features of the previous encoder layer are represented as follows: The features from the output of the Modal Complementary Cue (MCP) are These two features will be used for feature fusion and processing in the next layer. The encoder input prompts and dominant RGB modal features of the current layer are represented as follows: This is actually a feature representation obtained by combining the input prompts from the current layer encoder with RGB feature information. In this process, the output of the previous layer encoder and the features generated by the prompts jointly influence the feature extraction of the current layer, thereby enhancing the integration of multimodal information and improving the encoder's ability to perceive and express features of different modalities.

[0081] S4. Construct a time state observer based on the time information of historical frames, including generating the current time node marker, using time series features to improve the tracking process, and promoting the generation of subsequent time node markers;

[0082] When constructing a temporal state observer based on historical frame time information, the first step is to select a state-space model that allows the model's parameters to change dynamically, enabling it to adaptively adjust according to the characteristics of the input data. Specifically, the two input modalities... =[ ]and =[ It contains spatiotemporal features from RGB and thermal infrared modes, among which and These represent the visible light characteristics of each mode, while and These represent their infrared characteristics. By extracting information from the characteristics of these two modalities, a state vector containing historical frame time information can be generated.

[0083] At this point, the model generates two temporal information vectors using historical information from the previous n frames. and These correspond to RGB and TIR modes, respectively. This historical time information will serve as a guide to help predict the target state at the current time point.

[0084] To ensure that the target information at the current time point can be correctly imported, the system will insert an empty token into the time information. The purpose of this empty token is to store the target state information at the current time point, which is then used as new time information in subsequent frames.

[0085] Next, the system processes the input features. First, it processes the input features... =[ ]and =[ and historical time information and Discretization is performed. The discretization process involves converting continuous time information into discrete time steps, thereby enabling the model to more clearly track changes in the time state.

[0086] On the discretized features, the system uses one-dimensional convolution to extract local spatiotemporal features, a step that helps enhance the temporal dependence of the features. Next, a linear projection operation is performed to map the convolutional features to a high-dimensional space, generating feature matrices B, C, and D. These matrices carry the relationship between the current frame and historical frames, as well as cross-modal information.

[0087] Matrix C is further exchanged and fused with supplementary information from another modality (such as TIR or RGB modality). This process helps enhance the complementarity of features and the ability to convey information. Finally, the fused information generates a time token for the current time point, which accurately reflects the target state at the current time. The discrete expression is as follows:

[0088]

[0089]

[0090]

[0091]

[0092] in, These are predefined time-scale parameters, continuous parameters. and The corresponding discrete parameters mapped to the discrete space and , ,and It is the length of the visual marker. The discrete process is the same as described above. and The final output is obtained by adding them together.

[0093] Finally, the system adds the generated current time token to the total time set T, forming a time information sequence containing the current frame and historical frames, such as... Figure 3 As shown, this total time set is input into the processing of the next frame, serving as the basis for the next state prediction. Through this series of operations, the model, guided by multimodal inputs and historical time information, can more accurately predict and adjust the target state, achieving more refined temporal state modeling.

[0094] S5. An adaptive dynamic spatiotemporal module is constructed based on the spatiotemporal features of a parallel modal complementarity cue and a temporal state observer, cleverly integrating the dual advantages of temporal and spatial features. This includes introducing an adaptive weighting mechanism to effectively balance the features of the spatiotemporal branches;

[0095] First, the input RGB and TIR modal features are defined as a set x=[ ],in and These are the RGB and TIR modal feature representations of the current layer, respectively. For example... Figure 4 As shown, in order to effectively fuse these two modalities, the features of each modality are first linearized. The features of each modality are passed through a linear mapping layer to calculate the average value of each channel (i.e., global average pooling). This step aims to reduce the computational complexity of high-dimensional features while preserving important spatiotemporal information.

[0096] Next, these averaged features are fed in parallel into a three-branch network. The three-branch network processes the RGB and TIR features separately and extracts their respective spatiotemporal information. Each branch is mapped through two cascaded linear layers and then nonlinearly transformed using the ReLU activation function to generate different feature representations. The output of this stage is a preliminary spatiotemporal representation of the features of each modality.

[0097] After obtaining these initial spatiotemporal features, the crucial adaptive weight calculation process follows. The goal of this process is to dynamically assign different weight values ​​to the MCP and TSO branches, allowing for flexible adjustment of their contributions during fusion. Specifically, the model calculates the adaptive weight y for each branch based on the specific content of the input features. These weights represent the contribution of each modality to target tracking at the current time point.

[0098]

[0099] To calculate the adaptive weights, the system uses a softmax operation to normalize the original weights of the two branches, ensuring that the weight values ​​z fall between 0 and 1, thereby ensuring the reasonableness of the weights.

[0100]

[0101] The output of this softmax processing is the adaptive weight value for each branch, denoted as a and b, which represent the relative importance of the MCP and TSO branches, respectively.

[0102]

[0103] Where i, j, k, l represent the indices of each dimension of the input x.

[0104] Finally, after obtaining the normalized adaptive weights, the system performs a weighted fusion of the outputs of the MCP and TSO branches based on these weight values. The weighted features are then passed to the next encoder layer as input for the next step of feature processing and target state prediction. In this way, the system can flexibly adjust the weights of each branch according to the input features at different time points and in different modalities, thereby improving tracking accuracy and robustness.

[0105]

[0106] In summary, this method effectively integrates the spatiotemporal features of RGB and TIR modes by introducing an adaptive weighting mechanism, making full use of the advantages of the temporal and spatial domains, improving the feature representation and dynamic response capabilities during target tracking, and thus enhancing tracking performance in complex environments.

[0107] S6. Input the final fused features into the autoregressive decoder, convert the bounding box into discrete labels, and autoregressively predict the final target position to complete target tracking.

[0108] First, the target bounding box is represented as a discrete sequence of tokens, containing the center coordinates of the target. and ,width and height Four values. For example... Figure 5 As shown, special tokens "start" and "end" are introduced to represent the beginning and end of the sequence, respectively. To transform continuous image coordinate values ​​into discrete values ​​suitable for sequence model processing, the method discretizes these coordinate values ​​into a set of vocabulary units, each corresponding to a different coordinate point on the image. Furthermore, to preserve the sequence's order information, learnable positional encodings are superimposed, ensuring the order of each token is explicitly identified.

[0109] The decoder first indexes information from the positional encoding and then generates four coordinates step by step according to the positional order. In this process, the input token sequence first passes through a masked multi-head self-attention mechanism. This mechanism ensures that the model considers information from previous tokens during generation by calculating the relationships between tokens in the sequence, thus avoiding information leakage. Next, the token sequence undergoes further feature optimization through a standard multi-head self-attention mechanism to capture global dependencies between different tokens. Then, it is further processed by a feedforward network to extract more refined features, and finally processed and optimized using a multilayer perceptron.

[0110] After processing by the feedforward network, the decoder outputs a hidden layer representation, showing the prediction of the four coordinate values ​​of the target bounding box. This prediction is mapped to the output space through a linear projection layer, and the most likely token for the current step is calculated using the Softmax function. This process continues until all four coordinate tokens representing the bounding box are generated. The sequence ends with an "end" token.

[0111] In this way, the model can dynamically adjust the information flow during the generation process, accurately capture the temporal dependencies in the sequence using the self-attention mechanism, and combine position encoding to ensure that the order information of the coordinates is effectively transmitted, thereby accurately generating the four coordinates of the target bounding box.

[0112] Example 2

[0113] The difference between this embodiment and Embodiment 1 is that this embodiment provides a specific experiment of an RGB-T tracking method based on adaptive spatiotemporal cues:

[0114] The experimental environment configuration for this invention is shown in Table 1. This invention was implemented on Ubuntu 20.04 using PyTorch 1.13.0 and Python 3.9. It was trained for 60 rounds on two NVIDIA RTX 4090 GPUs. The search region size was adjusted to 256×256, and the template size was adjusted to 128×128. In each training round, the batch size was 16, and the total batch size was 32. The AdamW optimizer was used, and the learning rate of other modules was set to 4× during training. The learning rate during decoder training is 4× After 48 training rounds, the number of samples decreased by 10 times, with 60k samples collected in each round.

[0115] Table 1 Experimental Environment Configuration

[0116]

[0117] The experimental datasets used were the LasHeR dataset, the RGBT234 dataset, the GTOT dataset, and the RGBT210 dataset.

[0118] LasHeR is a large-scale RGB-T dataset, comprising 979 pairs of training video sequences and 245 pairs of test video sequences. It includes 19 different challenge attributes: no occlusion (NO), partial occlusion (PO), complete occlusion (TO), transparent occlusion (HO), out of view (OV), low light (LI), high light (HI), sudden light change (AIV), low resolution (LR), distortion (DEF), background clutter (BC), similar appearance (SA), hot crossover (TC), motion blur (MB), camera movement (CM), frame loss (FL), fast motion (FM), aspect ratio change (SV), and aspect ratio change (ARC). The main evaluation metrics are precision (PR), normalized precision (NPR), and success rate (SR).

[0119] RGBT234 contains 234 pairs of video sequences with 12 challenging attributes: No Occlusion (NO), Partial Occlusion (PO), Heavy Occlusion (HO), Low Light (LI), Low Resolution (LR), Hot Crossover (TC), Deformation (DEF), Fast Motion (FM), Scale Variation (SV), Motion Blur (MB), Camera Movement (CM), and Background Clutter (BC). The main evaluation metrics are Precision (PR) and Success Rate (SR).

[0120] GTOT is a classic RGB-T tracking benchmark consisting of 50 short-term sequences. The main evaluation metrics are accuracy (PR) and success rate (SR).

[0121] The RGBT210 dataset contains 210 video sequences, the longest of which contains 8k frames, making it suitable for long-term tracking tasks. The video sequences are aligned without any pre- or post-processing to ensure perfectly aligned images. Detailed occlusion annotations are also provided to test the occlusion robustness of various object tracking algorithms. The main evaluation metrics are precision (PR) and success rate (SR).

[0122] Table 2 and Figures 6-8 The results are the training and testing results of each algorithm on four datasets.

[0123] On the LasHeR dataset, as shown in Table 2, our invention outperforms both Siam-based and MDNet-based trackers. Compared to Transformer-based trackers, our invention achieves 3.3% and 3.4% higher PR and SR than QueryTrack, respectively; 2.1% and 1.8% higher PR and SR than MIXRGBX, respectively; and 2.8% and 1.3% higher PR and SR than SDSTrack, respectively. Compared to VIPT, our invention achieves 4.2% and 2.9% higher PR and SR, respectively. Furthermore, our invention exhibits significantly better tracking performance than other unmentioned Transformer-based methods. Using the LasHeR toolkit, we plotted evaluation curves of our method (PR, NPR, and SR) against four other open-source methods, as shown in Table 2. Figure 6 As shown in the figures, these curves visually demonstrate the superior competitive performance of this invention.

[0124] This invention was compared with several excellent RGB-T trackers on the RGBT234 dataset, as shown in Table 2. This invention demonstrates superior performance compared to Siam-based and MDNet-based trackers. Compared to trackers based on the Tracnsformer framework, this invention exhibits superior performance, with PR and SR exceeding VIPT by 3.4% and 2.3%, respectively, and exceeding SDSTrack by 2.2% and 1.6%, respectively, in PR and SR. Compared to STMT, this invention improves PR and SR by 0.5% and 0.3%, respectively. Furthermore, using the LasHeR toolbox, we plotted the evaluation curves of this invention (PR and SR) against four other open-source methods, as shown in Table 2. Figure 7 As shown in the figures, these curves visually demonstrate the superior competitive performance of this invention. This invention exhibits significantly better tracking performance than other unmentioned Transformer-based methods.

[0125] Table 2 compares the algorithms on the LasHeR dataset, RGBT234 dataset, and RGBT210 dataset;

[0126]

[0127] This invention was compared with many excellent RGB-T trackers on the RGBT210 dataset, as shown in Table 2. Compared with SiamMGT and CAT++, this invention improved PR and SR by 4.8% / 1.8% and 2.9% / 5.4%, respectively. Experimental results show that this invention has superior tracking performance in all aspects.

[0128] This invention was evaluated against many excellent RGB-T trackers on the GTOT dataset, such as... Figure 8 As shown. Compared to MANet, the present invention improves PR and SR by 2.0% and 4.4%, respectively. Compared to DAPNet, the present invention improves PR and SR by 3.2% and 6.1%, respectively, demonstrating its excellent tracking performance.

[0129] Table 3 compares and evaluates the LasHeR dataset across five other methods using 18 attributes.

[0130]

[0131] Table 4 compares and evaluates the LasHeR dataset across 18 attributes with five other methods.

[0132]

[0133] Table 3 evaluates 18 attributes on the LasHeR dataset. This invention evaluates ASTP, TransT, UnTrack, SDSTrack, BAT, and VIPT on 18 challenging attributes of the LasHeR dataset. Compared to VIPT, ASTP outperforms VIPT in PR and SR by 3.1% and 3.0%, 2.2% and 1.4%, and 7.9% and 5.4% respectively in challenges such as partial occlusion (PO), background clutter (BC), and low resolution (LR), demonstrating more competitive performance under extreme conditions. Compared to UnTrack, ASTP outperforms VIPT in attributes such as high illumination (HI), similar appearance (SA), and fast motion (FM) by 4.5% and 4.6%, 8.1% and 5.8%, and 6.5% and 4.8% respectively, demonstrating a significant advantage in tracking performance even under abnormal background conditions. Furthermore, our method exhibits similar tracking performance to BAT across multiple challenging attributes, demonstrating the superior tracking performance of ASTP.

[0134] Table 4 evaluates 12 attributes of the RGBT234 dataset. This invention evaluates ASTP, TBSI, APFNet, STMT, mfDiMP, and VIPT on 12 challenging attributes of the RGBT234 dataset. ASTP outperforms other trackers on many attributes. Compared to VIPT, ASTP improves PR / SR metrics by 4.2% / 2.7%, 5.1% / 4.2%, and 2.4% / 1.7% under challenging conditions such as complete occlusion (NO), low illumination (LI), and camera movement (CM), respectively, indicating stronger competitiveness and tracking performance when the target encounters non-ideal conditions. Compared to the STMT method, ASTP improves PR / SR metrics by 4.4% / 2.0% and 4.7% / 2.7% under challenging conditions such as heavy occlusion (HO) and background clutter (BC), respectively, indicating that it maintains superior performance even under complex background conditions. However, ASTP's PR and SR performance were relatively low under the fast-moving (MB) condition, indicating that the model still needs to further improve its robustness and feature discrimination ability in such scenarios.

[0135] To verify the effectiveness of each module of the model, this invention conducted a detailed ablation study on each module on the LasHeR dataset using precision (PR), normalized precision (NPR), and success rate (SR) metrics. The results are shown in Table 5.

[0136] Table 5 compares the number of parameters and indicators for each module in the model;

[0137]

[0138] The impact of the S-Decoder module, TSO module, and ADST module on tracker performance was verified separately. Table 4 shows that the model improved overall PR, NPR, and SR by 4.2%, 3.7%, and 2.9% compared to the baseline, respectively. Specifically, the S-Decoder module and TSO module contributed 1.4%, 1.2%, and 1.0% to the model's PR, NPR, and SR, respectively, and 2.3%, 1.8%, and 1.4% to the model's SR, respectively. It is also evident that relying solely on the S-Decoder is far from sufficient, but the cascading of the TSO and ADST modules yielded significant results, contributing a total of 3.8%, 2.9%, and 2.7% to the model's performance. This underscores the necessity of using the ADST module to dynamically adapt to the fusion of multimodal features in the search region.

[0139] We also conducted a detailed ablation study on each module on the RGBT234 dataset using the precision (PR) and success rate (SR) metrics, and the results are shown in Table 5.

[0140] Table 6 shows the ablation experiments of ASTP on the RGBT234 dataset.

[0141]

[0142] We verified the impact of the S-Decoder module, TSO module, and ADST module on tracker performance. Table 5 shows that the model improved overall PR and SR by 3.4% and 2.3% compared to the baseline. Specifically, the TSO module contributed 1.5% and 0.4% to the model's PR and SR, respectively. It is also evident that relying solely on the S-Decoder and TSO modules is insufficient; the cascaded ADST module achieved significant results, contributing 1.9% and 1.0% to the model's PR and 2.5% and 1.7%, respectively. This demonstrates the crucial role of the ADST module in dynamically adapting the spatiotemporal relationships of the search region.

[0143] like Figure 9 As shown, to realistically and intuitively demonstrate the applicability of our method in real-world tracking scenarios, this paper selects four representative video sequence pairs from the LasHeR dataset and compares the tracking results of the latest trackers SDSTrack and VIPT with ASTP, mainly covering complex scenarios such as dense crowds, occlusion, and extreme conditions. In video sequence 1, facing insufficient lighting and occlusion issues, ASTP exhibits superior performance compared to other trackers. In video sequence 2, due to dense crowds and complex environments, some trackers experience brief target loss in certain frames, failing to accurately track the target position, while ASTP remains stable, fully demonstrating its excellent robustness. In video sequence 3, although the target is occluded by other objects, some trackers experience target loss, but ASTP consistently tracks the target accurately. In video sequence 4, other trackers experience brief target loss due to the target being far and small, but ASTP still maintains good tracking ability, demonstrating its advantage under small target conditions. In summary, these experiments verify the effectiveness and broad applicability of our method under various complex conditions.

[0144] This invention proposes an RGB-T method based on adaptive spatiotemporal cues (ASTP). First, TSO combines current target features with historical time information and uses an autoregressive mechanism to generate new time stamps, effectively providing temporal information for subsequent target tracking while using temporal features to guide the tracking process. Second, ADST utilizes an adaptive weighting mechanism to more accurately capture and understand the continuous changes and complex characteristics of input features in the temporal and spatiotemporal domains, achieving an optimal balance between the two branches for accurate and stable tracking. Finally, the S-Decoder receives output features through the decoder to predict the target position, transforming the bounding box into a discrete marker sequence. It then combines RGB visual features and TIR features using a self-attention mechanism, employing an autoregressive approach for target position prediction, improving tracking accuracy and efficiency. Experimental results show that ASTP achieves PR / SR of 69.3% / 55.4% and 87.0% / 64.1% on the LasHeR and RGBT234 datasets, and PR / SR of 91.4% / 76.8% and 85.1% / 61.5% on the GTOT and RGBT210 datasets, respectively, providing a reliable solution to the challenges in RGB-T tracking.

[0145] Example 3

[0146] This embodiment provides an RGB-T tracking system based on adaptive spatiotemporal cues;

[0147] The data acquisition module is configured to acquire image sequences containing both RGB and TIR modes, and to preprocess the acquired image sequences.

[0148] The feature extraction module is configured to perform basic feature extraction based on the preprocessed image sequence, including constructing a backbone network consisting of multiple cascaded encoder blocks based on the Vision Transformer framework;

[0149] The prompting learning module is configured to: generate effective prompt words based on the prompting learning method, extract semantic features at different levels, including parallel embedding of modal complementary prompters into the encoder blocks of the backbone network;

[0150] The time feature module is configured to: construct a time state observer based on the time information of historical frames, including generating the current time node marker, using time series features to improve the tracking process, and promoting the generation of subsequent time node markers;

[0151] The adaptive balancing module is configured to: construct an adaptive dynamic spatiotemporal module based on the spatiotemporal features of a parallel modal complementarity cue and a temporal state observer, cleverly integrating the dual advantages of temporal and spatial features. This includes introducing an adaptive weighting mechanism to effectively balance the features of the spatiotemporal branches;

[0152] The output module is configured to input the final fused features into the autoregressive decoder, convert the bounding box into discrete labels, and autoregressively predict the final target location to complete target tracking.

[0153] The above are all preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for RGB-T tracking based on adaptive spatio-temporal cues, characterized in that, The method comprises the following specific steps: S1, obtaining an image sequence containing two modalities of RGB and TIR, and preprocessing the obtained image sequence; S2, performing basic feature extraction based on the preprocessed image sequence, comprising constructing a backbone network composed of multiple series of encoder blocks based on a Vision Transformer framework; S3, generating effective prompt words based on a prompt learning method, and extracting different levels of semantic features, comprising embedding a modal complementary prompter in parallel to the encoder blocks of the backbone network; S4, constructing a time state observer based on the time information of the historical frames, comprising generating a current time node label, improving the tracking process using time sequence features, and promoting the generation of subsequent time node labels; S5, constructing an adaptive dynamic space-time module based on the space-time features of the parallel modal complementary prompter and the time state observer, and skillfully combining the dual advantages of time domain features and spatial domain features; It includes introducing an adaptive weight mechanism to effectively balance the features of the space-time branches; S6, inputting the final fused features into a self-recursive decoder to convert the bounding box into a discrete label and recursively predict the final target position to complete target tracking. 2.The RGB-T tracking method based on adaptive spatio-temporal cues according to claim 1, wherein, The preprocessing of the obtained image sequence comprises spatial alignment processing of the RGB and TIR image sequences, extracting template images and search images from the image sequence, dividing the processed RGB template images, RGB search images, TIR template images and TIR search images into the same number of patches, encoding the patches through linear projection, converting two-dimensional patches into one-dimensional feature vectors, and generating visible light modality and infrared modality feature sequences. 3.The RGB-T tracking method based on adaptive spatio-temporal cues according to claim 1, wherein, The basic feature extraction based on the preprocessed image sequence includes inputting the RGB dominant modality patch sequence obtained by the patch embedding layer into the encoder block of the backbone network; firstly, multi-head self-attention calculation is performed on the input features; layer normalization is performed on the output of the multi-head self-attention; residual concatenation is performed between the layer-normalized features and the input features to obtain the first intermediate features; the first intermediate features are input into a feedforward convolutional network for nonlinear transformation; residual concatenation is performed between the output of the feedforward convolutional network and the first intermediate features to obtain the output features of the encoder block; and so on. The encoder block and the first encoder block The output relationship of the encoder block can be expressed as: ； ； wherein, is the output feature of the th encoder block, is the layer normalization, is the multi-head self-attention, is the feed-forward convolutional network, is the intermediate feature after the multi-head self-attention and the residual connection. 4.The RGB-T tracking method based on adaptive spatio-temporal cues according to claim 1, wherein, The prompt learning method generates effective prompt words and extracts different levels of semantic features, including fixing all parameters in the encoder blocks of the backbone network to not participate in gradient update, and embedding a modal complementary prompter in parallel to the encoder blocks of the backbone network; the modal complementary prompter performs channel deformation on the output features of the previous layer encoder and the prompter, then performs spatial focusing on the deformed RGB features and fuses them with TIR features, and then performs channel deformation again to restore the original feature form and generate channel prompts, which are input into the original input of the encoder; the process expression is: ； ； wherein, the output from the MCP, the outputs from the previous layer encoder and the MCP, respectively, denotes the current encoder input prompt words and the features of the dominant RGB modality.

5. The RGB-T tracking method based on adaptive spatio-temporal cues according to claim 1, characterized in that, The time information based on historical frames constructs a time state observer, including selecting a state space model to make the parameters of the model dependent on the input data, extracting features from the input from two modalities [ ] and [ ] to generate time information learned from the previous n frames and Guide the target state of the current time node, while inserting an empty token , for storing the target information of the current time node; then and Discretize the time information, and generate a feature matrix by one-dimensional convolution and linear projection on the input , generate the current time token by combining the supplementary information from another modality through the exchange matrix C; Then add the current time information token to the total time set T, and input it into the next frame; the discretization processing expression is: ； ； ； ； wherein, is a predefined time scale parameter, a continuous parameter and are mapped to respective discrete parameters in a discrete space and , , and is the length of the visual marker; the discrete process is the same as above, and are added to obtain the final output.

6. The RGB-T tracking method based on adaptive spatio-temporal cues according to claim 1, characterized in that, The parallel modality complementary prompter and time state observer based space-time feature construction adaptive dynamic space-time module includes introducing an adaptive weight mechanism, skillfully combining the dual advantages of time domain features and spatial domain features, and the adaptive weight method defines two modal input feature tokens as a set x=[ ] is input into the linearization layer and the average value of each channel is generated, then three branches are input in parallel, then the above features are input into a sequence module composed of two linear layers (Linear) and ReLU activation functions, the adaptive weights allocated to the MCP branch and the TSO branch are dynamically calculated, and the adaptive weight y is subjected to softmax processing to normalize it into a weight value z between 0 and 1, and finally the weights a and b of the two branches are dynamically extracted through the adaptive weight mechanism and added to the next layer encoder; the branch weight calculation and output expression is: ； ； ； ； ； Wherein, the output y of the adaptive weight is a 2-dimensional vector representing the weight proportion, z represents the normalized weight value, and i, j, k, l represent the index of each dimension of the input x.

7. The RGB-T tracking method based on adaptive spatio-temporal cues according to claim 1, characterized in that, The final fusion feature is input into an autoregressive decoder instead of a traditional tracking head to predict the final target position autoregressively, completing target tracking; including converting the bounding box into a discrete token sequence And introducing special tokens "start" and "end" to represent the beginning and end of the sequence respectively, while converting the continuous coordinate values on the search image into discrete vocabulary units and superimposing learnable position encodings to inject the sequence order information, then the decoder indexes and sequentially generates and processes the Value of the target position, which is sequentially input into the masked multi-head self-attention mechanism, multi-head self-attention mechanism, feedforward network and multi-layer perceptron, and finally outputs a hidden layer representation after feedforward network processing, and through a linear projection layer and Softmax function, the current most likely token is predicted from the preset position encoding, until four coordinate tokens representing the bounding box are generated and terminated by the special token "end".

8. An adaptive spatio-temporal cueing based RGB-T tracking system, performing the method of any of claims 1-7, characterized in that, It includes: Data acquisition module: obtaining an image sequence containing two modalities of RGB and TIR, and preprocessing the obtained image sequence; Feature extraction module: performing basic feature extraction based on the preprocessed image sequence, comprising constructing a backbone network composed of multiple series of encoder blocks based on a Vision Transformer framework; Prompt learning module: generating effective prompt words based on a prompt learning method, and extracting different levels of semantic features, comprising embedding a modal complementary prompter in parallel to the encoder blocks of the backbone network; The time feature module is configured to construct a time state observer based on time information of historical frames, including generating a current time node label, improving a tracking process by using time sequence features, and promoting generation of a subsequent time node label. The adaptive balance module is configured to construct an adaptive dynamic space-time module based on space-time features of a parallel modal complementary prompter and the time state observer, and to skillfully combine dual advantages of time domain features and space domain features. The adaptive balance module includes introducing an adaptive weight mechanism to effectively balance features of the space-time branches. The output module is configured to input the final fused features into a self-recurrent decoder, convert a bounding box into a discrete label, and self-recursively predict a final target position to complete target tracking.

Images