An RGB-T Target Tracking Method and System Based on Spatiotemporal State Evolution

Abstract

This invention relates to an RGB-T target tracking method and system based on spatiotemporal state evolution. The method constructs a dual-branch RGB-T target tracking model, where the RGB and TIR branches share ViT encoder weights. An iterative processing framework is employed, using updatable context memory tokens to transmit spatiotemporal context information between frames. In each iteration, RGB and TIR modal features are extracted separately, and then cross-modal temporal context modeling is performed via a modality-aware temporal Mamba module. This module achieves long-term, target-aware representation learning through a cross-modal coupled state transition mechanism and cue guidance strategy. Finally, a cross-modal Mamba aggregation module fuses intra-modal and inter-modal features in the spatial dimension, and the target position is output via a prediction head. This invention achieves persistent cross-modal state evolution with linear complexity, effectively fusing spatiotemporal features of visible light and infrared light, and improving the robustness and accuracy of tracking in complex scenarios such as rapid target movement, occlusion, or modal degradation.

Description

Technical Field

[0001] This invention belongs to the field of image processing and analysis technology, specifically relating to an RGB-T target tracking method and system based on spatiotemporal state evolution. Background Technology

[0002] Visual object tracking is one of the fundamental tasks in computer vision, especially under challenging conditions such as low light, occlusion, and cluttered backgrounds, where traditional single-modal RGB tracking methods often suffer from performance limitations. To improve robustness, RGB-T tracking technology offers an effective solution by fusing the complementary characteristics of RGB (visible light) and thermal infrared (TIR) ​​modes. The RGB mode provides rich color and texture information, while the TIR mode performs well in low-light or thermally significant scenes. Combining the two enables the utilization of spatial diversity and temporally complementary contexts.

[0003] Early RGB-T tracking methods primarily focused on spatial complementarity fusion. These methods relied solely on the appearance information of the target object, enhancing the target representation through spatial feature interactions between modalities to perform tracking matching and prediction tasks. For example, L. Zhang et al., in their paper "Multi-modal fusion for end-to-end RGBT tracking" (Proceedings of the IEEE / CVF International Conference on Computer Vision workshops, 2019), constructed three end-to-end fusion frameworks by concatenating multimodal features to facilitate the fusion of multimodal data. C. Li et al., in their paper "Challenge-aware RGBT tracking" (European Conference on Computer Vision, pp. 222-237), designed fusion strategies for specific multimodalities to address challenges such as illumination variations or occlusion. In their paper "RGBT tracking via multi-adapter network with hierarchical divergence loss" (IEEE Transactions on Image Processing, 2021, vol. 30, pp. 5613-5625), A. Lu et al. employed a multi-adapter network and hierarchical divergence loss to fuse modal features. Y. Xiao et al., in their paper "Multi-zone transformer based on self-distillation for facial attribute recognition" (IEEE International Conference on Automatic Face and Gesture Recognition, 2023, pp. 1-7), introduced an attribute fusion mechanism, thereby enhancing its performance in specific scenarios.

[0004] While these methods optimize modal interactions in the spatial dimension, they neglect deep modeling of the temporal context, leading to unstable performance in sequences with frequent target appearance drift, trajectory changes, or environmental interactions. To address these issues, RGB-T methods incorporating temporal modeling have emerged in recent years. For example, H. Wang et al., in their paper "Temporaladaptive RGBT tracking with modality prompt" (Proceedings of the AAAIConference on Artificial Intelligence. 2024, pp.5436-5444), incorporated historical information through online branching to enhance tracking robustness. Z. Zeng et al., in their paper "Cross Fusion RGB-TT Tracking with Bi-directional Adapter" (arXiv preprint arXiv:2408.16979, 2024), employed cross-fusion RGB-T tracking and a bidirectional adapter to update the latest appearance information of the target object and improve model performance. In their paper "Exploiting multimodal spatial temporal patterns for video object tracking" (Proceedings of the AAAI Conference on Artificial Intelligence, 3581-3589), X. Hu et al. generated multimodal temporal information tokens using a temporal state generator to convey contextual information. However, these methods are often limited to short-term window temporal modeling, constrained by computational limitations, and unable to achieve long-range semantic propagation. Furthermore, existing temporal modeling is mostly performed in later stages of the network (such as the head layer), leading to the loss of fine-grained modality-specific cues in the early backbone layers, thus limiting the persistence and robustness of cross-modal tracking.

[0005] While Transformer-based models excel at capturing long-range dependencies, their quadratic complexity limits their extension to long video sequences. In contrast, State-space models (SSMs) such as Mamba maintain continuously evolving latent states with linear complexity, providing an efficient spatiotemporal modeling framework. However, standard Mamba's unidirectional causal design and modality-independent state updates cannot support bidirectional reasoning and cross-modal interaction, making it difficult to meet the requirements of persistent semantic propagation in RGB-T tracking. Existing Mamba-based tracking methods, such as the paper "MambaVT: Spatio-Temporal Contextual Modeling for Robust RGB-T Tracking" by S. Lai et al. (IEEE Transactions on Circuits and Systems for Video Technology, pp. 1-1), which first introduced Mamba for target tracking, utilizes Mamba to model remote contextual information, achieving comprehensive utilization of spatiotemporal information. In their paper "Exploiting multimodal spatial temporal patterns for video object tracking" (Proceedings of the AAAI Conference on Artificial Intelligence, 2025, pp. 3581-3589), X. Hu et al. combined complementary information from another modality by exchanging the C matrix parameters in the state-space equation. X. Li et al., in their paper "MambaLCT: Boosting tracking via long-term context statespace model" (Proceedings of the AAAI Conference on Artificial Intelligence, 2025, pp. 4986-4994), introduced a contextual Mamba modeling module to explore the complete changes in the target object. However, existing methods are only based on simple utilization of Mamba, and the fusion of Mamba-based multimodal tracking methods is relatively crude, lacking in-depth modeling of complex dependencies between modalities. Furthermore, the limitations of the inherent SSM mechanism in multimodal target tracking have not been fully explored.

[0006] Existing RGB-T tracking methods are limited to short-term window time modeling, resulting in context truncation and weakening of long-range dependencies; modal interactions are mostly static or late-stage fusion, causing the loss of early fine-grained modal-specific context; at the same time, the computational efficiency of Transformer cannot efficiently handle the cross-modal semantic evolution of long sequences; and methods based on Mamba modeling lack dynamic adaptation mechanisms, making it difficult to cope with modal imbalance and noise interference. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the prior art and provide an RGB-T target tracking method and system based on spatiotemporal state evolution. This invention can effectively integrate the spatiotemporal features of visible light and infrared images and achieve robust target tracking under conditions of rapid target movement, partial occlusion, or modal degradation.

[0008] To achieve the above objectives, the technical solution adopted by this invention is: an RGB-T target tracking method based on spatiotemporal state evolution, which constructs an RGB-T target tracking model. The RGB-T target tracking model adopts a dual-branch architecture including an RGB branch and a TIR branch, wherein the RGB branch and the TIR branch share the same ViT encoder weights. The RGB-T target tracking model employs an iterative processing framework throughout the tracking process, performing tracking iterations frame by frame along the time series. Each iteration corresponds to the input of a search image frame, and spatiotemporal context information is propagated through context memory tokens between iterations. The method includes the following steps:

[0009] 1) Input Acquisition Stage: Acquire RGB modal data and TIR modal data as input for subsequent input model feature extraction; each modal input includes template image sequence, search image and context memory tokens corresponding to each modality. Context memory tokens are initialized in the first frame, encode local historical context within the modality and global cues across modalities, and are updated and passed throughout the tracking process to guide the spatiotemporal propagation of subsequent iterations;

[0010] 2) Iterative Input Preparation: In subsequent iterations, the RGB modal search image and TIR modal search image of the current frame are input along the timeline; the template image sequence remains unchanged in each iteration and is passed as a reference template; the context memory tokens are passed from the previous iteration to the current iteration after being updated, and combined with the template image sequence and the current search image to form a complete modal input including the template image sequence, the search image and the context memory tokens, so as to capture the continuous motion and appearance changes of the target in the time series;

[0011] 3) Feature extraction stage: In each iteration, the RGB branch receives RGB modal input including RGB template image sequence, RGB search image and context memory tokens, and the TIR branch receives TIR modal input including TIR template image sequence, TIR search image and context memory tokens; the ViT encoder performs feature extraction on the RGB modal input and TIR modal input respectively, and outputs RGB feature sequence and TIR feature sequence;

[0012] 4) Temporal Feature Interaction Enhancement Stage: The modality-aware temporal Mamba module performs cross-modal temporal context modeling and propagation on RGB feature sequences and TIR feature sequences to generate temporally enhanced RGB feature sequences and temporally enhanced TIR feature sequences. The modality-aware temporal Mamba module captures local historical context within a modality and fuses cross-modal global change cues through a modality-aware temporal state space model to achieve persistent cross-modal context propagation.

[0013] 5) Spatial feature fusion stage: The temporally enhanced RGB feature sequence and the temporally enhanced TIR feature sequence are fused in the spatial dimension through the cross-modal Mamba aggregation module to generate a fused feature sequence;

[0014] 6) Prediction stage: The prediction head is used to perform convolution processing on the fused feature sequence to output the bounding box prediction result of the target;

[0015] 7) Context memory tokens update and iterative propagation: In each iteration, the context memory tokens learn the spatiotemporal context information from the initial frame to the current frame based on the output of the modality-aware temporal Mamba module, and are updated based on the output of the cross-modality Mamba aggregation module before being propagated to the next iteration. The entire iterative process is executed continuously along the timeline until the tracking ends, ensuring that the model maintains the evolution consistency of cross-modality spatiotemporal state in multi-frame sequences.

[0016] Further, in step 4), the processing procedure of the modality-aware temporal Mamba module includes:

[0017] The RGB feature sequence F obtained from feature extraction RGB and TIR characteristic sequence F TIR The template feature sequence, search feature sequence, and context memory feature sequence corresponding to each modality are separated. Then, the search feature sequence and context memory feature sequence corresponding to each modality are concatenated to obtain the input sequence of the modality-aware temporal Mamba module.

[0018] The input sequences at the t-th time step of the RGB mode and the TIR mode are respectively represented as follows: and Each modality's input sequence contains a search feature sequence and a context memory feature sequence. The input sequence for each modality is processed using linear projection, 1D convolution, and SiLU activation function. A modality-aware temporal state space model is used to jointly process the input sequences of RGB and TIR modalities, enabling cross-modal temporal information interaction and intra-modal context information storage and updating. The modality-aware temporal state space model includes a cross-modal coupled state transition mechanism and a cue-guided state space modeling strategy. Through the cross-modal coupled state transition mechanism, a shared coupled hidden state is constructed to fuse cross-modal temporal context information. ; where the hidden state of each modality , Initialize at the first time step, and update based on the current input in subsequent time steps; that is, based on the input sequence of each modality. and Update the hidden state of the corresponding modality at the current time step to store the temporal context information from frame 1 to the current frame, represented as:

[0019]

[0020] Where, m {RGB, TIR} For the input weight matrix, Let be the state transition matrix for modal sensing, used to ensure the independence of intramodal state processes while performing cross-modal context information interaction, and expressed as:

[0021]

[0022] in, Let m be the discretization time step of mode m. Let m be the continuous-time transition matrix of mode m. Let n be the discretization time step of mode n. {RGB, TIR}, n≠m, It is an exponential function;

[0023] By employing a prompt-guided state-space modeling strategy, a learnable prompt pool p is introduced to dynamically select prompt vectors. ,in The projection weight matrix integrates the cue vectors into the state output matrix. In this process, the output projection of the modality-aware temporal Mamba module is obtained, which is expressed as:

[0024]

[0025] in, This is the feedforward weight matrix; Used to update context memory sequences, enabling non-causal, goal-aware representation learning. for , Represents a temporally enhanced RGB feature sequence. This represents a time-enhanced TIR feature sequence.

[0026] Furthermore, in step 5), the processing of the cross-modal Mamba aggregation module includes two stages: intra-modal aggregation and inter-modal aggregation.

[0027] The intramodal aggregation stage provides features F for each modality. m (m {RGB, TIR}) are used to obtain the first feature through parallel state space processing. Second feature ; The operations include linear projection, 1D convolution, and the SiLU activation function; The operations include linear projection and activation operations; the transition features of each mode are obtained through intra-modal aggregation, represented as:

[0028]

[0029] in, Represents independent state-space model operations; by concatenating the transition features of each mode, the concatenated features are obtained. Then, through linear projection, the intra-modal aggregated features of each modality are separated, as follows:

[0030]

[0031] in, This represents a linear projection operation. Indicates a separation operation;

[0032] The intermodal aggregation stage aggregates intramodal features for each mode. and conduct Operation and The operations, after being concatenated, are input into a shared state-space model to obtain inter-modal aggregated features. , represented as:

[0033]

[0034] Intermodal aggregation features The final output of the cross-modal Mamba aggregation module is obtained through linear projection and modal separation.

[0035] Further, in step 6), the prediction head uses a fully convolutional network, which includes L stacked convolutional blocks. Each convolutional block consists of a convolutional layer, a BatchNorm, and a ReLU activation function, used to convert feature tokens into feature maps and output target bounding boxes.

[0036] The present invention also provides an RGB-T target tracking system based on spatiotemporal state evolution, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the above-described method steps.

[0037] Compared with existing technologies, the present invention has the following advantages: The present invention provides an RGB-T target tracking method and system based on spatiotemporal state evolution. It achieves persistent cross-modal context propagation through spatiotemporal state space evolution, captures fine-grained local temporal cues and overall semantic evolution, alleviates the short-term window limitation of traditional methods, and integrates coupled state transition and cue-guided modeling to support dynamic semantic reasoning and non-causal target perception. It fuses cross-modal features with linear complexity to ensure robust adaptive integration, and fuses and transmits cross-modal context information with fewer training parameters, thereby improving training efficiency and effectiveness. Attached Figure Description

[0038] Figure 1 This is an architecture diagram of the RGB-T target tracking network model in an embodiment of the present invention.

[0039] Figure 2 This is a schematic diagram illustrating the implementation principle of the modality-aware timing Mamba module in this embodiment of the invention.

[0040] Figure 3 This is a schematic diagram illustrating the implementation principle of the cross-modal Mamba aggregation module in this embodiment of the invention. Detailed Implementation

[0041] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0042] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0043] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0044] This embodiment provides an RGB-T target tracking method based on spatiotemporal state evolution, which constructs an RGB-T target tracking model, such as... Figure 1 As shown, the RGB-T target tracking model adopts a dual-branch architecture including an RGB branch and a TIR branch. The RGB branch and the TIR branch share the same ViT (Vision Transformer) encoder weights to achieve uniformity in feature extraction across modalities. The RGB-T target tracking model employs an iterative processing framework throughout the tracking process, performing tracking iterations frame by frame along the time series. Each iteration corresponds to the input of a search image frame, and spatiotemporal context information is propagated through context memory tokens between iterations. The specific implementation steps of this method are as follows.

[0045] 1) Input Acquisition Stage: RGB and TIR modal data are acquired as input for subsequent feature extraction in the input model. Each modal input includes a template image sequence, a search image, and context memory tokens corresponding to each modality. The context memory tokens are initialized in the first frame, encoding local historical context within the modality and global cues across modalities. They are updated and passed throughout the tracking process to guide the spatiotemporal propagation of subsequent iterations.

[0046] 2) Iterative Input Preparation: In subsequent iterations (t=2,3,...), the RGB modal search image and TIR modal search image of the current frame are input along the timeline. The template image sequence remains unchanged in each iteration and is passed as a reference template; the context memory tokens are updated from the previous iteration (t-1) and passed to the current iteration. Combined with the template image sequence and the current search image, a complete modal input (template image sequence + search image + context memory tokens) is formed to capture the continuous motion and appearance changes of the target over time.

[0047] 3) Feature Extraction Stage: In each iteration, the RGB branch receives the RGB modal input (RGB template image sequence + RGB search image + context memory tokens), and the TIR branch receives the TIR modal input (TIR template image sequence + TIR search image + context memory tokens). The weighted ViT encoder extracts features from the RGB modal input and outputs an RGB feature sequence. Similarly, the ViT encoder extracts features from the TIR modal input and outputs a TIR feature sequence.

[0048] 4) Temporal Feature Interaction Enhancement Stage: The modality-aware temporal Mamba module performs cross-modal temporal context modeling and propagation on the RGB and TIR feature sequences, generating temporally enhanced RGB and TIR feature sequences. The modality-aware temporal Mamba module includes a modality-aware temporal state-space model improved from a general state-space model. This model captures local historical context within a modality and fuses cross-modal global change cues, enabling persistent cross-modal context propagation.

[0049] 5) Spatial feature fusion stage: The temporally enhanced RGB feature sequence and the temporally enhanced TIR feature sequence are fused in the spatial dimension through the cross-modal Mamba aggregation module to generate a fused feature sequence.

[0050] 6) Prediction stage: The prediction head is used to perform convolution processing on the fused feature sequence to output the bounding box prediction result of the target;

[0051] 7) Context Memory Token Update and Iterative Transmission: During each iteration, the context memory tokens learn the spatiotemporal context information from the initial frame to the current frame based on the output of the modality-aware temporal Mamba module, update it based on the output of the cross-modality Mamba aggregation module, and then pass it to the next iteration. The entire iterative process is executed continuously along the timeline until the tracking ends, ensuring that the model maintains the consistency of cross-modal spatiotemporal state evolution across multiple frame sequences.

[0052] like Figure 2 As shown, the processing procedure of the modality-aware temporal Mamba module is as follows.

[0053] The RGB feature sequence F obtained from feature extraction RGB and TIR characteristic sequence F TIR The template feature sequence, search feature sequence, and context memory feature sequence corresponding to each modality are separated. Then, the search feature sequence and context memory feature sequence corresponding to each modality are concatenated to obtain the input sequence of the modality-aware temporal Mamba module.

[0054] The input sequences at the t-th time step of the RGB mode and the TIR mode are respectively represented as follows: and Each modality's input sequence contains a search feature sequence and a context memory feature sequence. The input sequence for each modality is processed using linear projection, 1D convolution, and the SiLU activation function. A modality-aware temporal state space model is used to jointly process the input sequences of RGB and TIR modalities, enabling cross-modal temporal information interaction and intra-modal context information storage and updating. The modality-aware temporal state space model includes a cross-modal coupled state transition mechanism and a cue-guided state space modeling strategy. Through the cross-modal coupled state transition mechanism, shared coupled hidden states are constructed to fuse cross-modal temporal context information. The hidden state of each modality. , Initialize at the first time step, and update based on the current input in subsequent time steps; that is, based on the input sequence of each modality. and Update the hidden state of the corresponding modality at the current time step, so that the temporal context information from frame 1 to the current frame can be stored, represented as:

[0055]

[0056] Where, m {RGB, TIR}; The input weight matrix is ​​used to weight the input sequence at the current time step. A linear mapping is applied to the hidden state space to selectively weight the current input information, thereby affecting the update of the hidden state. Let be the state transition matrix for modal sensing. Its function is to model the state space after summing the states, ensuring the independence of the internal state processes of a mode while enabling cross-modal context information interaction. It can be represented as:

[0057]

[0058] in, Let m be the discretization time step of mode m. Let m be the continuous-time transition matrix of mode m. Let n be the discretization time step of mode n. {RGB, TIR}, n≠m, It is an exponential function.

[0059] By employing a prompt-guided state-space modeling strategy, a learnable prompt pool p is introduced to dynamically select prompt vectors. ;in, The projection weight matrix, dynamically calculated via an attention router, is used to select cue vectors relevant to the current modality from a shared pool of learnable cues p. This enables dynamic semantic guidance for different modalities.

[0060] Integrate the cue vector into the state output matrix In this process, the output projection of the modality-aware temporal Mamba module is obtained, which is expressed as:

[0061]

[0062] in, This is the feedforward weight matrix, used to weight the current input. Mapping to output This is equivalent to a direct path in residual connections, enhancing the model's ability to retain and transmit current input information. Used to update context memory sequences, enabling non-causal, goal-aware representation learning. for , Represents a temporally enhanced RGB feature sequence. This represents a time-enhanced TIR feature sequence.

[0063] By employing a prompt-guided state-space modeling strategy, a learnable prompt pool p is introduced to dynamically select prompt vectors. ,in The cue vectors are integrated into the state output matrix through the attention router computation. In the process, the output projection of the modality-aware temporal Mamba module is obtained, denoted as: This is used to update the context memory sequence, enabling non-causal, goal-aware representation learning. for , Represents a temporally enhanced RGB feature sequence. This represents a time-enhanced TIR feature sequence.

[0064] like Figure 3 As shown, the processing of the cross-modal Mamba aggregation module includes two stages: intra-modal aggregation and inter-modal aggregation.

[0065] The intramodal aggregation stage provides features F for each modality. m (m {RGB, TIR}) are used to obtain the first feature through parallel state space processing. Second feature ; The operations include linear projection, 1D convolution, and the SiLU activation function; The operations include linear projection and activation operations. Transition features for each mode are obtained through intra-modal aggregation, and can be represented as:

[0066]

[0067] in, This represents independent state-space model operations. The concatenated features are obtained by concatenating the transition features of each mode. Then, through linear projection, the intra-modal aggregated features of each modality are separated, as follows:

[0068]

[0069] in, This represents a linear projection operation. This indicates a separation operation.

[0070] The intermodal aggregation stage aggregates intramodal features for each mode. and conduct Operation and The operations, after being concatenated, are input into a shared state-space model to obtain inter-modal aggregated features. , represented as:

[0071]

[0072] Intermodal aggregation features The final output of the cross-modal Mamba aggregation module is obtained through linear projection and modal separation. This module can efficiently simulate the interaction of internal tokens across modalities with linear complexity.

[0073] In this embodiment, the prediction head uses a fully convolutional network, which includes L stacked convolutional blocks. Each convolutional block consists of a convolutional layer, a BatchNorm, and a ReLU activation function, used to convert feature tokens into feature maps and output target bounding boxes.

[0074] In this embodiment, the LasHeR dataset was used for comparative verification under visible light and infrared light image settings of the target object. Table 1 shows the comparison results of the proposed method with other multimodal target tracking methods on the LasHeR dataset. As can be seen from Table 1, the method of the present invention has higher accuracy and robustness compared with other RGB-T target tracking methods, specifically in terms of optimal precision and accuracy.

[0075] Table 1

[0076]

[0077] In Table 1, mfDiMP corresponds to the method proposed by L. Zhang et al. in their paper "Multi-modal fusion for end-to-end RGBT tracking" (Proceedings of the IEEE / CVF International Conference on Computer Vision workshops, 2019).

[0078] DAPNeT corresponds to the method proposed by Y. Gao et al. in their paper "Deep adaptive fusion network for high performance RGBT tracking" (Proceedings of the IEEE International Conference on Computer Vision workshops, 2019).

[0079] CAT corresponds to the method proposed by C. Li et al. in their paper "Challenge-aware RGBT tracking" (European Conference on Computer Vision, pp. 222-237).

[0080] MANet++ corresponds to the method proposed by A. Lu et al. in their paper "RGBT tracking via multi-adapternetwork with hierarchical divergence loss" (IEEE Transactions on Image Processing, 2021, vol. 30, pp. 5613-5625).

[0081] APFNet corresponds to the method proposed by Y. Xiao et al. in their paper "Multi-zone transformer based on self-distillation for facial attribute recognition" (IEEE International Conference on Automatic Face and Gesture Recognition, 2023, pp. 1-7).

[0082] ProTrack corresponds to the method proposed by J. Yang et al. in their paper "Prompting for Multi-ModalTracking" (Proceedings of the 30th ACM international conference on multimedia, 2022, pp.3492-3500).

[0083] ViPT corresponds to the method proposed by J. Zhu et al. in their paper "Visual Prompt Multi-ModalTracking" (IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2023, pp.9516-9526).

[0084] BAT refers to the method proposed by Y. Shu et al. in their paper "Bi-directional adapter for multimodal tracking" (Proceedings of the AAAI Conference on Artificial Intelligence. 2024, pp.927-935).

[0085] STMT corresponds to the method proposed by D. Sun et al. in their paper "Transformer RGBT Tracking with Spatio-Temporal Multimodal Tokens" (arXiv preprint arXiv:2401.01674, 2024).

[0086] TATrack corresponds to the method proposed by H. Wang et al. in their paper "Temporal adaptive RGBTtracking with modality prompt" (Proceedings of the AAAI Conference on Artificial Intelligence. 2024, pp.5436-5444).

[0087] AINet corresponds to the method proposed by A. Lu et al. in their paper "RGB-T tracking via all-layer multimodal interactions with progressive fusion mamba" (Proceedings of the AAAI Conference on Artificial Intelligence. 2025, pp.5793–5801).

[0088] CAFormer corresponds to the method proposed by Y. Xiao et al. in their paper "Cross-modulated AttentionTransformer for RGBT Tracking" (Proceedings of the AAAI Conference on Artificial Intelligence. 2025, pp.8682–8690).

[0089] This embodiment also provides an RGB-T target tracking system based on spatiotemporal state evolution, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the above-described method steps.

[0090] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0091] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0092] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0093] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0094] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. An RGB-T target tracking method based on spatiotemporal state evolution, characterized in that, An RGB-T target tracking model is constructed, which adopts a dual-branch architecture including an RGB branch and a TIR branch, wherein the RGB branch and the TIR branch share the same ViT encoder weights; The RGB-T target tracking model employs an iterative processing framework throughout the tracking process, performing tracking iterations frame by frame along the time series. Each iteration corresponds to the input of a search image frame, and spatiotemporal context information is propagated between iterations through context memory tokens. The method includes the following steps: 1) Input Acquisition Stage: Acquire RGB modal data and TIR modal data as input for subsequent input model feature extraction; each modal input includes template image sequence, search image and context memory tokens corresponding to each modality. Context memory tokens are initialized in the first frame, encode local historical context within the modality and global cues across modalities, and are updated and passed throughout the tracking process to guide the spatiotemporal propagation of subsequent iterations; 2) Iterative Input Preparation: In subsequent iterations, the RGB modal search image and TIR modal search image of the current frame are input along the timeline; the template image sequence remains unchanged in each iteration and is passed as a reference template; the context memory tokens are passed from the previous iteration to the current iteration after being updated, and combined with the template image sequence and the current search image to form a complete modal input including the template image sequence, the search image and the context memory tokens, so as to capture the continuous motion and appearance changes of the target in the time series; 3) Feature extraction stage: In each iteration, the RGB branch receives RGB modal input including RGB template image sequence, RGB search image and context memory tokens, and the TIR branch receives TIR modal input including TIR template image sequence, TIR search image and context memory tokens; the ViT encoder performs feature extraction on the RGB modal input and TIR modal input respectively, and outputs RGB feature sequence and TIR feature sequence; 4) Temporal Feature Interaction Enhancement Stage: The modality-aware temporal Mamba module performs cross-modal temporal context modeling and propagation on RGB feature sequences and TIR feature sequences to generate temporally enhanced RGB feature sequences and temporally enhanced TIR feature sequences. The modality-aware temporal Mamba module captures local historical context within a modality and fuses cross-modal global change cues through a modality-aware temporal state space model to achieve persistent cross-modal context propagation. 5) Spatial feature fusion stage: The temporally enhanced RGB feature sequence and the temporally enhanced TIR feature sequence are fused in the spatial dimension through the cross-modal Mamba aggregation module to generate a fused feature sequence; 6) Prediction stage: The prediction head is used to perform convolution processing on the fused feature sequence to output the bounding box prediction result of the target; 7) Context memory tokens update and iterative propagation: In each iteration, the context memory tokens learn the spatiotemporal context information from the initial frame to the current frame based on the output of the modality-aware temporal Mamba module, and are updated based on the output of the cross-modality Mamba aggregation module before being propagated to the next iteration. The entire iterative process is executed continuously along the timeline until the tracking ends, ensuring that the model maintains the evolution consistency of cross-modality spatiotemporal state in multi-frame sequences.

2. The RGB-T target tracking method based on spatiotemporal state evolution according to claim 1, characterized in that, In step 4), the processing procedure of the modality-aware temporal Mamba module includes: The RGB feature sequence F obtained from feature extraction RGB and TIR characteristic sequence F TIR The template feature sequence, search feature sequence, and context memory feature sequence corresponding to each modality are separated. Then, the search feature sequence and context memory feature sequence corresponding to each modality are concatenated to obtain the input sequence of the modality-aware temporal Mamba module. The input sequences at the t-th time step of the RGB mode and the TIR mode are respectively represented as follows: and Each modality's input sequence contains a search feature sequence and a context memory feature sequence. The input sequence for each modality is processed using linear projection, 1D convolution, and SiLU activation function. A modality-aware temporal state space model is used to jointly process the input sequences of RGB and TIR modalities, enabling cross-modal temporal information interaction and intra-modal context information storage and updating. The modality-aware temporal state space model includes a cross-modal coupled state transition mechanism and a cue-guided state space modeling strategy. Through the cross-modal coupled state transition mechanism, a shared coupled hidden state is constructed to fuse cross-modal temporal context information. ; where the hidden state of each modality , Initialize at the first time step, and update based on the current input in subsequent time steps; that is, based on the input sequence of each modality. and Update the hidden state of the corresponding modality at the current time step to store the temporal context information from frame 1 to the current frame, represented as: Where, m {RGB, TIR} For the input weight matrix, Let be the state transition matrix for modal sensing, used to ensure the independence of intramodal state processes while performing cross-modal context information interaction, and expressed as: in, Let m be the discretization time step of mode m. Let m be the continuous-time transition matrix of mode m. Let n be the discretization time step of mode n. {RGB, TIR}, n≠m, It is an exponential function; By employing a prompt-guided state-space modeling strategy, a learnable prompt pool p is introduced to dynamically select prompt vectors. ,in The projection weight matrix integrates the cue vectors into the state output matrix. In this process, the output projection of the modality-aware temporal Mamba module is obtained, which is expressed as: in, This is the feedforward weight matrix; Used to update context memory sequences, enabling non-causal, goal-aware representation learning. for , Represents a temporally enhanced RGB feature sequence. This represents a time-enhanced TIR feature sequence.

3. The RGB-T target tracking method based on spatiotemporal state evolution according to claim 1, characterized in that, In step 5), the processing of the cross-modal Mamba aggregation module includes two stages: intra-modal aggregation and inter-modal aggregation. The intramodal aggregation stage provides features F for each modality. m (m {RGB, TIR}) are used to obtain the first feature through parallel state space processing. Second feature ; The operations include linear projection, 1D convolution, and the SiLU activation function; The operations include linear projection and activation operations; the transition features of each mode are obtained through intra-modal aggregation, represented as: in, Represents independent state-space model operations; by concatenating the transition features of each mode, the concatenated features are obtained. Then, through linear projection, the intra-modal aggregated features of each modality are separated, as follows: in, This represents a linear projection operation. Indicates a separation operation; The intermodal aggregation stage aggregates intramodal features for each mode. and conduct Operation and The operations, after being concatenated, are input into a shared state-space model to obtain inter-modal aggregated features. , represented as: Intermodal aggregation features The final output of the cross-modal Mamba aggregation module is obtained through linear projection and modal separation.

4. The RGB-T target tracking method based on spatiotemporal state evolution according to claim 1, characterized in that, In step 6), the prediction head uses a fully convolutional network, which includes L stacked convolutional blocks. Each convolutional block consists of a convolutional layer, a BatchNorm, and a ReLU activation function, used to convert feature tokens into feature maps and output target bounding boxes.

5. An RGB-T target tracking system based on spatiotemporal state evolution, characterized in that, It includes a memory, a processor, and computer program instructions stored in the memory and executable by the processor, which, when executed by the processor, enable the implementation of the method as described in any one of claims 1-4.

Images