A method and system for RGB-T target tracking that combines high-order modal and temporal dynamics modeling

Abstract

This invention relates to a method and system for RGB-T target tracking that jointly models higher-order modalities and temporal dynamics. The method includes: constructing an RGB-T target tracking model with a dual-branch architecture of RGB modal branches and TIR modal branches, including a Transformer backbone network, a cross-modal target query generator, a spatiotemporal higher-order interactive cue module, and a prediction head; dynamically fusing the current template label with the previous frame's spatiotemporal label through the cross-modal target query generator to output a time-aware target query; using the spatiotemporal higher-order interactive cue module, approximating second-order attention in the frequency domain through Fast Fourier Transform and expanding it into recursive higher-order interaction, and generating higher-order spatiotemporal cue by combining a spatiotemporal diffraction injection mechanism; injecting the higher-order cue into the Transformer backbone network and passing it to the next frame to enhance long-distance dependencies and temporal continuity. This invention can fully utilize cross-modal higher-order spatiotemporal interaction and semantic alignment to improve the accuracy and robustness of target tracking in complex scenes.

Description

Technical Field

[0001] This invention belongs to the field of image processing and analysis technology, specifically relating to an RGB-T target tracking method and system that jointly models higher-order modalities and temporal dynamics. Background Technology

[0002] The integration of thermal infrared (TIR) ​​data provides complementary information to the visible light (RGB) modality, greatly benefiting visual target tracking in complex environments. By utilizing both modalities, RGB-T target tracking holds promise for improved robustness under challenging conditions. However, heterogeneous sources introduce new challenges beyond single-modal modeling, particularly in maintaining cross-modal semantic and temporal consistency.

[0003] Existing RGB-T object tracking methods can be broadly categorized into two types based on how they model cross-modal and temporal information. Early methods focused solely on single-frame modal interactions, applying unidirectional or bidirectional interactions between RGB and TIR features. While state-of-the-art backbone architectures, from CNNs to Transformers, have enhanced feature representations, these interactions lack cross-modal semantic alignment across time. Consequently, the model is vulnerable to occlusion, deformation, or illumination variations when current-frame cues are insufficient. To incorporate temporal information, the second type of work introduces token-level propagation across frames, where search tokens are updated or reused over time. However, since object cues are often entangled with complex semantic contexts (e.g., cluttered backgrounds or modal inconsistencies), token representations tend to degrade during propagation, especially in the absence of fine-grained temporal refinement. This leads to time-varying drift and weak object discriminability.

[0004] Compared to updating time tokens, Wang et al. recently proposed a prompt-based mechanism in their paper "Temporal adaptive rgbt tracking with modality prompt" (Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(6): 5436-5444), which provides more targeted and controllable representations by injecting compact and learnable prompts. While promising, existing methods typically rely on first- or second-order local attention of modal interactions, which limits the semantic depth and global consistency of the prompts. Furthermore, their temporal updates are often conditionally triggered or limited to short sliding windows, which restricts the model's ability to capture continuous and long-term target dynamics across different scenarios. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and system for RGB-T tracking that jointly models higher-order modalities and temporal dynamics, so as to make full use of cross-modal higher-order spatiotemporal interactions and improve the accuracy and robustness of RGB-T tracking.

[0006] To achieve the above objectives, the technical solution adopted by this invention is: an RGB-T target tracking method that jointly models higher-order modalities and temporal dynamics, comprising the following steps:

[0007] 1) Obtain the training dataset and training set annotation files;

[0008] 2) Construct a joint modeling high-order modality and temporal dynamics RGB-T target tracking model. The RGB-T target tracking model is a dual-branch architecture with RGB modal branches and TIR modal branches, including a Transformer backbone network, a cross-modal target query generator, a spatiotemporal high-order interactive cue module, and a prediction head. For each modal branch, the input frame is first encoded into template markers and search markers through patch embedding and positional encoding. Then, the features of each modal branch are extracted by the Transformer. The cross-modal target query generator processes the current template marker and the spatiotemporal marker of the previous frame. The system dynamically fuses multimodal information and outputs cross-modal and temporal fusion features as a time-aware target query. This time-aware target query is then passed to the spatiotemporal higher-order interaction prompt module. This module integrates the search tags from the previous frame and the current frame to generate updated higher-order prompts for multi-location interactions in the current frame, providing guidance for the model's Transformer and temporal evolution. The updated higher-order prompts are then injected into the Transformer backbone network to enhance the modeling of long-distance dependencies. Simultaneously, the updated higher-order prompts are propagated to the next frame to maintain temporal continuity.

[0009] The RGB-T target tracking model was trained using the training dataset and the training set annotation file to obtain generalizable model parameters.

[0010] 3) Use the trained RGB-T target tracking model to track targets and achieve robust and stable tracking results.

[0011] Further, in step 1), the training dataset and the corresponding training set annotation file are obtained from the open source dataset, wherein the open source dataset is RGBT234, RGBT210 and LasHeR tracking dataset.

[0012] Further, in step 2), the specific implementation of the RGB-T target tracking model includes:

[0013] The RGB-T target tracking model includes two modal branches: an RGB modal branch for RGB modal feature extraction and a TIR modal branch for TIR modal feature extraction. Each modal branch consists of L stacked Transformer modules. Each Transformer module is composed of multi-head self-attention and a multilayer perceptron through residual connections and layer normalization. The Transformer modules of the two modal branches form a Transformer backbone network. The cross-modal target query generator and the spatiotemporal high-order interactive prompting module are located between the two modal branches and are processed in parallel with the two modal branches, while the cross-modal target query generator and the spatiotemporal high-order interactive prompting module are processed serially.

[0014] First, the reference video sequences for the two modal branches, namely the template image and the search image, are input into the RGB-T target tracking model. Template labels for the two modal branches are obtained through patch embedding and positional encoding. , and search tags , Then, the Transformer module extracts features from the RGB modality branch and the TIR modality branch respectively. Simultaneously, template tags and spatiotemporal tags initialized with template tags are input into a cross-modal target query generator. This generator employs a dynamic sparse ghost fusion module to construct an adaptive, target-related query by jointly modeling spatiotemporal cues of the RGB and TIR modalities. The cross-modal target query generator contains two parallel instances of the dynamic sparse ghost fusion module: one for fusing multimodal template tags and the other for fusing multimodal spatiotemporal tags. Each dynamic sparse ghost fusion module selects a sparse but information-rich token set to achieve multimodal information fusion. After the output of the dynamic sparse ghost fusion module, a multi-head self-attention mechanism is introduced. This mechanism concatenates the multimodal template tag fusion result and the multimodal spatiotemporal tag fusion result output by the dynamic sparse ghost fusion module along the channel dimension as input, achieving dense, content-aware interaction in modality, time, and space dimensions, ultimately outputting a time-aware target query.

[0015] The time-aware target query output by the cross-modal target query generator, along with search tags obtained from patch embedding and positional encoding of the current and previous frame images, is input into the spatiotemporal higher-order interaction prompting module. This module utilizes a second-order interaction mechanism in the frequency domain, approximating self-attention through Fast Fourier Transform, and extends this second-order interaction to a recursive higher-order interaction. A spatiotemporal diffraction injection scheme is employed, allowing early context injection into subsequent features. The next iteration injects refined higher-order cues to enhance spatiotemporal relevance and modal alignment. Finally, the output of the spatiotemporal higher-order interactive cue module, i.e., the higher-order cues, is decomposed into features corresponding to the template tag length to form new spatiotemporal tags, which are then used as higher-order cues in subsequent inference.

[0016] Furthermore, the implementation method of the dynamic sparse ghost fusion module is as follows:

[0017] First, a modality-aware sparse sampling strategy suitable for template fusion and a temporal-aware sparse sampling strategy suitable for temporal fusion are adopted. For template markers, the feature magnitude at each marker position is calculated as the l2 norm across channels. The Top-k indices are sampled as follows:

[0018]

[0019] Among them, Top-k This represents the position indices corresponding to the k largest amplitude values; This represents the calculated characteristic amplitude; Indicates a location in time and space. Location, passage eigenvectors; Indicates the total number of channels; Indicates the first value with the largest amplitude. A location index is used to sample salient features;

[0020] For time stamps, the rate of change of features is measured by the mean absolute difference across channels between consecutive time steps:

[0021]

[0022] in, Indicates at time step Location, passage eigenvalues; This represents the calculated characteristic rate of change; The first one with the largest rate of change A location index is used to highlight areas that change significantly over time;

[0023] Sampling sparse index and Cross-modal tagging applied to splicing and In the initial frame, the spatiotemporal marker , Use template tags respectively , Perform initialization; where This represents channel-level splicing, thereby obtaining a sparse, information-rich representation; for The sparse features are obtained as follows: :

[0024]

[0025] in, Indicates based on the obtained index from Sparse features are obtained by mid-sampling ;

[0026] Generate sparse ghost features as follows:

[0027]

[0028] in, This represents a point-to-point convolution that extracts intrinsic features from sparse input. Indicates the generation of the first Depth convolution of ghost feature branches, Indicates the number of ghost feature branches; each ghost feature Capture different information while maintaining computational efficiency;

[0029] In obtaining sparse ghost characteristics Subsequently, lightweight ghost branch interactions are performed to facilitate cross-branch communication and recover latent semantics lost during sparse sampling; each ghost feature Processed as:

[0030]

[0031] in, This indicates detailed ghost features. and For two learnable 1s 1 convolutional layer; The ReLU activation function is used to filter out noisy or invalid interactions, retaining only meaningful responses; additionally, A residual embedding is provided to preserve modality-invariant semantic information;

[0032] For a refined set of ghost features Adaptive fusion is performed using predicted weights; first, all branches are concatenated along the channel dimension, and global average pooling is applied to obtain a compact global descriptor. Then, point-wise convolution is used, followed by softmax to generate a standardized importance score; its representation is as follows:

[0033]

[0034] in, Indicates global average pooling. This indicates the importance weights that include all ghost branches;

[0035] Then, the fused ghost features are obtained by weighted summation, represented as:

[0036]

[0037] Finally, the inherent sparse features are along the channel dimension. With the characteristics of the fused ghost By piecing them together, the final output is generated. , represented as:

[0038] .

[0039] Furthermore, the implementation method of the cross-modal target query generator is as follows:

[0040] The cross-modal target query generator comprises two parallel instances of a dynamic sparse ghost fusion module: one for fusing multimodal template tags and the other for fusing multimodal spatiotemporal tags, denoted as:

[0041]

[0042] in, , The outputs of two parallel instances of the Dynamic Sparse Ghost Fusion module. Encapsulate the modal complementary spatial signal from the template, and Capture temporal dynamic signals related to the target's recent motion; This indicates a dynamic sparse ghost fusion module;

[0043] To further expand the perception range, a multi-head self-attention mechanism is introduced into the output of the dynamic sparse ghost fusion module to enable time-aware target querying. exist The ability to adaptively emphasize target-relevant regions while maintaining modal alignment and temporal awareness is expressed as:

[0044]

[0045] in, This represents a multi-head self-attention mechanism.

[0046] Furthermore, the implementation method of the spatiotemporal higher-order interactive prompt module is as follows:

[0047] The spatiotemporal high-order interactive prompting module will generate a time-aware target query from the cross-modal target query generator. and multimodal search tags from the current frame. Multimodal search markers from the previous frame As input;

[0048] To capture dense correlations, a second-order interaction is approximated in the frequency domain using the Fast Fourier Transform (FFT); given a query and each search tag ,time Modality Second-order interaction in the frequency domain Described as:

[0049]

[0050] in, and Represents FFT and inverse FFT, express The transpose operation, Represents element-wise multiplication; They represent queries respectively. ,key Sum Learnable projection, and This represents the query and key obtained after performing an FFT, respectively. This represents the attention matrix obtained after inverse FFT. Representation layer normalization;

[0051] To capture richer semantics beyond second-order interactions, the above frequency domain second-order interaction process is extended to recursive higher-order interactions. A simplified cross-modal fusion recursively updates the features of the original domain, causing the intra-feature positions obtained in each recursion to change, thus considering more positional dependencies during interactions. The recursive update is represented as:

[0052]

[0053] in, ;

[0054] Recursive query is represented as:

[0055]

[0056] in, This represents a dynamic query representation that updates across recursive layers. Indicates capture in The time modality is of Interaction at different levels;

[0057] To enhance spatiotemporal correlation, a spatiotemporal diffraction injection scheme is adopted, allowing early features to diffract into later features; firstly, time-aware target queries are integrated through residual fusion. Merged into each higher-order feature, i.e. Then, according to the diffraction sequence and The features injected in the query are cascaded and refined to achieve perceptual alignment between the RGB and TIR modalities; at each step, the current feature is updated by the previous feature through lightweight residual projection, as shown below:

[0058]

[0059] in, Indicates the current feature in the cascade. Indicates the guiding feature of the previous step; through The refined features obtained in the next iteration are denoted as... and ;

[0060] After spatiotemporal diffraction injection, merge time-aware target query And refactor the final enhanced higher-order hints as , Finally, the higher-order hints are broken down into features corresponding to the template token length to form new spacetime tokens. This serves as a clue to the dynamic and spatiotemporally rich nature of subsequent reasoning.

[0061] The present invention also provides an RGB-T target tracking system that jointly models higher-order modes and temporal dynamics, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the above-described method.

[0062] The present invention also provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the above-described method.

[0063] Compared with existing technologies, this invention has the following advantages: This method and system overcome the limitations of low-level labeling or cues (caused by shallow or low-level cross-modal and temporal interactions), and constructs more robust and semantically aligned target-aware spatiotemporal cues by exploring the synergy between cross-modal fusion and temporal modeling through high-order spatiotemporal interactions. It can ensure robustness in dynamic and uncertain environments. Attached Figure Description

[0064] Figure 1 This is an architecture diagram of the RGB-T target tracking model that jointly models higher-order modalities and temporal dynamics in an embodiment of the present invention;

[0065] Figure 2 This is a schematic diagram illustrating the implementation principle of the cross-modal target query generator in this embodiment of the invention.

[0066] Figure 3 This is a schematic diagram illustrating the implementation principle of the spatiotemporal higher-order interactive prompt module in this embodiment of the invention. Detailed Implementation

[0067] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0068] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0069] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0070] This embodiment provides a method for RGB-T target tracking that jointly models higher-order modalities and temporal dynamics, including the following steps:

[0071] 1) Obtain the training dataset and training set annotation files;

[0072] 2) Construct a joint modeling high-order modality and temporal dynamic RGB-T target tracking model. The RGB-T target tracking model is a dual-branch architecture with RGB modal branches and TIR modal branches, including a Transformer backbone network, a cross-modal target query generator, a spatiotemporal high-order interactive prompting module, and a prediction head. For each modal branch, the input frame is first encoded into template markers and search markers through patch embedding and positional encoding. Then, the Transformer extracts the features of each modal branch. The cross-modal target query generator dynamically fuses multimodal information by processing the current template marker and the spatiotemporal marker of the previous frame, outputting the fused features of cross-modality and time as a time-aware target query. The time-aware target query is then passed to the spatiotemporal high-order interactive prompting module. The spatiotemporal high-order interactive prompting module integrates the search marker of the previous frame as a key. To reduce information loss, and at the same time integrate the search tag of the current frame as a key. This process generates updated higher-order cues for multi-location interactions in the current frame. These higher-order cues encode rich spatiotemporal semantic information, providing guidance for the Transformer and temporal evolution of the model. The updated higher-order cues are then injected into the Transformer backbone to enhance the modeling of long-distance dependencies. Simultaneously, the updated higher-order cues are propagated to the next frame to maintain temporal continuity.

[0073] The RGB-T target tracking model was trained using the training dataset and the training set annotation file to obtain generalizable model parameters.

[0074] 3) Use the trained RGB-T target tracking model to track targets and achieve robust and stable tracking results.

[0075] In this embodiment, the training dataset and the corresponding training set annotation file are obtained from open source datasets, namely RGBT234, RGBT210 and LasHeR tracking datasets.

[0076] Figure 2 This is an architecture diagram of the RGB-T target tracking model in this embodiment. Figure 2 As shown, the specific implementation method of the RGB-T target tracking model is as follows.

[0077] The RGB-T target tracking model includes two modal branches: an RGB modal branch for RGB modal feature extraction and a TIR modal branch for TIR modal feature extraction. Each modal branch consists of L stacked Transformer modules. Each Transformer module is composed of multi-head self-attention and a multilayer perceptron through residual connections and layer normalization. The Transformer modules of the two modal branches form a Transformer backbone network. The cross-modal target query generator and the spatiotemporal high-order interactive prompting module are located between the two modal branches and are processed in parallel with the two modal branches, while the cross-modal target query generator and the spatiotemporal high-order interactive prompting module are processed serially.

[0078] First, the reference video sequences (i.e., template images) and search images of the two modal branches are input into the RGB-T target tracking model. Template labels for the two modal branches are obtained through patch embedding and positional encoding. , and search tags , Then, the Transformer module extracts features from the RGB modal branches and TIR modal branches respectively. Simultaneously, template markers and spatiotemporal markers initialized with the template markers (the spatiotemporal markers in subsequent frames are propagated from higher-order cues generated in the previous frame) are input into a cross-modal target query generator. This cross-modal target query generator employs a Dynamic Sparse Ghost Fusion Module (DSGF) to construct adaptive, target-related queries by jointly modeling spatiotemporal cues from the RGB and TIR modalities. The cross-modal target query generator contains two parallel implementations of the Dynamic Sparse Ghost Fusion Module. Example: One module is used for the fusion of multimodal template tags, and another is used for the fusion of multimodal spatiotemporal tags; each dynamic sparse ghost fusion module selects a sparse but information-rich token set to achieve multimodal information fusion; after the output of the dynamic sparse ghost fusion module, a multi-head self-attention (MHSA) mechanism is introduced; MHSA takes the multimodal template tag fusion result and the multimodal spatiotemporal tag fusion result output by the dynamic sparse ghost fusion module respectively and concatenates them along the channel dimension as input, realizing dense, content-aware interaction in modality, time and space dimensions, and finally outputting a time-aware target query.

[0079] The time-aware target query output by the cross-modal target query generator, along with search tags obtained from patch embedding and positional encoding of the current and previous frame images, is input into the spatiotemporal higher-order interaction prompting module. This module utilizes a second-order interaction mechanism in the frequency domain, approximating self-attention through Fast Fourier Transform, and extends this second-order interaction to a recursive higher-order interaction. A spatiotemporal diffraction injection scheme is employed, allowing early context injection into subsequent features. The next iteration injects refined higher-order cues to enhance spatiotemporal relevance and modal alignment. Finally, the output of the spatiotemporal higher-order interactive cue module (i.e., higher-order cues) is split into features corresponding to the template tag length to form new spatiotemporal tags, which are then used as higher-order cues in subsequent inference.

[0080] The specific implementation method of the dynamic sparse ghost fusion module is as follows.

[0081] First, a modality-aware sparse sampling strategy suitable for template fusion and a temporal-aware sparse sampling strategy suitable for temporal fusion are adopted. For template markers, the feature magnitude at each marker position is calculated as the l2 norm across channels. The Top-k indices are sampled as follows:

[0082]

[0083] Among them, Top-k This represents the position indices corresponding to the k largest amplitude values; This represents the calculated characteristic amplitude; Indicates a location in time and space. Location, passage eigenvectors; Indicates the total number of channels; Indicates the first value with the largest amplitude. A location index is used to sample salient features.

[0084] For time stamps, the rate of change of features is measured by the mean absolute difference across channels between consecutive time steps:

[0085]

[0086] in, Indicates at time step Location, passage eigenvalues; This represents the calculated characteristic rate of change; The first one with the largest rate of change A location index is used to highlight areas that change significantly over time.

[0087] Sampling sparse index and Cross-modal tagging applied to splicing and In the initial frame, the spatiotemporal marker , Use template tags respectively , Perform initialization; where This represents channel-level splicing, thereby obtaining a sparse, information-rich representation; with For example ( Following the same process, sparse features are obtained as follows: :

[0088]

[0089] in, Indicates based on the obtained index from Sparse features are obtained by mid-sampling .

[0090] To learn more features with low-cost operations, we leverage ghost modules. However, it treats each branch independently, lacking inter-branch communication, which is suboptimal for cross-modal fusion where semantic redundancy must be suppressed while preserving complementarity. First, sparse ghost features are generated. as follows:

[0091]

[0092] in, This represents a point-to-point convolution that extracts intrinsic features from sparse input. Indicates the generation of the first Depth convolution of ghost feature branches, Indicates the number of ghost feature branches; each ghost feature Capture diverse information while maintaining computational efficiency.

[0093] Secondly, in obtaining sparse ghost features Subsequently, lightweight ghost branch interactions are performed to facilitate cross-branch communication and recover latent semantics lost during sparse sampling; each ghost feature Processed as:

[0094]

[0095] in, This indicates detailed ghost features. and For two learnable 1s 1 convolutional layer; The ReLU activation function is used to filter out noisy or invalid interactions, retaining only meaningful responses; additionally, A residual embedding is provided to preserve modality-invariant semantic information.

[0096] Furthermore, for the refined set of ghost features Ghost features in the set are adaptively fused using predicted weights. To capture global cues, all branches are first concatenated along the channel dimension, and global average pooling is applied to obtain a compact global descriptor. Then, pointwise convolutions are used, followed by softmax to generate standardized importance scores; its representation is as follows:

[0097]

[0098] in, Indicates global average pooling. This indicates the importance weights that include all ghost branches.

[0099] Then, the fused ghost features are obtained by weighted summation, represented as:

[0100]

[0101] Finally, the inherent sparse features are along the channel dimension. With the characteristics of the fused ghost By piecing them together, the final output is generated. , represented as:

[0102] .

[0103] Figure 2 This is a schematic diagram illustrating the implementation principle of the cross-modal target query generator in this embodiment. For example... Figure 2 As shown, the implementation method of the cross-modal target query generator is as follows.

[0104] The cross-modal target query generator comprises two parallel instances of a dynamic sparse ghost fusion module: one for fusing multimodal template tags and the other for fusing multimodal spatiotemporal tags, denoted as:

[0105]

[0106] in, , The outputs of two parallel instances of the Dynamic Sparse Ghost Fusion module. Encapsulate the modal complementary spatial signal from the template, and Capture temporal dynamic signals related to the target's recent motion; This indicates a dynamic sparse ghost fusion module.

[0107] To further expand the sensing range, a multi-head self-attention (MHSA) mechanism is introduced into the output of the dynamic sparse ghost fusion module. This allows for the final time-aware target query... exist The ability to adaptively emphasize target-relevant regions while maintaining modal alignment and temporal awareness is expressed as:

[0108]

[0109] in, This represents a multi-head self-attention mechanism.

[0110] Figure 3 This is a schematic diagram illustrating the implementation principle of the spatiotemporal higher-order interactive prompt module in this embodiment. For example... Figure 3 As shown, the implementation method of the spatiotemporal higher-order interactive prompt module is as follows.

[0111] The spatiotemporal high-order interactive prompting module will generate a time-aware target query from the cross-modal target query generator. And multimodal search tags from the current frame: (i.e., key) ) and the multimodal search markers of the previous frame: (i.e., key) () as input.

[0112] First, to capture dense correlations, a second-order interaction is approximated in the frequency domain using a Fast Fourier Transform (FFT); given a query and each search tag (time Modality ), second-order interaction in the frequency domain Described as:

[0113]

[0114] in, and Represents FFT and inverse FFT, express The transpose operation, Represents element-wise multiplication; They represent queries ( ),key( ) and value ( Learnable projections, and (time Modality () represents the query and key obtained after FFT respectively. This represents the attention matrix obtained after inverse FFT. Representation layer normalization. Compared to the self-attention mechanism, the above formula approximates spatial correlation while reducing complexity from... Reduce to .

[0115] To capture richer semantics beyond second-order interactions, the above frequency domain second-order interaction process is extended to recursive higher-order interactions. Instead of applying FFT at each layer, simplified cross-modal fusion is used to recursively update the features of the original domain, allowing the intra-feature positions to change with each recursion, thus considering more positional dependencies during interactions. The recursive update is represented as:

[0116]

[0117] in, .

[0118] Recursive query is represented as:

[0119]

[0120] in, As a dynamic query representation for updates across recursive layers, Indicates capture in The time modality is of Interaction at different levels.

[0121] Although recursive updates capture time clues, each branch All updates are independent; to enhance spatiotemporal correlation, a spatiotemporal diffraction injection scheme is adopted, allowing early features to "diffuse" into later features; firstly, time-aware target queries are integrated through residual fusion. Merged into each higher-order feature, i.e. Then, according to the diffraction sequence ( and These query-injected features are cascaded and refined to achieve perceptual alignment between RGB and TIR modalities; at each step, the current feature is updated by the previous feature through lightweight residual projection, as shown below:

[0122]

[0123] in, Indicates the current feature in the cascade. This represents the guiding features of the previous step; this sequential update injects earlier context into subsequent features and progressively refines them with complementary modalities and temporal cues. Therefore, through... The refined features obtained in the next iteration are denoted as... and ;

[0124] After spatiotemporal diffraction injection, merge time-aware target query And refactor the final enhanced higher-order hints as , Finally, the higher-order hints are broken down into features corresponding to the template token length to form new spacetime tokens. This serves as a clue to the dynamic and spatiotemporally rich nature of subsequent reasoning.

[0125] In this embodiment, the RGBT234, RGBT210, and LasHeR datasets were used for comparative verification. Table 1 shows the comparison results of the proposed method with other multimodal tracking methods on the RGBT234, RGBT210, and LasHeR datasets. As can be seen from Table 1, the proposed method exhibits higher accuracy and robustness compared to other multimodal tracking methods, specifically reflected in PR, NPR, SR, MPR, and MSR. In the evaluation metrics of target tracking, PR represents Precision Rate, and SR represents Success Rate, serving as the main evaluation indicators. NPR represents Normalized Precision Rate, and MPR and MSR represent Maximum Precision Rate and Maximum Success Rate, respectively.

[0126] Table 1

[0127]

[0128] This embodiment also provides an RGB-T target tracking system that jointly models higher-order modes and temporal dynamics, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the above-described method.

[0129] This embodiment also provides a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the above-described method.

[0130] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0131] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0132] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0133] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0134] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for RGB-T target tracking that jointly models higher-order modalities and temporal dynamics, characterized in that, Includes the following steps: 1) Obtain the training dataset and training set annotation files; 2) Construct a joint modeling high-order modality and temporal dynamics RGB-T target tracking model. The RGB-T target tracking model is a dual-branch architecture with RGB modality branches and TIR modality branches, including a Transformer backbone network, a cross-modal target query generator, a spatiotemporal high-order interactive cue module, and a prediction head. For each modality branch, the input frame is first encoded into template tags and search tags through patch embedding and position encoding, and then the features of each modality branch are extracted by Transformer. The cross-modal target query generator dynamically fuses multimodal information by processing the current template marker and the spatiotemporal marker of the previous frame, outputs the cross-modal and temporal fusion features as a time-aware target query, and then passes the time-aware target query to the spatiotemporal higher-order interactive prompt module; The spatiotemporal higher-order interaction cue module integrates the search tags of the previous frame and the current frame to generate updated higher-order cue for multi-location interactions in the current frame, providing guidance for the Transformer and temporal evolution of the model. Then, the updated higher-order cue is injected into the Transformer backbone network to enhance the modeling of long-distance dependencies. At the same time, the updated higher-order cue is propagated to the next frame to maintain temporal continuity. The RGB-T target tracking model was trained using the training dataset and the training set annotation file to obtain generalizable model parameters. 3) The trained RGB-T target tracking model is used to track targets to achieve robust and stable tracking results; The cross-modal target query generator uses a dynamic sparse ghost fusion module to construct adaptive, target-related queries by jointly modeling spatiotemporal cues of RGB and TIR modalities. The cross-modal target query generator contains two parallel instances of the dynamic sparse ghost fusion module: one for the fusion of multimodal template labels and the other for the fusion of multimodal spatiotemporal labels. Each dynamic sparse ghost fusion module selects a sparse but information-rich token set to achieve multimodal information fusion. After the output of the dynamic sparse ghost fusion module, a multi-head self-attention mechanism is introduced. The multi-head self-attention mechanism concatenates the multimodal template tag fusion result and the multimodal spatiotemporal tag fusion result output by the dynamic sparse ghost fusion module along the channel dimension and uses them as input. It realizes dense, content-aware interaction in the modality, time and space dimensions, and finally outputs a time-aware target query. The implementation method of the cross-modal target query generator is as follows: The cross-modal target query generator comprises two parallel instances of a dynamic sparse ghost fusion module: one for fusing multimodal template tags and the other for fusing multimodal spatiotemporal tags, denoted as: in, , The outputs of two parallel instances of the Dynamic Sparse Ghost Fusion module. Encapsulate the modal complementary spatial signal from the template, and Capture temporal dynamic signals related to the target's recent motion; This indicates a dynamic sparse ghost fusion module; , Template markers representing two modal branches, , This represents a spatiotemporal marker in the initial frame; To further expand the perception range, a multi-head self-attention mechanism is introduced into the output of the dynamic sparse ghost fusion module to enable time-aware target querying. exist The ability to adaptively emphasize target-relevant regions while maintaining modal alignment and temporal awareness is expressed as: in, This represents a multi-head self-attention mechanism.

2. The RGB-T target tracking method for jointly modeling higher-order modes and temporal dynamics according to claim 1, characterized in that, In step 1), the training dataset and the corresponding training set annotation file are obtained from the open source dataset, which are RGBT234, RGBT210 and LasHeR tracking datasets.

3. The RGB-T target tracking method for jointly modeling higher-order modes and temporal dynamics according to claim 1, characterized in that, In step 2), the specific implementation of the RGB-T target tracking model includes: The RGB-T target tracking model includes two modal branches: an RGB modal branch for RGB modal feature extraction and a TIR modal branch for TIR modal feature extraction. Each modal branch consists of L stacked Transformer modules. Each Transformer module is composed of multi-head self-attention and a multilayer perceptron through residual connections and layer normalization. The Transformer modules of the two modal branches form a Transformer backbone network. The cross-modal target query generator and the spatiotemporal high-order interactive prompting module are located between the two modal branches and are processed in parallel with the two modal branches, while the cross-modal target query generator and the spatiotemporal high-order interactive prompting module are processed serially. First, the reference video sequences for the two modal branches, namely the template image and the search image, are input into the RGB-T target tracking model. Template labels for the two modal branches are obtained through patch embedding and positional encoding. , and search tags , Then, the features of the RGB modal branch and the TIR modal branch are extracted by the Transformer module, and the template label and the spatiotemporal label initialized with the template label are input into the cross-modal target query generator. The time-aware target query output by the cross-modal target query generator, along with search tags obtained from patch embedding and positional encoding of the current and previous frame images, is input into the spatiotemporal higher-order interaction prompting module. This module utilizes a second-order interaction mechanism in the frequency domain, approximating self-attention through Fast Fourier Transform, and extends this second-order interaction to a recursive higher-order interaction. A spatiotemporal diffraction injection scheme is employed, allowing early context injection into subsequent features. The next iteration injects refined higher-order cues to enhance spatiotemporal relevance and modal alignment. Finally, the output of the spatiotemporal higher-order interactive cue module, i.e., the higher-order cues, is decomposed into features corresponding to the template tag length to form new spatiotemporal tags, which are then used as higher-order cues in subsequent inference.

4. The RGB-T target tracking method for jointly modeling higher-order modes and temporal dynamics according to claim 3, characterized in that, The implementation method of the dynamic sparse ghost fusion module is as follows: First, a modality-aware sparse sampling strategy suitable for template fusion and a temporal-aware sparse sampling strategy suitable for temporal fusion are adopted. For template markers, the feature magnitude at each marker position is calculated as the l2 norm across channels. The Top-k indices are sampled as follows: Among them, Top-k This represents the position indices corresponding to the k largest amplitude values; This represents the calculated characteristic amplitude; Indicates a location in time and space. Location, passage eigenvectors; Indicates the total number of channels; Indicates the first value with the largest amplitude. A location index is used to sample significant features; For time stamps, the rate of change of features is measured by the mean absolute difference across channels between consecutive time steps: in, Indicates at time step Location, passage eigenvalues; This represents the calculated characteristic rate of change; The first one with the largest rate of change A location index is used to highlight areas that change significantly over time; Sampling sparse index and Cross-modal tagging applied to splicing and In the initial frame, the spatiotemporal marker , Use template tags respectively , Perform initialization; where This represents channel-level splicing, thereby obtaining a sparse, information-rich representation; for The sparse features are obtained as follows: : in, Indicates based on the obtained index from Sparse features are obtained by mid-sampling ; Generate sparse ghost features as follows: in, This represents a point-to-point convolution that extracts intrinsic features from sparse input. Indicates the generation of the first Depth convolution of ghost feature branches, Indicates the number of ghost feature branches; each ghost feature Capture different information while maintaining computational efficiency; In obtaining sparse ghost characteristics Subsequently, lightweight ghost branch interactions are performed to facilitate cross-branch communication and recover latent semantics lost during sparse sampling; each ghost feature Processed as: in, This indicates detailed ghost features. and For two learnable 1s 1 convolutional layer; The ReLU activation function is used to filter out noisy or invalid interactions, retaining only meaningful responses; additionally, A residual embedding is provided to preserve modality-invariant semantic information; For a refined set of ghost features Adaptive fusion is performed using predicted weights; first, all branches are concatenated along the channel dimension, and global average pooling is applied to obtain a compact global descriptor. Then, point-wise convolution is used, followed by softmax to generate a standardized importance score; its representation is as follows: in, Indicates global average pooling. This indicates the importance weights that include all ghost branches; Then, the fused ghost features are obtained by weighted summation, represented as: Finally, the inherent sparse features are along the channel dimension. With the characteristics of the fused ghost By piecing them together, the final output is generated. , represented as: 。 5. The RGB-T target tracking method for jointly modeling higher-order modes and temporal dynamics according to claim 3, characterized in that, The implementation method of the aforementioned spatiotemporal higher-order interactive prompt module is as follows: The spatiotemporal high-order interactive prompting module will generate a time-aware target query from the cross-modal target query generator. and multimodal search tags from the current frame. Multimodal search markers from the previous frame As input; To capture dense correlations, a second-order interaction is approximated in the frequency domain using the Fast Fourier Transform (FFT); given a query and each search tag ,time Modality Second-order interaction in the frequency domain Described as: in, and Represents FFT and inverse FFT, express The transpose operation, Represents element-wise multiplication; They represent queries respectively. ,key Sum Learnable projection, and This represents the query and key obtained after performing an FFT, respectively. This represents the attention matrix obtained after inverse FFT. Representation layer normalization; To capture richer semantics beyond second-order interactions, the above frequency domain second-order interaction process is extended to recursive higher-order interactions. A simplified cross-modal fusion recursively updates the features of the original domain, causing the intra-feature positions obtained in each recursion to change, thus considering more positional dependencies during interactions. The recursive update is represented as: in, ; Recursive query is represented as: in, This represents a dynamic query representation that updates across recursive layers. Indicates capture in The time-mode is of Interaction at different levels; To enhance spatiotemporal correlation, a spatiotemporal diffraction injection scheme is adopted, allowing early features to diffract into later features; firstly, time-aware target queries are integrated through residual fusion. Merged into each higher-order feature, i.e. Then, according to the diffraction sequence and The features injected in the query are cascaded and refined to achieve perceptual alignment between the RGB and TIR modalities; at each step, the current feature is updated by the previous feature through lightweight residual projection, as shown below: in, Indicates the current feature in the cascade. Indicates the guiding feature of the previous step; through The refined features obtained in the next iteration are denoted as... and ; After spatiotemporal diffraction injection, merge time-aware target query And refactor the final enhanced higher-order hints as , Finally, the higher-order hints are broken down into features corresponding to the template token length to form new spacetime tokens. This serves as a clue to the dynamic and spatiotemporally rich nature of subsequent reasoning.

6. An RGB-T target tracking system that jointly models higher-order modalities and temporal dynamics, characterized in that, It includes a memory, a processor, and computer program instructions stored in the memory and executable by the processor, wherein when the processor executes the computer program instructions, it can implement the method as described in any one of claims 1-5.

7. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by a processor, the method described in any one of claims 1-5 is implemented.

Images