Structured attribute guided multi-scale context interaction RGB-T sparse fusion tracking method and system

Abstract

The invention belongs to the technical field of computer vision and multi-modal fusion, and discloses a structured attribute guided multi-scale context interaction RGB-T sparse fusion tracking method and system, and the method comprises the steps: obtaining an RGB-T video pair; performing block embedding on the visible light / infrared template and the search area image to obtain a token sequence; based on the CLIP model and a preset discrete attribute template library, dynamically generating a category and color structured attribute prompt of the target, and injecting the category and color structured attribute prompt into a double-branch visual Transform trunk; performing joint modeling on global context dependence and multi-scale space information through a multi-modal context-scale perception interaction integration module; a multi-modal sparse cross fusion module is used, and a self-adaptive Top-K selection mechanism is adopted to realize cross-modal sparse attention fusion; and finally outputting a target bounding box to finish tracking. The problems of semantic guidance deficiency, multi-modal interaction shallow layer, fusion calculation redundancy and the like in existing RGB-T tracking are solved, and tracking is achieved in extreme illumination, scale dramatic change and complex shielding scenes.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and multimodal fusion technology, specifically a structured attribute-guided multi-scale contextual interactive RGB-T sparse fusion tracking method and system. Background Technology

[0002] Visual target tracking, a core task in computer vision, aims to accurately predict the state information of a target in subsequent video frames based on a given bounding box in an initial frame. Due to its crucial role in dynamic scene analysis, visual tracking has been widely applied in intelligent applications such as autonomous driving, intelligent surveillance, and drone navigation. However, unpredictable dynamic changes in real-world environments, such as extreme lighting variations, severe weather, and occlusion, pose significant challenges to single-modal tracking in practical applications. Therefore, the significant information complementarity between multi-sensor data has made multimodal tracking a research hotspot, particularly the RGB-T tracking method combining RGB and thermal infrared images. By fusing the rich color texture information of RGB images with the thermal radiation advantages of TIR images under extreme conditions, this method has proven to achieve more stable and robust tracking performance.

[0003] Despite significant progress in RGB-T tracking technology, existing methods still face several key technical challenges when dealing with complex and ever-changing real-world application scenarios:

[0004] 1. Lack or Insufficient Guidance from Structured Attributes: With the rapid development of visual language models such as CLIP, linguistic text information has shown great potential in various visual tasks, but its application in RGB-T tracking is still in the exploratory stage. Existing methods such as VLCTrack require pre-generating target categories and static text descriptions for specific sequences; this fixed pattern lacks flexibility and versatility. MKFTracker only applies textual cues during the tracker's prediction stage, failing to fully leverage the guiding role of prior linguistic knowledge in multimodal feature extraction and fusion. This limitation prevents the tracker from effectively utilizing structured attributes such as target category and color to enhance the semantic expressiveness of the target, especially when facing semantically similar but visually different targets, lacking effective semantic discrimination capabilities.

[0005] 2. Insufficient Multimodal Deep Interaction and Scale Adaptation Capabilities: Existing RGB-T tracking methods focus more on efficiently integrating multimodal information, but often neglect the importance of deep interactions between multimodal nonlocal contextual information and multiscale information in improving feature representation capabilities. This design flaw leads to poor performance of trackers in complex scenarios such as long-term occlusion, rapid target deformation, or large scale changes, failing to fully utilize the complementary information of RGB and TIR modalities at different contextual and scale levels. In particular, when the target undergoes rapid scale changes, existing methods lack effective scale adaptation mechanisms, making it difficult to maintain stable tracking performance.

[0006] 3. The Efficiency and Accuracy Balance of Multimodal Fusion Strategies: Effective multimodal interaction fusion has become a core focus of current RGB-T tracking research, but existing fusion strategies have significant limitations. On the one hand, simple concatenation or summation operations assign the same weights to different modalities, failing to dynamically adjust modal importance based on the scenario. On the other hand, channel-aware aggregation networks typically only focus on weight allocation along the channel dimension, ignoring the complementary nature of features in the spatial dimension. Furthermore, while traditional global cross-attention mechanisms can achieve good fusion results, they are computationally intensive, significantly increasing computational overhead and potentially diluting key target features with background information, leading to information redundancy and the introduction of irrelevant features, making it difficult to achieve high accuracy while maintaining high computational efficiency. Summary of the Invention

[0007] In view of the shortcomings of the prior art, the present invention addresses the following main issues:

[0008] 1) Existing RGB-T tracking methods either ignore semantic attributes or use static or generative text descriptions, lacking a structured, dynamically constructible target attribute guidance mechanism, thus failing to provide stable and accurate semantic priors for multimodal feature extraction. 2) Existing methods lack effective multimodal deep interaction mechanisms, failing to fully utilize non-local context and multi-scale information, resulting in insufficient robustness and adaptability in complex scenarios such as target occlusion and scale changes. 3) Existing multimodal fusion strategies struggle to achieve high computational efficiency while maintaining high fusion accuracy, failing to achieve efficient and accurate information selection at the cross-modal level, and failing to balance computational efficiency and fusion quality. 4) Existing methods lack a unified collaborative optimization framework, and the lack of effective information transmission and coordination mechanisms between various functional modules limits the improvement of overall system performance.

[0009] This invention addresses key issues in existing technologies, such as the lack of structured target attribute guidance, insufficient multimodal deep interaction and scale adaptation capabilities, difficulty in balancing computational efficiency and accuracy in fusion strategies, and the absence of a unified framework for collaborative optimization. It proposes a systematic and innovative solution. By constructing a structured attribute guidance module, enhancing multimodal deep interaction capabilities, developing efficient sparse fusion strategies, and establishing a unified collaborative optimization framework, this invention provides an RGB-T target tracking system with stronger semantic understanding, higher computational efficiency, and better robustness in complex environments. In particular, it maintains highly reliable and efficient target tracking performance even in challenging scenarios such as extreme lighting changes, rapid target scale changes, and complex background interference.

[0010] The technical solution adopted in this invention is:

[0011] In a first aspect, the present invention provides a structured attribute-guided multi-scale contextual interactive RGB-T sparse fusion tracking method, comprising the following steps:

[0012] S10: Acquire RGB-T video pairs, including visible light template image, infrared template image, visible light search area image, and infrared search area image;

[0013] S20: Block embedding is performed on the template image and the search area image respectively to obtain the corresponding token sequence; S30: Based on the visible light template image, the target structured attribute prompt is dynamically generated using the pre-trained visual language model CLIP and the preset discrete attribute template library, wherein the attributes include at least category attributes and color attributes;

[0014] S40: The structured attribute hints are additively fused with the RGB token sequence and the TIR token sequence to form a semantically guided multimodal token;

[0015] S50 inputs semantically guided multimodal tokens into the dual-branch visual Transformer backbone to extract visible light and infrared features;

[0016] S60: The visible light features and infrared features are sent to the multimodal context-scale perception interactive integration module to obtain enhanced features that simultaneously encode global context dependence and multi-scale spatial information;

[0017] S70: The enhanced visible light features and infrared features are fed into the multimodal sparse cross-fusion module, and cross-modal sparse attention fusion is achieved through an adaptive Top-K selection mechanism to generate fused features;

[0018] S80: Based on the fused features, the target bounding box is output through the prediction head to complete the tracking.

[0019] Preferably, the "dynamic generation of target structured attribute prompts" in S30 includes: extracting visual features from a visible light template image using a CLIP visual encoder; mapping a preset category dictionary and color dictionary to text features using a CLIP text encoder; processing the visual features and text features respectively through a lightweight multimodal adapter before similarity calculation; calculating the similarity between the adapted visual features and text features, selecting the category and color with the highest similarity, and splicing them together to form a structured attribute prompt.

[0020] Preferably, the lightweight multimodal adapter consists of upsampling convolution, ReLU activation function, and downsampling convolution.

[0021] Preferably, the "multimodal context-scale-aware interactive integration module" in S60 includes: a nonlocal context attention submodule for capturing global semantic dependencies; a scale-aware multidimensional spatial attention submodule for extracting multi-scale spatial structure information in parallel along the height and width dimensions; and parallel processing of the fused visible light-infrared-attribute features to output enhanced features.

[0022] Preferably, the scale-aware multidimensional spatial attention submodule employs depthwise separable convolution to perform multi-scale convolution on the height and width dimensions respectively, and suppresses semantic interference between sub-features through group normalization.

[0023] Preferably, the "multimodal sparse cross-fusion module" in S70 includes: using depthwise separable convolution to map visible light features and infrared features to queries, keys, and values ​​respectively; calculating a bidirectional cross-attention matrix; retaining the Top-K maximum weights for each row of attention vectors and setting the rest to zero to generate a sparse attention matrix; and using the sparse attention matrix to complete cross-modal feature fusion.

[0024] Preferably, K is a learnable parameter that can be automatically adjusted during end-to-end training.

[0025] Secondly, this invention provides a structured attribute-guided multi-scale context-interactive RGB-T sparse fusion tracking system, comprising:

[0026] Image input interface for receiving RGB-T video pairs;

[0027] Block embedding unit, used to convert an image into a token sequence;

[0028] The structured attribute hint building unit is used to perform step S30 in claim 1;

[0029] A dual-branch vision Transformer backbone is used to perform step S50 in claim 1;

[0030] A multimodal context-scale-aware interactive integration unit is used to perform step S60 in claim 1;

[0031] A multimodal sparse cross-fusion unit is used to perform step S70 in claim 1;

[0032] The prediction head is used to output the target bounding box.

[0033] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the first aspects.

[0034] Fourthly, the present invention also provides a processor for running a computer program, wherein the computer program performs the steps of the method described in the first aspect when it is executed.

[0035] Compared with the prior art, the present invention has the following advantages:

[0036] 1. Dynamic Construction of Target Structured Attribute Guidance Mechanism: This invention proposes a dynamic construction module for structured attribute prompts. This module, based on RGB template images and predefined language description templates such as "a photo of {class}" and "a photo of {color}", cleverly integrates the pre-trained visual language model CLIP with a specially designed multimodal adapter. By calculating the semantic similarity between the text and the template image, it dynamically selects the category and color text prompts that best match the target characteristics. This adaptive attribute text prompt generation mechanism enables the tracker to automatically extract and filter high-quality semantic descriptions, significantly enhancing its ability to perceive the essential features of the target. It provides rich and accurate semantic prior knowledge for the RGB-T tracking process, effectively solving the problem of insufficient utilization of language information in existing methods.

[0037] 2. Achieving Multimodal Deep Interaction and Scale Adaptation: Addressing the shortcomings of existing methods in multimodal information interaction and scale adaptation, this invention designs a multimodal context-scale-aware interaction integration module and strategically embeds it into the visual Transformer backbone network. This module innovatively combines nonlocal contextual attention and scale-aware multidimensional spatial attention, simultaneously capturing global semantic associations and multi-scale spatial information. This enables deep interaction of cross-modal features at different spatial scales and semantic levels, significantly improving the tracker's adaptability to target scale changes, deformations, and complex scenes.

[0038] 3. Development of an Efficient Sparse Fusion Strategy: To address the balance between efficiency and accuracy in traditional fusion methods, this invention proposes a Multi-modal Sparse Cross Fusion (MMSCF) module. This module innovatively introduces an adaptive Top-K selection mechanism on top of traditional cross-attention, enabling each modality to accurately identify and selectively absorb the most relevant complementary information from another modality, while effectively filtering redundant information and suppressing interference from irrelevant features. This sparse fusion paradigm significantly reduces computational complexity while preserving the most discriminative complementary information, providing an efficient and accurate multi-modal fusion solution for RGB-T tracking.

[0039] By organically combining the above-mentioned innovative technologies, this invention aims to build a multimodal target tracking system that can perform well under various complex environmental conditions (such as extreme lighting changes, severe weather, target occlusion, scale changes, etc.), ensuring the high reliability and accuracy of the tracking task under language guidance, while maintaining good computational efficiency, thereby providing a better technical solution for practical intelligent applications.

[0040] Specifically as follows:

[0041] 1) This invention proposes a structured attribute prompting module, based on the pre-trained visual language model CLIP and a predefined attribute template library, such as "a photo of {class}" and "a photo of {color}", dynamically combining the category and color attributes that best match the target through semantic similarity retrieval. This mechanism avoids free text generation, provides interpretable, task-customizable, and dynamically adaptable structured semantic guidance, and significantly enhances the semantic consistency and discriminative ability of the target in RGB and TIR modalities.

[0042] 2) This invention designs a multimodal context-scale-aware interaction integration module, which innovatively combines nonlocal contextual attention and scale-aware multidimensional spatial attention to achieve deep interaction of cross-modal features at different semantic levels and spatial scales. This module can effectively capture long-distance dependencies and multi-scale spatial information, significantly enhancing the tracker's adaptability and robustness to complex scenes.

[0043] 3) This invention proposes a multimodal sparse cross-fusion module, which introduces an adaptive Top-K selection mechanism to achieve computationally efficient selective fusion of cross-modal features. This module can accurately identify and selectively fuse the most relevant cross-modal information, significantly reducing computational overhead while effectively suppressing redundant and interfering features, providing an efficient and accurate multimodal fusion solution for RGB-T tracking.

[0044] 4) This invention organically integrates three key modules—language guidance, deep interaction, and sparse fusion—into a unified end-to-end training framework, and designs multi-level feature interaction paths to ensure that each module can work together to optimize and promote each other, thereby maximizing the overall performance of the system. Attached Figure Description

[0045] Figure 1 This is a diagram illustrating the overall framework of the RGB-T target tracking method of the present invention.

[0046] Figure 2 This is the structured attribute hint building module of the present invention;

[0047] Figure 3 This is the multimodal context-scale-aware interactive integration module of the present invention;

[0048] Figure 4 This is the multimodal sparse cross-fusion module of the present invention. Detailed Implementation

[0049] With the rapid development of artificial intelligence technology, visual target tracking, as a core research direction in the field of computer vision, plays an increasingly important role in practical applications such as intelligent surveillance, autonomous driving, and drone navigation. The essential task of target tracking is to continuously and accurately locate and track a specific target in a continuous video stream, given an initial target position. However, modern target tracking systems still face challenges from various aspects, including changes in lighting, inclement weather, and dynamic occlusion, when dealing with complex and ever-changing real-world environments.

[0050] Traditional unimodal target tracking methods primarily rely on RGB visible light images for feature extraction and target localization. Early research used correlation filtering-based trackers to achieve target tracking by constructing a target appearance model and performing efficient matching in the frequency domain. Subsequently, deep learning-based tracking methods emerged, from early fully convolutional networks to later Siamese network structures, significantly improving feature representation capabilities and tracking accuracy. However, these unimodal methods exhibit significant limitations in extreme environmental conditions, particularly in scenarios with poor visual conditions such as nighttime, fog, and strong light. The significant degradation of RGB image quality directly restricts the reliability and practicality of the tracking system.

[0051] To overcome the inherent limitations of single-modal tracking, RGB-T (visible and thermal infrared) multimodal fusion tracking technology has emerged. RGB-T tracking leverages the rich texture details of RGB images and the stable temperature characteristics of thermal infrared images simultaneously, enabling more robust and accurate target tracking in complex environments. Due to its insensitivity to ambient light, thermal infrared images provide reliable target thermal radiation information in low-light and nighttime scenes, effectively compensating for the shortcomings of RGB images in these scenarios. This complementarity gives RGB-T tracking stronger environmental adaptability and tracking stability in practical applications.

[0052] Early RGB-T tracking research was primarily based on traditional handcrafted features and shallow neural network architectures. Inspired by multi-domain network frameworks (MDNet, RT-MDNet) in single-modal tracking, researchers began extending the single-branch structure to a two-branch architecture, handling feature extraction from RGB and TIR images separately. MANet, as an early representative work, designed multiple adapters to separately acquire shared target representations and modality-specific information, laying the foundation for subsequent multimodal fusion. Building upon MANet, MANet++ further introduced multi-kernel maximum mean divergence loss to enhance the learning effect of shared features and modality-specific features. The CAT method, from the perspective of challenge attributes, designed a challenge-aware network to specifically handle modality-sharing challenges such as rapid motion and occlusion, as well as modality-specific challenges such as illumination changes and thermal crossover.

[0053] With a deeper understanding of multimodal information interaction, more research is focusing on how to more effectively achieve cross-modal feature fusion. DMCNet proposed a mutual conditional module, using discriminative information from one modality as a condition to guide feature learning of the target's appearance in another modality, while introducing a dual-gating mechanism to further suppress noise interference. Considering the importance of the challenging attribute information implicit in RGB-T tracking for improving the model's discriminative ability, ADRNet introduced an attribute-driven residual branch, while APFNet constructed an attribute-based progressive fusion network specifically designed to mitigate typical challenges such as occlusion, scale variations, and hot crossover.

[0054] While the aforementioned shallow CNN-based trackers have achieved good results in specific scenarios, they have inherent limitations in target feature extraction, struggling to effectively model long-range dependencies and capture rich contextual information. With the superior performance of Vision Transformer (ViT) in image recognition and single-modal visual tracking, researchers have begun exploring its extension to RGB-T tracking tasks. This shift in technical approach not only enhances feature extraction capabilities but also provides new possibilities for designing more efficient cross-modal fusion mechanisms.

[0055] TBSI, as an early exploration of introducing Transformer into RGB-T tracking, proposed a template bridging search region interaction module, which bridges cross-modal interactions between multimodal search regions by collecting task-related target and environmental context. However, this interaction module does not fully consider the scale variation of the target, and traditional global cross-attention mechanisms risk the dilution of key target features by background information.

[0056] In recent years, inspired by the successful application of prompt learning and adapter techniques in the field of natural language processing, a class of RGB-T tracking methods based on visual cues or adapters has emerged and achieved significant performance improvements. ProTrack transforms multimodal input into single-modal processing by providing prompt examples, which simplifies the tracking process to some extent, but this design inevitably sacrifices the inherent characteristics and advantages of other modalities. Methods such as ViPT, OneTracker, and Un-Track construct general multimodal tracking frameworks by introducing more refined prompt learning mechanisms. ViPT, for example, designs multiple modal complementary cues, learning modality-related cue information to enable the base model to effectively adapt to various downstream multimodal tracking tasks. OneTracker further proposes a unified cue embedding mechanism, modeling auxiliary modalities such as infrared, event, and depth into a unified cue form, and introduces cross-modal tracking cues to achieve effective fusion of additional information.

[0057] However, existing unified trackers generally employ a single backbone structure, overly relying on RGB modality as the dominant information source. This unbalanced dependency may lead to shallow feature mining of specific auxiliary modalities, and the complementary information between modalities is not fully utilized. The BAT method addresses this issue by proposing a bidirectional adapter to learn cross-modal cue information and achieve effective fusion of visual cues. However, its symmetrical design assumes that all modalities have the same reliability, ignoring the inherent differences and complementarities between modalities under different environmental conditions. SDSTrack, on the other hand, designs a symmetrical multimodal adapter, aiming to transfer feature extraction capabilities from RGB modality to other modalities while simultaneously facilitating information exchange between multimodal tokens. However, the limitations of the adapter's own structure make it difficult to support deeper interactions and fusion needs between modalities.

[0058] Meanwhile, the application of semantic prior information in computer vision tasks is rapidly developing. The success of visual language models such as CLIP demonstrates the enormous potential of textual information in enhancing visual understanding. In the field of object tracking, a few works have attempted to introduce linguistic information into RGB-T tracking. VLCTrack provides semantic guidance by pre-generating object categories and generating static text descriptions for specific sequences, but this fixed text generation pattern lacks sufficient flexibility and versatility, making it difficult to adapt to dynamically changing tracking scenarios. MKFTracker, on the other hand, only applies textual cues during the tracker's prediction phase. This design prevents linguistic prior knowledge from playing a guiding role in the early stages of multimodal feature extraction and fusion, thus limiting further improvements in tracking performance.

[0059] Of particular note are recently published patents that attempt to introduce natural language descriptions into target tracking tasks. For example, CN120013992A proposes a UAV target tracking method based on natural language descriptions. This method only uses visible light images and manually provided text descriptions for visual-linguistic alignment, without addressing thermal infrared modalities. Therefore, it cannot solve the failure problem of RGB images under extreme lighting conditions and is not applicable to true RGB-T multimodal scenarios, fundamentally different from the RGB-T multimodal tracking task addressed by this invention. Another work, CN120031918A, while addressing RGB-T tracking and introducing text guidance, proposes an RGB-T tracking method based on a BLIP generative model. This scheme automatically generates a natural language description of the target (e.g., "a red car moving left") through BLIP and combines it with a sparse attention mechanism for feature enhancement. However, this method has certain limitations: (1) it relies on open-domain text generation, the semantic content is uncontrollable, and it is easily affected by image quality, resulting in semantic drift; (2) its sparse attention only acts on the text branch and does not involve cross-modal sparse interaction between RGB and TIR modes; (3) it does not consider target scale changes and context modeling problems, and lacks multi-scale spatial structure perception capabilities. On the other hand, although patent CN120013992A introduces natural language description, its application scenario is drone traffic accident tracking, and it only uses visible light images and manually provided text descriptions for visual-language alignment, without involving thermal infrared (TIR) ​​modes. Therefore, this method is essentially a single-modal visual-language tracking method, which cannot cope with the failure of RGB images under extreme lighting conditions such as night and haze. Therefore, the above methods cannot solve the core problems of structured target attribute guidance, multi-scale context interaction and efficient cross-modal sparse fusion targeted by this invention.

[0060] In terms of multimodal information fusion strategies, existing methods can be roughly divided into three categories: The first category, such as MANet and ADRNet, uses simple feature concatenation or weighted summation and other fixed fusion operations. These methods assign the same importance weights to different modalities and cannot dynamically adjust the contribution of modalities according to specific scenarios. The second category, such as MTNet and SiamMLAA, designs channel-aware aggregation networks to achieve RGB and TIR feature fusion, but usually only focuses on the weight allocation of the channel dimension and ignores the complementary characteristics of features in the spatial dimension. The third category uses the traditional global cross-attention mechanism to achieve cross-modal information interaction. Although it can achieve good fusion results, the computational cost is huge and it is easy to cause key target features to be diluted by irrelevant background information.

[0061] Although the above methods have made some progress under their respective design ideas, they still have the following significant shortcomings:

[0062] 1. Lack of target structured attribute guidance mechanism: Existing methods either completely ignore the potential value of semantic priors or adopt static, generative text description methods, which cannot achieve dynamic and controllable semantic guidance based on discrete attributes such as target category and color, severely limiting the role of semantic information in improving tracking performance.

[0063] 2. Insufficient ability to achieve deep multimodal interaction and multiscale context awareness: Most existing methods focus on how to integrate multimodal information simply and efficiently, but often ignore the importance of non-local context information and multiscale and spatial structure information in tracking tasks. They lack effective mechanisms to simultaneously model global semantic dependencies and local scale changes, and fail to fully capture and utilize these key information.

[0064] 3. It is difficult to balance efficiency and accuracy in fusion strategies: Traditional fusion methods are either too simple, resulting in limited fusion effects, or have excessively high computational complexity and are prone to introducing noise interference, making it difficult to achieve efficient computation while maintaining high accuracy.

[0065] like Figure 1 As shown, this invention proposes a structured attribute-guided multi-scale contextual interactive RGB-T sparse fusion tracking method, which includes the following steps:

[0066] S0: Input video image data

[0067] Description: This method uses video images as input and receives four types of image data: visible light search area images. With infrared search area image and the corresponding visible light template image. With infrared template image To ensure consistency in data processing, the input image data was standardized by adjusting the size of the search area image to 256×256 and the template image to 128×128, thus establishing a standardized data foundation for subsequent processing.

[0068] S1: Block embedding module implements image feature transformation and mapping

[0069] Description: In the initial stage of feature extraction, the system employs block embedding technology to preprocess all input image data. Specifically, this module first divides the complete image into a grid according to a fixed scale, with each sub-region having a pixel size of 16×16. Subsequently, a linear projection layer is used to convert these discrete image blocks into fixed-dimensional vector representations, achieving a mapping from pixel space to feature space.

[0070] To preserve the spatial topological structure of the image, the system incorporates corresponding positional encoding into each feature vector, enabling the model to perceive the relative positional relationships between different image patches. Through this processing, the original image is ultimately converted into a series of position-aware token sequences, forming the initial feature representation.

[0071]

[0072] in It's the Patch Embed module. The initial features of the token form representing the RGB and TIR search regions. The number of patches for the search area. It is the length of the token. The initial features represent the token form of the RGB and TIR templates. This represents the number of template patches. and These represent the location embedding of the search area and the template, respectively.

[0073] S2: Dynamically generated structured attribute hints for specific targets.

[0074] Description: In target tracking, target attribute guidance can effectively improve feature representation and target discrimination capabilities.

[0075] To address this, the present invention proposes a structured attribute hint construction module. While performing block embedding transformation on the image, this generator directly uses a visible light template image and a predefined language template with target attributes as input to dynamically generate highly targeted attribute hints, thereby guiding subsequent networks to focus on more discriminative target features.

[0076] The structured attribute hint construction module proposed in this invention is as follows: Figure 2 As shown, a multi-path structure is employed to process visual information and attribute information separately. In the visual path, the RGB template image is input into the visual encoder of the visual language pre-trained model CLIP to extract high-level semantic visual features:

[0077]

[0078] To enhance the expressive power of visual features, this invention designs a lightweight visual adapter in the generator, consisting of upsampling convolution, ReLU activation function, and downsampling convolution, to finely adjust the extracted visual features so that the image features output by the CLIP visual encoder are adapted to the downstream target tracking task.

[0079]

[0080] Next, the features output by the adapter are added to the visual features that have been processed by residual connection and high-level semantic visual features, and then the generator outputs the visual features after standardization.

[0081]

[0082] In the target category and target color attribute path, this invention refers to the categories in the COCO dataset in the field of computer vision and constructs a category set containing 80 common target categories. Examples include kite, person, dog, and car. Simultaneously, a color tag set containing 12 colors was constructed. Examples include black, red, white, and blue. Based on this, we further employ unified natural language templates "a photo of {class}" and "a photo of {color}" to construct candidate category text description templates and color text representation templates:

[0083]

[0084] The CLIP text encoder maps category and color text descriptions to a high-dimensional semantic space. To ensure consistency with the visual path and maintain feature space uniformity, a lightweight text adapter with the same structure is applied to the text features. After residual connection and standardization, the category and color attribute text features are obtained.

[0085]

[0086]

[0087]

[0088] To determine the category text description and color text description that best match the current target, this invention calculates the similarity between the adapted visual features and all category text features, and also calculates the similarity between the adapted visual features and all color text features.

[0089] Specifically, the image features are multiplied by the text feature matrices of all categories and the color text feature matrix, respectively, to calculate the matching score between the target image and the semantic description of each category, and the matching score between the target image and each color description, and then... The function is normalized on the last dimension to obtain the probability distribution of the image for target attributes such as text cues of various categories and color text cues. and This reflects the degree of matching between the target image and the text descriptions of each category and color. Then, using... Function to obtain probability distribution and The index corresponding to the maximum value in the middle and .

[0090]

[0091]

[0092] Next, select the index from the text features. and The corresponding category and color semantic vectors, i.e., the category and color features with the highest similarity, are then concatenated to form a dynamically generated target attribute cue. This cue integrates both category and color semantic information of the target, effectively enhancing the model's discrimination ability in complex backgrounds.

[0093]

[0094]

[0095] S3: Dual-branch backbone extracts visible light and infrared image features

[0096] Description: After obtaining the initial image features and target structured attribute hints through the above steps, the template tokens of the visible light image are concatenated with the search region tokens to obtain the RGB branch tokens, i.e. The template tokens and search region tokens of the infrared image are concatenated to obtain the tokens for the TIR branch, i.e. Then, the text prompts generated by the dynamic text prompt generator will be used. and and Perform additive fusion to obtain RGB features with voice prompts. and TIR characteristics .

[0097] The fused features are input into a dual-branch backbone network composed of a visual Transformer encoder for joint feature extraction and matching. Considering the similarity of the feature processing procedures for RGB and TIR modalities, RGB tokens that incorporate structured target attribute cues are used here. Let's take an example to illustrate the processing flow of the ViT backbone network. In each encoder, features are transformed into queries through a linear mapping. ,key Sum The attention weight matrix is ​​generated through matrix multiplication:

[0098]

[0099] in, This represents the matrix transpose operation. This represents the normalization function. Through this mechanism, search region tokens and template tokens not only enhance their respective feature representations but also achieve feature fusion. Furthermore, the fused language cues further enhance the semantic expressiveness of the features. Through successive visual Transformer encoder layers, the features of the search region and template tokens are progressively extracted and optimized, and their matching relationships are effectively captured, thus laying the foundation for subsequent multi-modal target localization.

[0100] S4: Multimodal Context-Scale Aware Interaction Integration

[0101] Description: To obtain a more comprehensive and discriminative multimodal target representation, this invention embeds a proposed multimodal context-scale-aware interactive integration module after a portion of the visual Transformer encoder layer. The module structure is as follows: Figure 3 As shown, the modal tokens extracted from the RGB and TIR branches are used as basic inputs, and attribute guidance information dynamically generated by the structured attribute hint building module is integrated to form a three-modal collaborative enhancement framework.

[0102] Specifically, the extracted RGB and TIR token features are added element-wise with attribute hints to achieve semantic enhancement, and then concatenated to achieve multimodal fusion and joint processing. Simultaneously, a linear layer is applied to unify the feature space and promote intermodal information exchange. Finally, the tokens are reshaped into tokens. The shape features facilitate spatial manipulation.

[0103]

[0104]

[0105]

[0106] Subsequently, we applied the proposed nonlocal contextual attention and scale-aware multidimensional spatial attention processing fusion features in parallel. The former effectively captures global contextual dependencies to enhance the semantic understanding of the target, while the latter enhances the spatial attention distribution in both wide and high dimensions. At the same time, we utilize multi-scale deep separable convolution to extract key target structural information at different scales.

[0107] The structure of nonlocal contextual attention is as follows Figure 3 As shown in (b), we employ a series of convolutional layers and The function performs nonlocal transformations on the fused features to achieve global context modeling across spatial locations. To capture and process global context information more efficiently, we use adaptive average pooling to compress the features appropriately, preserving the main structural information with strong semantic discriminative power.

[0108]

[0109] Then, a multilayer perceptron consisting of convolution, layer normalization, and activation functions is used to perform nonlinear transformations on the context features to enhance expressive power and discriminativeness.

[0110]

[0111] Finally, we will integrate features and processes. The attention maps of the context channels activated by the function are multiplied to enhance the feature response of semantically relevant regions while suppressing background interference, and this is used as the output feature of the module.

[0112]

[0113] The structure of the scale-aware multidimensional spatial attention module is as follows: Figure 3 As shown in (c), the features are analyzed along both the height and width dimensions. Global average pooling is performed to obtain the distribution information of the target in different spatial dimensions, resulting in two independent spatial description sequences. To learn more refined spatial distribution and multi-scale information, each sequence is divided into multiple independent sub-features (set to four in this invention), as follows:

[0114]

[0115]

[0116] To extract multi-scale structural spatial information, a convolution kernel size of [size missing] was used. Depthwise separable convolutions are used to process these sub-features in parallel to extract target structural features at different scales from high and wide dimensional spaces. This multi-scale design can simultaneously focus on the local details and global morphology of the target, adapting to targets of different sizes and shapes.

[0117]

[0118] After decomposing sub-features and capturing spatial information at different scales, we concatenate sub-features with different scale spatial structures to construct a complete multi-scale spatial representation. Then, we use grouped normalization to independently normalize each sub-feature to alleviate semantic interference between them and avoid attention dilution. Then we use... The functions are processed separately to generate multi-scale spatial attention maps in the height and width directions.

[0119]

[0120] Finally, we multiply the two spatial attentions obtained with the fusion features to obtain the output of the module, realizing multi-scale and spatial dimension interaction.

[0121]

[0122] S5: Multimodal Sparse Cross-Fusion

[0123] Description: In RGBT tracking tasks, effectively fusing complementary information from different modalities is crucial, but it also requires addressing issues of redundant information interference and computational complexity. To address this, this invention proposes a multimodal sparse cross-fusion module, the structure of which is as follows: Figure 4 As shown, efficient selective fusion of cross-modal information is achieved through adaptive feature selection and dynamic sparse attention computation.

[0124] This module enriches the RGB features output by the multimodal context-scale awareness module. and TIR characteristics As input, to enhance feature representation and maintain the stability of information flow, we first implement a feature preprocessing strategy: applying a multilayer perceptron (MLP) to capture complex nonlinear feature transformations, combining layer normalization to eliminate feature distribution differences, and retaining the original feature information through residual connections. Subsequently, we apply convolution operations to further extract spatially local correlated features, thereby building a high-quality feature foundation for subsequent cross-modal interactions.

[0125]

[0126] To efficiently construct query, key, and value representations, we employ depthwise separable convolutions, which are more computationally efficient and effectively preserve spatial context information, to map the processed features to the corresponding query, key, and value for each modality, i.e., the RGB modality. and TIR modes This provides the necessary components for cross-attention computation. Subsequently, we compute the cross-modal initial attention matrix. and :

[0127]

[0128]

[0129] in For depthwise separable convolution, For feature dimension, This indicates the transpose operation. and These represent attention mappings from RGB to TIR and from TIR to RGB, respectively.

[0130] Dynamic Sparse Attention Mechanism. Unlike directly using a complete attention matrix, we introduce a dynamically learnable sparse attention mechanism that automatically determines the optimal attention matrix through end-to-end training. The value is used to select the most relevant and complementary cross-modal feature correspondences from the attention matrix. The specific implementation is as follows: First, an initial mask matrix with all elements equal to 0 is constructed. Then, combined with the results obtained from dynamic learning Values ​​are used to generate a Top-K sparse mask binary matrix. and .

[0131]

[0132] Where TopK represents the sparse mask matrix generation function. This function is applied to the attention matrix. and For each row, select the row with the highest attention value. The indexes of each element are used to set the corresponding positions in the initial all-zero mask matrix to 1. This design allows the model to focus on the most informative cross-modal correspondences while dynamically adjusting the sparsity to suit the needs of different scenarios.

[0133] Subsequently, we perform element-wise multiplication of the initial attention matrix with the corresponding Top-K sparse mask matrix to obtain the sparse attention matrix:

[0134]

[0135] in This indicates element-wise multiplication. This step effectively preserves the most critical cross-modal attention weights while setting the remaining positions to 0, significantly reducing computation and suppressing interference from irrelevant features.

[0136] After obtaining the sparse attention matrix, execute Normalization is performed and multiplied with the corresponding value matrix to achieve sparse cross-fusion between RGB and TIR features.

[0137]

[0138] Finally, MLP operations are performed on the fused features to enhance their expressive power, and the original information is preserved through convolutional residual connections. Finally, batch normalization is performed to stabilize the feature distribution and generate high-quality fused features.

[0139]

[0140] Fusion features and As the output of this module, it is fed into the next stage, the ViT block, to further extract high-level semantic features and enhance the synergistic and complementary effects between modalities.

[0141] S6: Predictive Head

[0142] Description: Finally, the network feeds the deeply fused multimodal feature vectors into the prediction network module, which undertakes the key output task. The prediction network, through classification and regression branches, transforms the abstract representation in the high-dimensional feature space into a two-dimensional image coordinate system, thereby determining the spatial location information of the target object. Specifically, this module can output accurate bounding box parameters, achieving precise localization and continuous tracking of the target object, providing reliable tracking performance for the system.

[0143] Through the above steps, this invention provides an efficient and robust multimodal target tracking method. This method not only enhances the ability to capture multi-scale information but also achieves effective fusion of different modal information through a multimodal cue interaction learning module and a cross-fusion adaptor. Ultimately, the system can provide high-precision target tracking results in various complex environments, meeting the needs of practical applications. In the final stage, the system feeds the deeply fused multimodal feature vectors to the prediction network module, which undertakes the key output task. The prediction network, through classification and regression branches, transforms the abstract representation in the high-dimensional feature space back into a two-dimensional image coordinate system, thereby determining the spatial location information of the target object. Specifically, this module can output accurate bounding box parameters, achieving precise positioning and continuous tracking of the target object, providing reliable tracking performance assurance for the system.

[0144] In summary, this invention constructs a high-performance and highly adaptable multimodal target tracking technology solution. The core advantage of this solution lies in its powerful multi-level information mining capabilities. Through a structured attribute hinting module, it accurately captures the multi-attribute features of the target, providing effective semantic guidance for subsequent tracking. Simultaneously, the multimodal context-scale-aware enrichment module successfully achieves full interaction of heterogeneous modal data across multiple scales and spatial dimensions. Combined with a multimodal sparse fusion module, it achieves efficient fusion of multimodal features. In complex and ever-changing real-world application scenarios, this system exhibits excellent environmental adaptability, capable of outputting high-quality tracking results under various complex conditions, fully meeting the stringent requirements for accuracy and robustness in engineering practice.

[0145] This unified, collaborative design ensures that the three innovations—structured attribute guidance, deep multimodal feature interaction, and sparse fusion—can mutually promote and work synergistically, rather than simply being a stack of modules. Early injection of language information provides better semantic guidance for subsequent multimodal interactions; deep modal interactions provide richer feature representations for sparse fusion; and the efficiency of sparse fusion enables the entire system to support more complex language guidance and interaction mechanisms. This synergistic effect manifests in practical applications as stronger tracking stability and higher computational efficiency, especially in handling long-term tracking tasks, maintaining consistently high-precision tracking performance.

[0146] Through the deep integration of the aforementioned key technological innovations, this invention successfully constructs a multimodal target tracking system with strong environmental adaptability. This system not only effectively overcomes the technical bottlenecks of traditional methods in areas such as language information utilization, deep modal interaction, and efficient fusion at the theoretical level, but also demonstrates outstanding performance in practical application scenarios. Especially when facing challenging scenarios such as extreme lighting changes, severe weather conditions, rapid changes in target scale, and complex occlusion, this invention maintains consistently stable high-precision tracking results, significantly improving the system's reliability and practicality. These technological breakthroughs not only provide new research paradigms and solutions for the field of multimodal visual tracking, but also offer more flexible, efficient, and robust technical solutions for practical applications such as intelligent monitoring, autonomous driving, and drone navigation, possessing significant academic value and broad prospects for industrial application.

[0147] Example 2

[0148] Structured attribute-guided target semantic enhancement capabilities

[0149] Technical Solution: A structured semantic cues are dynamically constructed using a module that leverages CLIP's visual-linguistic alignment capabilities. This module automatically estimates the target's category and primary color attributes from RGB template images and matches them against predefined discrete attribute templates (e.g., "a photo of a {class}" and "a photo of {color}") to dynamically build structured semantic cues. These cues serve as prior injections to the tracking backbone, enabling precise semantic guidance.

[0150] Possible alternatives:

[0151] (1) Use manually labeled categories / color tags to directly construct prompts (requires additional labeling and is not suitable for actual tracking scenarios);

[0152] (2) Generative models (such as BLIP, LLaVA) are used to generate free text descriptions;

[0153] (3) Use only category hints and ignore auxiliary attributes such as color.

[0154] Limitations of the alternative:

[0155] (1) Manual annotation is not scalable, which violates the setting of the tracking task that "only depends on the initial template";

[0156] (2) Generative text semantics are uncontrollable and easily affected by image quality, and are prone to hallucinations when color information is missing in the infrared modality;

[0157] (3) Single-category prompts have limited ability to distinguish between target categories in dense scenarios (such as multiple "car").

[0158] Advantages of this invention: Through automatic, discrete, and multi-attribute structured prompts, it achieves interpretable, robust, and task-oriented semantic guidance, which is significantly superior to existing alternative paths.

[0159] Example 3

[0160] 2. Multimodal deep interaction and multi-scale context awareness capabilities

[0161] Technical Solution: By designing a multimodal context-scale-aware interactive integration module, this invention achieves deep interaction of cross-modal features and capture of multi-scale information. This module innovatively combines nonlocal context attention and scale-aware multidimensional spatial attention. The former captures long-distance dependencies through global context modeling, while the latter extracts key structural information from different spatial dimensions through multi-scale deep separable convolution.

[0162] Possible alternatives:

[0163] (1) Modality fusion is performed using simple feature splicing or weighted averaging;

[0164] (2) Use standard cross-attention or channel attention mechanisms;

[0165] (3) Only one fusion is performed at the end of the network, lacking intermediate layer interaction.

[0166] Limitations of the alternative:

[0167] (1) Shallow fusion cannot model long-distance dependencies and fails when occluded or deformed;

[0168] (2) Single-scale convolution is difficult to adapt to drastic changes in target scale;

[0169] (3) The lack of multi-level interaction leads to the insufficient exploration of modal complementarity.

[0170] Advantages of this invention: Through joint modeling of context, multi-scale, multi-dimensional space and multi-level, it can effectively handle complex scenes such as target occlusion and rapid scale changes. Through trimodal collaborative enhancement (RGB, TIR, attribute text), it achieves a more comprehensive feature understanding and significantly improves the robustness and adaptability of the tracker.

[0171] 3. Efficient and sparse cross-modal fusion strategy

[0172] Technical Solution: A multimodal sparse cross-fusion module was developed. This module introduces an adaptive Top-K selection mechanism and dynamic sparse attention computation to achieve computationally efficient selective fusion of cross-modal features. By accurately identifying the most relevant cross-modal correspondences, MSCF significantly reduces computational overhead while effectively suppressing redundant and interfering features, solving the computational complexity and noise interference problems of traditional global attention mechanisms.

[0173] Advantages: Compared with traditional global cross-attention or simple feature stitching methods, MSCF achieves the best balance between computational efficiency and fusion accuracy. Through dynamic and learnable sparsity adjustment, it ensures that modalities can effectively complement each other and reduce negative interference, providing an efficient and accurate fusion solution for RGB-T tracking.

[0174] Example 4

[0175] Overall Methodology and System Architecture

[0176] Technical Solution: This paper proposes an RGB-T multimodal target tracking method based on dynamic language guidance, covering the complete technical chain from text prompt generation, feature extraction, deep interaction to sparse fusion. This method constructs an innovative and powerful end-to-end target tracking system that can maintain highly reliable tracking performance under challenging scenarios such as extreme lighting changes, rapid changes in target scale, and complex background interference.

[0177] Advantages: By introducing dynamic language priors, enhancing deep multimodal interaction capabilities, and optimizing fusion computation efficiency, the entire method achieves a comprehensive improvement in semantic understanding, computational efficiency, and robustness, forming a complete solution from input RGB-T image pairs to accurate target localization.

[0178] Summarize

[0179] This invention provides an efficient and robust RGB-T multimodal target tracking method by introducing a structured attribute prompting dynamic construction module, a multimodal context-scale-aware interaction integration module, a multimodal sparse cross-fusion module, and a unified collaborative optimization framework. These innovations significantly surpass existing alternatives in semantic understanding, deep interaction, efficient fusion, and collaborative optimization. While some traditional alternative technologies exist, they fall short of the solution proposed in this invention in terms of semantic guidance capabilities, cross-modal interaction depth, computational efficiency, and overall coordination, and cannot simultaneously meet the comprehensive performance requirements of high accuracy, high efficiency, and strong robustness.

[0180] Terminology Explanation

[0181] 1. Object Tracking: Object tracking is a core technology in computer vision, aiming to continuously locate and track the position and state changes of a specific target object in a continuous sequence of video frames. Given the position information of a target, typically marked as a bounding box in the initial frame, the object tracking algorithm needs to automatically estimate the accurate position of the target in subsequent frames and output the corresponding bounding box coordinates. Object tracking technology is widely used in video surveillance, autonomous driving, human-computer interaction, augmented reality, and other fields. Modern object tracking methods often face challenges such as changes in target appearance, occlusion, lighting changes, scale changes, and background interference, requiring strong robustness and real-time performance.

[0182] 2. Language Prompt: Language prompts are a technique that uses natural language descriptions to provide additional semantic guidance to computer vision models. In multimodal learning frameworks, language prompts typically take the form of structured text templates, such as descriptive statements like "a photo of a {class}", where {class} represents the specific category name of the target to be identified. In this way, language prompts can embed rich prior semantic knowledge into the visual processing flow, guiding the model to focus on specific target features and attributes. The core advantage of language prompts lies in bridging high-level semantic understanding with low-level visual features, enabling the model to combine human cognitive priors to enhance its understanding of image content, thereby significantly improving the accuracy of target recognition and tracking. Compared to purely visual methods, the semantic constraints introduced by language prompts can effectively alleviate visual ambiguity problems and improve the model's discrimination ability in complex scenes. In this invention, the dynamic text prompt generator can adaptively generate the most suitable language prompt based on the actual visual features of the target, achieving a breakthrough from static predefined descriptions to dynamic semantic guidance, providing the RGB-T tracking system with more accurate and flexible target understanding capabilities.

[0183] 3. Visual Language Pre-training Model: CLIP (Contrastive Language-Image Pre-training) is a large-scale visual language pre-training model developed by OpenAI. It is trained on massive image-text pair data through contrastive learning. The CLIP model consists of two core components: a visual encoder and a text encoder, which map images and text to a unified high-dimensional semantic space, respectively. By maximizing the similarity of matching image-text pairs in the embedding space while minimizing the similarity of mismatched pairs, CLIP achieves deep alignment between visual and language modalities. This gives CLIP powerful zero-shot classification and cross-modal retrieval capabilities, enabling it to understand the correspondence between image content and natural language descriptions. In this invention, CLIP is used to build a dynamic text prompt generator to achieve automatic language prompt generation based on visual content.

[0184] 4. Multimodal Fusion: Multimodal fusion refers to the technical process of effectively integrating information from different perceptual modalities or data sources, aiming to improve overall system performance by leveraging the complementarity and synergy between modalities. In RGB-T target tracking, multimodal fusion mainly involves the integration of information from three modalities: visible light (RGB) images, thermal infrared (TIR) ​​images, and verbal descriptions. Different modalities have their own advantages and limitations: RGB images provide rich texture and color information, TIR images have advantages in lighting changes and nighttime scenes, and verbal descriptions provide high-level semantic guidance. Effective multimodal fusion needs to address key issues such as the differentiation between modalities, the utilization of complementarity, and the suppression of redundant information. Common methods include feature-level fusion, decision-level fusion, and attention-guided fusion.

[0185] 5. Top-K Selection Mechanism: Top-K selection is a feature or information filtering strategy based on importance ranking. It achieves data sparsity and key information extraction by retaining the top K elements with the highest values. In attention mechanisms, Top-K selection is typically applied to the attention weight matrix, identifying and retaining the K positions with the highest attention scores while setting the remaining positions to zero or ignoring them. The core advantage of this mechanism is that it significantly reduces computational complexity while focusing on the most relevant information interactions, effectively suppressing interference from noise and redundant information. In the multimodal sparse cross-fusion module of this invention, the Top-K selection mechanism is used to dynamically determine the most important cross-modal feature correspondences, improving both fusion efficiency and fusion quality, achieving the optimal balance between computational efficiency and performance accuracy.

Claims

1. A method of structured attribute-guided multi-scale contextual interactive RGB-T sparse fusion tracking, characterized in that, The method comprises the following steps: S10 obtains an RGB-T video pair, including a visible light template image, an infrared template image, a visible light search area image, and an infrared search area image; S20 respectively performs block embedding on the template image and the search area image to obtain corresponding token sequences; S30, based on the visible light template image, dynamically generates a target structured attribute prompt using a pre-trained visual language model CLIP and a preset discrete attribute template library, the attribute at least including a category attribute and a color attribute; S40 adds the structured attribute prompt to the RGB token sequence and the TIR token sequence for additive fusion to form a multi-modal token with semantic guidance; S50 inputs the multi-modal token with semantic guidance into a double-branch visual Transformer main body to extract visible light features and infrared features; S60 inputs the visible light features and the infrared features into a multi-modal context-scale perception interaction integration module to obtain enhanced features that simultaneously encode global context dependence and multi-scale spatial information; S70 inputs the enhanced visible light features and infrared features into a multi-modal sparse cross-fusion module to realize cross-modal sparse attention fusion through an adaptive Top-K selection mechanism to generate fusion features; S80 outputs a target bounding box through a prediction head based on the fusion features to complete tracking.

2. The method of claim 1, wherein, The dynamic generation of the target structured attribute prompt in S30 comprises: extracting visual features of the visible light template image using a CLIP visual encoder; mapping a preset category word library and a color word library into text features through a CLIP text encoder; the visual features and the text features are processed through a lightweight multi-modal adapter before similarity calculation; calculating the similarity between the adapted visual features and the text features to select the category and the color with the highest similarity to form the structured attribute prompt.

3. The method of claim 2, wherein the method is a structured attribute guided multi-scale context interactive RGB-T sparse fusion tracking method. The lightweight multi-modal adapter is composed of an up-sampling convolution, a ReLU activation function, and a down-sampling convolution.

4. The method of claim 1, wherein, The multi-modal context-scale perception interaction integration module in S60 comprises: a non-local context attention sub-module for capturing global semantic dependence; a scale perception multi-dimensional spatial attention sub-module for extracting multi-scale spatial structure information along the height and width dimensions in parallel; and parallel processing the fused visible-infrared-attribute features to output enhanced features.

5. The method of claim 4, wherein, The scale perception multi-dimensional spatial attention sub-module uses a depth separable convolution to perform multi-scale convolution on the height and width dimensions respectively, and suppresses semantic interference between sub-features through grouped normalization.

6. The method of claim 1, wherein, The multi-modal sparse cross-fusion module in S70 comprises: using a depth separable convolution to map the visible light features and the infrared features into queries, keys, and values respectively; calculating a bidirectional cross-attention matrix; reserving Top-K maximum weights for each row of attention vectors and setting the rest to zero to generate a sparse attention matrix; using the sparse attention matrix to complete cross-modal feature fusion.

7. The method of claim 6, wherein, K is a learnable parameter that can be automatically adjusted in end-to-end training.

8. A structured attribute-guided multi-scale contextual interactive RGB-T sparse fusion tracking system, characterized in that, It comprises: an image input interface for receiving an RGB-T video pair; a block embedding unit configured to convert the image into a token sequence; a structured attribute prompt construction unit configured to perform the step S30 in claim 1; a dual-branch visual Transformer backbone configured to perform the step S50 in claim 1; a multi-modal context-scale perception interaction integration unit configured to perform the step S60 in claim 1; a multi-modal sparse cross fusion unit configured to perform the step S70 in claim 1; a prediction head configured to output the target bounding box. 9.A computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the steps of the method of any one of claims 1 to 7, and is applied to intelligent monitoring, autonomous driving, and unmanned aerial vehicle navigation. 10.A processor configured to run a computer program, which, when running, performs the steps of the method of any one of claims 1 to 7.

Images