RGBT target tracking method based on medium-term fusion element framework and composite visual prompt

By adopting an RGBT target tracking method based on a mid-term fusion meta-framework and composite visual cues, the robustness problem of existing methods under extreme lighting and severe weather conditions is solved, achieving efficient tracking in complex scenes and improving robustness and accuracy.

CN118967747APending Publication Date: 2024-11-15BEIJING INST OF TECH
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411023340.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Existing RGBT tracking methods exhibit reduced robustness in real-world scenarios such as extreme lighting and severe weather. Furthermore, existing cue-based tracking methods struggle to fully explore modal independence and dynamic dominance, limiting their application in complex scenarios.

Method used

An RGBT target tracking method based on a mid-term fusion meta-framework and composite visual cues is adopted. Through an adjustable mid-term fusion meta-framework and multi-form, multi-stage composite cueing strategies, it achieves efficient parameter transfer of upstream knowledge to the current task, including single-modal and cross-modal exploration cues, modality and stage-aware cues, reducing the burden on training equipment and fully releasing the cueing learning potential.

Benefits of technology

Robust RGBT tracking is achieved in complex open scenarios with modality-dominant dynamic changes, reducing the burden on training equipment, improving tracking stability and accuracy, and surpassing the performance of existing methods.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118967747A_ABST
    Figure CN118967747A_ABST
Patent Text Reader

Abstract

The invention discloses an RGBT (Red Green Blue Tracking) target tracking method based on a medium-term fusion element framework and composite visual cue, which relates to the technical field of visual target tracking and comprises the following steps: pre-training an upstream RGB tracking network, embedding the upstream RGB tracking network into a medium-term fusion multi-modal tracking element framework and constructing an RGBT cue tracking network by using three visual cue strategies, then prompt fine tuning is carried out on the network on the RGBT tracking training set; the prompt tracking method comprises a middle-term fusion meta-framework, a single-mode and cross-mode exploration prompt strategy, a middle fusion prompt strategy and a mode and stage perception prompt strategy. According to the method, efficient parameter knowledge migration of upstream knowledge on the current task is realized through a very small number of additional parameters needing fine tuning, so that the burden of training equipment is greatly reduced, and the huge potential of prompt learning in the RGBT tracking task is fully released; and robust RGBT tracking is realized in a plurality of complex open scenes with dynamic change of modal dominance.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of visual target tracking, and in particular to an RGBT target tracking method based on a mid-term fusion meta-framework and composite visual cues. Background Art

[0002] As a basic research direction in the field of computer vision, visual target tracking technology aims to predict the target state of the entire video sequence based on the target state of the first frame of the video, and has broad application prospects in the fields of intelligent monitoring, autonomous driving, and robot perception. However, in real-life scenarios such as extreme lighting and bad weather, the imaging quality of the visible light modality (also known as the RGB modality) is severely degraded, and the reliability of tracking methods based on this modality is severely reduced. At the same time, considering that the infrared modality (also known as the T modality) is highly robust to lighting changes, researchers proposed RGBT tracking to utilize the complementarity between the two modalities to achieve robust all-weather tracking.

[0003] On the one hand, in order to fully realize the complementary information fusion between the two modalities, the current mainstream RGBT tracking methods are mostly based on three fusion meta-frameworks: image-level (such as ECMD and ProTrack), feature-level (such as SiamCDA) and decision-level fusion frameworks (such as JMMAC). Among them, the image-level fusion framework has a concise structure and high efficiency, but lacks independent feature modeling of the two modalities, so the performance is very poor; on the contrary, the decision-level and feature-level fusion frameworks fully represent the features of the two modalities respectively, and have high performance, but the dual-stream skeleton structure introduces redundant calculations, which seriously limits its reasoning efficiency. Therefore, it is difficult for the above-mentioned fusion meta-frameworks to achieve a good balance between tracking performance and efficiency.

[0004] On the other hand, due to the high collection cost, labeling cost and alignment cost, the labeled data of RGBT tracking task is far less than that of upstream RGB tracking task, which seriously hinders the data-driven model from learning reliable knowledge from the current task data. To this end, considering that the current RGBT tracking framework is mostly extended from the RGB tracking framework, some studies such as MFNet, HMFT, DMCNet and TBSI use the pre-trained RGB tracking model parameters to initialize the current model and perform full parameter fine-tuning on the task data to use the upstream knowledge to guide the model to converge faster and better. Although effective, such methods require too many parameters to be fine-tuned, which not only places extremely high demands on the storage and computing power of the training equipment, but also may lead to the forgetting of upstream knowledge and model overfitting.

[0005] Considering the limitations of full-parameter fine-tuning methods, recent studies such as ProTrack, ViPT, and UnTrack have introduced the idea of ​​cue learning to RGBT tracking to better utilize the huge knowledge inheritance between upstream and downstream tasks. These methods freeze the parameters from the upstream model and only learn a small number of additional parameters to generate task-specific cues, thereby achieving efficient transfer of upstream knowledge to downstream tasks. This type of parameter-efficient cue fine-tuning method requires much less training equipment than the full-parameter fine-tuning method, and can achieve performance close to or even exceeding that of most full-parameter fine-tuning methods. However, existing cue-based tracking methods still have several key deficiencies:

[0006] (1) Most of these methods generate task cues through direct / indirect image-level fusion, which makes it difficult to fully explore rich modality-independent information and discriminative patterns;

[0007] (2) These methods treat infrared modality images as additional information and assign fixed weights, while ignoring the dynamic dominance of different modalities in different scenes, which limits the application of tracking methods in complex scenes;

[0008] (3) These methods only use a single weighted addition cue strategy and do not fully explore the cue strategy, making it difficult to fully unleash the huge potential of cue learning methods in RGBT tracking tasks.

[0009] Based on the above shortcomings, how to achieve robust RGBT tracking in a variety of complex open scenes with dynamic changes in modality dominance and release the potential of cue learning in RGBT tracking tasks is an urgent problem to be solved. Summary of the invention

[0010] In view of this, the present invention provides an RGBT target tracking method based on a mid-term fusion meta-framework and composite visual cues. Through an adjustable mid-term fusion meta-framework and a multi-form, multi-stage composite cue strategy, efficient knowledge transfer of upstream knowledge on the current task is achieved with very few additional parameters that require fine-tuning, which greatly reduces the burden on training equipment, fully releases the huge potential of cue learning in RGBT tracking tasks, and realizes robust RGBT tracking in a variety of complex open scenes with dynamic changes in modal dominance.

[0011] To achieve the above object, the technical solution of the present invention comprises the following steps:

[0012] Step S1: Collect the annotated and aligned RGBT dataset and construct the training set and validation set from it.

[0013] Step S2: Select the pre-trained RGB tracking network OSTrack as the upstream knowledge source and build the RGBT target tracking network based on its network.

[0014] Step S3: Load the pre-trained upstream RGB tracking model parameters and perform prompt fine-tuning training on the tracker on the training set.

[0015] Step S4: Perform online tracking, load the trained RGBT tracking model, read the given video sequence and input it into the model frame by frame for tracking.

[0016] Further, step S2, the specific steps are:

[0017] S2.1: Embed the pre-trained upstream RGB tracking network into the mid-term fusion RGBT tracking meta-framework to build a base tracker with a two-stage skeleton network.

[0018] S2.2: Apply the unimodal and cross-modal exploration prompt strategies to the skeleton network of the first stage, that is, configure the unimodal exploration prompter UEP and the cross-modal adaptive prompter IP into the skeleton network of the first stage to help the skeleton of the first stage to perform sufficient modality-independent modeling and cross-modal interaction.

[0019] S2.3: The intermediate fusion prompt strategy is applied between the skeleton networks of the two stages, that is, an intermediate fusion prompter MFP is configured between the skeleton networks of the first stage and the second stage to provide reliable fusion modal feature prompts for the skeleton of the second stage.

[0020] S2.4: Apply the modality and stage-aware cue strategy to the entire tracker, that is, introduce the three modality and stage-aware cue words into different positions of the skeleton network to help the skeleton network at each stage perceive the current modality and modeling stage.

[0021] Furthermore, in step 2, the structure of the pre-trained RGB tracking network OSTrack includes a single-stream transformer skeleton network and a head network: the transformer skeleton network includes a linear layer consisting of a convolution layer using a 16×16 convolution kernel and a step size of 16 and 12 encoder block layers with a hidden layer dimension of 768; the head network includes three groups of convolutional neural networks, each group consisting of 3 stacked convolution-BN-ReLU layers.

[0022] Furthermore, in step 2.1, the proposed mid-term fusion meta-framework inserts the mid-term fusion module into the middle of the skeleton network of the upstream framework, thereby dividing the skeleton network into two parts, namely the first-stage skeleton network and the second-stage skeleton network:

[0023] The skeleton network of the first stage is a dual parallel branch with weight sharing, which is used to represent the features of the two modalities respectively.

[0024] The skeleton network of the second stage is a single branch, which is used to further represent the fused features processed by the mid-term fusion module. The embedding layer and the head network are exactly the same as the upstream framework.

[0025] Furthermore, in step 2.2, the single-modal and cross-modal exploration prompt strategy configures the UEP module that is not shared by the modalities in parallel with the transformer encoder blocks of different branches in the first stage layer by layer, and configures the IP module shared by the modalities between the encoder blocks of different branches; the entire process of the single-modal and cross-modal exploration prompt strategy is as follows:

[0026] The UEP module first converts the tokens sequence into a feature map, uses a 1×1 convolution to compress the redundant channels and reduce the dimension of the input modal features, and then uses parallel point-by-point convolution, depth-wise separable point-by-point convolution and 3×3 dilated convolution to fully model the modal independent information of low-dimensional features from different angles and add them element-by-element. Then, a point-by-point convolution with a GELU activation function is used to process the weighted features, and residual connections are configured synchronously. Finally, a 1×1 convolution is used to map the features to the original dimension and restore them to a tokens sequence.

[0027] The IP module is also configured with dimensionality reduction and dimensionality increase operations and residual connections; the IP module compresses the reduced modal features for prompting into single-channel prompts through channel average pooling, and adds them to the reduced modal features of prompting in the form of additional channels, and then restores the prompted modal features to the reduced dimension through a point-by-point convolution, and finally maps this feature to the original dimension and restores it to a tokens sequence.

[0028] The unimodal and cross-modal exploration prompt strategy inputs the unimodal features input to the encoder block of this layer into the UEP module of this layer at the same time, and adds the modality-independent features output by the UEP module to the output of the encoder block in an element-by-element manner, so as to obtain rich modality sharing and independent information through intra-modal prompts; then, the prompt strategy regards the modality as the prompt modality, takes the template tokens of this modality and another modality and inputs them into the IP module together for cross-modal adaptive interactive prompts, and finally injects the prompted modality features into the encoder block of the next layer.

[0029] Furthermore, in step 2.3, the intermediate fusion prompt strategy inserts MFP between the two stages of the skeleton network to adaptively fuse the two modal features output by the skeleton network of the first stage and provide fusion feature prompts for the skeleton network of the second stage. The processing flow of MFP is as follows:

[0030] MFP first reduces the dimensionality of the two modal features, and then obtains the total features, modality-independent features, and modality-shared features respectively, and connects these features at the channel level. Then, a fully connected layer MLP and a softmax function are used to generate element-by-element weights for the two modalities to weightedly add the two modal features, and then a residual operation is configured. Finally, the fused modal features are mapped back to the original feature space, and the fusion prompt is output. The process of generating weights and weighted addition in the low-dimensional space is expressed by the following formula:

[0031] F′ S =(F V,L ×F T,L )

[0032] F′ A =(F V,L +F T,L )

[0033] F′ V =Sigmoid(F V,L -F T,L )

[0034] F′ T =Sigmoid(F T,L -F V,L )

[0035] F = concaten(F′ V ,F′ S ,F′ A ,F′ S ,F′ T )

[0036] [weight V ,weight T ]=Softmax(fc(F))

[0037] F F,L =F′ S +F V,L ×weight V +F T,L ×weight T

[0038] Among them, F V,L and F T,L Respectively represent the RGB mode and T mode features of the low-dimensional space, F S ′、F A ′、F V ′、F T′ represents modality shared features, modality total features, RGB modality independent features, and T modality independent features, concaten represents channel-level connection operations, Sigmoid and Sigmax represent sigmoid and softmax functions, respectively, and weight V and weight T Represents the weights of RGB and T modes respectively, F F,L Represents the fused modal features in a low-dimensional space.

[0039] Furthermore, in step 2.4, the modality and stage awareness prompt strategy configures three learnable tokens as modality and stage awareness prompt words to guide the skeleton networks of different stages and modality branches to better perceive the current modality and stage, as follows:

[0040] (1) The entire strategy is configured with three types of prompt words: T-modality-aware prompt words, RGB-modality-aware prompt words, and fusion-modality-aware prompt words. These prompt words learn and store fixed patterns of different modalities. The dimension of each prompt word is set to be consistent with the hidden layer dimension of the skeleton network.

[0041] (2) The prompt strategy attaches the three prompt words to the T-modality branch, RGB-modality branch of the first-stage skeleton, and the input feature tokens of the second-stage skeleton in a channel-level connection manner.

[0042] (3) All cue words only participate in the forward propagation process of the upstream skeleton branch and do not participate in the processing of other cue strategies. When feature tokens are input into other cue branches, these cue words will be separated and reattached to the feature tokens after other cue processes are completed.

[0043] Beneficial effects:

[0044] The present invention provides an RGBT target tracking method based on a mid-term fusion meta-framework and composite visual cues. The method aims to achieve parameter-efficient knowledge transfer of upstream knowledge on the current task with very few additional parameters that need to be fine-tuned through an adjustable mid-term fusion meta-framework and a multi-modal, multi-stage composite cue strategy, greatly reducing the burden on training equipment, fully releasing the huge potential of cue learning in RGBT tracking tasks, and achieving robust RGBT tracking in a variety of complex open scenes with dynamic changes in modality dominance.

[0045] The present invention realizes independent modal representation, cross-modal scene adaptive interaction, adaptive fusion and modality and stage perception through a composite prompt strategy that includes different prompt forms and is applied to different stages, thereby helping the upstream model to achieve parameter-efficient knowledge transfer on limited downstream task data; and helps tracking to better balance speed and performance through a flexible and adjustable mid-term fusion meta-framework. The developed RGBT prompt tracking method only needs to fine-tune additional parameters that are less than 0.4% of the upstream model parameters, which is less than all previous similar methods in terms of the amount of fine-tuning parameters, and at the same time, it surpasses all previous similar prompt tracking methods in terms of performance. In addition to showing leading performance in general scenarios, this method also shows good tracking stability and accuracy in a variety of challenging scenarios such as light intensity mutations, low light, temperature crossover and other modality-dominated dynamic change scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a schematic diagram of the mid-term fusion tracking meta-framework of the present invention.

[0047] Figure 2 It is a schematic diagram of the RGBT tracking network structure of the present invention.

[0048] Figure 3 It is a schematic diagram of the single-mode and cross-modal exploration prompt strategy and UEP and IP network structure in the present invention.

[0049] Figure 4 It is a schematic diagram of the MFP network structure in the present invention. DETAILED DESCRIPTION

[0050] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0051] The present invention provides an RGBT target tracking method based on a mid-term fusion meta-framework and composite visual cues, which specifically comprises the following steps:

[0052] Step S1: Obtain RGBT image pairs that are manually collected, aligned and annotated with true values ​​from the training set of the visible light and infrared RGBT tracking dataset LasHer, and divide them into a training set and a validation set.

[0053] Step S2: Select the weights of the pre-trained RGB tracking network OSTrack256 as the source of reliable upstream knowledge (i.e., the weight parameters store upstream knowledge). The OSTrack network consists of a single-stream transformer skeleton network and a head network: the transformer skeleton network contains 1 linear embedding layer and 12 encoderblock layers with a hidden dimension of 768; the head network contains three sets of convolutional neural networks. The RGBT cue tracking network is built on this basis, such as Figure 2As shown, the specific method is:

[0054] S2.1: According to Figure 1 The RGBT tracking meta-framework shown embeds the pre-trained upstream RGB tracking network blocks into the mid-term fused RGBT tracking meta-framework, thereby constructing a basic tracker network consisting of a two-stage skeleton network.

[0055] S2.2: If Figure 2 and Figure 3 As shown, the unimodal and cross-modal exploration prompt strategies are applied to the upstream transformer encoders skeleton network of the first stage, that is, the designed unimodal exploration prompter UEP and cross-modal adaptive prompter IP are configured into this part of the skeleton to help the upstream skeleton network of this stage to perform modality-independent information modeling and cross-modal information interaction for the two modalities.

[0056] like Figure 1 As shown, enter Z V , X V are visible light image template frame and search frame, respectively, Z T , X T They are the template frame and search frame of infrared mode; the template image and Embed layer of visible light form, and the output E V 、E T , add prompt word P V , P T , where the prompt word can be added at any position and input into the Encoder after adding the prompt word.

[0057] S2.3: If Figure 2 As shown, an intermediate fusion prompt strategy is applied between the two stages of skeleton networks, that is, an intermediate fusion prompter MFP is configured after the first stage skeleton network to generate reliable fusion modality feature prompts and provide them to the upstream skeleton network of the second stage.

[0058] S2.4: If Figure 2 As shown in the figure, the modality and stage-aware cue strategies are applied to the skeleton networks of the two stages, that is, the three modalities and the stage-aware cue words P V , P T They are introduced into different positions of the skeleton network to help the skeleton network at each stage perceive the current mode and modeling stage.

[0059] Step S3: Load the pre-trained OSTrack256 parameters and perform prompt fine-tuning training on the tracker on the LasHer training set.

[0060] Step S4: Perform online tracking, load the trained RGBT tracking model, read the given video sequence and input it into the model frame by frame for tracking.

[0061] In step 2.1 above, the proposed mid-term fusion meta-framework divides the transformer skeleton network into two parts, namely the first-stage skeleton network and the second-stage skeleton network: the first-stage skeleton network is expanded into two parallel branches with weight sharing, each branch processes one modality; the second-stage skeleton network further represents the fused features. The embedding layer and the head network are consistent with the use of the upstream framework.

[0062] In this embodiment, the skeleton network of the first stage includes the first 10 layers of transformer encoder blocks, and the skeleton network of the second stage includes the last 2 layers.

[0063] In the above step 2.2, if Figure 2 and Figure 3 As shown, the single-modal and cross-modal exploration prompt strategy configures the UEP module that is not shared by the modality and the transformer encoder block of different branches in the first stage in parallel layer by layer, and configures the IP module shared by the modality between the encoder blocks of different branches. In order to save parameters, in this embodiment, the UEP module is only configured on the 2nd, 5th and 8th layer encoder blocks. The process of the entire single-modal and cross-modal exploration prompt strategy is as follows:

[0064] like Figure 3 As shown, UEP first converts the tokens sequence into a feature map, and the low-dimensional projection module uses a 1×1 convolution to perform redundant channel compression and dimensionality reduction on the input modal features. In this embodiment, the number of low-dimensional space channels of all UEPs is set to 8.

[0065] Then, parallel point-by-point convolution, depth-wise separable point-by-point convolution, and 3×3 dilated convolution are used to fully model the modal independent information of low-dimensional features from different angles and add them element-by-element to obtain weighted features.

[0066] Then, a point-by-point convolution with a GELU activation function is used to process the weighted features. Finally, a 1×1 convolution is used to map the features to the original dimension, and a residual connection is configured simultaneously to restore them to a tokens sequence.

[0067] like Figure 3As shown, the IP is also configured with dimensionality reduction and dimensionality increase operations and residual connections; in this embodiment, the number of low-dimensional space channels of all IPs is set to 8, and the IP compresses the modal features used for prompting after dimensionality reduction into single-channel prompts through channel average pooling, and adds them to the prompted modal features in the form of additional channels, and then restores the prompted modal features to the reduced dimension through a point-by-point convolution, and finally maps this feature to the original dimension and restores it to a tokens sequence.

[0068] The unimodal and cross-modal exploration prompt strategy inputs the unimodal features input to the encoder block of this layer into the UEP of this layer at the same time, and adds the modality-independent features output by the UEP to the output of the encoder block in an element-by-element manner, so as to obtain rich modality sharing and independent information through intra-modal prompts; then, the prompt strategy regards the modality as the prompt modality, takes the template tokens of this modality and another modality and inputs them into the IP together for cross-modal adaptive interactive prompts, and finally injects the prompted modality features into the encoder block of the next layer.

[0069] Figure 4 In, Ep,z is the template data in the form of a token queue used to generate the prompt; Fp,z is the feature map form of Ep,z; Ex,z is the template data in the form of a token queue to be prompted; and Fx,z is the feature map form of Ex,z.

[0070] In the above step 2.3, the intermediate fusion prompt strategy inserts MFP between the skeletons of the two stages to adaptively fuse the two modal features output by the skeleton of the first stage and provide fusion feature prompts for the skeleton of the second stage. The processing flow of MFP is as follows:

[0071] like Figure 3 As shown in the figure, MFP first reduces the dimension of the two modal features. In this embodiment, the low-dimensional space dimension of MFP is set to 16. Then, the total features, modality-independent features, and modality-shared features are obtained respectively, and these features are connected at the channel level. Then, a fully connected layer MLP and a softmax function are used to generate element-by-element weights for the two modalities to weightedly add the two modal features. Then, a residual operation is configured, and finally, the fused modal features are mapped back to the original feature space, and the fusion prompt is output. The process of generating weights and weighted addition in the low-dimensional space is expressed by the following formula:

[0072] F′ S =(F V,L ×F T,L )

[0073] F′ A =(F V,L +FT,L )

[0074] F′ V =Sigmoid(F V,L -F T,L )

[0075] F′ T =Sigmoid(F T,L -F V,L )

[0076] F = concaten(F′ V ,F′ S ,F′ A ,F′ s ,F′ T )

[0077] [weight V ,weight T ]=Softmax(fc(F))

[0078] F F,L =F′ S +F V,L ×weight V +F T,L ×weight T

[0079] Among them, F V,L and F T,L Respectively represent the RGB mode and T mode features of the low-dimensional space, F S ′、F A ′、F V ′、F T ′ represents modality shared features, modality total features, RGB modality independent features, and T modality independent features, concaten represents channel-level connection operations, Sigmoid and Softmax represent sigmoid and softmax functions, respectively, and weight V and weight T Represents the weights of RGB and T modes respectively, F F,L Represents the fused modal features in a low-dimensional space.

[0080] In step 2.4 above, the modality and stage awareness prompt strategy configures three learnable tokens as modality and stage awareness prompt words to guide the skeleton networks of different stages and modality branches to better perceive the current modality and stage, such as Figure 2 As shown, the details are as follows:

[0081] (1) The entire strategy is configured with three types of prompt words: T-modality-aware prompt words, RGB-modality-aware prompt words, and fusion-modality-aware prompt words. These prompt words learn and store fixed low-level information of different modalities. The dimension of each prompt word is set to be consistent with the hidden layer dimension of the skeleton network.

[0082] (2) The hint strategy attaches the three hint words to the T modality branch, RGB modality branch of the first-stage skeleton, and the input feature tokens of the second-stage skeleton in a channel-level connection manner to guide the upstream skeleton to better recognize the current modality and stage.

[0083] (3) All cue words only participate in the forward propagation process of the upstream skeleton branch and do not participate in the processing of other cue strategies. When feature tokens are input into other cue branches, these cue words will be separated and reattached to the feature tokens after other cue processes are completed.

[0084] (4) In this embodiment, the number of tokens contained in each prompt word is set to 2.

[0085] In this embodiment, the designed RGBT cue tracking network was trained on the training set for 60 rounds with a learning rate of 0.0004. Starting from the 48th round, the learning rate decayed to 0.00004.

[0086] Table 1 compares the invented method with the best current RGBT tracking methods with fine-tuning parameters and tracking performance on the LasHer test set. To ensure a fair comparison, all methods use the same upstream RGB tracking network pre-training parameters for parameter initialization. As can be seen from Table 1, the invented method requires fewer parameters than all existing similar methods, and shows an absolute advantage in three authoritative tracking evaluation indicators, which demonstrates the superiority of the invented method.

[0087] Table 1 Comparison with existing similar methods on the LasHer test set

[0088] method years Total parameters Fine-tune parameter amount PR NPR SR ViPT-shaw 2023 92.73M 0.61M 59.0 55.7 47.6 ViPT-deep 2023 92.96M 0.84M 64.8 61.3 52.2 UnTrack 2024 98.72M 6.60M 66.7 62.9 53.6 OneTrack 2024 94.92M 2.80M 67.1 63.3 54.1 This method 2024 92.46M 0.34M 70.0 66.3 56.1

[0089] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An RGBT target tracking method based on a mid-term fusion meta-framework and composite visual cues, characterized in that: The steps include: Step S1: Collect the annotated and aligned RGBT dataset and construct the training set and validation set from it; Step S2: Select the pre-trained RGB tracking network OSTrack as the upstream knowledge source, and build the RGBT target tracking network based on its network; Step S3: Load the pre-trained upstream RGB tracking model parameters, and perform prompt fine-tuning training on the tracker on the training set; Step S4: Perform online tracking, load the trained RGBT tracking model, read the given video sequence and input it into the model frame by frame for tracking.

2. The RGBT target tracking method based on the mid-term fusion meta-framework and composite visual cues as claimed in claim 1, characterized in that: The step S2 comprises the following specific steps: S2.1: Embed the pre-trained upstream RGB tracking network into the mid-term fusion RGBT tracking meta-framework to build a base tracker with a two-stage skeleton network; S2.2: Applying the unimodal and cross-modal exploration prompt strategies to the first-stage skeleton network, that is, configuring the unimodal exploration prompter UEP and the cross-modal adaptive prompter IP to the first-stage skeleton network to help the first-stage skeleton to perform sufficient modality-independent modeling and cross-modal interaction; S2.3: Apply the intermediate fusion prompt strategy between the skeleton networks of the two stages, that is, configure the intermediate fusion prompter MFP between the skeleton networks of the first stage and the second stage to provide reliable fusion modality feature prompts for the skeleton of the second stage; S2.4: Apply the modality and stage-aware cue strategy to the entire tracker, that is, introduce the three modality and stage-aware cue words into different positions of the skeleton network to help the skeleton network at each stage perceive the current modality and modeling stage.

3. The RGBT target tracking method based on the mid-term fusion meta-framework and composite visual cues as claimed in claim 2, characterized in that: In step 2, the structure of the pre-trained RGB tracking network OSTrack includes a single-stream transformer skeleton network and a head network: the transformer skeleton network includes a linear layer consisting of a convolution layer using a 16×16 convolution kernel and a step size of 16 and 12 encoder block layers with a hidden layer dimension of 768; the head network includes three groups of convolutional neural networks, each group consists of 3 stacked convolution-BN-ReLU layers.

4. The RGBT target tracking method based on mid-term fusion meta-framework and composite visual cues as claimed in claim 2, characterized in that: In step 2.1, the proposed mid-term fusion meta-framework inserts the mid-term fusion module into the middle of the skeleton network of the upstream framework, thereby dividing the skeleton network into two parts, namely the skeleton network of the first stage and the skeleton network of the second stage: The skeleton network in the first stage is a weight-shared dual parallel branch, which is used to represent the features of the two modalities respectively; The skeleton network of the second stage is a single branch, which is used to further represent the fused features processed by the mid-term fusion module. The embedding layer and the head network are exactly the same as the upstream framework.

5. The RGBT target tracking method based on mid-term fusion meta-framework and composite visual cues as claimed in claim 2, characterized in that: In step 2.2, the single-modal and cross-modal exploration prompt strategy configures the UEP module that is not shared by the modalities and the transformer encoder block of different branches in the first stage in parallel layer by layer, and configures the IP module shared by the modalities between the encoder blocks of different branches; the process of the entire single-modal and cross-modal exploration prompt strategy is as follows: The UEP module first converts the tokens sequence into a feature map, uses a 1×1 convolution to perform redundant channel compression and dimensionality reduction on the input modal features, then uses parallel point-by-point convolution, depth-wise separable point-by-point convolution, and 3×3 dilated convolution to fully model the modal independent information of low-dimensional features from different angles and add them element-by-element, then uses a point-by-point convolution with a GELU activation function to process the weighted features, and simultaneously configures the residual connection, and finally uses a 1×1 convolution to map the features to the original dimension and restore them to a tokens sequence; The IP module is also configured with dimensionality reduction and dimensionality increase operations and residual connections; the IP module compresses the modal features used for prompting after dimensionality reduction into a single-channel prompt through channel average pooling, and adds an additional channel to the prompted modal features after dimensionality reduction, and then restores the prompted modal features to the reduced dimension through a point-by-point convolution, and finally maps this feature to the original dimension and restores it to a tokens sequence; The unimodal and cross-modal exploration prompt strategy inputs the unimodal features input to the encoder block of this layer to the UEP module of this layer at the same time, and adds the modality-independent features output by the UEP module to the output of the encoderblock in an element-by-element manner, so as to obtain rich modality sharing and independence information through intra-modal prompts; Subsequently, the prompt strategy regards the modality as the prompt modality, takes the template tokens of this modality and another modality and inputs them into the IP module for cross-modal adaptive interactive prompts, and finally injects the prompted modal features into the encoder block of the next layer.

6. The RGBT target tracking method based on mid-term fusion meta-framework and composite visual cues as claimed in claim 2, characterized in that: In step 2.3, the intermediate fusion prompt strategy inserts MFP between the two stages of the skeleton network to adaptively fuse the two modal features output by the skeleton network of the first stage and provide fusion feature prompts for the skeleton network of the second stage. The processing flow of MFP is as follows: MFP first reduces the dimensionality of the two modal features, and then obtains the total features, modality-independent features, and modality-shared features respectively, and connects these features at the channel level. Then, a fully connected layer MLP and a softmax function are used to generate element-by-element weights for the two modalities to weightedly add the two modal features, and then a residual operation is configured. Finally, the fused modal features are mapped back to the original feature space, and the fusion prompt is output. The process of generating weights and weighted addition in the low-dimensional space is expressed by the following formula: F′ S =(F V,L ×F T,L ) F′ A =(F V,L +F T,L ) F′ V =Sigmoid(F V,L -F T,L ) F′ T =Sigmoid(F T,L -F V,L ) F=concaten(F′ V ,F′ S ,F′ A ,F′ S ,F′ T ) [weight V ,weight T ]=Softmax(fc(F)) F F,L =F′ S +F V,L ×weight V +F T,L ×weight T Among them, F V,L and F T,L Respectively represent the RGB mode and T mode features of the low-dimensional space, F′ S , F′ A , F′ V , F′ T They represent modality shared features, modality total features, RGB modality independent features, and T modality independent features, respectively. concaten represents the channel-level connection operation. Sigmoid and Softmax represent the sigmoid and softmax functions, respectively. weight V and weight T Represent the weights of RGB and T modes respectively, F F,L Represents the fused modal features in a low-dimensional space.

7. The RGBT target tracking method based on mid-term fusion meta-framework and composite visual cues as claimed in claim 2, characterized in that: In step 2.4, the modality and stage perception prompt strategy configures three learnable tokens as modality and stage perception prompt words to guide the skeleton networks of different stages and modality branches to better perceive the current modality and stage, as follows: (1) The entire strategy is configured with three types of prompt words: T-modality-aware prompt words, RGB-modality-aware prompt words, and fusion-modality-aware prompt words. These prompt words learn and store fixed patterns of different modalities. The dimension of each prompt word is set to be consistent with the hidden layer dimension of the skeleton network. (2) The prompt strategy attaches the three prompt words to the T-modality branch, RGB-modality branch of the first-stage skeleton, and the input feature tokens of the second-stage skeleton in a channel-level connection manner; (3) All cue words only participate in the forward propagation process of the upstream skeleton branch and do not participate in the processing of other cue strategies. When feature tokens are input into other cue branches, these cue words will be separated and reattached to the feature tokens after other cue processes are completed.

Citation Information

Cited By

  • Multi-modal target tracking method and system based on time sequence modeling and prompt fine tuning

    CN120510480A

  • Multimodal object tracking method and system based on temporal modeling and prompt fine-tuning

    CN120510480B