RGB-t tracking method and device with dual modality enhancement mechanism

Abstract

The present application relates to the technical field of multi-modal learning, in particular to a kind of RGB-T tracking method and device of dual modal enhancement mechanism, method includes: the initial multi-modal target tracking model is constructed, initial multi-modal target tracking model includes two parallel ViT backbone networks, initial infrared perception adapter, initial cross-modal fusion adapter, RGB prediction head, infrared prediction head and response graph decoupling distillation response module;According to training sample and the preset loss function, the initial multi-modal target tracking model is trained and optimized, and the trained multi-modal target tracking model is obtained, including Transform backbone network, infrared perception adapter, cross-modal fusion adapter, RGB prediction head and infrared prediction head;Obtain to be identified data, and to be identified RGB image and to be identified infrared image are input into multi-modal target tracking model, and the tracking result corresponding to to be identified data is obtained. By using the present application, the accuracy of target tracking can be improved.

Description

Technical Field

[0001] The present invention relates to the field of multimodal learning technology, and in particular to an RGB-T tracking method and device with a dual-modality enhancement mechanism. Background Art

[0002] Single object tracking is a fundamental problem in computer vision, which aims to robustly detect the target object in subsequent frames based on its initial state in the first frame. Currently, object tracking using pure visible light (RGB) data has shown encouraging results, but it still faces difficulties in extreme and complex scenarios, such as illumination changes, bad weather, object occlusion, and motion blur. Compared with RGB data that provides detailed color information, thermal infrared (TIR or T) images excel in object recognition and scene understanding under low light and occlusion conditions. Therefore, with the complementary advantages of visible light and infrared (RGB-T), RGB-T tracking has gained increasing attention in various challenging applications such as autonomous driving and video surveillance.

[0003] Traditional single object tracking (SOT) methods are based on visible light modalities and have difficulty coping with extreme imaging conditions such as darkness, rain, and fog. In contrast, thermal infrared is insensitive to illumination changes and can serve as a meaningful complementary representation in the tracking process. Therefore, RGB-T tracking, as a branch of the tracking task, has become a popular solution for achieving reliable semantic scene understanding under adverse weather and lighting conditions. Existing methods mainly regard cross-modal interaction as a key component. Recently, RGB-T trackers have focused on utilizing multiple fusion strategies to fully leverage the advantages of different fusion methods. However, these methods ignore the impact of networks pre-trained using large-scale RGB datasets on the infrared modality and do not address the differences between the responses of the RGB and infrared branches.

[0004] In addition, due to the lack of large-scale infrared data and backbone networks specifically designed for TIR mode, most existing RGB-T tracking models typically rely on backbone networks pre-trained on large-scale RGB datasets (such as ImageNet and SOT), following RGB-based vision tasks. However, this reliance on RGB pre-trained backbone networks introduces inherent biases towards TIR mode, which can be divided into two levels: (1) Feature-level bias. Models pre-trained on RGB data cannot fully extract and understand TIR data because the imaging mechanism of TIR is different from RGB. Therefore, after learning the specific pattern of RGB data, feature bias is introduced when processing TIR mode data. This bias may weaken the performance of the model, making it difficult to fully capture the key patterns and feature information unique to the TIR mode. (2) Decision-level bias. During the object localization process, trackers of different modes show inconsistent responses to regions with different attributes, resulting in imbalance in tracking decisions. For example, TIR mode performs better in low-light or occluded conditions, while RGB mode is more sensitive to areas with sufficient light, bright colors, rich textures and shapes. Existing methods usually rely on simple cross-modal response fusion, neglecting the distribution difference between RGB and TIR tracker responses, which may lead to significant decision bias in the tracking process. Summary of the Invention

[0005] In order to solve the technical problem of target tracking decision deviation in the existing technology, the embodiment of the present invention provides an RGB-T tracking method and device with a dual modality enhancement mechanism. The technical solution is as follows:

[0006] In one aspect, a dual-modality enhanced RGB-T tracking method is provided. The method is implemented by an RGB-T tracking device with a dual-modality enhanced mechanism. The method includes:

[0007] S1. Constructing an initial multimodal target tracking model, wherein the initial multimodal target tracking model includes two parallel pre-trained ViT backbone networks, an initial infrared perception adapter, an initial cross-modal fusion adapter, an RGB prediction head, an infrared prediction head, and a response graph decoupling distillation response module, wherein the two parallel pre-trained ViT backbone networks are respectively a first ViT backbone network and a second ViT backbone network;

[0008] S2. Based on the RGB training samples, the infrared training samples, and the preset loss function, the initial infrared perception adapter and the initial cross-modal fusion adapter are trained and optimized to obtain a trained multimodal target tracking model. The multimodal target tracking model includes a Transformer backbone network, an infrared perception adapter, a cross-modal fusion adapter, an RGB prediction head, and an infrared prediction head.

[0009] S3. Acquire data to be identified, where the data to be identified includes an RGB image to be identified and an infrared image to be identified;

[0010] S4. Input the RGB image to be identified and the infrared image to be identified into the multimodal target tracking model to obtain a tracking result corresponding to the data to be identified.

[0011] On the other hand, an RGB-T tracking device with a dual-modality enhancement mechanism is provided, which is applied to an RGB-T tracking method with a dual-modality enhancement mechanism, and includes:

[0012] A construction unit, configured to construct an initial multimodal target tracking model, wherein the initial multimodal target tracking model includes two parallel pre-trained ViT backbone networks, an initial infrared perception adapter, an initial cross-modal fusion adapter, an RGB prediction head, an infrared prediction head, and a response graph decoupling distillation response module, wherein the two parallel pre-trained ViT backbone networks are respectively a first ViT backbone network and a second ViT backbone network;

[0013] A training unit is used to train and optimize the initial infrared perception adapter and the initial cross-modal fusion adapter based on RGB training samples, infrared training samples, and a preset loss function to obtain a trained multimodal target tracking model. The multimodal target tracking model includes a Transformer backbone network, an infrared perception adapter, a cross-modal fusion adapter, an RGB prediction head, and an infrared prediction head.

[0014] An acquisition unit, configured to acquire data to be identified, wherein the data to be identified includes an RGB image to be identified and an infrared image to be identified;

[0015] The tracking unit is used to input the RGB image to be identified and the infrared image to be identified into the multimodal target tracking model to obtain a tracking result corresponding to the data to be identified.

[0016] On the other hand, an RGB-T tracking device with a dual-modal enhancement mechanism is provided, comprising: a processor; and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the RGB-T tracking methods with the dual-modal enhancement mechanism described above is implemented.

[0017] On the other hand, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement any one of the RGB-T tracking methods of the above-mentioned dual-modal enhancement mechanism.

[0018] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0019] To address feature-level bias, this embodiment proposes a joint infrared fusion adapter module to capture debiased TIR and cross-modal representations. This module includes an infrared perception adapter (IA) and a cross-modal fusion adapter (CFA). Specifically, compared to the RGB modality, the TIR modality relies on thermal radiation characteristics, making it difficult to distinguish between target and background at low contrast. To address this issue, this embodiment employs the infrared perception adapter and introduces a target-related selector mechanism to filter out irrelevant intra-modal information. To further enhance the capabilities of the infrared branch, this embodiment proposes a cross-modal fusion adapter to leverage the complementary information between the two modalities. Specifically, this embodiment implements a modality-related guidance that uses TIR and RGB regional features as queries to learn complementary modal information from the TIR and RGB guidance. Notably, the cross-modal fusion adapter can also mutually enhance feature learning in the RGB modality, thereby bridging the gap between it and the infrared modality. To address decision-level bias, this embodiment proposes a knowledge distillation strategy to ensure consistent response distributions between the TIR and fusion branches. However, in tracking tasks, only a small part of the search area is the target area, and most of it is non-target (background), resulting in an unbalanced response distribution. To address this problem, an embodiment of the present invention decouples the response map into positive and negative parts, with the positive part representing the area most relevant to the target. Then, an embodiment of the present invention designs a response map decoupling distillation loss function to ensure that the model pays more attention to the target area and achieves a consistent response map distribution, thereby improving the accuracy of target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 This is a flow chart of an RGB-T tracking method with a dual-modality enhancement mechanism provided by an embodiment of the present invention;

[0022] Figure 2 Schematic diagram of the overall structure of an initial multimodal target tracking model provided by an embodiment of the present invention;

[0023] Figure 3 This is a block diagram of an RGB-T tracking device with a dual-modality enhancement mechanism provided by an embodiment of the present invention;

[0024] Figure 4 It is a structural diagram of an RGB-T tracking device with a dual-modality enhancement mechanism provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0026] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as an "exemplary" in the present invention should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner. Furthermore, in the embodiments of the present invention, "and / or" can mean both or either of the two.

[0027] In the embodiments of the present invention, the terms "image" and "picture" may be used interchangeably. It should be noted that, when the distinction between them is not emphasized, their intended meanings are the same. The terms "of," "corresponding," and "corresponding" may be used interchangeably. It should be noted that, when the distinction between them is not emphasized, their intended meanings are the same.

[0028] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0029] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0030] The embodiment of the present invention provides an RGB-T tracking method with a dual-modality enhancement mechanism, which can be implemented by an RGB-T tracking device with a dual-modality enhancement mechanism, and the RGB-T tracking device with a dual-modality enhancement mechanism can be a terminal or a server. Figure 1 The flowchart of the RGB-T tracking method of the dual-modality enhancement mechanism is shown. The processing flow of the method may include the following steps:

[0031] S1. Construct an initial multimodal target tracking model. The initial multimodal target tracking model includes two parallel pre-trained ViT backbone networks, an initial infrared perception adapter, an initial cross-modal fusion adapter, an RGB prediction head, an infrared prediction head, and a response graph decoupling distillation response module. The two parallel pre-trained ViT backbone networks are the first ViT backbone network and the second ViT backbone network.

[0032] Optionally, the first ViT backbone network and the second ViT backbone network have the same structure.

[0033] The first ViT backbone network includes an embedding layer, a first normalization layer, a multi-head attention mechanism, a second normalization layer, and a multi-layer perceptron.

[0034] Optionally, the initial infrared perception adapter includes multiple convolutional layers, Gaussian error linear units, and a channel selector.

[0035] Optionally, the initial cross-modal fusion adapter includes a T-guided module, an R-guided module, a first attention bias learning module, a second attention bias learning module, a linear projection function, a focus operation module, a multi-channel attention mechanism, and a 1×1 convolution module.

[0036] In one possible implementation, given a video and the initial target box B0 of the target object in the first frame, the goal of the RGB-T tracking task is to develop a tracker that can reliably predict the position and size of the target object in all subsequent frames, which can be expressed as Here, (For simplicity, the temporal index is omitted) The search image is represented as RGB and TIR modalities. Tracker It can be decomposed into a feature extraction function and a target box prediction head. The tracking process is expressed as:

[0037]

[0038] Considering the powerful representation ability of Vision Transformer (ViT) in visual tasks, the proposed DMD uses ViT as a tracker The multimodal backbone network of is a pre-trained Transformer backbone network. Figure 2 As shown in Fig. 2, two parallel ViT backbone networks are used, one for RGB and one for infrared modalities, called RGB branch and TIR branch, and each stream shares the same parameters. Specifically, the RGB and TIR search images X are first r ,X t and its corresponding template frame Divide into patches and flatten into 1D tokens where N x ,N z Indicates the number of search area patches, and D is the dimension of the token. The RGB and TIR tokens are concatenated as

[0039] S2. Based on the RGB training samples, infrared training samples and the preset loss function, the initial infrared perception adapter and the initial cross-modal fusion adapter are trained and optimized to obtain a trained multimodal target tracking model. The multimodal target tracking model includes a Transformer backbone network, an infrared perception adapter, a cross-modal fusion adapter, an RGB prediction head and an infrared prediction head.

[0040] The RGB training samples include RGB training sample images and RGB training sample true labels, and the infrared training samples include infrared training sample images and infrared training sample true labels.

[0041] Optionally, the specific operation of S2 may include the following steps S21-S25:

[0042] S21. Input the infrared training sample image into the first ViT backbone network and the initial infrared perception adapter, accumulate the output of the initial infrared perception adapter with the output of the first ViT backbone network, input the RGB training sample image into the second ViT backbone network, and input the accumulated result and the output of the second ViT backbone network into the initial cross-modal fusion adapter.

[0043] Optionally, the specific operation of inputting the infrared training sample image into the first ViT backbone network and the initial infrared sensing adapter in S21 may include steps S211-S214:

[0044] S211. Input the infrared training sample image into the embedding layer, and input the output of the embedding layer into the first normalization layer.

[0045] S212. Input the output of the first normalization layer into the multi-head attention mechanism and the initial infrared perception adapter respectively, and accumulate the output of the multi-head attention mechanism and the output of the initial infrared perception adapter.

[0046] Optionally, the specific operation of S212 may be as follows:

[0047] The output of the first normalization layer is input into the multi-head attention mechanism and the 1×1 convolution layer of the initial infrared perception adapter respectively, the output of the 1×1 convolution layer is input into the Gaussian error linear unit, the output of the Gaussian error linear unit is input into the 3×3 convolution layer, the output of the 3×3 convolution layer is input into the Gaussian error linear unit, the output of the Gaussian error linear unit is input into the channel selector and the 1×1 convolution layer respectively, the output of the channel selector is input into the 1×1 convolution layer, and the output of the 1×1 convolution layer is used as the output of the initial infrared perception adapter.

[0048] S213: Input the accumulated results into the second normalization layer and the initial infrared sensing adapter respectively, and input the output of the second normalization layer into the multi-layer perceptron.

[0049] S214, accumulating the output of the multi-layer perception and the output of the initial infrared perception adapter.

[0050] Optionally, the specific operation of inputting the accumulated result and the output of the second ViT backbone network to the initial cross-modal fusion adapter in S21 can include steps S215-S219:

[0051] S215, determining the RGB search token and the TIR search token respectively according to the accumulated result and the output of the second ViT backbone network.

[0052] S216, determining the first attention output according to the accumulated result, the output of the second ViT backbone network and the first attention bias learning module, and determining the second attention output according to the accumulated result, the output of the second ViT backbone network and the second attention bias learning module.

[0053] S217, inputting the TIR search token to the T guide module, accumulating the output of the T guide module and the first attention output, inputting the accumulated result to the linear projection function, accumulating the first output obtained by the linear projection function and the TIR search token, inputting the accumulated result to the focal operation module to obtain focal operation data, and inputting the focal operation data to the multi-channel attention mechanism.

[0054] S218, inputting the RGB search token to the R guide module, accumulating the output of the R guide module and the second attention output, inputting the accumulated result to the linear projection function, accumulating the second output obtained by the linear projection function and the RGB search token, accumulating the accumulated result and the focal operation data, and inputting the accumulated result to the multi-channel attention mechanism.

[0055] S219, inputting the output of the multi-channel attention mechanism to the 1x1 convolution layer, and taking the output of the 1x1 convolution layer as the output of the initial cross-modal fusion adapter.

[0056] S22, inputting the output of the initial cross-modal fusion adapter to the infrared prediction head, accumulating the output of the second ViT backbone network and the output of the initial cross-modal fusion adapter, and inputting the accumulated result to the RGB prediction head.

[0057] In a feasible implementation, the Token sequence As the input of the first layer visual Transformer encoder, it can be expressed as follows:

[0058]

[0059] Wherein is the output of the next encoder layer. The Transformer encoder layer consists of a multi-head self-attention mechanism (MHA), a feed-forward network (FFN), a layer normalization (LN), and a multi-layer perceptron (MLP).

[0060] Subsequently, a joint infrared fusion adapter is designed as a trainable modular modification of the Transformer backbone network to adjust the bias of the TIR modality. Specifically, the joint infrared fusion adapter module consists of an infrared-aware adapter (IA) and a cross-fusion adapter (CFA). The IA is inserted into the MHA and MLP of each Transformer encoder layer in the TIR branch, while the CFA is embedded in the Transformer encoder at layer 1 for the interaction of the encoders of the two modalities. The learned adapter in each encoder can be expressed as:

[0061]

[0062] in Represents the two-stream backbone network equipped with the proposed joint infrared fusion adapter.

[0063] After L Transformer encoding layers, the multimodal features of the tracked object are extracted. The TIR and RGB modality features are then fed into a prediction head to generate a response. The following describes the initial infrared sensing adapter and the initial cross-modal fusion adapter:

[0064] 1. Infrared sensing adapter:

[0065] The employed two-stream backbone network is pre-trained on large-scale RGB datasets such as ImageNet and SOT, leveraging common patterns and features in large amounts of data to improve the model's training efficiency and performance on new tasks. However, the infrared modality exhibits unique image patterns, making weights trained on RGB data unsuitable for adaptation to the infrared modality. To address this limitation, an infrared-aware adapter (IA) is proposed that adaptively adjusts the TIR branch to mitigate the feature-level bias inherent in the TIR modality.

[0066] The proposed IA consists of a convolutional layer, a Gaussian error linear unit (GELU), and a channel selector G s where the selector is used to update the intermediate hidden state in the IA module and filter out irrelevant intra-modal information in the TIR modality. The IA process can be expressed as:

[0067]

[0068] Among them G sIt is a fully connected layer that converts the fused local-global features into a channel selector with a Sigmoid activation function. Avg(·) is the average operation performed along the Token dimension. Represents the i-th Token in the TIR modality. f g (·) represents a function consisting of three convolutional layers and two GELU layers. ⊙ represents the Hadamard product, which is used to select object-related information and reject irrelevant information within the modality.

[0069] The Infrared Perception Adapter (IA) adopts a modular design and is applied in a bypass configuration in the Multi-Head Attention (MHA) and Multi-Layer Perceptron (MLP) stages. The output of the IA is added as a residual to the original backbone network, thereby enhancing the overall processing effect without interfering with the main information flow.

[0070] 2. Cross-modal fusion adapter:

[0071] To further enhance the capabilities of the infrared branch, a Cross-Modal Fusion Adapter (CFA) is proposed to leverage the complementary information between the two modalities. Specifically, a modality-dependent guidance strategy is proposed, which consists of a T-guided attention module and an R-guided attention module. First, an attention bias is learned for each modality to facilitate cross-modal guidance and thus identify regions of interest. The process can be described as follows:

[0072]

[0073] Where V r / t =f v (P r / t ) and Q r / t =f q (P r / t ) represent the query and key of RGB / TIR modality respectively. k (·) and f v (·) is a linear projection function. Considering the high computational cost of directly operating the original Token, the input Token and Reshape into And reduce their channel dimension through a 1×1 convolution operation.

[0074] A t =f p (softmax(Q t K r +U1)V r )

[0075] A r =f p (softmax(Qr K t +U2)V t )

[0076] Among them, f p Represents the linear projection function Proj, K r / t =f k (P r / t ) represents the key of RGB / TIR mode. t and A r are the outputs of the T-guided and R-guided attention modules, respectively. Specifically, U1 utilizes the RGB modality as the query, while the T-guided attention module uses the TIR modality as the query. By summing U1 with the T-guided module output, CFA achieves more effective cross-modal guidance, resulting in more robust modality complementary features. Conversely, the same operation is applied to the R-guided attention module. It should be noted that although symmetric operations are applied to the RGB and infrared modalities here, the feature-level bias in the infrared modality will be eliminated next. In addition, the learned matrix A t It will be directly used to assist in enhancing RGB features.

[0077] To reduce the bias in the TIR modality and bridge the gap between RGB and TIR modalities, the RGB information of the TIR branch is enhanced by reintroducing the output of the R-guided attention module.

[0078]

[0079] Among them, f fov Indicates Fovea operation, f c Is a channel attention function used to weigh different channels for effective information selection. As a byproduct of the CFA module, A t It can be used to enhance the feature representation of the RGB branch. Therefore, this feature is incorporated into the RGB feature as a residual.

[0080] S23. Input the output of the infrared prediction head and the output of the RGB prediction head into the response graph decoupling and distillation response module to obtain the target area and non-target area after the response graph is decoupled.

[0081] In one feasible implementation, to adjust for feature-level bias in the TIR modality, this embodiment of the present invention proposes modality-specific and modality fusion adapters. To further eliminate decision-level bias in the TIR modality, the goal is to ensure consistency between the response distributions of the TIR and RGB branches. Specifically, a response decoupling distillation strategy is proposed that leverages the relationship between the RGB and TIR modalities to make more accurate response decisions.

[0082] Considering the inherent nature of the tracking task, where only a small portion of the search region is the target region, while the majority is non-target (background), resulting in an unbalanced response distribution, the response map is decoupled into positive and negative parts according to the true value. A response decoupled distillation loss is designed to apply weights to these regions separately for target region distillation and non-target region distillation.

[0083] Optionally, decouple the distillation loss The formula is shown in the following formula (2):

[0084]

[0085] Among them, y i Represents the predicted response map output by the RGB prediction head, represents the response map output by the infrared prediction head, N represents the total number of samples, n represents the number of positive samples, α represents the weight of the target area, and β represents the weight of the non-target area.

[0086] S24. Based on the target area, non-target area, true labels of RGB training samples, true labels of infrared training samples, and a preset loss function, the initial infrared perception adapter and the initial cross-modal fusion adapter are trained and optimized to obtain a trained infrared perception adapter and a cross-modal fusion adapter.

[0087] Optionally, the preset loss function is as follows:

[0088]

[0089] in, represents the total loss, represents the classification loss, Used to measure the intersection-over-union loss, represents the L1 loss, represents the decoupled distillation loss, and the hyperparameters λ1, λ2, and λ3 are weight factors, respectively.

[0090] in, is calculated as follows:

[0091]

[0092] Among them, p ij Represents the prediction score, y ij Represents the score label.

[0093] is calculated as follows:

[0094]

[0095] Among them, I represents the intersection area of ​​the predicted box and the real box, U represents the union area of ​​the predicted box and the real box, and A c Represents the area of ​​the minimum enclosing rectangle of the predicted box and the true box.

[0096] is calculated as follows:

[0097]

[0098] Among them, y i represents the prediction box, Represents the real frame.

[0099] Repeat the above steps for multiple optimization trainings until the preset conditions are met, stop the training, and use the infrared sensing adapter and cross-modal fusion adapter at this time as the trained infrared sensing adapter and cross-modal fusion adapter.

[0100] S25. Remove the response graph decoupling and distillation response module of the initial multimodal target tracking model including the trained infrared perception adapter and the cross-modal fusion adapter to obtain a trained multimodal target tracking model.

[0101] In a feasible implementation, the response graph decoupling and distillation response module is applied to the module of the training model. Therefore, after the model training is completed, the response graph decoupling and distillation response module is removed to obtain a trained multimodal target tracking model. The trained multimodal target tracking model includes a Transformer backbone network, an infrared perception adapter, a cross-modal fusion adapter, an RGB prediction head, and an infrared prediction head.

[0102] S3. Obtain data to be identified, where the data to be identified includes an RGB image to be identified and an infrared image to be identified.

[0103] S4. Input the RGB image to be identified and the infrared image to be identified into the multimodal target tracking model to obtain the tracking result corresponding to the data to be identified.

[0104] In a feasible implementation, the RGB image to be identified and the infrared image to be identified are input into the multimodal target tracking model. The process of obtaining the output through the multimodal target tracking model is roughly similar to the first half of the process of training the initial multimodal target tracking model mentioned above. The steps S21-S22 mentioned above can be referred to. After obtaining the output of the RGB prediction head and the output of the infrared prediction head, the output of the RGB prediction head and the output of the infrared prediction head are averaged to obtain the final tracking result.

[0105] The following is an explanation of the experimental process and results:

[0106] The model of the embodiment of the present invention is implemented using PyTorch, and the experiments are carried out on two NVIDIA RTX3090 GPUs. For input, the template patch is adjusted to 128×128, and the search area is adjusted to 256×256. The training process adopts a batch size of 24, trains 15 epochs, and processes 60,000 pairs of images per epoch. AdamW is used as the optimizer, the weight decay is set to 1×10^(-4), and the momentum is 0.9. The learning rate of IA and CFA parameters is set to 1×10^(-4), and the learning rate of other parameters is 7.5×10^(-5). After 10 epochs, the learning rate of all parameters is reduced to one tenth of the original value. IA is inserted in the multi-head self-attention stage and MLP stage of each layer of the backbone network. In the CFA module, the number of attention heads is set to 4. The CFA module is inserted into each layer of the backbone network. Set L=12, in the formula In

[15] , α = 1, β = 2. The model is trained on the LasHeR training set with 60k image pairs per epoch and evaluated on the LasHeR test set, RGBT210, and RGBT234 datasets without further fine-tuning.

[0107] The present invention compares the DMD of the present invention with state-of-the-art methods using three different metrics: precision rate (PR), normalized precision rate (NPR), and success rate (SR) to evaluate its effectiveness on four RGB-T tracking benchmarks: GTOT, RGBT210, RGBT234, and LasHeR, as shown in Table 1.

[0108] Table 1

[0109]

[0110]

[0111]

[0112]

[0113] Evaluation on GTOT: In Table 1, the comparison of DMD with the latest methods on the GTOT dataset is reported. GTOT is the first standard RGBT tracking dataset. Note that before the introduction of the LasHeR dataset in 2021, the evaluation of GTOT was performed by randomly selecting 70 videos from the RGBT234 dataset as the training set. To ensure fairness, the method of the embodiment of the present invention is compared with the state-of-the-art methods to verify its effectiveness. The results show that the method of the embodiment of the present invention outperforms these models in all evaluation metrics. Specifically, the model of the embodiment of the present invention improves the MPR by 1.0% and the MSR by 3.2% over the current state-of-the-art method LMINet. Unlike the methods that recently gave up evaluating on the GTOT dataset, the method of the embodiment of the present invention continues to use GTOT for verification, highlighting the generalization ability of DMD on small datasets.

[0114] Evaluation on RGBT210: Table 1 shows the comparison results of DMD with the latest methods on the RGBT210 dataset. The method of the embodiment of the present invention surpasses all the latest RGB-T tracking methods in all indicators. In particular, the method of the embodiment of the present invention achieves comprehensive performance advantages in PR and SR indicators, reaching 87.0% and 63.7% respectively. Compared with the best CNN-based method CAT++, PR is improved by 4.8% and SR is improved by 7.6%. Compared with the advanced Transformer-based method TBSI, DMD improves PR and SR by 1.9% and 1.2% respectively. Please note that the public experimental settings of TBSI are repeated, and the results that exceed the original results presented in the TBSI paper are reported to ensure fair comparison. These results further demonstrate the excellent performance and generalization ability of the method of the embodiment of the present invention on small datasets.

[0115] Evaluation on RGBT234: The RGBT234 dataset is the cornerstone of the RGB-T tracking field, and almost all RGB-T algorithms use it as the main evaluation benchmark. Please note that for the RGBT234 dataset, the true labels of RGB and infrared are not exactly the same. To ensure a fair comparison, the embodiment of the present invention uses maximum precision rate (MPR) and maximum success rate (MSR) as evaluation indicators instead of accuracy and success rate. As shown in Table 1, compared with other RGB-T tracking methods, the DMD of the embodiment of the present invention sets new state-of-the-art results in all indicators, reaching 89.3% on MPR and 66.7% on MSR. The latest CNN-based method CAT++ is 84.0% and 59.2% respectively. In addition, compared with the best Transformer-based RGB-T tracker TBSI, the method of the embodiment of the present invention improves by 1.3% on MPR and 0.9% on SR.

[0116] Evaluation on LasHeR: The effectiveness of DMD was evaluated on 245 test video sequences in the LasHeR dataset, and the detailed results are shown in Table 1. The proposed DMD achieved state-of-the-art results in all indicators. For example, the model of the embodiment of the present invention achieved 72.6%, 68.6%, and 57.6% in PR, NPR, and SR indicators, respectively, while the latest CNN-based method CMD achieved 59.0%, 54.6%, and 46.6%, respectively. For the best Transformer-based method TBSI, DMD outperformed the tracker by 2.1%, 2%, and 1.3%, respectively. The LasHeR dataset is a commonly used large-scale public RGB-T tracking dataset. Compared with other RGB-T tracking datasets, LasHeR poses greater tracking challenges due to its large scale and complexity. Its performance verifies the excellent ability of DMD in handling large-scale tracking data.

[0117] To address feature-level bias, this embodiment proposes a joint infrared fusion adapter module to capture debiased TIR and cross-modal representations. This module includes an infrared perception adapter (IA) and a cross-modal fusion adapter (CFA). Specifically, compared to the RGB modality, the TIR modality relies on thermal radiation characteristics, making it difficult to distinguish between target and background at low contrast. To address this issue, this embodiment employs the infrared perception adapter and introduces a target-related selector mechanism to filter out irrelevant intra-modal information. To further enhance the capabilities of the infrared branch, this embodiment proposes a cross-modal fusion adapter to leverage the complementary information between the two modalities. Specifically, this embodiment implements a modality-related guidance that uses TIR and RGB regional features as queries to learn complementary modal information from the TIR and RGB guidance. Notably, the cross-modal fusion adapter can also mutually enhance feature learning in the RGB modality, thereby bridging the gap between it and the infrared modality. To address decision-level bias, this embodiment proposes a knowledge distillation strategy to ensure consistent response distributions between the TIR and fusion branches. However, in tracking tasks, only a small portion of the search area is the target area, while the majority is non-target (background), resulting in an unbalanced response distribution. To address this issue, embodiments of the present invention decouple the response map into positive and negative components, with the positive component representing the area most relevant to the target. Embodiments of the present invention then design a response map decoupling distillation loss function to ensure that the model focuses more on the target area and achieves a consistent response map distribution.

[0118] Figure 3This is a block diagram of an RGB-T tracking device with a dual-modality enhancement mechanism according to an exemplary embodiment. The device is used in an RGB-T tracking method with a dual-modality enhancement mechanism. Figure 3 , the device comprises:

[0119] A construction unit 310 is used to construct an initial multimodal target tracking model, wherein the initial multimodal target tracking model includes two parallel pre-trained ViT backbone networks, an initial infrared perception adapter, an initial cross-modal fusion adapter, an RGB prediction head, an infrared prediction head, and a response graph decoupling distillation response module, wherein the two parallel pre-trained ViT backbone networks are respectively a first ViT backbone network and a second ViT backbone network;

[0120] A training unit 320 is configured to train and optimize the initial infrared perception adapter and the initial cross-modal fusion adapter based on the RGB training samples, the infrared training samples, and a preset loss function to obtain a trained multimodal target tracking model, wherein the multimodal target tracking model includes a Transformer backbone network, an infrared perception adapter, a cross-modal fusion adapter, an RGB prediction head, and an infrared prediction head;

[0121] An acquisition unit 330 is configured to acquire data to be identified, wherein the data to be identified includes an RGB image to be identified and an infrared image to be identified;

[0122] The tracking unit 340 is configured to input the RGB image to be identified and the infrared image to be identified into the multimodal target tracking model to obtain a tracking result corresponding to the data to be identified.

[0123] Optionally, the RGB training samples include RGB training sample images and RGB training sample true labels, and the infrared training samples include infrared training sample images and infrared training sample true labels;

[0124] The training unit 320 is configured to:

[0125] S21, inputting the infrared training sample image into the first ViT backbone network and the initial infrared perception adapter, accumulating the output of the initial infrared perception adapter with the output of the first ViT backbone network, inputting the RGB training sample image into the second ViT backbone network, and inputting the accumulated result and the output of the second ViT backbone network into the initial cross-modal fusion adapter;

[0126] S22: Input the output of the initial cross-modal fusion adapter to the infrared prediction head, accumulate the output of the second ViT backbone network and the output of the initial cross-modal fusion adapter, and input the accumulated result to the RGB prediction head;

[0127] S23, inputting the output of the infrared prediction head and the output of the RGB prediction head into the response graph decoupling and distillation response module to obtain the target area and non-target area after the response graph is decoupled;

[0128] S24, training and optimizing the initial infrared sensing adapter and the initial cross-modal fusion adapter based on the target area, the non-target area, the true labels of the RGB training samples, the true labels of the infrared training samples, and a preset loss function, to obtain a trained infrared sensing adapter and a cross-modal fusion adapter;

[0129] S25. Remove the response graph decoupling and distillation response module of the initial multimodal target tracking model including the trained infrared perception adapter and the cross-modal fusion adapter to obtain a trained multimodal target tracking model.

[0130] Optionally, the first ViT backbone network and the second ViT backbone network have the same structure;

[0131] The first ViT backbone network includes an embedding layer, a first normalization layer, a multi-head attention mechanism, a second normalization layer, and a multi-layer perceptron.

[0132] Optionally, the training unit 320 is configured to:

[0133] S211, inputting the infrared training sample image into the embedding layer, and inputting the output of the embedding layer into the first normalization layer;

[0134] S212, inputting the output of the first normalization layer into the multi-head attention mechanism and the initial infrared perception adapter respectively, and accumulating the output of the multi-head attention mechanism and the output of the initial infrared perception adapter;

[0135] S213, inputting the accumulated results into the second normalization layer and the initial infrared sensing adapter respectively, and inputting the output of the second normalization layer into the multilayer perceptron;

[0136] S214: Accumulate the output of the multilayer perceptron and the output of the initial infrared sensing adapter.

[0137] Optionally, the initial infrared sensing adapter includes a plurality of convolutional layers, a Gaussian error linear unit, and a channel selector;

[0138] The training unit 320 is configured to:

[0139] The output of the first normalization layer is input into the multi-head attention mechanism and the 1×1 convolution layer of the initial infrared perception adapter respectively, the output of the 1×1 convolution layer is input into the Gaussian error linear unit, the output of the Gaussian error linear unit is input into the 3×3 convolution layer, the output of the 3×3 convolution layer is input into the Gaussian error linear unit, the output of the Gaussian error linear unit is input into the channel selector and the 1×1 convolution layer respectively, the output of the channel selector is input into the 1×1 convolution layer, and the output of the 1×1 convolution layer is used as the output of the initial infrared perception adapter.

[0140] Optionally, the initial cross-modal fusion adapter includes a T-guided module, an R-guided module, a first attention bias learning module, a second attention bias learning module, a linear projection function, a focus operation module, a multi-channel attention mechanism, and a 1×1 convolution module;

[0141] The training unit 320 is configured to:

[0142] S215. Determine the RGB search token and the TIR search token respectively according to the accumulated result and the output of the second ViT backbone network;

[0143] S216: Determine a first attention output based on the accumulated result, the output of the second ViT backbone network, and the first attention bias learning module; and determine a second attention output based on the accumulated result, the output of the second ViT backbone network, and the second attention bias learning module.

[0144] S217: Input the TIR search token into the T guidance module, accumulate the output of the T guidance module with the first attention output, input the accumulated result into the linear projection function, accumulate the first output obtained by the linear projection function with the TIR search token, input the accumulated result into the focus operation module to obtain focus operation data, and input the added operation data into the multi-channel attention mechanism;

[0145] S218: Input the RGB search token into the R guidance module, accumulate the output of the R guidance module with the second attention output, input the accumulated result into the linear projection function, accumulate the second output obtained by the linear projection function with the RGB search token, accumulate the accumulated result with the focus operation data, and input the accumulated result into the multi-channel attention mechanism;

[0146] S219. Input the output of the multi-channel attention mechanism into the 1×1 convolutional layer, and use the output of the 1×1 convolutional layer as the output of the initial cross-modal fusion adapter.

[0147] Optionally, the preset loss function is as follows:

[0148]

[0149] in, represents the total loss, represents the classification loss, Used to measure the intersection-over-union loss, represents the L1 loss, represents the decoupled distillation loss, and the hyperparameters λ1, λ2, and λ3 are weight factors respectively;

[0150] in, The formula is shown in the following formula (2):

[0151]

[0152] Among them, y i Represents the predicted response map output by the RGB prediction head, represents the response map output by the infrared prediction head, N represents the total number of samples, n represents the number of positive samples, α represents the weight of the target area, and β represents the weight of the non-target area.

[0153] To address feature-level bias, this embodiment proposes a joint infrared fusion adapter module to capture debiased TIR and cross-modal representations. This module includes an infrared perception adapter (IA) and a cross-modal fusion adapter (CFA). Specifically, compared to the RGB modality, the TIR modality relies on thermal radiation characteristics, making it difficult to distinguish between target and background at low contrast. To address this issue, this embodiment employs the infrared perception adapter and introduces a target-related selector mechanism to filter out irrelevant intra-modal information. To further enhance the capabilities of the infrared branch, this embodiment proposes a cross-modal fusion adapter to leverage the complementary information between the two modalities. Specifically, this embodiment implements a modality-related guidance that uses TIR and RGB regional features as queries to learn complementary modal information from the TIR and RGB guidance. Notably, the cross-modal fusion adapter can also mutually enhance feature learning in the RGB modality, thereby bridging the gap between it and the infrared modality. To address decision-level bias, this embodiment proposes a knowledge distillation strategy to ensure consistent response distributions between the TIR and fusion branches. However, in tracking tasks, only a small portion of the search area is the target area, while the majority is non-target (background), resulting in an unbalanced response distribution. To address this issue, embodiments of the present invention decouple the response map into positive and negative components, with the positive component representing the area most relevant to the target. Embodiments of the present invention then design a response map decoupling distillation loss function to ensure that the model focuses more on the target area and achieves a consistent response map distribution.

[0154] Figure 4FIG is a structural diagram of an RGB-T tracking device with a dual-modality enhancement mechanism provided by an embodiment of the present invention. Figure 4 As shown, the RGB-T tracking device with dual modality enhancement mechanism can include the above Figure 3 RGB-T tracking device with dual-modality enhancement mechanism as shown. Optionally, the RGB-T tracking device with dual-modality enhancement mechanism 410 may include a first processor 2001 .

[0155] Optionally, the RGB-T tracking device 410 with dual-modality enhancement mechanism may further include a memory 2002 and a transceiver 2003 .

[0156] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0157] The following combination Figure 4 The components of the RGB-T tracking device 410 with dual-modality enhancement mechanism are described in detail:

[0158] The first processor 2001 is the control center of the RGB-T tracking device 410 with a dual-modality enhancement mechanism, and can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement an embodiment of the present invention, such as one or more digital signal processors (DSPs) or one or more field programmable gate arrays (FPGAs).

[0159] Optionally, the first processor 2001 may perform various functions of the RGB-T tracking device 410 with a dual-modality enhancement mechanism by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0160] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 4 CPU0 and CPU1 are shown in FIG.

[0161] In a specific implementation, as an embodiment, the RGB-T tracking device 410 with dual-modality enhancement mechanism may also include multiple processors, such as Figure 41 and 2. The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0162] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled by the first processor 2001 for execution. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0163] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001 or exist independently and be connected to the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0164] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0165] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 4 (not shown separately in the figure). The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0166] Optionally, the transceiver 2003 may be integrated with the first processor 2001 or may exist independently and be connected to the interface circuit of the RGB-T tracking device 410 with a dual-modality enhancement mechanism ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0167] It should be noted that, Figure 4 The structure of the RGB-T tracking device 410 with the dual-modal enhancement mechanism shown in the figures is not intended to limit the router, and the actual knowledge structure recognition device can include more or fewer components than the illustration, or combine certain components, or different component arrangements.

[0168] In addition, the technical effects of the RGB-T tracking device 410 with the dual-modal enhancement mechanism can refer to the technical effects of the RGB-T tracking method with the dual-modal enhancement mechanism described in the above method embodiments, which will not be repeated here.

[0169] It should be understood that the first processor 2001 in the embodiments of the present application can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0170] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0171] The above embodiments can be implemented in whole or in part through software, hardware (such as circuits), firmware, or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired method (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, or magnetic tape), an optical medium (such as a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0172] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0173] In this disclosure, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, "at least one of a, b, or c" can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0174] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0175] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0176] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0177] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of the device or unit, which can be electrical, mechanical or other forms.

[0178] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0179] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0180] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0181] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A dual-modality enhancement mechanism RGB-T tracking method, characterized in that: The method comprises: S1. Constructing an initial multimodal target tracking model, wherein the initial multimodal target tracking model includes two parallel pre-trained ViT backbone networks, an initial infrared perception adapter, an initial cross-modal fusion adapter, an RGB prediction head, an infrared prediction head, and a response graph decoupling distillation response module, wherein the two parallel pre-trained ViT backbone networks are respectively a first ViT backbone network and a second ViT backbone network; S2. According to the RGB training samples, the infrared training samples and the preset loss function, the initial infrared perception adapter and the initial cross-modal fusion adapter are trained and optimized to obtain a trained multimodal target tracking model, wherein the multimodal target tracking model includes a Transformer backbone network, an infrared perception adapter, a cross-modal fusion adapter, an RGB prediction head and an infrared prediction head; wherein, the infrared image block is embedded and input into the Transformer backbone network, and the Transformer backbone network includes the infrared perception adapter; the red, green and blue image blocks are embedded and input into the Transformer backbone network, the infrared image block is embedded into the corresponding Transformer output and the red, green and blue image block is embedded into the corresponding Transformer output and input into the cross-modal fusion adapter, and the output data is input into the infrared prediction head; the red, green and blue image blocks are embedded into the corresponding Transformer output and the output of the cross-modal fusion adapter are added and input into the RGB prediction head; S3. Acquire data to be identified, where the data to be identified includes an RGB image to be identified and an infrared image to be identified; S4. Input the RGB image to be identified and the infrared image to be identified into the multimodal target tracking model to obtain a tracking result corresponding to the data to be identified.

2. The RGB-T tracking method with dual modality enhancement mechanism according to claim 1, characterized in that: The RGB training samples include RGB training sample images and RGB training sample true labels, and the infrared training samples include infrared training sample images and infrared training sample true labels; The S2 trains and optimizes the initial infrared perception adapter and the initial cross-modal fusion adapter based on the RGB training samples, the infrared training samples, and the preset loss function to obtain a trained multimodal target tracking model, including: S21, inputting the infrared training sample image into the first ViT backbone network and the initial infrared perception adapter, accumulating the output of the initial infrared perception adapter with the output of the first ViT backbone network, inputting the RGB training sample image into the second ViT backbone network, and inputting the accumulated result and the output of the second ViT backbone network into the initial cross-modal fusion adapter; S22: Input the output of the initial cross-modal fusion adapter to the infrared prediction head, accumulate the output of the second ViT backbone network and the output of the initial cross-modal fusion adapter, and input the accumulated result to the RGB prediction head; S23, inputting the output of the infrared prediction head and the output of the RGB prediction head into the response graph decoupling and distillation response module to obtain the target area and non-target area after the response graph is decoupled; S24, training and optimizing the initial infrared sensing adapter and the initial cross-modal fusion adapter based on the target area, the non-target area, the true labels of the RGB training samples, the true labels of the infrared training samples, and a preset loss function, to obtain a trained infrared sensing adapter and a cross-modal fusion adapter; S25. Remove the response graph decoupling and distillation response module of the initial multimodal target tracking model including the trained infrared perception adapter and the cross-modal fusion adapter to obtain a trained multimodal target tracking model.

3. The RGB-T tracking method with dual modality enhancement mechanism according to claim 2, characterized in that: The first ViT backbone network and the second ViT backbone network have the same structure; The first ViT backbone network includes an embedding layer, a first normalization layer, a multi-head attention mechanism, a second normalization layer, and a multi-layer perceptron.

4. The RGB-T tracking method with dual modality enhancement mechanism according to claim 3, characterized in that: The step S21 of inputting the infrared training sample image into the first ViT backbone network and the initial infrared sensing adapter, and accumulating the output of the initial infrared sensing adapter and the output of the first ViT backbone network includes: S211, inputting the infrared training sample image into the embedding layer, and inputting the output of the embedding layer into the first normalization layer; S212, inputting the output of the first normalization layer into the multi-head attention mechanism and the initial infrared perception adapter respectively, and accumulating the output of the multi-head attention mechanism and the output of the initial infrared perception adapter; S213, inputting the accumulated results into the second normalization layer and the initial infrared sensing adapter respectively, and inputting the output of the second normalization layer into the multilayer perceptron; S214: Accumulate the output of the multilayer perceptron and the output of the initial infrared sensing adapter.

5. The RGB-T tracking method with dual modality enhancement mechanism according to claim 4, characterized in that: The initial infrared sensing adapter includes a plurality of convolutional layers, a Gaussian error linear unit and a channel selector; The S212 inputs the output of the first normalization layer into the multi-head attention mechanism and the initial infrared perception adapter respectively, including: The output of the first normalization layer is input into the multi-head attention mechanism and the 1×1 convolution layer of the initial infrared perception adapter respectively, the output of the 1×1 convolution layer is input into the Gaussian error linear unit, the output of the Gaussian error linear unit is input into the 3×3 convolution layer, the output of the 3×3 convolution layer is input into the Gaussian error linear unit, the output of the Gaussian error linear unit is input into the channel selector and the 1×1 convolution layer respectively, the output of the channel selector is input into the 1×1 convolution layer, and the output of the 1×1 convolution layer is used as the output of the initial infrared perception adapter.

6. The RGB-T tracking method with dual modality enhancement mechanism according to claim 2, characterized in that: The initial cross-modal fusion adapter includes a T-guided module, an R-guided module, a first attention bias learning module, a second attention bias learning module, a linear projection function, a focus operation module, a multi-channel attention mechanism, and a 1×1 convolution module; The S21 inputs the accumulated results and the output of the second ViT backbone network into the initial cross-modal fusion adapter, including: S215. Determine the RGB search token and the TIR search token respectively according to the accumulated result and the output of the second ViT backbone network; S216: Determine a first attention output based on the accumulated result, the output of the second ViT backbone network, and the first attention bias learning module; and determine a second attention output based on the accumulated result, the output of the second ViT backbone network, and the second attention bias learning module. S217: Input the TIR search token into the T guidance module, accumulate the output of the T guidance module with the first attention output, input the accumulated result into the linear projection function, accumulate the first output obtained by the linear projection function with the TIR search token, input the accumulated result into the focus operation module to obtain focus operation data, and input the added operation data into the multi-channel attention mechanism; S218: Input the RGB search token into the R guidance module, accumulate the output of the R guidance module with the second attention output, input the accumulated result into the linear projection function, accumulate the second output obtained by the linear projection function with the RGB search token, accumulate the accumulated result with the focus operation data, and input the accumulated result into the multi-channel attention mechanism; S219. Input the output of the multi-channel attention mechanism into the 1×1 convolutional layer, and use the output of the 1×1 convolutional layer as the output of the initial cross-modal fusion adapter.

7. The RGB-T tracking method with dual modality enhancement mechanism according to claim 1, characterized in that: The preset loss function is as follows: in, represents the total loss, represents the classification loss, Used to measure the intersection-over-union loss, represents the L1 loss, represents the decoupled distillation loss, and the hyperparameters λ1, λ2, and λ3 are weight factors respectively; in, The formula is shown in the following formula (2): Among them, y i Represents the predicted response map output by the RGB prediction head, represents the response map output by the infrared prediction head, N represents the total number of samples, n represents the number of positive samples, α represents the weight of the target area, and β represents the weight of the non-target area.

8. An RGB-T tracking device with a dual-modal enhancement mechanism, wherein the RGB-T tracking device with a dual-modal enhancement mechanism is used to implement the RGB-T tracking method with a dual-modal enhancement mechanism as described in any one of claims 1 to 7, characterized in that: The device comprises: A construction unit, configured to construct an initial multimodal target tracking model, wherein the initial multimodal target tracking model includes two parallel pre-trained ViT backbone networks, an initial infrared perception adapter, an initial cross-modal fusion adapter, an RGB prediction head, an infrared prediction head, and a response graph decoupling distillation response module, wherein the two parallel pre-trained ViT backbone networks are respectively a first ViT backbone network and a second ViT backbone network; A training unit is used to train and optimize the initial infrared perception adapter and the initial cross-modal fusion adapter based on RGB training samples, infrared training samples, and a preset loss function to obtain a trained multimodal target tracking model. The multimodal target tracking model includes a Transformer backbone network, an infrared perception adapter, a cross-modal fusion adapter, an RGB prediction head, and an infrared prediction head. An acquisition unit, configured to acquire data to be identified, wherein the data to be identified includes an RGB image to be identified and an infrared image to be identified; The tracking unit is used to input the RGB image to be identified and the infrared image to be identified into the multimodal target tracking model to obtain a tracking result corresponding to the data to be identified.

9. An RGB-T tracking device with a dual-modality enhancement mechanism, characterized in that: The RGB-T tracking device with dual-modality enhancement mechanism includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images