RGB-T tracking method and device of dual-mode enhancement mechanism

Abstract

The invention relates to the technical field of multi-mode learning, in particular to an RGB-T tracking method and device of a dual-mode enhancement mechanism, and the method comprises the steps: constructing an initial multi-mode target tracking model, the initial multi-modal target tracking model comprises two parallel ViT backbone networks, an initial infrared sensing adapter, an initial cross-modal fusion adapter, an RGB prediction head, an infrared prediction head and a response diagram decoupling distillation response module; according to the training sample and a preset loss function, training optimization is carried out on the initial multi-modal target tracking model to obtain a trained multi-modal target tracking model, and the trained multi-modal target tracking model comprises a Transform backbone network, an infrared sensing adapter, a cross-modal fusion adapter, an RGB prediction head and an infrared prediction head; and obtaining to-be-identified data, and inputting the to-be-identified RGB image and the to-be-identified infrared image into the multi-modal target tracking model to obtain a tracking result corresponding to the to-be-identified data. According to the invention, the accuracy of target tracking can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal learning, and in particular to an RGB-T tracking method and device with a dual-modal enhancement mechanism. Background Art

[0002] Single object tracking is a fundamental problem in computer vision, which aims to robustly detect a target object in subsequent frames based on its initial state in the first frame. Currently, object tracking using pure visible light (RGB) data has shown promising results, but still faces difficulties in extreme and complex scenarios, such as illumination changes, bad weather, object occlusion, and motion blur. Compared with RGB data that provides detailed color information, thermal infrared (TIR or T) images excel in object recognition and scene understanding in low-light and occluded conditions. Therefore, with the complementary advantages of visible light and infrared (RGB-T), RGB-T tracking has gained increasing attention in various challenging applications, such as autonomous driving and video surveillance.

[0003] Traditional single object tracking (SOT) methods are based on visible light modalities and have difficulty coping with extreme imaging conditions, such as darkness, rain, fog, etc. In contrast, thermal infrared is insensitive to illumination changes and can be used as a meaningful complementary representation in the tracking process. Therefore, RGB-T tracking, as a branch of the tracking task, has become a popular solution for achieving reliable semantic scene understanding under adverse weather and lighting conditions. Existing methods mainly regard cross-modal interaction as a key component. Recently, RGB-T trackers have focused on utilizing multiple fusion strategies to fully exploit the advantages of different fusion methods. However, these methods ignore the impact of networks pre-trained using large-scale RGB datasets on the infrared modality and do not address the differences between the responses of the RGB and infrared branches.

[0004] In addition, due to the lack of large-scale infrared data and backbone networks specifically designed for TIR mode, most existing RGB-T tracking models usually rely on backbone networks pre-trained on large-scale RGB datasets (such as ImageNet and SOT) following RGB-based visual tasks. However, this reliance on RGB pre-trained backbone networks introduces inherent biases towards TIR mode, which can be divided into two levels: (1) Feature-level bias. Models pre-trained on RGB data cannot fully extract and understand TIR data because the imaging mechanism of TIR is different from RGB. Therefore, after learning the specific pattern of RGB data, feature bias is introduced when processing TIR mode data. This bias may weaken the performance of the model, making it difficult to fully capture the key patterns and feature information unique to the TIR mode. (2) Decision-level bias. During the object localization process, trackers of different modes show inconsistent responses to regions with different attributes, resulting in imbalance in tracking decisions. For example, the TIR mode performs better in low-light or occluded conditions, while the RGB mode is more sensitive to areas with bright light, bright colors, rich textures, and rich shapes. Existing methods usually rely on simple cross-modal response fusion, neglecting the distribution difference between RGB and TIR tracker responses, which may lead to significant decision bias in the tracking process. Summary of the invention

[0005] In order to solve the technical problem of target tracking decision deviation in the prior art, the embodiment of the present invention provides an RGB-T tracking method and device with a dual-modality enhancement mechanism. The technical solution is as follows:

[0006] On the one hand, a dual-modality enhancement mechanism RGB-T tracking method is provided, the method is implemented by a dual-modality enhancement mechanism RGB-T tracking device, the method comprising:

[0007] S1. construct an initial multimodal target tracking model, wherein the initial multimodal target tracking model includes two parallel pre-trained ViT backbone networks, an initial infrared perception adapter, an initial cross-modal fusion adapter, an RGB prediction head, an infrared prediction head, and a response graph decoupling distillation response module, wherein the two parallel pre-trained ViT backbone networks are respectively a first ViT backbone network and a second ViT backbone network;

[0008] S2. According to the RGB training samples, the infrared training samples and the preset loss function, the initial infrared perception adapter and the initial cross-modal fusion adapter are trained and optimized to obtain a trained multi-modal target tracking model, wherein the multi-modal target tracking model includes a Transformer backbone network, an infrared perception adapter, a cross-modal fusion adapter, an RGB prediction head and an infrared prediction head;

[0009] S3, obtaining data to be identified, wherein the data to be identified includes an RGB image to be identified and an infrared image to be identified;

[0010] S4. Input the RGB image to be identified and the infrared image to be identified into the multimodal target tracking model to obtain the tracking result corresponding to the data to be identified.

[0011] On the other hand, a dual-modality enhancement mechanism RGB-T tracking device is provided, and the device is applied to the dual-modality enhancement mechanism RGB-T tracking method, and the device includes:

[0012] A construction unit, used to construct an initial multimodal target tracking model, wherein the initial multimodal target tracking model includes two parallel pre-trained ViT backbone networks, an initial infrared perception adapter, an initial cross-modal fusion adapter, an RGB prediction head, an infrared prediction head, and a response graph decoupling distillation response module, wherein the two parallel pre-trained ViT backbone networks are respectively a first ViT backbone network and a second ViT backbone network;

[0013] A training unit is used to train and optimize the initial infrared perception adapter and the initial cross-modal fusion adapter according to the RGB training samples, the infrared training samples and the preset loss function to obtain a trained multi-modal target tracking model, wherein the multi-modal target tracking model includes a Transformer backbone network, an infrared perception adapter, a cross-modal fusion adapter, an RGB prediction head and an infrared prediction head;

[0014] An acquisition unit, used for acquiring data to be identified, wherein the data to be identified includes an RGB image to be identified and an infrared image to be identified;

[0015] The tracking unit is used to input the RGB image to be identified and the infrared image to be identified into the multimodal target tracking model to obtain the tracking result corresponding to the data to be identified.

[0016] On the other hand, an RGB-T tracking device with a dual modal enhancement mechanism is provided, and the RGB-T tracking device with a dual modal enhancement mechanism comprises: a processor; and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the RGB-T tracking methods with the dual modal enhancement mechanism described above is implemented.

[0017] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the RGB-T tracking methods of the above-mentioned dual-modality enhancement mechanism.

[0018] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0019] In the embodiment of the present invention, for the deviation at the feature level, the embodiment of the present invention proposes a joint infrared fusion adapter module to capture the debiased TIR and cross-modal representations, including an infrared perception adapter (IA) and a cross-modal fusion adapter (CFA). Specifically, compared with the RGB modality, since the TIR modality depends on the thermal radiation characteristics, it becomes difficult to distinguish between the target and the background when the contrast is low. To solve this problem, the embodiment of the present invention adopts the infrared perception adapter and introduces the target-related selector mechanism to filter out irrelevant intra-modal information. To further enhance the capability of the infrared branch, the embodiment of the present invention proposes a cross-modal fusion adapter to utilize the complementary information between the two modalities. Specifically, the embodiment of the present invention implements a modality-related guidance, using TIR and RGB regional features as queries to learn the complementary modality information of TIR guidance and RGB guidance. It is worth noting that the cross-modal fusion adapter can also mutually enhance the feature learning of the RGB modality, thereby bridging the gap between it and the infrared modality. To deal with the deviation at the decision level, the embodiment of the present invention proposes a knowledge distillation strategy to ensure that the response distributions of the TIR and fusion branches are consistent. However, in the tracking task, only a small part of the search area is the target area, and most of it is non-target (background), resulting in an unbalanced response distribution. To solve this problem, the embodiment of the present invention decouples the response map into positive and negative parts, and the positive part represents the area most relevant to the target. Then the embodiment of the present invention designs a response map decoupling distillation loss function to ensure that the model pays more attention to the target area and achieves a consistent response map distribution, thereby improving the accuracy of target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 is a flow chart of an RGB-T tracking method with a dual-modality enhancement mechanism provided by an embodiment of the present invention;

[0022] Figure 2 is a schematic diagram of the overall structure of an initial multi-modal target tracking model provided by an embodiment of the present invention;

[0023] Figure 3 It is a block diagram of an RGB-T tracking device with a dual-modality enhancement mechanism provided by an embodiment of the present invention;

[0024] Figure 4 It is a structural schematic diagram of an RGB-T tracking device with a dual-modality enhancement mechanism provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0026] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0027] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0028] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0029] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0030] The embodiment of the present invention provides an RGB-T tracking method with a dual-modality enhancement mechanism, which can be implemented by an RGB-T tracking device with a dual-modality enhancement mechanism, and the RGB-T tracking device with a dual-modality enhancement mechanism can be a terminal or a server. Figure 1 The flowchart of the RGB-T tracking method of the dual-modality enhancement mechanism shown in the figure, the processing flow of the method may include the following steps:

[0031] S1. Construct an initial multimodal target tracking model. The initial multimodal target tracking model includes two parallel pre-trained ViT backbone networks, an initial infrared perception adapter, an initial cross-modal fusion adapter, an RGB prediction head, an infrared prediction head, and a response graph decoupling distillation response module. The two parallel pre-trained ViT backbone networks are the first ViT backbone network and the second ViT backbone network.

[0032] Optionally, the first ViT backbone network and the second ViT backbone network have the same structure.

[0033] The first ViT backbone network includes an embedding layer, a first normalization layer, a multi-head attention mechanism, a second normalization layer, and a multi-layer perceptron.

[0034] Optionally, the initial infrared sensing adapter includes a plurality of convolutional layers, a Gaussian error linear unit, and a channel selector.

[0035] Optionally, the initial cross-modal fusion adapter includes a T-guided module, an R-guided module, a first attention bias learning module, a second attention bias learning module, a linear projection function, a focus operation module, a multi-channel attention mechanism, and a 1×1 convolution module.

[0036] In one possible implementation, given a video and the initial target box B0 of the target object in the first frame, the goal of the RGB-T tracking task is to develop a tracker that can reliably predict the position and size of the target object in all subsequent frames, which can be expressed as Here, (For simplicity, the temporal index is omitted) The search image is represented as RGB and TIR modalities. Tracker It can be decomposed into a feature extraction function and a target box prediction head. The tracking process is expressed as:

[0037]

[0038] Considering the powerful representation ability of Vision Transformer (ViT) in visual tasks, the proposed DMD uses ViT as a tracker The multimodal backbone network is is a pre-trained Transformer backbone network. Figure 2 As shown in Fig. 1, two parallel ViT backbone networks are used, one for RGB and one for infrared modalities, called RGB branch and TIR branch, and each stream shares the same parameters. Specifically, the RGB and TIR search images X are first r ,X t and its corresponding template frame Divide into patches and flatten into 1D tokens Where N x ,N z represents the number of search area patches, and D is the dimension of the token. The RGB and TIR tokens are concatenated as

[0039] S2. According to the RGB training samples, the infrared training samples and the preset loss function, the initial infrared perception adapter and the initial cross-modal fusion adapter are trained and optimized to obtain a trained multimodal target tracking model. The multimodal target tracking model includes a Transformer backbone network, an infrared perception adapter, a cross-modal fusion adapter, an RGB prediction head and an infrared prediction head.

[0040] Among them, the RGB training samples include RGB training sample images and RGB training sample true labels, and the infrared training samples include infrared training sample images and infrared training sample true labels.

[0041] Optionally, the specific operation of S2 may include the following steps S21-S25:

[0042] S21. Input the infrared training sample image into the first ViT backbone network and the initial infrared sensing adapter, accumulate the output of the initial infrared sensing adapter with the output of the first ViT backbone network, input the RGB training sample image into the second ViT backbone network, and input the accumulated result and the output of the second ViT backbone network into the initial cross-modal fusion adapter.

[0043] Optionally, the specific operation of inputting the infrared training sample image into the first ViT backbone network and the initial infrared sensing adapter in S21 may include steps S211-S214:

[0044] S211, inputting the infrared training sample image into the embedding layer, and inputting the output of the embedding layer into the first normalization layer.

[0045] S212. Input the output of the first normalization layer into the multi-head attention mechanism and the initial infrared sensing adapter respectively, and accumulate the output of the multi-head attention mechanism and the output of the initial infrared sensing adapter.

[0046] Optionally, the specific operation of S212 may be as follows:

[0047] The output of the first normalization layer is input into the multi-head attention mechanism and the 1×1 convolution layer of the initial infrared perception adapter respectively, the output of the 1×1 convolution layer is input into the Gaussian error linear unit, the output of the Gaussian error linear unit is input into the 3×3 convolution layer, the output of the 3×3 convolution layer is input into the Gaussian error linear unit, the output of the Gaussian error linear unit is input into the channel selector and the 1×1 convolution layer respectively, the output of the channel selector is input into the 1×1 convolution layer, and the output of the 1×1 convolution layer is used as the output of the initial infrared perception adapter.

[0048] S213, inputting the accumulated results into the second normalization layer and the initial infrared sensing adapter respectively, and inputting the output of the second normalization layer into the multi-layer perceptron.

[0049] S214, accumulating the output of the multilayer perceptron and the output of the initial infrared sensing adapter.

[0050] Optionally, the specific operation of inputting the accumulated result and the output of the second ViT backbone network into the initial cross-modal fusion adapter in S21 may include steps S215-S219:

[0051] S215. Determine the RGB search token and the TIR search token respectively according to the accumulated result and the output of the second ViT backbone network.

[0052] S216. Determine the first attention output based on the accumulated results, the output of the second ViT backbone network, and the first attention bias learning module, and determine the second attention output based on the accumulated results, the output of the second ViT backbone network, and the second attention bias learning module.

[0053] S217. Input the TIR search token into the T guidance module, accumulate the output of the T guidance module with the first attention output, input the accumulated result into the linear projection function, accumulate the first output obtained by the linear projection function with the TIR search token, input the accumulated result into the focus operation module, obtain the focus operation data, and input the added operation data into the multi-channel attention mechanism.

[0054] S218. Input the RGB search token into the R guidance module, accumulate the output of the R guidance module with the second attention output, input the accumulated result into the linear projection function, accumulate the second output obtained by the linear projection function with the RGB search token, accumulate the accumulated result with the focus operation data, and input the accumulated result into the multi-channel attention mechanism.

[0055] S219. Input the output of the multi-channel attention mechanism into the 1×1 convolutional layer, and use the output of the 1×1 convolutional layer as the output of the initial cross-modal fusion adapter.

[0056] S22. Input the output of the initial cross-modal fusion adapter to the infrared prediction head, accumulate the output of the second ViT backbone network with the output of the initial cross-modal fusion adapter, and input the accumulated result to the RGB prediction head.

[0057] In one feasible implementation, the Token sequence As the input of the l-th layer visual Transformer encoder, it can be expressed as follows:

[0058]

[0059] in is the output of the next encoder layer. The Transformer encoder layer consists of a multi-head self-attention mechanism (MHA), a feed-forward network (FFN), a layer normalization (LN), and a multi-layer perceptron (MLP).

[0060] Subsequently, a joint-IR fusion adapter is designed as a trainable modular modification of the Transformer backbone network to adjust the bias of the TIR modality. Specifically, the joint-IR fusion adapter module consists of an IR-aware adapter (IA) and a cross-fusion adapter (CFA). The IA is inserted into the MHA and MLP of each Transformer encoder layer in the TIR branch, while the CFA is embedded in the Transformer encoder at layer 1 for the interaction of the encoders of the two modalities. The learned adapter in each encoder can be expressed as:

[0061]

[0062] in Represents the two-stream backbone network equipped with the proposed joint infrared fusion adapter.

[0063] After L Transformer encoding layers, the multimodal features of the tracked object are extracted. Then, the features of TIR and RGB modalities are input into a prediction head for generating responses. The following introduces the initial infrared perception adapter and the initial cross-modal fusion adapter respectively:

[0064] 1. Infrared sensing adapter:

[0065] The adopted two-stream backbone network is pre-trained on large-scale RGB datasets such as ImageNet and SOT, leveraging common patterns and features in large amounts of data to improve the training efficiency and performance of the model on new tasks. However, the infrared modality exhibits unique image patterns, making the weights trained on RGB data inappropriate for adapting to the infrared modality. To address this limitation, an infrared-aware adapter (IA) is proposed that adaptively adjusts the TIR branch to mitigate the feature-level bias inherent in the TIR modality.

[0066] The proposed IA consists of a convolutional layer, a Gaussian error linear unit (GELU), and a channel selector G s The IA process can be expressed as:

[0067]

[0068] Among them G sis a fully connected layer that transforms the fused local-global features into a channel selector with a Sigmoid activation function. Avg(·) is the average operation performed along the Token dimension. Represents the i-th Token in the TIR modality. g (·) represents a function consisting of three convolutional layers and two GELU layers. ⊙ represents the Hadamard product, which is used to select object-related information and reject irrelevant information within the modality.

[0069] The IR Perception Adapter (IA) is modularly designed and applied in a bypass configuration at the Multi-Head Attention (MHA) and Multi-Layer Perceptron (MLP) stages. The output of the IA is added as a residual to the original backbone network, thereby enhancing the overall processing effect without disturbing the main information flow.

[0070] 2. Cross-modal fusion adapter:

[0071] To further enhance the capability of the infrared branch, a cross-modal fusion adapter (CFA) is proposed to exploit the complementary information between the two modalities. Specifically, a modality-dependent guidance strategy is proposed, which includes a T-guided attention module and an R-guided attention module. First, attention biases are learned for each modality to facilitate cross-modal guidance and thus identify regions of interest. The process can be described as follows:

[0072]

[0073] Where V r / t =f v (P r / t ) and Q r / t =f q (P r / t ) represent the query and key of RGB / TIR modality respectively. k (·) and f v (·) is a linear projection function. Considering the high computational cost of directly operating the original Token, the input Token and Reshape And reduce their channel dimension through a 1×1 convolution operation.

[0074] A t =f p (softmax(Q t K r +U1)V r )

[0075] A r =f p (softmax(Qr K t +U2)V t )

[0076] Among them, f p Represents the linear projection function Proj, K r / t =f k (P r / t ) represents the key of RGB / TIR mode. t and A r are the outputs of the T-guided and R-guided attention modules, respectively. Specifically, U1 utilizes the RGB modality as the query, while the T-guided attention module uses the TIR modality as the query. By summing U1 with the T-guided module output, CFA achieves more effective cross-modal guidance, resulting in more robust modality complementary features. Conversely, the same operation is applied to the R-guided attention module. It should be noted that although symmetric operations are applied to the RGB and infrared modalities here, the feature-level bias in the infrared modality will be eliminated next. In addition, the learned matrix A t It will be directly used to assist in enhancing RGB features.

[0077] To reduce the bias in the TIR modality and bridge the gap between RGB and TIR modalities, the RGB information of the TIR branch is enhanced by reintroducing the output of the R-guided attention module.

[0078]

[0079] Among them, f fov Indicates Fovea operation, f c is a channel attention function used to weigh different channels for effective information selection. As a byproduct of the CFA module, A t It can be used to enhance the feature representation of the RGB branch. Therefore, this feature is incorporated into the RGB feature as a residual.

[0080] S23. Input the output of the infrared prediction head and the output of the RGB prediction head into the response graph decoupling distillation response module to obtain the target area and the non-target area after the response graph is decoupled.

[0081] In a feasible implementation, in order to adjust the feature-level deviation in the TIR modality, the embodiment of the present invention proposes a specific modality and a modality fusion adapter. In order to further eliminate the decision-level deviation in the TIR modality, it is intended to ensure the consistency between the response distributions of the TIR and RGB branches. Specifically, a response decoupling distillation strategy is proposed to make more accurate response decisions by utilizing the relationship between the RGB and TIR modalities.

[0082] Considering the inherent nature of the tracking task, where only a small portion of the search region is the target region, while the majority is non-target (background), resulting in an unbalanced response distribution, the response map is decoupled into positive and negative parts according to the true value. A response-decoupled distillation loss is designed to apply weights to these regions separately for target region distillation and non-target region distillation.

[0083] Optionally, decouple the distillation loss The formula is shown in the following formula (2):

[0084]

[0085] Among them, y i represents the predicted response map output by the RGB prediction head, represents the response map output by the infrared prediction head, N represents the total number of samples, n represents the number of positive samples, α represents the weight of the target area, and β represents the weight of the non-target area.

[0086] S24. According to the target area, non-target area, the true label of the RGB training sample, the true label of the infrared training sample and the preset loss function, the initial infrared sensing adapter and the initial cross-modal fusion adapter are trained and optimized to obtain the trained infrared sensing adapter and the cross-modal fusion adapter.

[0087] Optionally, the preset loss function is as follows:

[0088]

[0089] in, represents the total loss, represents the classification loss, Used to measure the intersection-over-union loss, represents L1 loss, represents the decoupled distillation loss, and the hyperparameters λ1, λ2, and λ3 are weight factors, respectively.

[0090] in, The calculation method is as follows:

[0091]

[0092] Among them, p ij Represents the prediction score, y ij Represents the score label.

[0093] The calculation method is as follows:

[0094]

[0095] Among them, I represents the intersection area of ​​the predicted box and the real box, U represents the union area of ​​the predicted box and the real box, and A c Represents the area of ​​the minimum enclosing rectangle of the predicted box and the true box.

[0096] The calculation method is as follows:

[0097]

[0098] Among them, y i represents the prediction box, Represents the real frame.

[0099] Repeat the above steps to perform multiple optimization trainings until the preset conditions are met, stop the training, and use the infrared sensing adapter and cross-modal fusion adapter at this time as the trained infrared sensing adapter and cross-modal fusion adapter.

[0100] S25. Remove the response module of the response graph decoupling distillation of the initial multimodal target tracking model including the trained infrared perception adapter and the cross-modal fusion adapter to obtain a trained multimodal target tracking model.

[0101] In a feasible implementation, the response graph decoupling distillation response module is applied to the module of the training model. Therefore, after completing the model training, the response graph decoupling distillation response module is removed to obtain a trained multimodal target tracking model. The trained multimodal target tracking model includes a Transformer backbone network, an infrared perception adapter, a cross-modal fusion adapter, an RGB prediction head, and an infrared prediction head.

[0102] S3. Obtaining data to be identified, where the data to be identified includes an RGB image to be identified and an infrared image to be identified.

[0103] S4. Input the RGB image to be identified and the infrared image to be identified into the multimodal target tracking model to obtain the tracking result corresponding to the data to be identified.

[0104] In a feasible implementation manner, the RGB image to be identified and the infrared image to be identified are input into the multimodal target tracking model. The process of obtaining the output through the multimodal target tracking model is roughly similar to the first half of the process of training the initial multimodal target tracking model mentioned above. The above steps S21-S22 can be referred to. After obtaining the output of the RGB prediction head and the output of the infrared prediction head, the output of the RGB prediction head and the output of the infrared prediction head are averaged to obtain the final tracking result.

[0105] The experimental process and results are described below:

[0106] The model of the embodiment of the present invention is implemented using PyTorch, and the experiments are carried out on two NVIDIA RTX3090 GPUs. For input, the template patch is adjusted to 128×128, and the search area is adjusted to 256×256. The training process adopts a batch size of 24, trains 15 epochs, and processes 60,000 pairs of images in each epoch. AdamW is used as the optimizer, the weight decay is set to 1×10^(-4), and the momentum is 0.9. The learning rate of IA and CFA parameters is set to 1×10^(-4), and the learning rate of other parameters is 7.5×10^(-5). After 10 epochs, the learning rate of all parameters is reduced to one tenth of the original value. IA is inserted in the multi-head self-attention stage and MLP stage of each layer of the backbone network. In the CFA module, the number of attention heads is set to 4. The CFA module is inserted into each layer of the backbone network. Set L=12, in the formula where α = 1, β = 2. The model is trained on the LasHeR training set using 60k image pairs per epoch and evaluated on the LasHeR test set, RGBT210 and RGBT234 datasets without further fine-tuning.

[0107] The present invention compares the DMD of the present invention with the latest methods on four RGB-T tracking benchmarks using three different metrics: precision rate (PR), normalized precision rate (NPR), and success rate (SR) to evaluate its effectiveness. The four benchmarks include GTOT, RGBT210, RGBT234, and LasHeR, as shown in Table 1 below.

[0108] Table 1

[0109]

[0110]

[0111]

[0112]

[0113] Evaluation on GTOT: In Table 1, the comparison of DMD with the state-of-the-art methods on the GTOT dataset is reported. GTOT is the first standard RGBT tracking dataset. Note that before the introduction of the LasHeR dataset in 2021, the evaluation of GTOT was performed by randomly selecting 70 videos from the RGBT234 dataset as the training set. To ensure fairness, the method of the embodiment of the present invention is compared with the state-of-the-art methods to verify its effectiveness. The results show that the method of the embodiment of the present invention outperforms these models in all evaluation metrics. Specifically, the model of the embodiment of the present invention improves the MPR by 1.0% and the MSR by 3.2% over the current state-of-the-art method LMINet. Unlike the recent methods that gave up evaluating on the GTOT dataset, the method of the embodiment of the present invention continues to use GTOT for verification, highlighting the generalization ability of DMD on small datasets.

[0114] Evaluation on RGBT210: In Table 1, the comparison results of DMD and the latest methods on the RGBT210 dataset are shown. The method of the embodiment of the present invention surpasses all the latest RGB-T tracking methods in all indicators. In particular, the method of the embodiment of the present invention achieves comprehensive performance advantages in PR and SR indicators, reaching 87.0% and 63.7%, respectively. Compared with the best CNN-based method CAT++, PR is improved by 4.8% and SR is improved by 7.6%. Compared with the advanced Transformer-based method TBSI, DMD improves PR and SR by 1.9% and 1.2%, respectively. Please note that the public experimental settings of TBSI are repeated, and the results that exceed those originally presented in the TBSI paper are reported to ensure fair comparison. These results further demonstrate the excellent performance and generalization ability of the method of the embodiment of the present invention on small datasets.

[0115] Evaluation on RGBT234: The RGBT234 dataset is the cornerstone of the RGB-T tracking field, and almost all RGB-T algorithms use it as the main evaluation benchmark. Please note that for the RGBT234 dataset, the true labels of RGB and infrared are not exactly the same. In order to ensure a fair comparison, the embodiment of the present invention uses the maximum precision rate (MPR) and the maximum success rate (MSR) as evaluation indicators instead of accuracy and success rate. As shown in Table 1, compared with other RGB-T tracking methods, the DMD of the embodiment of the present invention sets new state-of-the-art results in all indicators, reaching 89.3% on MPR and 66.7% on MSR. The latest CNN-based method CAT++ is 84.0% and 59.2% respectively. In addition, compared with the best Transformer-based RGB-T tracker TBSI, the method of the embodiment of the present invention improves by 1.3% on MPR and 0.9% on SR.

[0116] Evaluation on LasHeR: The effectiveness of DMD is evaluated on 245 test video sequences in the LasHeR dataset, and the detailed results are shown in Table 1. The proposed DMD achieves state-of-the-art results in all indicators. For example, the model of the embodiment of the present invention achieves 72.6%, 68.6% and 57.6% in PR, NPR and SR indicators, respectively, while the latest CNN-based method CMD achieves 59.0%, 54.6% and 46.6%, respectively. For the best Transformer-based method TBSI, DMD outperforms the tracker by 2.1%, 2% and 1.3%, respectively. The LasHeR dataset is a commonly used large-scale public RGB-T tracking dataset. Compared with other RGB-T tracking datasets, LasHeR brings greater tracking challenges due to its large scale and complexity. Its performance verifies the excellent ability of DMD in handling large-scale tracking data.

[0117] In the embodiment of the present invention, for the deviation at the feature level, the embodiment of the present invention proposes a joint infrared fusion adapter module to capture the debiased TIR and cross-modal representations, including an infrared perception adapter (IA) and a cross-modal fusion adapter (CFA). Specifically, compared with the RGB modality, since the TIR modality depends on the thermal radiation characteristics, it becomes difficult to distinguish between the target and the background when the contrast is low. To solve this problem, the embodiment of the present invention adopts the infrared perception adapter and introduces the target-related selector mechanism to filter out irrelevant intra-modal information. To further enhance the capability of the infrared branch, the embodiment of the present invention proposes a cross-modal fusion adapter to utilize the complementary information between the two modalities. Specifically, the embodiment of the present invention implements a modality-related guidance, using TIR and RGB regional features as queries to learn the complementary modality information of TIR guidance and RGB guidance. It is worth noting that the cross-modal fusion adapter can also mutually enhance the feature learning of the RGB modality, thereby bridging the gap between it and the infrared modality. To deal with the deviation at the decision level, the embodiment of the present invention proposes a knowledge distillation strategy to ensure that the response distributions of the TIR and fusion branches are consistent. However, in tracking tasks, only a small part of the search area is the target area, and most of it is non-target (background), resulting in an unbalanced response distribution. To solve this problem, the embodiment of the present invention decouples the response map into positive and negative parts, and the positive part represents the area most relevant to the target. Then the embodiment of the present invention designs a response map decoupling distillation loss function to ensure that the model pays more attention to the target area and achieves a consistent response map distribution.

[0118] Figure 31 is a block diagram of an RGB-T tracking device with a dual-modality enhancement mechanism according to an exemplary embodiment, wherein the device is used in an RGB-T tracking method with a dual-modality enhancement mechanism. Figure 3 , the device comprises:

[0119] A construction unit 310 is used to construct an initial multimodal target tracking model, wherein the initial multimodal target tracking model includes two parallel pre-trained ViT backbone networks, an initial infrared perception adapter, an initial cross-modal fusion adapter, an RGB prediction head, an infrared prediction head, and a response graph decoupling distillation response module, wherein the two parallel pre-trained ViT backbone networks are respectively a first ViT backbone network and a second ViT backbone network;

[0120] A training unit 320 is used to train and optimize the initial infrared perception adapter and the initial cross-modal fusion adapter according to the RGB training samples, the infrared training samples and the preset loss function to obtain a trained multi-modal target tracking model, wherein the multi-modal target tracking model includes a Transformer backbone network, an infrared perception adapter, a cross-modal fusion adapter, an RGB prediction head and an infrared prediction head;

[0121] An acquisition unit 330 is used to acquire data to be identified, wherein the data to be identified includes an RGB image to be identified and an infrared image to be identified;

[0122] The tracking unit 340 is used to input the RGB image to be identified and the infrared image to be identified into the multimodal target tracking model to obtain the tracking result corresponding to the data to be identified.

[0123] Optionally, the RGB training samples include RGB training sample images and RGB training sample true labels, and the infrared training samples include infrared training sample images and infrared training sample true labels;

[0124] The training unit 320 is used to:

[0125] S21, inputting the infrared training sample image into the first ViT backbone network and the initial infrared perception adapter, accumulating the output of the initial infrared perception adapter with the output of the first ViT backbone network, inputting the RGB training sample image into the second ViT backbone network, and inputting the accumulated result and the output of the second ViT backbone network into the initial cross-modal fusion adapter;

[0126] S22, inputting the output of the initial cross-modal fusion adapter to the infrared prediction head, accumulating the output of the second ViT backbone network and the output of the initial cross-modal fusion adapter, and inputting the accumulated result to the RGB prediction head;

[0127] S23, inputting the output of the infrared prediction head and the output of the RGB prediction head into the response graph decoupling distillation response module to obtain the target area and the non-target area after the response graph is decoupled;

[0128] S24, training and optimizing the initial infrared sensing adapter and the initial cross-modal fusion adapter according to the target area, the non-target area, the true label of the RGB training sample, the true label of the infrared training sample, and the preset loss function, to obtain a trained infrared sensing adapter and a cross-modal fusion adapter;

[0129] S25. Remove the response module of the response graph decoupling distillation of the initial multimodal target tracking model including the trained infrared perception adapter and the cross-modal fusion adapter to obtain a trained multimodal target tracking model.

[0130] Optionally, the first ViT backbone network has the same structure as the second ViT backbone network;

[0131] The first ViT backbone network includes an embedding layer, a first normalization layer, a multi-head attention mechanism, a second normalization layer and a multi-layer perceptron.

[0132] Optionally, the training unit 320 is used to:

[0133] S211, inputting the infrared training sample image into the embedding layer, and inputting the output of the embedding layer into the first normalization layer;

[0134] S212, inputting the output of the first normalization layer into the multi-head attention mechanism and the initial infrared perception adapter respectively, and accumulating the output of the multi-head attention mechanism and the output of the initial infrared perception adapter;

[0135] S213, inputting the accumulated results into the second normalization layer and the initial infrared sensing adapter respectively, and inputting the output of the second normalization layer into the multi-layer perceptron;

[0136] S214, accumulating the output of the multilayer perceptron and the output of the initial infrared sensing adapter.

[0137] Optionally, the initial infrared sensing adapter includes a plurality of convolutional layers, a Gaussian error linear unit, and a channel selector;

[0138] The training unit 320 is used to:

[0139] The output of the first normalization layer is input into the multi-head attention mechanism and the 1×1 convolution layer of the initial infrared perception adapter respectively, the output of the 1×1 convolution layer is input into the Gaussian error linear unit, the output of the Gaussian error linear unit is input into the 3×3 convolution layer, the output of the 3×3 convolution layer is input into the Gaussian error linear unit, the output of the Gaussian error linear unit is input into the channel selector and the 1×1 convolution layer respectively, the output of the channel selector is input into the 1×1 convolution layer, and the output of the 1×1 convolution layer is used as the output of the initial infrared perception adapter.

[0140] Optionally, the initial cross-modal fusion adapter includes a T-guided module, an R-guided module, a first attention bias learning module, a second attention bias learning module, a linear projection function, a focus operation module, a multi-channel attention mechanism, and a 1×1 convolution module;

[0141] The training unit 320 is used to:

[0142] S215, determining the RGB search token and the TIR search token respectively according to the accumulated result and the output of the second ViT backbone network;

[0143] S216, determining the first attention output according to the accumulated result, the output of the second ViT backbone network, and the first attention bias learning module, and determining the second attention output according to the accumulated result, the output of the second ViT backbone network, and the second attention bias learning module;

[0144] S217, input the TIR search token into the T guidance module, accumulate the output of the T guidance module and the first attention output, input the accumulated result into the linear projection function, accumulate the first output obtained by the linear projection function and the TIR search token, input the accumulated result into the focus operation module, obtain the focus operation data, and input the added operation data into the multi-channel attention mechanism;

[0145] S218, input the RGB search token into the R guidance module, accumulate the output of the R guidance module with the second attention output, input the accumulated result into the linear projection function, accumulate the second output obtained by the linear projection function with the RGB search token, accumulate the accumulated result with the focus operation data, and input the accumulated result into the multi-channel attention mechanism;

[0146] S219. Input the output of the multi-channel attention mechanism into the 1×1 convolutional layer, and use the output of the 1×1 convolutional layer as the output of the initial cross-modal fusion adapter.

[0147] Optionally, the preset loss function is as follows:

[0148]

[0149] in, represents the total loss, represents the classification loss, Used to measure the intersection-over-union loss, represents L1 loss, represents the decoupled distillation loss, and the hyperparameters λ1, λ2, and λ3 are weight factors respectively;

[0150] in, The formula is shown in the following formula (2):

[0151]

[0152] Among them, y i represents the predicted response map output by the RGB prediction head, represents the response map output by the infrared prediction head, N represents the total number of samples, n represents the number of positive samples, α represents the weight of the target area, and β represents the weight of the non-target area.

[0153] In the embodiment of the present invention, for the deviation at the feature level, the embodiment of the present invention proposes a joint infrared fusion adapter module to capture the debiased TIR and cross-modal representations, including an infrared perception adapter (IA) and a cross-modal fusion adapter (CFA). Specifically, compared with the RGB modality, since the TIR modality depends on the thermal radiation characteristics, it becomes difficult to distinguish between the target and the background when the contrast is low. To solve this problem, the embodiment of the present invention adopts the infrared perception adapter and introduces the target-related selector mechanism to filter out irrelevant intra-modal information. To further enhance the capability of the infrared branch, the embodiment of the present invention proposes a cross-modal fusion adapter to utilize the complementary information between the two modalities. Specifically, the embodiment of the present invention implements a modality-related guidance, using TIR and RGB regional features as queries to learn the complementary modality information of TIR guidance and RGB guidance. It is worth noting that the cross-modal fusion adapter can also mutually enhance the feature learning of the RGB modality, thereby bridging the gap between it and the infrared modality. To deal with the deviation at the decision level, the embodiment of the present invention proposes a knowledge distillation strategy to ensure that the response distributions of the TIR and fusion branches are consistent. However, in tracking tasks, only a small part of the search area is the target area, and most of it is non-target (background), resulting in an unbalanced response distribution. To solve this problem, the embodiment of the present invention decouples the response map into positive and negative parts, and the positive part represents the area most relevant to the target. Then the embodiment of the present invention designs a response map decoupling distillation loss function to ensure that the model pays more attention to the target area and achieves a consistent response map distribution.

[0154] Figure 4is a schematic diagram of the structure of an RGB-T tracking device with a dual-modality enhancement mechanism provided by an embodiment of the present invention, such as Figure 4 As shown, the RGB-T tracking device with dual-modality enhancement mechanism may include the above Figure 3 The RGB-T tracking device with dual-modality enhancement mechanism shown. Optionally, the RGB-T tracking device with dual-modality enhancement mechanism 410 may include a first processor 2001 .

[0155] Optionally, the RGB-T tracking device 410 with dual modality enhancement mechanism may further include a memory 2002 and a transceiver 2003 .

[0156] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0157] Combine the following Figure 4 The components of the RGB-T tracking device 410 with dual-modality enhancement mechanism are specifically introduced as follows:

[0158] The first processor 2001 is the control center of the RGB-T tracking device 410 with dual modality enhancement mechanism, and can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (FPGAs).

[0159] Optionally, the first processor 2001 can perform various functions of the RGB-T tracking device 410 with a dual-modality enhancement mechanism by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0160] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 4 CPU0 and CPU1 are shown in FIG.

[0161] In a specific implementation, as an embodiment, the RGB-T tracking device 410 with dual-modality enhancement mechanism may also include multiple processors, such as Figure 4The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0162] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0163] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be connected to the RGB-T tracking device 410 through the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0164] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0165] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 4 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0166] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and communicate with the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0167] It should be noted that Figure 4 The structure of the RGB-T tracking device 410 with a dual-modality enhancement mechanism shown in the figure does not constitute a limitation on the router, and the actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0168] In addition, the technical effects of the RGB-T tracking device 410 with dual-modality enhancement mechanism can refer to the technical effects of the RGB-T tracking method with dual-modality enhancement mechanism described in the above method embodiment, which will not be repeated here.

[0169] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0170] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0171] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0172] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0173] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0174] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0175] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0176] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0177] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0178] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0179] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0180] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0181] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A dual-modality enhancement mechanism RGB-T tracking method, characterized in that: The method comprises: S1. construct an initial multimodal target tracking model, wherein the initial multimodal target tracking model includes two parallel pre-trained ViT backbone networks, an initial infrared perception adapter, an initial cross-modal fusion adapter, an RGB prediction head, an infrared prediction head, and a response graph decoupling distillation response module, wherein the two parallel pre-trained ViT backbone networks are respectively a first ViT backbone network and a second ViT backbone network; S2. According to the RGB training samples, the infrared training samples and the preset loss function, the initial infrared perception adapter and the initial cross-modal fusion adapter are trained and optimized to obtain a trained multi-modal target tracking model, wherein the multi-modal target tracking model includes a Transformer backbone network, an infrared perception adapter, a cross-modal fusion adapter, an RGB prediction head and an infrared prediction head; S3, obtaining data to be identified, wherein the data to be identified includes an RGB image to be identified and an infrared image to be identified; S4. Input the RGB image to be identified and the infrared image to be identified into the multimodal target tracking model to obtain the tracking result corresponding to the data to be identified.

2. The RGB-T tracking method with dual modality enhancement mechanism according to claim 1, characterized in that: The RGB training samples include RGB training sample images and RGB training sample true labels, and the infrared training samples include infrared training sample images and infrared training sample true labels; The S2 trains and optimizes the initial infrared perception adapter and the initial cross-modal fusion adapter according to the RGB training samples, the infrared training samples and the preset loss function to obtain a trained multi-modal target tracking model, including: S21, inputting the infrared training sample image into the first ViT backbone network and the initial infrared perception adapter, accumulating the output of the initial infrared perception adapter with the output of the first ViT backbone network, inputting the RGB training sample image into the second ViT backbone network, and inputting the accumulated result and the output of the second ViT backbone network into the initial cross-modal fusion adapter; S22, inputting the output of the initial cross-modal fusion adapter to the infrared prediction head, accumulating the output of the second ViT backbone network and the output of the initial cross-modal fusion adapter, and inputting the accumulated result to the RGB prediction head; S23, inputting the output of the infrared prediction head and the output of the RGB prediction head into the response graph decoupling distillation response module to obtain the target area and the non-target area after the response graph is decoupled; S24, training and optimizing the initial infrared sensing adapter and the initial cross-modal fusion adapter according to the target area, the non-target area, the true label of the RGB training sample, the true label of the infrared training sample, and the preset loss function, to obtain a trained infrared sensing adapter and a cross-modal fusion adapter; S25. Remove the response module of the response graph decoupling distillation of the initial multimodal target tracking model including the trained infrared perception adapter and the cross-modal fusion adapter to obtain a trained multimodal target tracking model.

3. The RGB-T tracking method with dual modality enhancement mechanism according to claim 2, characterized in that: The structure of the first ViT backbone network is the same as that of the second ViT backbone network; The first ViT backbone network includes an embedding layer, a first normalization layer, a multi-head attention mechanism, a second normalization layer and a multi-layer perceptron.

4. The RGB-T tracking method with dual modality enhancement mechanism according to claim 3, characterized in that: The step S21 of inputting the infrared training sample image into the first ViT backbone network and the initial infrared sensing adapter, and accumulating the output of the initial infrared sensing adapter and the output of the first ViT backbone network includes: S211, inputting the infrared training sample image into the embedding layer, and inputting the output of the embedding layer into the first normalization layer; S212, inputting the output of the first normalization layer into the multi-head attention mechanism and the initial infrared perception adapter respectively, and accumulating the output of the multi-head attention mechanism and the output of the initial infrared perception adapter; S213, inputting the accumulated results into the second normalization layer and the initial infrared sensing adapter respectively, and inputting the output of the second normalization layer into the multi-layer perceptron; S214, accumulating the output of the multilayer perceptron and the output of the initial infrared sensing adapter.

5. The RGB-T tracking method with dual modality enhancement mechanism according to claim 4, characterized in that: The initial infrared sensing adapter includes a plurality of convolutional layers, a Gaussian error linear unit and a channel selector; The S212 inputs the output of the first normalization layer into the multi-head attention mechanism and the initial infrared perception adapter respectively, including: The output of the first normalization layer is input into the multi-head attention mechanism and the 1×1 convolution layer of the initial infrared perception adapter respectively, the output of the 1×1 convolution layer is input into the Gaussian error linear unit, the output of the Gaussian error linear unit is input into the 3×3 convolution layer, the output of the 3×3 convolution layer is input into the Gaussian error linear unit, the output of the Gaussian error linear unit is input into the channel selector and the 1×1 convolution layer respectively, the output of the channel selector is input into the 1×1 convolution layer, and the output of the 1×1 convolution layer is used as the output of the initial infrared perception adapter.

6. The RGB-T tracking method with dual modality enhancement mechanism according to claim 2, characterized in that: The initial cross-modal fusion adapter includes a T-guided module, an R-guided module, a first attention bias learning module, a second attention bias learning module, a linear projection function, a focus operation module, a multi-channel attention mechanism, and a 1×1 convolution module; The S21 inputs the accumulated result and the output of the second ViT backbone network into the initial cross-modal fusion adapter, including: S215, determining the RGB search token and the TIR search token respectively according to the accumulated result and the output of the second ViT backbone network; S216, determining the first attention output according to the accumulated result, the output of the second ViT backbone network, and the first attention bias learning module, and determining the second attention output according to the accumulated result, the output of the second ViT backbone network, and the second attention bias learning module; S217, input the TIR search token into the T guidance module, accumulate the output of the T guidance module and the first attention output, input the accumulated result into the linear projection function, accumulate the first output obtained by the linear projection function and the TIR search token, input the accumulated result into the focus operation module, obtain the focus operation data, and input the added operation data into the multi-channel attention mechanism; S218, input the RGB search token into the R guidance module, accumulate the output of the R guidance module with the second attention output, input the accumulated result into the linear projection function, accumulate the second output obtained by the linear projection function with the RGB search token, accumulate the accumulated result with the focus operation data, and input the accumulated result into the multi-channel attention mechanism; S219. Input the output of the multi-channel attention mechanism into the 1×1 convolutional layer, and use the output of the 1×1 convolutional layer as the output of the initial cross-modal fusion adapter.

7. The RGB-T tracking method with dual modality enhancement mechanism according to claim 1, characterized in that: The preset loss function is as follows: in, represents the total loss, represents the classification loss, Used to measure the intersection-over-union loss, represents L1 loss, represents the decoupled distillation loss, and the hyperparameters λ1, λ2, and λ3 are weight factors respectively; in, The formula is shown in the following formula (2): Among them, y i represents the predicted response map output by the RGB prediction head, represents the response map output by the infrared prediction head, N represents the total number of samples, n represents the number of positive samples, α represents the weight of the target area, and β represents the weight of the non-target area.

8. An RGB-T tracking device with dual-modality enhancement mechanism, the RGB-T tracking device with dual-modality enhancement mechanism is used to implement the RGB-T tracking method with dual-modality enhancement mechanism as claimed in any one of claims 1 to 7, characterized in that: The device comprises: A construction unit, used to construct an initial multimodal target tracking model, wherein the initial multimodal target tracking model includes two parallel pre-trained ViT backbone networks, an initial infrared perception adapter, an initial cross-modal fusion adapter, an RGB prediction head, an infrared prediction head, and a response graph decoupling distillation response module, wherein the two parallel pre-trained ViT backbone networks are respectively a first ViT backbone network and a second ViT backbone network; A training unit is used to train and optimize the initial infrared perception adapter and the initial cross-modal fusion adapter according to the RGB training samples, the infrared training samples and the preset loss function to obtain a trained multi-modal target tracking model, wherein the multi-modal target tracking model includes a Transformer backbone network, an infrared perception adapter, a cross-modal fusion adapter, an RGB prediction head and an infrared prediction head; An acquisition unit, used for acquiring data to be identified, wherein the data to be identified includes an RGB image to be identified and an infrared image to be identified; The tracking unit is used to input the RGB image to be identified and the infrared image to be identified into the multimodal target tracking model to obtain the tracking result corresponding to the data to be identified.

9. A dual-modality enhanced RGB-T tracking device, characterized in that: The RGB-T tracking device with dual-modality enhancement mechanism includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images