Rapid RGBT target tracking method based on cross-modal interaction enhancement and knowledge distillation

Abstract

The embodiment of the invention relates to the technical field of computer vision, in particular to a rapid RGBT target tracking method based on cross-modal interaction enhancement and knowledge distillation, and the method comprises the steps: building a training sample set based on template images of a visible light mode and a thermal infrared mode, a search region and a corresponding real label; constructing an RGBT target tracking model composed of a visible light feature extraction network, a thermal infrared feature extraction network, a cross-modal interaction enhancement network and a multi-modal tracking head; selecting a teacher model, and taking the RGBT target tracking model as a student model; the samples in the training sample set are correspondingly input into the student model, the visible light teacher model and the thermal infrared teacher model, inter-modal distillation and intra-modal distillation are carried out, and an overall loss function is constructed and trained until convergence is achieved in combination with output of a real label and a multi-modal tracking head, so that a trained RGBT target tracking model is obtained; and deploying the trained RGBT target tracking model to realize real-time RGBT target tracking.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer vision, and in particular to a fast RGBT target tracking method based on cross-modal interaction enhancement and knowledge distillation. BACKGROUND

[0002] In recent years, with the continuous development of deep learning technology, many single-spectrum-based visible light target tracking methods have been widely applied. However, the current single-spectrum-based visible light target tracking method has many problems, especially in bad environments such as rain, fog, light change, occlusion, and low contrast, the tracking effect of single modal will be significantly reduced. In extreme conditions, the quality of visible light imaging is easily affected by external factors. For example, at night or in weak light, the camera is difficult to give a clear image, which makes it difficult for the tracker to accurately identify and track it. Compared with visible light images, thermal infrared images can better reflect the thermal radiation characteristics of objects in the scene, so in the night, dim light, and other scenes, stable image quality can still be obtained. These characteristics make it have very high application value in night monitoring, fog driving, and other low-visibility conditions. However, thermal infrared images lack obvious texture, color, and other detail characteristics, which makes it difficult for the target tracking model based on the thermal infrared modal to distinguish similar targets. In addition, due to the low resolution of thermal infrared imaging, it is easily affected by environmental factors, and relying solely on the thermal infrared modal to achieve target tracking still has great limitations.

[0003] Visible light modal and thermal infrared modal each have advantages and disadvantages, therefore, the RGBT target tracking method using the complementarity of the two modalities has emerged. Visible light images can provide rich color and texture information, and thermal infrared images can still have good imaging ability in extreme light or bad weather conditions, and the RGBT target tracking method effectively combines these two advantages, so that it can still maintain stable tracking effect in complex environments. In the field of intelligent video monitoring, the RGBT target tracking technology can effectively improve the target tracking performance, especially in the case of insufficient light or large occlusion, by introducing the thermal infrared modal to improve the robustness of the monitoring system. In the field of unmanned driving, the RGBT target tracking method can effectively improve the ability of vehicles to identify targets in complex scenes. In the field of unmanned aerial vehicles and robot vision, unmanned aerial vehicles often need to track ground targets, and the RGBT target tracking method helps to improve the stable tracking ability of unmanned aerial vehicles to targets, and promotes the application of unmanned aerial vehicles in original forest emergency rescue, forest patrol, mountain disaster monitoring, and other aspects.

[0004] While ensuring the high-performance tracking ability of the RGBT target tracking model, the increase of the computational complexity and the parameter amount seriously limits the upper limit of the model, and the tracking performance and real-time performance are difficult to balance. Therefore, many research teams introduce knowledge distillation into the RGBT target tracking method to balance the tracking accuracy and efficiency. For example, SDSTrack adopts a self-distillation strategy to perform knowledge migration within a single model, thereby improving the feature expression ability of the model, i.e., using high-level semantic information to guide low-level feature learning, so that the model maintains consistency in different levels of feature space. The self-distillation mechanism can effectively compress the redundant information of the model, so that the single-flow tracking can more efficiently process cross-modal data without the need for an additional teacher model. Yang et al. use an online distillation strategy during the training process, using a double-flow model to guide a single-flow model that fuses RGBT features, to realize information compression and optimization. The core idea is to use mutual prompt learning to enhance the feature interaction of the double-flow model, and to migrate the knowledge of the double-flow model to a lightweight single-flow model during the knowledge distillation stage, thereby achieving the effect of reducing the computational cost.

[0005] However, SDSTrack only uses self-distillation technology to align the features within the model, and does not directly use the complementary information between the two modalities. The feature difference between the modalities still exists, and the tracking robustness has not actually been significantly improved. Although using a double-flow model to guide a single-flow model can achieve optimization to a certain extent, the knowledge migration is still one-way, and the modal difference has not been truly broken. The model is still difficult to achieve a good balance between tracking performance and real-time performance. SUMMARY

[0006] Therefore, the embodiments of the present application propose a fast RGBT target tracking method based on cross-modal interaction enhancement and knowledge distillation, which adopts a cross-modal knowledge distillation scheme to reduce the parameter amount of the model while ensuring that the student model can extract the key features of the teacher model, realize effective fusion of cross-modal information, reduce the feature difference between the modalities, and achieve a good balance between tracking performance and real-time performance.

[0007] To achieve the above object, the embodiment of the present application proposes a fast RGBT target tracking method based on cross-modal interaction enhancement and knowledge distillation, which is realized based on an RGBT target tracking model and includes the following steps: based on the template image, search region and corresponding real label of the visible light modality and thermal infrared modality, a training sample set is constructed; an RGBT target tracking model composed of a visible light feature extraction network, a thermal infrared feature extraction network, a cross-modal interaction enhancement network and a multi-modal tracking head is constructed; the cross-modal interaction enhancement network is composed of a cross-modal semantic double enhancement module and a cross-modal spatial double enhancement module; a teacher model including a visible light teacher model and a thermal infrared teacher model is selected, the RGBT target tracking model is taken as a student model, the visible light feature extraction network is taken as a visible light student model branch, and the thermal infrared feature extraction network is taken as a thermal infrared student model branch; the training samples in the training sample set are input into the student model, the visible light teacher model and the thermal infrared teacher model, inter-modal distillation is performed based on the visible light student model branch and the thermal infrared student model branch, intra-modal distillation is performed based on the visible light student model branch and the visible light teacher model and the thermal infrared student model branch and the thermal infrared teacher model, and then the overall loss function is constructed in combination with the output of the real label and the multi-modal tracking head, so that the student model, the visible light teacher model and the thermal infrared teacher model are jointly trained based on the overall loss function until the student model converges, and the trained RGBT target tracking model is obtained; the trained RGBT target tracking model is deployed to realize real-time RGBT target tracking.

[0008] To achieve the above object, the embodiment of the present application also proposes a fast RGBT target tracking system based on cross-modal interaction enhancement and knowledge distillation, which is realized based on an RGBT target tracking model and includes: a sample set construction module, configured to construct a training sample set based on template images, search regions and corresponding real labels of a visible light modality and a thermal infrared modality; an RGBT target tracking model construction module, configured to construct an RGBT target tracking model composed of a visible light feature extraction network, a thermal infrared feature extraction network, a cross-modal interaction enhancement network and a multi-modal tracking head, the cross-modal interaction enhancement network being composed of a cross-modal semantic double enhancement module and a cross-modal spatial double enhancement module; a distillation preparation module, configured to select teacher models including a visible light teacher model and a thermal infrared teacher model, take the RGBT target tracking model as a student model, take the visible light feature extraction network as a visible light student model branch and take the thermal infrared feature extraction network as a thermal infrared student model branch; a training module, configured to input training samples in the training sample set to the student model, the visible light teacher model and the thermal infrared teacher model, perform inter-modal distillation based on the visible light student model branch and the thermal infrared student model branch, perform intra-modal distillation based on the visible light student model branch and the visible light teacher model and the thermal infrared student model branch and the thermal infrared teacher model, and then combine the real labels and the output of the multi-modal tracking head to construct an overall loss function, so as to perform common training on the student model, the visible light teacher model and the thermal infrared teacher model based on the overall loss function until the student model converges, thereby obtaining a trained RGBT target tracking model; and a deployment and use module, configured to deploy the trained RGBT target tracking model to realize real-time RGBT target tracking.

[0009] To achieve the above object, the embodiment of the present application also proposes an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform a fast RGBT target tracking method based on cross-modal interaction enhancement and knowledge distillation as described in the first aspect above.

[0010] To achieve the above object, the embodiment of the present application proposes a computer readable storage medium storing a computer program, which can be executed by a processor to implement a fast RGBT target tracking method based on cross-modal interaction enhancement and knowledge distillation as described in the first aspect above.

[0011] Optionally, the inter-modal distillation based on the visible light student model branch and the thermal infrared student model branch includes:

[0012] The visible light intermediate feature in the visible light student model branch is obtained and thermal infrared intermediate features in the thermal infrared student model branch Among them, l represents the feature of the lth layer, and n represents the nth training sample;

[0013] Through the following formulas, based on and Calculate the visible light feature mean, visible light feature standard deviation, thermal infrared feature mean, and thermal infrared feature standard deviation:

[0014]

[0015] Where N is the total number of training samples, is the characteristic mean of visible light, is the standard deviation of visible light characteristics, is the thermal infrared characteristic mean, is the standard deviation of thermal infrared characteristics;

[0016] based on and Quantify the mean squared error between the features of all layers of the visible light student model branch and the thermal infrared student model branch to calculate the inter-modal distillation loss;

[0017] The inter-modal distillation loss is expressed as:

[0018]

[0019] Where L represents the total number of layers of visible light student model branches and thermal infrared student model branches, L intra represents the inter-modal distillation loss.

[0020] Optionally, intra-modal distillation is performed based on the visible light student model branch and the visible light teacher model, the thermal infrared student model branch and the thermal infrared teacher model, including:

[0021] Get the visible light student token feature f in the visible light student model branch i rgb , the visible light teacher token feature F in the visible light teacher model i rgb , thermal infrared student token feature f in the thermal infrared student model branch i tir and thermal infrared teacher token features in the thermal infrared teacher model Where i represents the i-th token;

[0022] Based on f i rgb Calculate the covariance matrix of the visible light Student model branch Based on F irgb Computing the visible light teacher model covariance matrix

[0023] and is expressed by formula as:

[0024]

[0025] where I represents the total number of tokens, represents the visible light student token mean, represents the visible light teacher token mean, and T in the top right corner represents taking the transpose;

[0026] Based on feature modeling is performed on the visible light student model branch to obtain the high-strength feature distribution of the visible light student model branch and based on feature modeling is performed on the visible light teacher model to obtain the high-strength feature distribution of the visible light teacher model

[0027] and is expressed by formula as:

[0028]

[0029] Based on the square of the Wasserstein distance between and a visible light adjustment factor λ is introduced rgb to obtain the Wasserstein distance distillation loss as the visible light intra-modal distillation loss;

[0030] The visible light intra-modal distillation loss is expressed by formula as:

[0031]

[0032] where W(·) represents taking the Wasserstein distance, Tr(·) represents taking the trace of a matrix, and ‖·‖ 2 represents taking the Euclidean distance, represents the visible light intra-modal distillation loss;

[0033] Based on f i tir Computing the thermal infrared student model branch covariance matrix and based on computing the thermal infrared teacher model covariance matrix

[0034] and is expressed by formula as:

[0035]

[0036] in, represents the mean value of thermal infrared student tokens, represents the mean value of thermal infrared teacher token;

[0037] based on Perform feature modeling on the thermal infrared student model branch to obtain the Gaussian feature distribution of the thermal infrared student model branch Based on Perform feature modeling on the thermal infrared teacher model to obtain the Gaussian feature distribution of the thermal infrared teacher model

[0038] and It is expressed by the formula:

[0039]

[0040] based on and The square of the Wasserstein distance between the two is used to introduce the thermal infrared adjustment factor λ tir , we get the Wasserstein distance distillation loss as the distillation loss within the thermal infrared modality;

[0041] The distillation loss in the thermal infrared mode is expressed by the formula:

[0042]

[0043] in, represents the distillation loss within the thermal infrared mode;

[0044] Will and Add together to get the total intra-modal distillation loss L intra .

[0045] Optionally, the cross-modal semantic dual enhancement module is provided with a visible light channel weight calculation unit, a thermal infrared channel weight calculation unit and a cross-modal weight calculation unit, and the input of the cross-modal semantic dual enhancement module is the visible light information feature F output by the visible light feature extraction network. rgb0 And the thermal infrared information feature F output by the thermal infrared feature extraction network tir0 ;

[0046] Cross-modal semantic dual enhancement module for F rgb0 and F tir0 The processing includes:

[0047] Along the channel dimension, F rgb0 and F tir0The spliced features are restored to the original size through a convolution operation, redundant information is filtered, and thus the fusion features F of the two modalities are obtained mix0 ;

[0048] The cross-modal weight calculation unit is used to calculate the cross-modal semantic weight W of F mix0 mix0 , W mix0 is expressed by the formula as follows:

[0049] W mix0 =CMWE(F mix0 );

[0050] wherein, CMWE(·) represents the cross-modal weight calculation unit;

[0051] The visible light channel weight calculation unit is used to calculate the visible light channel weight W of F rgb0 rgb0 , the thermal infrared channel weight calculation unit is used to calculate the thermal infrared channel weight W of F tir0 tir0 , W rgb0 and W tir0 are expressed by the formula as follows:

[0052] W rgb0 =CWE rgb (F rgb0 );

[0053] W tir0 =CWE tir (F tir0 );

[0054] wherein, CWE rgb (·) represents the visible light channel weight calculation unit, and CWE tir (·) represents the thermal infrared channel weight calculation unit;

[0055] Based on W rgb0 , W tir0 and W mix0 , the secondary enhancement is performed on F rgb0 and F tir0 respectively, and thus the visible light enhanced feature F rgb1 and the thermal infrared enhanced feature F tir1 are obtained, F rgb1 and F tir1 are expressed by the formula as follows:

[0056] F rgb1 =F rgb0 ⊙W ENrgb0 ; ​​​

[0057] F tir1 =F tir0 ⊙W ENtir0 ;

[0058]

[0059] Among them, softmax(·) represents the softmax function, represents pixel-by-pixel multiplication, and ⊙ represents the Hadamard product.

[0060] Optionally, the input of the cross-modal spatial dual enhancement module is F rgb1 and F tir1 ;

[0061] Cross-modal space dual enhancement module for F rgb1 and F tir1 The processing includes:

[0062] Use convolution to F rgb1 and F tir1 Perform shape transformation and then multiply pixel by pixel to obtain the fusion feature F of the two modalities mix1 , F mix1 It is expressed by the formula:

[0063]

[0064] Among them, conv(·) represents convolution;

[0065] Use two convolutions of different scales to filter F from different scales rgb1 and F tir1 Extract pixel weights and then splice them in the channel dimension to obtain the visible light pixel weight W rgb1 and thermal infrared pixel weight W tir1 , W rgb1 and W tir1 It is expressed by the formula:

[0066] W rgb1 =concat[conv1(F rgb1 ),conv2(F rgb1 )];

[0067] W tir1 =concat[conv1(F tir1 ),conv2(F tir1 )];

[0068] Among them, conv1(·) and conv2(·) represent two convolutions of different scales, and concat(·) represents splicing in the channel dimension;

[0069] Based on Qrgb1 , Q tir1 and F mix1 , F rgb1 and F tir1 is enhanced twice to obtain visible light final feature F rgb2 and thermal infrared final feature F tir2 , F rgb2 and F tir2 are expressed by formulas as follows:

[0070] F rgb2 = F rgb1 ⊙W ENrgb1 ;

[0071] F tir2 = F tir1 ⊙W ENtir1 ;

[0072]

[0073] Optionally, the overall loss function of the student model is expressed by a formula as follows:

[0074]

[0075] wherein L Focal represents the adaptive weight Focal classification loss, L GIoU represents the IoU loss, L L1 represents the L1 loss, ξ intra and ξ inter are loss coefficients of L intra and L inter , respectively, represents the overall loss function of the student model.

[0076] Optionally, the overall loss function of the visible light teacher model is expressed by a formula as follows:

[0077]

[0078] wherein, represents the overall loss function of the visible light teacher model;

[0079] The overall loss function of the thermal infrared teacher model is expressed by a formula as follows:

[0080]

[0081] wherein, represents the overall loss function of the visible light teacher model.

[0082] The fast RGBT target tracking method based on cross-modal interaction enhancement and knowledge distillation proposed in this application has at least the following advantages compared with traditional knowledge distillation methods.

[0083] First, this application constructs, trains, and uses an RGBT target tracking model to achieve real-time RGBT target tracking. The RGBT target tracking model is specifically composed of a visible light feature extraction network, a thermal infrared feature extraction network, a cross-modal interaction enhancement network, and a multimodal tracking head. The cross-modal interaction enhancement network is equipped with a cross-modal semantic dual enhancement module and a cross-modal spatial dual enhancement module, which can deeply fuse the features output by the visible light feature extraction network and the thermal infrared feature extraction network from the semantic and spatial dimensions respectively, reducing the difference between the modalities, effectively improving the discriminability of the features, and thus improving the accuracy of RGBT target tracking.

[0084] Second, this application uses the RGBT target tracking model as the student model, the visible light feature extraction network as the visible light student model branch, and the thermal infrared feature extraction network as the thermal infrared student model branch. At the same time, the visible light teacher model and the thermal infrared teacher model are introduced, so that inter-modal distillation can be performed based on the visible light student model branch and the thermal infrared student model branch, and intra-modal distillation can be performed based on the visible light student model branch and the visible light teacher model, and the thermal infrared student model branch and the thermal infrared teacher model. Inter-modal distillation can minimize the modal gap between the two modalities, and intra-modal distillation can allow the student model to quickly inherit the core capabilities of the teacher model, reduce the complexity and difficulty of training, and reduce the time of model training.

[0085] In summary, the cross-modal knowledge distillation scheme proposed in this application can reduce the number of model parameters while ensuring that the student model can extract the key features of the teacher model, realize the effective fusion of cross-modal information, reduce the feature differences between modalities, and achieve a benign balance between tracking performance and real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the related technologies, the following is a brief introduction to the drawings required for use in the embodiments of the present application or the description of the related technologies. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0087] Figure 1 This is a flowchart of a fast RGBT target tracking method based on cross-modal interaction enhancement and knowledge distillation provided in one embodiment of the present application;

[0088] Figure 2is a structural schematic diagram of an RGBT target tracking model provided in an embodiment of the present application;

[0089] Figure 3 is a structural schematic diagram of a cross-modal interaction enhanced network provided in an embodiment of the present application;

[0090] Figure 4 is a structural schematic diagram of a fast RGBT target tracking system based on cross-modal interaction enhancement and knowledge distillation provided in another embodiment of the present application;

[0091] Figure 5 is a structural schematic diagram of an electronic device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0092] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below with reference to the drawings. In the various embodiments of the present application, many technical details are proposed in order to make the reader better understand the present application. However, the technical solutions claimed by the present application can be implemented even without these technical details and various changes and modifications based on the various embodiments below. The division of the various embodiments below is only for the convenience of description and should not constitute any limitation on the specific implementation of the present application. The various embodiments can be combined and referenced to each other without contradiction.

[0093] An embodiment of the present application proposes a fast RGBT target tracking method based on cross-modal interaction enhancement and knowledge distillation, which is realized based on an RGBT target tracking model and applied to an electronic device. The electronic device can be a terminal or a server. In the embodiments of the present application and the various embodiments below, the electronic device is taken as an example of a server. The implementation details of the fast RGBT target tracking method based on cross-modal interaction enhancement and knowledge distillation proposed in the embodiment will be described in detail below. The following content only provides related implementation details for the convenience of understanding and is not necessary for implementing the present solution.

[0094] The specific process of the fast RGBT target tracking method based on cross-modal interaction enhancement and knowledge distillation proposed in the embodiment can be as shown in Figure 1 , which includes:

[0095] Step 11, based on the template image, search region and corresponding real label of the visible light modality and thermal infrared modality, a training sample set is constructed.

[0096] In a specific implementation, the training sample set is a basis for model training. The server needs to construct the training sample set based on the template images of the visible light modality and the thermal infrared modality, the search region, and the corresponding real labels. The training samples exist in the form of groups. One group of training samples includes the template images and the search region of the visible light modality, the template images and the search region of the thermal infrared modality, and the corresponding real labels.

[0097] Step 12, constructing an RGBT target tracking model composed of a visible light feature extraction network, a thermal infrared feature extraction network, a cross-modal interactive enhancement network, and a multi-modal tracking head. The cross-modal interactive enhancement network is composed of a cross-modal semantic dual enhancement module and a cross-modal spatial dual enhancement module.

[0098] In a specific implementation, after the server constructs the training sample set, the RGBT target tracking model can be constructed. The structure of the constructed RGBT target tracking model can be as shown in Figure 2 , and specifically composed of a visible light feature extraction network, a thermal infrared feature extraction network, a cross-modal interactive enhancement network, and a multi-modal tracking head. The structure of the cross-modal interactive enhancement network can be as shown in Figure 3 , and specifically composed of a cross-modal semantic dual enhancement module and a cross-modal spatial dual enhancement module.

[0099] It should be noted that the input of the RGBT target tracking model is the visible light image and the thermal infrared image. The visible light feature extraction network extracts features from the visible light image, the thermal infrared feature extraction network extracts features from the thermal infrared image, the cross-modal interactive enhancement network performs cross-modal interactive enhancement on the outputs of the visible light feature extraction network and the thermal infrared feature extraction network, and the multi-modal tracking head is used for multi-modal tracking on the output of the cross-modal interactive enhancement network, thereby realizing RGBT target tracking.

[0100] One of the purposes of setting the cross-modal interactive enhancement network is to improve the robustness of the RGBT target tracking model through the fusion of features between modalities. However, the traditional feature fusion strategy may cause conflicts in the fusion results, such as the destruction of the rich texture of the visible light image by the textureless patches of the thermal infrared image, which has a certain negative impact on tracking accuracy. To deal with this situation, the embodiment sets two-dimensional enhancement mechanisms of cross-modal semantic dual enhancement (CSEDE) and cross-modal spatial dual enhancement (CSPDE) in the cross-modal interactive enhancement network. This design makes the fusion of feature information between the two modalities more accurate.

[0101] The cross-modal semantic dual enhancement mechanism is implemented by the cross-modal semantic dual enhancement module. In this module, dual enhancement refers to the enhancement of the bimodal fusion results between channels based on unimodal information, and the further enhancement of the unimodal through the complementary information of bimodal fusion. Therefore, CSEDE can assign weights according to the importance of each channel to emphasize semantic information. The first enhancement is to use unimodal features to improve the bimodal fusion results of visible light modality and infrared modality features, but these features may contain negative information that inhibits each other between modalities. Therefore, in the second enhancement, each feature of each modality is controlled to make full use of the benign features of the other modality to achieve improvement.

[0102] The cross-modal semantic dual enhancement module is equipped with a visible light channel weight calculation unit, a thermal infrared channel weight calculation unit, and a cross-modal weight calculation unit. These weight calculation units are responsible for different weight calculations. The input of the cross-modal semantic dual enhancement module is the visible light information feature F output by the visible light feature extraction network. rgb0 And the thermal infrared information feature F output by the thermal infrared feature extraction network tir0 .

[0103] In order to obtain the comprehensive information of the two modal features, the cross-modal semantic dual enhancement module first transforms the F rgb0 and F tir0 The spliced ​​features are then restored to their original size through convolution operations, while redundant information is filtered out to obtain the fusion feature F of the two modalities. mix0 .

[0104] F mix0 It is expressed by the formula:

[0105] F mix0 =conv[concat(F rgb0 ,F tir0 )];

[0106] Among them, conv(·) represents convolution and concat(·) represents concatenation in the channel dimension.

[0107] Next, the cross-modal weight calculation unit F mix0 Perform cross-modal semantic weight calculation to obtain the fusion semantic weight W mix0 , W mix0 It is expressed by the formula:

[0108] W mix0 =CMWE(F mix0 );

[0109] Where CMWE(·) represents the cross-modal weight calculation unit.

[0110] After that, the cross-modal semantic dual enhancement module needs to obtain the discriminative features of each modality. Inspired by the idea that SENet can improve the quality of feature representation by modeling the dependencies between the channels of its convolutional features, the cross-modal semantic dual enhancement module uses the visible light channel weight calculation unit to calculate the F rgb0 Calculate the visible light channel weight and obtain the visible light channel weight W rgb0 , using the thermal infrared channel weight calculation unit to calculate F tir0 Calculate the thermal infrared channel weight and obtain the thermal infrared channel weight W tir0 .

[0111] W rgb0 and W tir0 It is expressed by the formula:

[0112] W rgb0 =CWE rgb (F rgb0 );

[0113] W tir0 =CWE tir (F tir0 );

[0114] Among them, CWE rgb (·) represents the visible light channel weight calculation unit, CWE tir (·) represents the thermal infrared channel weight calculation unit;

[0115] Finally, the cross-modal semantic dual enhancement module is based on W rgb0 、W tir0 and W mix0 , respectively for F rgb0 and F tir0 Perform secondary enhancement to obtain the visible light enhancement feature F rgb1 and thermal infrared enhanced features F tir1 , F rgb1 and F tir1 It is expressed by the formula:

[0116] F rgb1 =F rgb0 ⊙W ENrgb0 ;

[0117] F tir1 =F tir0 ⊙W ENtir0 ;

[0118]

[0119]

[0120] wherein softmax(·) represents a softmax function, represents pixel-wise multiplication, and represents Hadamard product.

[0121] Similar to the cross-modal semantic dual enhancement module, the cross-modal spatial dual enhancement module also adopts a dual enhancement mechanism. CSPDE re-encodes the features in the spatial dimension, emphasizing important pixel positions while minimizing the importance of irrelevant pixel positions.

[0122] The input of the cross-modal spatial dual enhancement module is F rgb1 and F tir1 . In order to obtain the fusion features containing global information, the cross-modal spatial dual enhancement module first uses convolution to convert the shapes of F rgb1 and F tir1 , and then performs pixel-wise multiplication to obtain the fusion features F mix1 of the two modalities.

[0123] F mix1 is expressed by the formula as follows:

[0124]

[0125] wherein conv(·) represents convolution.

[0126] Next, in order to fully obtain the features specific to each modality in the spatial dimension, the cross-modal spatial dual enhancement module selects to use convolutions of different scales to learn the differences in spatial information, that is, two convolutions of different scales are used to extract pixel weights from F rgb1 and F tir1 in different scales, respectively, and then the channel dimension is spliced to obtain the visible light pixel weight W rgb1 and the thermal infrared pixel weight W tir1 .

[0127] W rgb1 and W tir1 are expressed by the formula as follows:

[0128] W rgb1 = concat[vonv1(F rgb1 ),conv2(F rgb1 )];

[0129] W tir1 = concat[conv1(F tir1 ),conv2(F tir1 )];

[0130] wherein conv1(·) and conv2(·) represent two convolutions of different scales, and concat(·) represents splicing in the channel dimension.

[0131] Finally, the cross-modal spatial double enhancement module reduces the channel number by half to compress the features, and based on W rgb1 , W tir1 and F mix1 , the second enhancement is performed on F rgb1 and F tir1 to obtain the final visible light feature F rgb2 and the final thermal infrared feature F tir2 .

[0132] F rgb2 and F tir2 are expressed by the formula as follows:

[0133] F rgb2 = F rgb1 ⊙W ENrgb1 ;

[0134] F tir2 = F tir1 ⊙W ENtir1 ;

[0135]

[0136] Step 13, selecting the teacher model including the visible light teacher model and the thermal infrared teacher model, taking the RGBT target tracking model as the student model, taking the visible light feature extraction network as the visible light student model branch, and taking the thermal infrared feature extraction network as the thermal infrared student model branch.

[0137] In a specific implementation, after the construction of the training sample set and the RGBT target tracking model is completed, the model training phase can be entered, the server selects the visible light teacher model and the thermal infrared teacher model, takes the RGBT target tracking model as the student model, that is, takes the visible light feature extraction network as the visible light student model branch, and takes the thermal infrared feature extraction network as the thermal infrared student model branch, so as to complete the preparation of knowledge distillation.

[0138] It should be noted that the visible light teacher model and the thermal infrared teacher model are respectively provided with a single-modal tracking head to realize target tracking.

[0139] Step 14, input the training samples in the training sample set into the student model, the visible light teacher model and the thermal infrared teacher model, perform inter-modal distillation based on the visible light student model branch and the thermal infrared student model branch, perform intra-modal distillation based on the visible light student model branch and the visible light teacher model, the thermal infrared student model branch and the thermal infrared teacher model, combine the real labels and the outputs of the multi-modal tracking head to construct an overall loss function, and train the student model, the visible light teacher model and the thermal infrared teacher model based on the overall loss function until the student model converges, thereby obtaining the trained RGBT target tracking model.

[0140] In a specific implementation, the server needs to input different groups of training samples in the training sample set into the student model, the visible light teacher model and the thermal infrared teacher model, perform inter-modal distillation based on the visible light student model branch and the thermal infrared student model branch, perform intra-modal distillation based on the visible light student model branch and the visible light teacher model, the thermal infrared student model branch and the thermal infrared teacher model, combine the real labels and the outputs of the multi-modal tracking head to construct an overall loss function, and train the student model, the visible light teacher model and the thermal infrared teacher model based on the overall loss function until the student model converges, thereby obtaining the trained RGBT target tracking model.

[0141] In one example, the purpose of inter-modal distillation is to distill between the student model branches of the two modalities, so as to minimize the modal gap between the two modalities.

[0142] When performing inter-modal distillation, first, the visible light intermediate features in the visible light student model branch and the thermal infrared intermediate features in the thermal infrared student model branch are obtained.

[0143] Next, the visible light feature mean, the visible light feature standard deviation, the thermal infrared feature mean and the thermal infrared feature standard deviation are calculated based on the following formulas:

[0144]

[0145] wherein N is the total number of training samples, is the visible light feature mean, is the visible light feature standard deviation, is the thermal infrared feature mean, and is the thermal infrared feature standard deviation.

[0146] Finally, the visible light feature mean and the thermal infrared feature mean are calculated based on the following formulas: and​​ The mean squared error between the features of all layers of the visible light student model branch and the thermal infrared student model branch is quantified to calculate the inter-modal distillation loss, which can be expressed as:

[0147]

[0148] Where L represents the total number of layers of visible light student model branches and thermal infrared student model branches, L intra represents the inter-modal distillation loss.

[0149] In one example, although the visible light features and thermal infrared features of the two student model branches can eliminate the modal gap through inter-modal distillation, this process may damage the feature content representation of the modalities themselves. To address this situation, intra-modal distillation is required between the teacher model and student model branches of the two modalities in the training room. The classic instance normalization operation is used to normalize the features of the teacher model and student model to obtain intra-modal features that are orthogonal to the inter-modal features. Traditional methods typically use L2 loss or KL divergence loss to calculate the similarity between the teacher model and the student model. However, L2 loss only calculates the point-to-point distance and easily ignores the overall distribution information. KL divergence loss requires that the probability distributions of the two model features cannot be completely staggered, otherwise it may not be effectively calculated. Therefore, inspired by the WKD proposed by Lv et al., this embodiment performs intra-modal distillation by calculating the Wasserstein distance of the intra-modal features of the two sets of teacher models and student models to compare the similarity between the features. Since the model features are high-dimensional and small-sized, this embodiment uses Gaussian distribution modeling for the token features of the intermediate layer.

[0150] The first step of intra-modal distillation is to obtain the visible student token features f in the visible student model branch i rgb , the visible light teacher token feature F in the visible light teacher model i rgb , thermal infrared student token feature f in the thermal infrared student model branch i tir and thermal infrared teacher token features in the thermal infrared teacher model Where i represents the i-th token.

[0151] Next, intra-modal distillation of the visible light mode is performed.

[0152] First, based on f i rgb Calculate the covariance matrix of the visible light Student model branch Based on F i rgb Calculate the covariance matrix of the visible light teacher model

[0153] and is expressed by formula as:

[0154]

[0155] where I denotes the total number of tokens, denotes the mean of the visible light student tokens, denotes the mean of the visible light teacher tokens, and T in the top right corner denotes taking the transpose.

[0156] Subsequently, the feature of the visible light student model branch is modeled based on to obtain the high-strength feature distribution of the visible light student model branch and the feature of the visible light teacher model is modeled based on to obtain the high-strength feature distribution of the visible light teacher model

[0157] and is expressed by formula as:

[0158]

[0159] Finally, based on the square of the Wasserstein distance between and , a visible light adjustment factor λ is introduced rgb to obtain the Wasserstein distance distillation loss as the intra-modal distillation loss of the visible light modal.

[0160] The intra-modal distillation loss of the visible light modal is expressed by formula as:

[0161]

[0162] where W(·) denotes taking the Wasserstein distance, Tr(·) denotes taking the trace of a matrix, and ‖·‖ 2 denotes taking the Euclidean distance, denotes the intra-modal distillation loss of the visible light modal.

[0163] To be followed is the intra-modal distillation of the thermal infrared modal.

[0164] First, based on f i tir the covariance matrix of the thermal infrared student model branch is calculated and based on the thermal infrared teacher model covariance matrix is calculated

[0165] and is expressed by formula as:

[0166]

[0167] wherein, denotes the thermal infrared student token mean, denotes the thermal infrared teacher token mean;

[0168] based on feature modeling the thermal infrared student model branch, obtaining the high-strength feature distribution of the thermal infrared student model branch and based on feature modeling the thermal infrared teacher model, obtaining the high-strength feature distribution of the thermal infrared teacher model

[0169] and is expressed by the formula:

[0170]

[0171] Finally, based on the square of the Wasserstein distance between and , a thermal infrared adjustment factor λ is introduced tir , to obtain the Wasserstein distance distillation loss as the thermal infrared intra-modal distillation loss.

[0172] The thermal infrared intra-modal distillation loss is expressed by the formula:

[0173]

[0174] wherein, denotes the thermal infrared intra-modal distillation loss.

[0175] It can be understood that adding and , the total intra-modal distillation loss L intra can be obtained.

[0176] In one example, the overall loss function of the student model is expressed by the formula:

[0177]

[0178] wherein, L Focal denotes the adaptive weight Focal classification loss, L GIoU denotes the intersection over union loss, L L1 denotes the L1 loss, ξ intra and ξ inter are the loss coefficients of L intra and L inter , respectively. a total loss function representing the student model.

[0179] In one example, the total loss function of the visible light teacher model is represented by the formula:

[0180]

[0181] wherein, the total loss function of the visible light teacher model.

[0182] In one example, the total loss function of the thermal infrared teacher model is represented by the formula:

[0183]

[0184] wherein, the total loss function of the visible light teacher model.

[0185] Step 15, deploy the trained RGBT target tracking model to realize real-time RGBT target tracking.

[0186] In a specific implementation, after obtaining the trained RGBT target tracking model, the server can deploy the trained RGBT target tracking model to the scene as needed, thereby realizing real-time RGBT target tracking based on the input visible light information and thermal infrared information.

[0187] The fast RGBT target tracking method based on cross-modal interaction enhancement and knowledge distillation proposed in this embodiment has at least the following advantages compared with traditional knowledge distillation methods.

[0188] First, the RGBT target tracking model is designed, constructed, trained and used to realize real-time RGBT target tracking. The RGBT target tracking model specifically consists of a visible light feature extraction network, a thermal infrared feature extraction network, a cross-modal interaction enhancement network and a multi-modal tracking head. The cross-modal interaction enhancement network is provided with a cross-modal semantic double enhancement module and a cross-modal spatial double enhancement module, which can respectively perform deep fusion on the features output by the visible light feature extraction network and the thermal infrared feature extraction network from the semantic and spatial dimensions, thereby effectively reducing the difference between the modalities, effectively improving the discriminability of the features, and further improving the accuracy of the RGBT target tracking.

[0189] Secondly, the RGBT target tracking model is taken as the student model, the visible light feature extraction network is taken as the visible light student model branch, and the thermal infrared feature extraction network is taken as the thermal infrared student model branch. The visible light teacher model and the thermal infrared teacher model are introduced, so that inter-modal distillation can be performed based on the visible light student model branch and the thermal infrared student model branch, and intra-modal distillation can be performed based on the visible light student model branch and the visible light teacher model and the thermal infrared student model branch and the thermal infrared teacher model. The inter-modal distillation can make the modal gap between the two modalities as small as possible, and the intra-modal distillation can make the student model quickly inherit the core ability of the teacher model, reduce the training complexity and difficulty, and reduce the model training time.

[0190] In summary, the cross-modal knowledge distillation scheme proposed in the embodiment can reduce the parameter amount of the model while ensuring that the student model can extract the key features of the teacher model, successfully realizing effective fusion of cross-modal information, reducing the feature difference between modalities, and achieving a balance between tracking performance and real-time performance.

[0191] The step division of the above methods is only for the purpose of clear description, and in implementation, one step can be combined or some steps can be divided into multiple steps, as long as the same logical relationship is included, which is within the protection scope of the present application. Adding insignificant modifications or introducing insignificant designs in the algorithm or process, but not changing the core design of the algorithm and process, are within the protection scope of the present application.

[0192] Another embodiment of the present application proposes a fast RGBT target tracking system based on cross-modal interaction enhancement and knowledge distillation, which is realized based on an RGBT target tracking model. The implementation details of the fast RGBT target tracking system based on cross-modal interaction enhancement and knowledge distillation proposed in the embodiment are described below. The following content is only the implementation details provided for the convenience of understanding, and is not necessary for implementing the embodiment.

[0193] Figure 4 is a structural schematic diagram of the fast RGBT target tracking system based on cross-modal interaction enhancement and knowledge distillation proposed in the embodiment. The system comprises a sample set construction module 21, an RGBT target tracking model construction module 22, a distillation preparation module 23, a training module 24, and a deployment and use module 25.

[0194] The sample set construction module 21 is configured to construct a training sample set based on the template images, the search regions, and the corresponding real labels of the visible light modalities and the thermal infrared modalities.

[0195] The RGBT target tracking model construction module 22 is configured to construct an RGBT target tracking model composed of a visible light feature extraction network, a thermal infrared feature extraction network, a cross-modal interaction enhancement network, and a multi-modal tracking head, wherein the cross-modal interaction enhancement network is composed of a cross-modal semantic dual enhancement module and a cross-modal spatial dual enhancement module.

[0196] The distillation preparation module 23 is configured to select teacher models including a visible light teacher model and a thermal infrared teacher model, take the RGBT target tracking model as a student model, take the visible light feature extraction network as a visible light student model branch, and take the thermal infrared feature extraction network as a thermal infrared student model branch.

[0197] The training module 24 is configured to input training samples in a training sample set to the student model, the visible light teacher model, and the thermal infrared teacher model, perform inter-modal distillation based on the visible light student model branch and the thermal infrared student model branch, perform intra-modal distillation based on the visible light student model branch and the visible light teacher model and the thermal infrared student model branch and the thermal infrared teacher model, combine a real label and an output of the multi-modal tracking head to construct an overall loss function, and perform joint training on the student model, the visible light teacher model, and the thermal infrared teacher model based on the overall loss function until the student model converges, thereby obtaining a trained RGBT target tracking model.

[0198] The deployment and use module 25 is configured to deploy the trained RGBT target tracking model to realize real-time RGBT target tracking.

[0199] It can be found that the present embodiment is a system embodiment corresponding to the above-mentioned method embodiment, and the present embodiment can be implemented in cooperation with the above-mentioned method embodiment. The related technical details and technical effects mentioned in the above-mentioned method embodiment are still valid in the present embodiment. In order to reduce repetition, they will not be described here again. Correspondingly, the related technical details mentioned in the present embodiment can also be applied to the above-mentioned method embodiment.

[0200] It is worth mentioning that each module and component involved in the present embodiment is a logical module. In actual application, one logical module can be one physical unit, or a part of one physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of the present application, units not closely related to solving the technical problems proposed in the present application are not introduced in the present embodiment, but this does not mean that there are no other units in the present embodiment.

[0201] Another embodiment of the present application proposes an electronic device, such as Figure 5As shown, the device comprises: at least one processor 31; and a memory 32 connected with the at least one processor 31 in communication; wherein the memory 32 stores instructions executable by the at least one processor 31, and the instructions are executed by the at least one processor 31 to enable the at least one processor 31 to perform a fast RGBT target tracking method based on cross-modal interaction enhancement and knowledge distillation as described in the above method embodiments.

[0202] The bus can be connected between the memory and the processor in a bus manner, and the bus includes any number of interconnected buses and bridges, and the bus connects various circuits of the one or more processors and the memory together. The bus can also connect various other circuits such as peripheral devices, voltage stabilizers, and power management circuits together, which are well known in the art, and thus further description thereof will not be given herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements such as multiple receivers and transmitters, and provides a unit for communicating with various other devices on a transmission medium. The data processed by the processor is transmitted on a wireless medium through the antenna, and further, the antenna also receives data and transmits the data to the processor.

[0203] The processor is responsible for managing the bus and general processing, and can also provide various functions including timing, peripheral interface, voltage regulation, power management, and other control functions. The memory can be used to store data used by the processor in performing operations.

[0204] Another embodiment of the present application provides a computer readable storage medium, wherein a computer program is stored in the computer readable storage medium, and the computer program is executable by a processor to implement a fast RGBT target tracking method based on cross-modal interaction enhancement and knowledge distillation as described in the above method embodiments.

[0205] That is, those skilled in the art can understand that all or part of the steps of the above method embodiments can be completed by program instruction related hardware, and the program is stored in a storage medium, and includes a plurality of instructions for causing a device (such as a single-chip microcomputer, a chip, etc.) or a processor to execute all steps or part of steps of the fast RGBT target tracking method based on cross-modal interaction enhancement and knowledge distillation as described in the method embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk, and various storage medium capable of storing program codes.

[0206] Those skilled in the art can understand that the above-mentioned embodiments are specific embodiments for realizing the present application, and various changes can be made in form and details in practical application without departing from the spirit and scope of the present application. Those skilled in the art can make several improvements and refinements without departing from the principles of the present application, and these improvements and refinements are also regarded as the protection scope of the present application.

Claims

1. A fast RGBT target tracking method based on cross-modal interaction enhancement and knowledge distillation, implemented based on the RGBT target tracking model, characterized by: The method comprises: Construct a training sample set based on template images of visible light modality and thermal infrared modality, search areas, and corresponding true labels; Construct an RGBT target tracking model consisting of a visible light feature extraction network, a thermal infrared feature extraction network, a cross-modal interaction enhancement network, and a multimodal tracking head. The cross-modal interaction enhancement network consists of a cross-modal semantic dual enhancement module and a cross-modal spatial dual enhancement module. Select teacher models including visible light teacher model and thermal infrared teacher model, use RGBT target tracking model as student model, use visible light feature extraction network as visible light student model branch, and use thermal infrared feature extraction network as thermal infrared student model branch; The training samples in the training sample set are input into the student model, visible light teacher model, and thermal infrared teacher model respectively. Inter-modal distillation is performed based on the visible light student model branch and the thermal infrared student model branch. Intra-modal distillation is performed based on the visible light student model branch and the visible light teacher model, and the thermal infrared student model branch and the thermal infrared teacher model. Then, an overall loss function is constructed based on the true label and the output of the multimodal tracking head. The student model, visible light teacher model, and thermal infrared teacher model are jointly trained based on the overall loss function until the student model converges, obtaining a trained RGBT target tracking model. Deploy the trained RGBT target tracking model to achieve real-time RGBT target tracking.

2. A fast RGBT target tracking method based on cross-modal interaction enhancement and knowledge distillation according to claim 1, characterized in that: Inter-modal distillation is performed based on the visible light student model branch and the thermal infrared student model branch, including: Obtain visible light intermediate features in the visible light student model branch and thermal infrared intermediate features in the thermal infrared student model branch Among them, l represents the feature of the lth layer, and n represents the nth training sample; Through the following formulas, based on and Calculate the visible light feature mean, visible light feature standard deviation, thermal infrared feature mean, and thermal infrared feature standard deviation: Where N is the total number of training samples, is the characteristic mean of visible light, is the standard deviation of visible light characteristics, is the thermal infrared characteristic mean, is the standard deviation of thermal infrared characteristics; based on and Quantify the mean squared error between the features of all layers of the visible light student model branch and the thermal infrared student model branch to calculate the inter-modal distillation loss; The inter-modal distillation loss is expressed as: Where L represents the total number of layers of visible light student model branches and thermal infrared student model branches, L intra represents the inter-modal distillation loss.

3. A fast RGBT target tracking method based on cross-modal interaction enhancement and knowledge distillation according to claim 2, characterized in that: Intra-modal distillation is performed based on the visible light student model branch and visible light teacher model, the thermal infrared student model branch and thermal infrared teacher model, including: Get the Visible Student token feature in the Visible Student model branch Visible light teacher token features in the visible light teacher model Thermal Infrared Student Token Features in the Thermal Infrared Student Model Branch and thermal infrared teacher token features in the thermal infrared teacher model Where i represents the i-th token; based on Calculate the covariance matrix of the visible light Student model branch Based on Calculate the covariance matrix of the visible light teacher model and It is expressed by the formula: Where I represents the total number of tokens, represents the mean value of visible light student tokens, It represents the mean of visible light teacher tokens, and T in the upper right corner means taking the transpose; based on Perform feature modeling on the visible light student model branch to obtain the Gaussian feature distribution of the visible light student model branch Based on Perform feature modeling on the visible light teacher model to obtain the Gaussian feature distribution of the visible light teacher model and It is expressed by the formula: based on and The square of the Wasserstein distance between the two, introducing the visible light adjustment factor λ rgb , we get the Wasserstein distance distillation loss as the distillation loss within the visible light modality; The distillation loss in the visible light mode is expressed by the formula: Where W(·) means to find the Wasserstein distance, Tr(·) means to find the trace of the matrix, ‖·‖ 2 Indicates the Euclidean distance. represents the distillation loss within the visible light mode; based on Calculate the covariance matrix of the thermal infrared student model branches Based on Calculate the covariance matrix of the thermal infrared teacher model and It is expressed by the formula: in, represents the mean value of thermal infrared student tokens, represents the mean value of thermal infrared teacher token; based on Perform feature modeling on the thermal infrared student model branch to obtain the Gaussian feature distribution of the thermal infrared student model branch Based on Perform feature modeling on the thermal infrared teacher model to obtain the Gaussian feature distribution of the thermal infrared teacher model and It is expressed by the formula: based on and The square of the Wasserstein distance between the two is used to introduce the thermal infrared adjustment factor λ tir , we get the Wasserstein distance distillation loss as the distillation loss within the thermal infrared modality; The distillation loss in the thermal infrared mode is expressed by the formula: in, represents the distillation loss within the thermal infrared mode; Will and Add together to get the total intra-modal distillation loss L intra .

4. A fast RGBT target tracking method based on cross-modal interaction enhancement and knowledge distillation according to claim 1, characterized in that: The cross-modal semantic dual enhancement module is equipped with a visible light channel weight calculation unit, a thermal infrared channel weight calculation unit, and a cross-modal weight calculation unit. The input of the cross-modal semantic dual enhancement module is the visible light information feature F output by the visible light feature extraction network. rgb0 And the thermal infrared information feature F output by the thermal infrared feature extraction network tir0 ; Cross-modal semantic dual enhancement module for F rgb0 and F tir0 The processing includes: Along the channel dimension, F rgb0 and F tir0 The spliced ​​features are then restored to their original size through convolution operations, while redundant information is filtered out to obtain the fusion feature F of the two modalities. mix0 ; Using cross-modal weights to calculate unit pairs F mix0 Perform cross-modal semantic weight calculation to obtain the fusion semantic weight W mix0 , W mix0 It is expressed by the formula: W mix0 =CMWE(F mix0 ); Among them, CMWE(·) represents the cross-modal weight calculation unit; Using the visible light channel weight calculation unit to calculate F rgb0 Calculate the visible light channel weight and obtain the visible light channel weight W rgb0 , using the thermal infrared channel weight calculation unit to calculate F tir0 Calculate the thermal infrared channel weight and obtain the thermal infrared channel weight W tir0 , W rgb0 and W tir0 It is expressed by the formula: W rgb0 =CWE rgb (F rgb0 ); W tir0 =CWE tir (F tir0 ); Among them, CWE rgb (·) represents the visible light channel weight calculation unit, CWE tir (·) represents the thermal infrared channel weight calculation unit; Based on W rgb0 、W tir0 and W mix0 , respectively for F rgb0 and F tir0 Perform secondary enhancement to obtain the visible light enhancement feature F rgb1 and thermal infrared enhanced features F tir1 , F rgb1 and F tir1 It is expressed by the formula: F rgb1 =F rgb0 ⊙W ENrgb0 ; F tir1 =F tir0 ⊙W ENtir0 ; Among them, softmax(·) represents the softmax function, represents pixel-by-pixel multiplication, and ⊙ represents the Hadamard product.

5. A fast RGBT target tracking method based on cross-modal interaction enhancement and knowledge distillation according to claim 4, characterized in that: The input of the cross-modal space dual enhancement module is F rgb1 and F tir1 ; Cross-modal space dual enhancement module for F rgb1 and F tir1 The processing includes: Use convolution to F rgb1 and F tir1 Perform shape transformation and then multiply pixel by pixel to obtain the fusion feature F of the two modalities mix1 , F mix1 It is expressed by the formula: Among them, conv(·) represents convolution; Use two convolutions of different scales to filter F from different scales rgb1 and F tir1 Extract pixel weights and then splice them in the channel dimension to obtain the visible light pixel weight W rgb1 and thermal infrared pixel weight W tir1 , W rgb1 and W tir1 It is expressed by the formula: W rgb1 =concat[conv1(F rgb1 ),conv2(F rgb1 )]; W tir1 =concat[concat1(F tir1 ),conv2(F tir1 )]; Among them, conv1(·) and conv2(·) represent two convolutions of different scales, and comcat(·) represents concatenation in the channel dimension; Based on W rgb1 、W tir1 and F mix1 F rgb1 and F tir1 Perform secondary enhancement to obtain the final visible light feature F rgb2 and thermal infrared final feature F tir2 , F rgb2 and F tir2 It is expressed by the formula: F rgb2 =F rgb1 ⊙W ENrgb1 ; F tir2 =F tir1 ⊙W ENtir1 ; 6. A fast RGBT target tracking method based on cross-modal interaction enhancement and knowledge distillation according to claim 3, characterized in that: The overall loss function of the student model is expressed as; Among them, L Focal represents the Focal classification loss of adaptive weights, L GIoU represents the intersection-over-union loss, L L1 represents the L1 loss, ξ intra and ξ inter L intra and L inter The loss coefficient, represents the overall loss function of the student model.

7. A fast RGBT target tracking method based on cross-modal interaction enhancement and knowledge distillation according to claim 6, characterized in that: The overall loss function of the visible light teacher model is expressed as; in, represents the overall loss function of the visible light teacher model; The overall loss function of the thermal infrared teacher model is expressed as; in, Represents the overall loss function of the visible light teacher model.

8. A fast RGBT target tracking system based on cross-modal interaction enhancement and knowledge distillation, implemented based on the RGBT target tracking model, characterized by: The system comprises: The sample set construction module is used to construct a training sample set based on template images of visible light modality and thermal infrared modality, search areas and corresponding true labels; RGBT target tracking model construction module, which is used to build an RGBT target tracking model consisting of a visible light feature extraction network, a thermal infrared feature extraction network, a cross-modal interaction enhancement network, and a multimodal tracking head. The cross-modal interaction enhancement network consists of a cross-modal semantic dual enhancement module and a cross-modal spatial dual enhancement module. A distillation preparation module is used to select teacher models including a visible light teacher model and a thermal infrared teacher model, use the RGBT target tracking model as the student model, use the visible light feature extraction network as the visible light student model branch, and use the thermal infrared feature extraction network as the thermal infrared student model branch; The training module is used to input the training samples in the training sample set into the student model, the visible light teacher model, and the thermal infrared teacher model, perform inter-modal distillation based on the visible light student model branch and the thermal infrared student model branch, and perform intra-modal distillation based on the visible light student model branch and the visible light teacher model, and the thermal infrared student model branch and the thermal infrared teacher model. Then, an overall loss function is constructed based on the true label and the output of the multimodal tracking head. The student model, the visible light teacher model, and the thermal infrared teacher model are jointly trained based on the overall loss function until the student model converges, thereby obtaining a trained RGBT target tracking model; The deployment and usage module is used to deploy the trained RGBT target tracking model to achieve real-time RGBT target tracking.

9. An electronic device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; In which, the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a fast RGBT target tracking method based on cross-modal interaction enhancement and knowledge distillation as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it can implement a fast RGBT target tracking method based on cross-modal interaction enhancement and knowledge distillation as described in any one of claims 1 to 7.

Images