RGBT target tracking method based on domain adaptation and spatio-temporal information fusion

Abstract

The invention provides an RGBT target tracking method based on domain adaptation and spatio-temporal information fusion. The problem that an existing target tracking scheme is poor in tracking effect under complex scenes such as target appearance change and shielding is mainly solved. Comprising the following steps: 1) constructing an RGBT data sample set; 2) constructing a dual-branch RGBT target tracking model based on a pre-trained RGB tracker and training the dual-branch RGBT target tracking model; 3) designing an online module updating strategy for dynamically updating the initial template image; 4) taking the initial template image and the search image as the input of the trained model, and taking the dynamic template image as the additional input of the model; and 5) loading the optimal parameter, predicting the motion state of the target in a subsequent frame by using the trained model, and obtaining and outputting a target tracking result. According to the method, the calculation cost is greatly reduced, and the time information of the target can be fully utilized, so that the performance of the feature extraction network is effectively improved, and the accuracy and robustness of target tracking are improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and further relates to target tracking technology, specifically an RGBT target tracking method based on domain adaptation and spatiotemporal information fusion, which can be used to track and locate targets in security monitoring, human-computer interaction, and autonomous driving. Background Art

[0002] Visual object tracking, as a basic task in computer vision, aims to locate the target in subsequent frames based on the initial position of the target in the first frame. In recent years, a large number of test benchmarks and excellent works have emerged, and have achieved considerable tracking performance. Due to the imaging mechanism of visible light images, RGB images have rich semantic information, but the performance of RGB trackers drops sharply under extreme lighting, rain, fog, snow and other bad weather. On the contrary, TIR images are insensitive to the above factors and have low resolution and insufficient texture information compared with RGB images. Therefore, in order to make up for the limitations of single-modal tracking methods, visible light-thermal infrared RGBT (RGB-Thremal) tracking came into being, aiming to achieve robust tracking through the mutual complementation of RGB image and TIR image information. RGBT target tracking has received more and more research attention and has broad application potential in security monitoring, human-computer interaction, and autonomous driving.

[0003] With the development of computer vision, RGBT object tracking usually adopts deep learning methods. First, the features of the initial template and search image are extracted. After the fusion module and prediction head, the target is located. Previous RGBT tracking methods (mfDiMP, APFNet, HMFT, etc.) use the RGB tracker as the base model and use the RGBT dataset to fine-tune the model's full parameters. Although this method is effective to a certain extent, the full parameter fine-tuning method has the disadvantages of high time cost and low efficiency. In addition, due to the limited annotated samples of the RGBT dataset, it is impossible to use the pre-trained knowledge of the base model trained on a large-scale dataset. Therefore, the full parameter fine-tuning method cannot obtain a generalized representation of the object.

[0004] In addition to the full parameter fine-tuning method, existing RGBT tracking methods (ViPT, SDSTrack, OneTracker, etc.) introduce parameter cue learning into the RGBT tracking method. These trackers freeze the backbone network parameters of the base model, add some additional trainable parameters, and generate modality-related cues, thereby adapting the frozen pre-trained model to the RGBT tracking method. This method reduces the computational burden to a certain extent, and the parameter cue learning-based approach achieves performance comparable to full parameter fine-tuning. However, the above RGBT tracker ignores temporal information. During the tracking process, the target and background are constantly changing, and it is difficult for the RGBT tracker to achieve good performance in complex tracking scenarios. Summary of the invention

[0005] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and propose an RGBT target tracking method based on domain adaptation and spatiotemporal information fusion to solve the problem of tracking failure in the existing target tracking technology under complex scenarios such as target appearance changes and occlusions.

[0006] The basic idea of ​​implementing the present invention is to transfer the model's feature extraction capability from the visible light domain to the multimodal domain by adding a small amount of trainable parameters, thereby improving the model's domain adaptability; design an online template update strategy to obtain a dynamic template with temporal information, which complements the spatial information provided by the initial template, thereby improving the tracker's target positioning capability in complex environments such as target appearance changes and occlusions. The present invention greatly reduces the computational cost and can make full use of the target's temporal information, thereby effectively improving the feature extraction network performance and enhancing the accuracy and robustness of the tracker.

[0007] The present invention achieves the above-mentioned purpose by the following specific steps:

[0008] (1) Construct a visible light-thermal infrared RGBT data sample set containing real labels and divide it into a training set and a test set;

[0009] (2) Using the pre-trained RGB tracker OSTrack as a benchmark, a dual-branch RGBT target tracking model is constructed:

[0010] (2.1) Expand the pre-trained RGB tracker OSTrack into a dual-branch RGBT tracker, where the dual branches are the visible light RGB modality branch and the thermal infrared TIR modality branch;

[0011] (2.2) Add an adapter to the transformer encoder of the dual-branch RGBT tracker and construct a fine-tuned transformer encoder to obtain a dual-branch RGBT target tracking model with domain adaptation capability;

[0012] (3) Import the model parameters of the pre-trained OSTrack, use the training set to train the dual-branch RGBT target tracking model to obtain the optimal parameters, and obtain the final RGBT target tracking model after training;

[0013] (4) Designing an online module update strategy for dynamically updating the initial template image; specifically, setting the classification score threshold as τ, when the maximum classification score of the target is greater than τ, updating the initial template image to obtain a dynamic template image, and executing step (5); otherwise, not updating the initial template image, and executing step (6);

[0014] (5) Using the initial template image, dynamic template image and search image as inputs of the final RGBT target tracking model, directly execute step (7);

[0015] (6) Taking the initial template image and the search image as inputs of the final RGBT target tracking model;

[0016] (7) Load the optimal parameters, use the final RGBT target tracking model to predict the motion state of the target in subsequent frames, obtain the target tracking results and output them.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] First, since the present invention replaces the backbone of the benchmark model with a fine-tuned transformer encoder in the feature extraction stage, the feature extraction capability of the benchmark model is migrated from the visible light domain to the multimodal domain by fine-tuning the transformer encoder, thereby effectively improving the domain adaptability of the model.

[0019] Second, the present invention designs an online template updating strategy during the tracking process, so that the temporal information provided by the dynamic template and the spatial information provided by the initial template are integrated with each other, so that the RGBT target tracking model based on domain adaptation and spatiotemporal information fusion can achieve robust tracking in complex scenes such as target appearance changes and occlusions. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flow chart for realizing the method of the present invention;

[0021] Figure 2 It is a schematic diagram of the network structure of the RGBT tracker constructed in the present invention;

[0022] Figure 3 It is a schematic diagram of the structure of the fine-tuning transformer encoder in the present invention. DETAILED DESCRIPTION

[0023] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation methods.

[0024] Example 1: Reference Figure 1 The present invention proposes an RGBT target tracking method based on domain adaptation and spatiotemporal information fusion, which is characterized by comprising the following steps:

[0025] Step 1: Construct a visible light-thermal infrared RGBT data sample set containing real labels and divide it into a training set and a test set;

[0026] Step 2. Using the pre-trained RGB tracker OSTrack as a baseline, build a dual-branch RGBT object tracking model:

[0027] (2.1) Expand the pre-trained RGB tracker OSTrack into a dual-branch RGBT tracker, where the dual branches are the visible light RGB modality branch and the thermal infrared TIR modality branch. The above expansion of the pre-trained RGB tracker OSTrack into a dual-branch RGBT tracker includes the following two stages:

[0028] The first stage is to expand OSTrack into weight-sharing parallel branches to extract features from the visible light RGB modality and thermal infrared TIR modality respectively;

[0029] The second stage is to add cross-modal interactions between parallel branches for information transfer between RGB and TIR modalities.

[0030] (2.2) Add an adapter to the transformer encoder of the dual-branch RGBT tracker. Specifically, an adapter is embedded in the multi-head attention layer and the fully connected layer of the transformer encoder in different branches in parallel to construct a fine-tuned transformer encoder; a dual-branch RGBT target tracking model with domain adaptation capability is obtained.

[0031] The fine-tuning transformer encoder, whose core component is the adapter; the adapter is an hourglass structure consisting of two linear layers and a scaling factor, and the hourglass structure is parallelly connected to the original multi-head attention layer branch and the fully connected layer branch through scaling of the scaling factor; the multi-head attention layer contains the interaction information of paired positions, and the fully connected layer contains the knowledge of specific tasks; the parallel connection is used to retain the original features of the independent branches, and the context is updated by scaling and aggregating the elements.

[0032] Assume that the i-1th encoder of different branches outputs the features of their respective modes as After the fine-tuning transformer encoder, the specific calculation process is expressed as follows:

[0033]

[0034] in, is the output of the i-th layer encoder of the RGB modality branch, is the output of the i-th layer encoder of the TIR modality branch; is the output of the multi-head attention layer of the i-th encoder of the RGB modality, It is the output of the multi-head attention layer of the i-th encoder of the TIR modality; MSA represents the multi-head attention layer, LN represents the normalization layer, MLP represents the fully connected layer, and Ada represents the adapter structure; s is the scale factor, and the scale factor is set to 1 in this embodiment.

[0035] Step 3. Import the model parameters of the pre-trained OSTrack, use the training set to train the dual-branch RGBT target tracking model to obtain the optimal parameters, and obtain the final RGBT target tracking model after training; the implementation steps are as follows:

[0036] (3.1) Input training frames including RGB and TIR modalities;

[0037] (3.2) Crop the training frames of the two modalities respectively to obtain their corresponding initial template images and search images as the input of the model;

[0038] (3.3) The initial template image and the search image are spliced ​​together and sent to the RGBT target tracking model for joint feature extraction;

[0039] (3.4) Input the extracted features into the bounding box prediction network to obtain the bounding box of the target and the confidence score of the bounding box;

[0040] (3.5) Use OSTrack’s loss function to calculate the tracking loss between the tracking result and the true label;

[0041] (3.6) The model parameters corresponding to the minimum tracking loss are taken as the optimal parameters to obtain the final RGBT target tracking model after training.

[0042] Step 4. Design an online module update strategy to dynamically update the initial template image; specifically: set the classification score threshold to τ, and when the maximum classification score of the target is greater than τ, update the initial template image to obtain a dynamic template image and execute step 5; otherwise, do not update the initial template image and execute step 6.

[0043] The dynamic template image is used to provide the time information of the target, and the initial template image is used to provide the spatial information of the target. When the maximum classification score of the target is greater than τ, the dynamic template image is obtained as an additional input of the RGBT target tracking model in step 1, and the dynamic template image and the initial template image are jointly input into the RGBT target tracking model. Otherwise, the initial template image is not updated, and only the initial template image is used as the input of the RGBT target tracking model.

[0044] Step 5. Use the initial template image, dynamic template image, and search image as the input of the final RGBT object tracking model and directly proceed to step 7.

[0045] The initial template image, dynamic template image and search image are used as the input of the final RGBT target tracking model to predict the motion state of the target in subsequent frames. The input image is first divided into P×P blocks by patch embedding and flattened into 6 patch sequences. Then, the features of the initial template, dynamic template and search image of the two modalities are obtained respectively through the linear projection layer. Finally, the marked tokens of the RGB and TIR modal branches are spliced ​​along the channel dimension. The specific implementation process is as follows; and then they are sent to the fine-tuned transformer encoder for processing.

[0046] The above-mentioned input image is divided into P×P blocks by patch embedding, and flattened into 6 patch sequences. Then, the initial template, dynamic template, and search image features of the two modalities are obtained through the linear projection layer. Finally, the label tokens of the RGB and TIR modal branches are spliced ​​along the channel dimension. The specific implementation process is shown as follows:

[0047]

[0048]

[0049] in Represent the learnable parameters of the linear projection layer of the RGB modality search image, the initial template image, and the dynamic template image respectively; Represent the learnable parameters of the linear projection layer of the TIR modality search image, the initial template image, and the dynamic template image respectively; They are patches of RGB and TIR modality search images respectively; They are patches of the initial template images in RGB mode and TIR mode respectively; They are patches of RGB modality and TIR modality dynamic template images respectively; They are the features of the RGB modality and TIR modality search images respectively; They are the features of the initial template images in RGB and TIR modalities respectively; They are the features of the dynamic template images in RGB and TIR modalities respectively; represents the input of the first-layer fine-tuned transformer encoder of the RGB modality branch, represents the input of the fine-tuned transformer encoder in the first layer of the TIR modality branch, and concat represents the concatenation operation along the channel.

[0050] Step 6. Use the initial template image and the search image as input to the final RGBT object tracking model;

[0051] Step 7. Load the optimal parameters, use the final RGBT target tracking model to predict the motion state of the target in subsequent frames, obtain the target tracking results and output them.

[0052] In this embodiment, the final RGBT target tracking model is used to predict the motion state of the target in subsequent frames. The specific implementation steps are as follows:

[0053] (7.1) Load the optimal parameters of the trained RGBT target tracking model;

[0054] (7.2) Input test frames including RGB and TIR modalities;

[0055] (7.3) Crop the test frames of the two modalities respectively to obtain their corresponding initial template images and search images as the input of the model;

[0056] (7.4) stitching the initial template image and the search image together to obtain a stitched image;

[0057] (7.5) Send the spliced ​​image to the final RGBT target tracking model for joint feature extraction;

[0058] (7.6) Input the extracted features into the bounding box prediction network to obtain the bounding box of the target and the confidence score of the bounding box. All confidence scores are judged as follows: if the maximum classification score of the target is lower than the set threshold, execute step (7.8); otherwise, execute step (7.7);

[0059] (7.7) Finally, the RGBT target tracking model updates the initial template based on the online template update strategy to obtain a dynamic template image, and splices the image with the initial template image and the search image as a spliced ​​image, and returns to execute step (7.5);

[0060] (7.8) The prediction process ends and the target tracking result is output.

[0061] Example 2: Reference Figure 1-3 The overall steps of the target tracking method in this embodiment are the same as those in Embodiment 1. Now a specific example is given to further describe the implementation process of the present invention in detail.

[0062] Step S1: Obtain RGBT image pairs containing true labels and alignment from the training set of the RGBT dataset LasHeR, and divide them into a training set and a validation set;

[0063] Step S2: Using the pre-trained RGB tracker OSTrack as a benchmark, expand it into the RGBT target tracking method provided by the present invention, such as Figure 1 As shown, the specific method is:

[0064] S2.1: Expand the pre-trained RGB tracker OSTrack into a two-branch RGBT tracker, which consists of two stages: The first stage is to expand the RGB tracker OSTrack into weight-sharing parallel branches to extract features from the visible light modality and the thermal infrared modality respectively; The second stage is to add cross-modal interaction between the parallel branches to promote the information transfer between the visible light modality and the thermal infrared modality and improve the target representation ability of the RGBT tracker.

[0065] S2.2: Embed a fine-tuning transformer encoding block into the two-branch RGBT tracker to construct an RGBT tracker with domain adaptation ability; As Figure 1 and Figure 2 shown, load the model parameters of the RGB tracker OSTrack and freeze them. The visible light branch and the thermal infrared branch share the parameters. Use the LasHeR training set to train the fine-tuning transformer encoding block, transfer the feature extraction ability of the model from the RGB domain to the RGBT domain, and improve the adaptation ability of the baseline model in RGBT tracking.

[0066] In the two-branch RGBT tracker, embed an adapter into the multi-head attention layer and the fully connected layer of the transformer encoding block in different branches to construct a fine-tuning transformer. The core component of the fine-tuning transformer encoder is the adapter, which is an hourglass structure composed of two linear layers and a scaling factor. The parameters of the first linear layer (FC Down) of the adapter are The parameters of the second linear layer (FC Up) are where d’ is the intermediate dimension, where d' << d. This hourglass structure is scaled by the scaling factor and connected in parallel to the original multi-head attention layer branch and the fully connected layer branch. The multi-head attention layer contains pairwise position interaction information, and the fully connected layer contains task-specific knowledge, which is beneficial for the model to adapt to new downstream tasks; The parallel design can retain the original features of the independent branches and update the context through element-wise scaling and aggregation.

[0067] S2.3: During the tracking process, the online module update strategy dynamically updates the initial template. The online template provides the temporal information of the target, and the initial template provides the spatial information of the target. The two templates are jointly input into the RGBT tracker. The templates provide the spatio-temporal information of the target to help the model achieve robust tracking in complex application environments.

[0068] As Figure 1As shown, during the tracking process, the online template update strategy dynamically updates the initial template. The dynamic template provides the time information of the target, and the initial template provides the spatial information of the target. The two templates are fused to provide the spatiotemporal information of the target to be tracked. The present invention is based on the maximum classification score of the target as the basis for our online template update, and the maximum classification score of the target reflects the confidence of the tracking result. This online update method maximizes the model's ability to distinguish between the target and the background without the need for additional training of the model. In this embodiment, the threshold is set to 0.5, and the maximum classification score of the target is lower than the given threshold, and the initial template is not updated. If the maximum classification score of the target is higher than the given threshold, the online template update updates the initial template to obtain the dynamic template. The input is the initial template, the dynamic template, and the search image. First, the patch embedding is divided into small blocks of size P×P, and it is flattened into a sequence of 6 patches. Then, the features of the search image, the initial template, and the dynamic template of the two modalities are obtained respectively through the linear projection layer. Finally, the marker embedding of the RGB modality branch and the TIR modality branch is spliced ​​along the channel dimension and sent to the fine-tuning transformer encoder.

[0069] Step S3: Import the model parameters of the pre-trained OSTrack and fine-tune the RGBT tracker using the LasHeR training set;

[0070] Step S4: Load the parameters of the trained RGBT tracking model. Given a video sequence, the first frame gives the spatial position of the target. The video sequence is fed into the tracking model frame by frame to predict the motion state of the target in subsequent frames.

[0071] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0072] The parts of the present invention that are not described in detail belong to the common knowledge of those skilled in the art. The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Obviously, after understanding the content and principle of the present invention, professionals in this field may make various modifications and changes in form and details without departing from the principle and structure of the present invention, but these modifications and changes based on the idea of ​​the present invention are still within the scope of protection of the claims of the present invention.

Claims

1. A RGBT target tracking method based on domain adaptation and spatiotemporal information fusion, characterized in that: The steps include: (1) Construct a visible light-thermal infrared RGBT data sample set containing real labels and divide it into a training set and a test set; (2) Using the pre-trained RGB tracker OSTrack as a benchmark, a dual-branch RGBT target tracking model is constructed: (2.1) Expand the pre-trained RGB tracker OSTrack into a dual-branch RGBT tracker, where the dual branches are the visible light RGB modality branch and the thermal infrared TIR modality branch; (2.2) Add an adapter to the transformer encoder of the dual-branch RGBT tracker and construct a fine-tuned transformer encoder to obtain a dual-branch RGBT target tracking model with domain adaptation capability; (3) Import the model parameters of the pre-trained OSTrack, use the training set to train the dual-branch RGBT target tracking model to obtain the optimal parameters, and obtain the final RGBT target tracking model after training; (4) Designing an online module update strategy for dynamically updating the initial template image; specifically, setting the classification score threshold as τ, when the maximum classification score of the target is greater than τ, updating the initial template image to obtain a dynamic template image, and executing step (5); otherwise, not updating the initial template image, and executing step (6); (5) Using the initial template image, dynamic template image and search image as inputs of the final RGBT target tracking model, directly execute step (7); (6) Taking the initial template image and the search image as inputs of the final RGBT target tracking model; (7) Load the optimal parameters, use the final RGBT target tracking model to predict the motion state of the target in subsequent frames, obtain the target tracking results and output them.

2. The method according to claim 1, characterized in that: Step (2.1) expands the pre-trained RGB tracker OSTrack into a dual-branch RGBT tracker, which includes the following two stages: The first stage is to expand OSTrack into weight-sharing parallel branches to extract features from the visible light RGB modality and thermal infrared TIR modality respectively; The second stage is to add cross-modal interactions between parallel branches for information transfer between RGB and TIR modalities.

3. The method according to claim 1, characterized in that: In step (2.2), an adapter is added to the transformer encoder of the dual-branch RGBT tracker. Specifically, an adapter is embedded in parallel in the multi-head attention layer and the fully connected layer of the transformer encoder in different branches to construct a fine-tuned transformer encoder.

4. The method according to claim 3, characterized in that: The fine-tuning transformer encoder, whose core component is the adapter; the adapter is an hourglass structure consisting of two linear layers and a scaling factor, and the hourglass structure is parallelly connected to the original multi-head attention layer branch and the fully connected layer branch through scaling of the scaling factor; the multi-head attention layer contains the interaction information of paired positions, and the fully connected layer contains the knowledge of specific tasks; the parallel connection is used to retain the original features of the independent branches, and the context is updated by scaling and aggregating the elements.

5. The method according to claim 4, characterized in that: Assume that the i-1th encoder of different branches outputs the features of their respective modes as After the fine-tuning transformer encoder, the specific calculation process is expressed as follows: in, is the output of the i-th layer encoder of the RGB modality branch, is the output of the i-th layer encoder of the TIR modality branch; is the output of the multi-head attention layer of the i-th encoder of the RGB modality, is the output of the multi-head attention layer of the i-th encoder of the TIR modality; MSA represents the multi-head attention layer, LN represents the normalization layer, MLP represents the fully connected layer, Ada represents the adapter structure; s is the scale factor.

6. The method according to claim 1, characterized in that: The steps for training the dual-branch RGBT target tracking model in step (3) are as follows: (3.1) Input training frames including RGB and TIR modalities; (3.2) Crop the training frames of the two modalities respectively to obtain their corresponding initial template images and search images as the input of the model; (3.3) The initial template image and the search image are spliced ​​together and sent to the RGBT target tracking model for joint feature extraction; (3.4) Input the extracted features into the bounding box prediction network to obtain the bounding box of the target and the confidence score of the bounding box; (3.5) Use OSTrack’s loss function to calculate the tracking loss between the tracking result and the true label; (3.6) The model parameters corresponding to the minimum tracking loss are taken as the optimal parameters to obtain the final RGBT target tracking model after training.

7. The method according to claim 1, characterized in that: The dynamic template image in step (4) is used to provide the time information of the target, and the initial template image is used to provide the spatial information of the target. When the maximum classification score of the target is greater than τ, the dynamic template image is obtained as an additional input of the RGBT target tracking model in step (4), and the dynamic template image and the initial template image are jointly input into the RGBT target tracking model. Otherwise, the initial template image is not updated, and only the initial template image is used as the input of the RGBT target tracking model.

8. The method according to claim 1, characterized in that: In step (5), the initial template image, dynamic template image and search image are used as the input of the final RGBT target tracking model to predict the motion state of the target in subsequent frames. The input image is first segmented into P×P blocks by patch embedding and flattened into 6 patch sequences. Then, the features of the initial template, dynamic template and search image of the two modalities are obtained respectively through the linear projection layer. Finally, the labeled tokens of the RGB and TIR modal branches are spliced ​​along the channel dimension and then sent to the fine-tuned transformer encoder for processing.

9. The method according to claim 7, characterized in that: The input image is divided into P×P blocks through patch embedding and flattened into 6 patch sequences. Then, the initial template, dynamic template, and search image features of the two modalities are obtained through the linear projection layer. Finally, the label tokens of the RGB and TIR modality branches are spliced ​​along the channel dimension. The specific implementation process is shown as follows: in Represent the learnable parameters of the linear projection layer of the RGB modality search image, the initial template image, and the dynamic template image respectively; Represent the learnable parameters of the linear projection layer of the TIR modality search image, the initial template image, and the dynamic template image respectively; They are patches of RGB and TIR modality search images respectively; They are patches of the initial template images in RGB mode and TIR mode respectively; They are patches of RGB modality and TIR modality dynamic template images respectively; They are the features of the RGB modality and TIR modality search images respectively; They are the features of the initial template images in RGB and TIR modalities respectively; They are the features of the dynamic template images in RGB and TIR modalities respectively; represents the input of the first-layer fine-tuned transformer encoder of the RGB modality branch, represents the input of the fine-tuned transformer encoder in the first layer of the TIR modality branch, and concat represents the concatenation operation along the channel.

10. The method according to claim 1, characterized in that: Step (7) uses the final RGBT target tracking model to predict the motion state of the target in subsequent frames, and the implementation steps are as follows: (7.1) Load the optimal parameters of the trained RGBT target tracking model; (7.2) Input test frames including RGB and TIR modalities; (7.3) Crop the test frames of the two modalities respectively to obtain their corresponding initial template images and search images as the input of the model; (7.4) stitching the initial template image and the search image together to obtain a stitched image; (7.5) Send the spliced ​​image to the final RGBT target tracking model for joint feature extraction; (7.6) Input the extracted features into the bounding box prediction network to obtain the bounding box of the target and the confidence score of the bounding box. All confidence scores are judged as follows: if the maximum classification score of the target is lower than the set threshold, execute step (7.8); otherwise, execute step (7.7); (7.7) Finally, the RGBT target tracking model updates the initial template based on the online template update strategy to obtain a dynamic template image, and splices the image with the initial template image and the search image as a spliced ​​image, and returns to execute step (7.5); (7.8) The prediction process ends and the target tracking result is output.

Images