A Multimodal Target Tracking Method and System Based on Template Update

Abstract

This invention relates to a multimodal target tracking method and system based on template updates. An initial target template image, a dynamic template image, and a search image are input as triplet images into a backbone feature extraction network to obtain color image feature vectors and thermal infrared image feature vectors. The color image feature vectors and thermal infrared image feature vectors are fused to obtain fused features. The fused features are input into the backbone feature extraction network to obtain deep semantic features corresponding to the triplet images. A feature sequence is determined. The feature sequence is input into an encoder to obtain a search region enhancement feature sequence. Output embeddings are obtained for all positions on the initial template, dynamic template, and search features. The bounding box of the tracked object is determined based on the search region enhancement feature sequence and the output embeddings. The target template is cropped and fused using the bounding box in the visible light image and thermal infrared image of the tracked object. The fused template is then set as the cross-modal dynamic template. This invention is unaffected by template information degradation and exhibits high tracking performance.

Description

Technical Field

[0001] This invention relates to the field of target tracking, and in particular to a multimodal target tracking method and system based on template updating. Background Technology

[0002] Visual object tracking is a fundamental research topic in computer vision. It involves identifying the object to be tracked in the first frame of a video sequence and continuously predicting its position in subsequent frames. In recent years, with the rapid development of deep learning, visual object tracking has moved beyond traditional machine learning, achieving breakthroughs by combining deep learning with methods such as correlation filters. Most tracking models use only RGB images, which, while providing rich target information in normal environments, offer limited information in extreme conditions such as occlusion, low light, fog, and nighttime. In contrast, thermal infrared images obtained from thermal infrared cameras are insensitive to changes in lighting and can complement RGB images in complex scenes, providing richer information. Because the complementarity of two modalities can effectively solve single-modal tracking problems, more and more researchers are focusing on the RGBT tracking field.

[0003] Based on their respective methods, proposed RGBT trackers can be broadly categorized into three types. Early RGBT trackers employed sparse representation-based algorithms. Li et al., for example, introduced reliable modal weight variables into a Bayesian framework, jointly optimizing the weights of sparse codes and reliable modalities online to achieve a collaborative sparse representation for adaptive modal fusion tracking. However, sparse representation-based models have high computational complexity and time-consuming online updates, making them unsuitable for practical applications. Therefore, Zhai et al. proposed applying correlation filters independently to the two modalities, then introducing low-rank constraints to collaboratively learn the two modal filters. Through consistent localization using similar filters of the two modalities, they achieved target object tracking in 277 frames, far exceeding real-time speed. However, correlation filter trackers often use manually designed image features, which are difficult to represent the target's appearance model in complex scenes.

[0004] The rapid development of deep learning has led to breakthroughs in object tracking. For example, Li's Manet algorithm uses three adapters to learn modality sharing, modality specificity, and instance awareness, respectively, to obtain powerful RGBT feature representations. To address the slow tracker speed, influenced by the strong competitiveness of Siamese networks in RGB tracking in recent years, many researchers have explored applying Siamese networks to RGBT trackers. For instance, Zhang et al. first proposed an RGBT tracker based on Siamese networks, achieving a tracking speed of approximately 30fps. However, the correlation operation based on Siamese networks only performs local linear calculations within the search region using a sliding window approach, lacking global information. Furthermore, the output is a similarity response map, losing the complete semantic information of the tracked target, thus creating a performance bottleneck. Simultaneously, during long-term tracking, template information decays with increasing tracking time, negatively impacting tracker performance. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal target tracking method and system based on template updates, which can improve tracking performance without being affected by template information degradation.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A template-updated multimodal target tracking method includes:

[0008] Obtain a triplet image, wherein the triplet image includes an initial target template image, a dynamic template image, and a search image;

[0009] The triplet image is input into the backbone feature extraction network to obtain the color image feature vector and the thermal infrared image feature vector;

[0010] The color image feature vector and the thermal infrared image feature vector are fused using a cross-modal feature difference complementary enhancement module to obtain fused features;

[0011] The fused features are input into the backbone feature extraction network to obtain the deep semantic features corresponding to the triplet image. The deep semantic features corresponding to the triplet image include the deep semantic features of the initial target template image, the deep semantic features of the dynamic template image, and the deep semantic features of the search image.

[0012] Based on the deep semantic features corresponding to the triplet image, a feature sequence is determined;

[0013] The feature sequence is input into the encoder to obtain the enhanced feature sequence of the search region;

[0014] The decoder can learn the query to obtain the initial template, dynamic template, and output embeddings at all positions on the search features.

[0015] The bounding box of the tracked object is determined based on the enhanced feature sequence of the search region and the output embedding;

[0016] The output is embedded into the prediction head to determine whether the prediction score is greater than the prediction threshold.

[0017] If so, the visible light image and thermal infrared image of the tracked object are respectively cropped with bounding boxes and fused, and the fused template is set as a cross-modal dynamic template;

[0018] If not, the cross-modal dynamic template will not be updated.

[0019] Optionally, the backbone feature extraction network adopts a ResNet50 network structure with the last stage and fully connected layers removed.

[0020] Optionally, the step of using a cross-modal feature difference complementary enhancement module to fuse the color image feature vector and the thermal infrared image feature vector to obtain fused features specifically includes:

[0021] Based on the color image feature vector and the thermal infrared image feature vector, the difference features are obtained;

[0022] Based on the aforementioned differential features, channel attention method is used to obtain differential complementary features;

[0023] Based on the color image feature vector and the thermal infrared image feature vector, channel complementary features are obtained through the cross-modal feature complementarity enhancement module;

[0024] Based on the difference complementary features and the channel complementary features, the fusion features are obtained.

[0025] Optionally, inputting the feature sequence into the encoder to obtain the enhanced feature sequence of the search region specifically includes:

[0026] The feature sequence is passed through a bottleneck network to obtain a triplet feature sequence with a reduced number of channels;

[0027] The triplet feature sequence after reducing the number of channels is expanded and concatenated along the spatial dimension to generate a feature sequence of a set dimension and a set length.

[0028] The feature sequence with the set dimension and length is input into an encoder constructed from n encoder layers to obtain the enhanced feature sequence of the search region.

[0029] Optionally, determining the bounding box of the tracked object based on the enhanced feature sequence of the search region and the output embedding specifically includes:

[0030] A similarity vector is obtained by multiplying the enhanced feature sequence of the search region and the output embedding.

[0031] The predicted feature sequence is obtained by multiplying the similarity vector and the enhanced feature sequence of the search region element by element.

[0032] The bounding box of the tracked object is determined based on the predicted feature sequence.

[0033] A template-updated multimodal target tracking system includes:

[0034] The triplet image acquisition module is used to acquire triplet images, which include an initial target template image, a dynamic template image, and a search image.

[0035] The modal feature vector determination module is used to input the triplet image into the backbone feature extraction network to obtain the color image feature vector and the thermal infrared image feature vector;

[0036] The fusion feature determination module is used to fuse the color image feature vector and the thermal infrared image feature vector using the cross-modal feature difference complementary enhancement module to obtain fused features;

[0037] The deep semantic feature determination module is used to input the fused features into the backbone feature extraction network to obtain the deep semantic features corresponding to the triplet image. The deep semantic features corresponding to the triplet image include the deep semantic features of the initial target template image, the deep semantic features of the dynamic template image, and the deep semantic features of the search image.

[0038] The feature sequence determination module is used to determine the feature sequence based on the deep semantic features corresponding to the triplet image;

[0039] The search region enhancement feature sequence determination module is used to input the feature sequence into the encoder to obtain the search region enhancement feature sequence;

[0040] The output embedding determination module is used to obtain the output embeddings of the initial template, dynamic template and all positions on the search features through a learnable query from the decoder.

[0041] The bounding box determination module is used to determine the bounding box of the tracked object based on the enhanced feature sequence of the search region and the output embedding;

[0042] The prediction score determination module is used to determine whether the prediction score is greater than the prediction threshold based on the output embedded into the prediction head.

[0043] The dynamic template update module is used to cut and fuse the target template in the visible light image and thermal infrared image of the tracked object with bounding boxes when the prediction score is greater than the prediction threshold, and set the fused template as the cross-modal dynamic template; when the prediction score is less than or equal to the prediction threshold, the cross-modal dynamic template is not updated.

[0044] Optionally, the backbone feature extraction network adopts a ResNet50 network structure with the last stage and fully connected layers removed.

[0045] Optionally, the fusion feature determination module specifically includes:

[0046] The difference feature determination unit is used to obtain difference features based on the color image feature vector and the thermal infrared image feature vector;

[0047] The difference-complementary feature determination unit is used to obtain the difference-complementary features based on the difference features using the channel attention method;

[0048] The channel complementary feature determination unit is used to obtain channel complementary features based on the color image feature vector and the thermal infrared image feature vector through the cross-modal feature complementary enhancement module;

[0049] The fusion feature determination unit is used to obtain fusion features based on the difference complementary features and the channel complementary features.

[0050] Optionally, the search region enhanced feature sequence determination module specifically includes:

[0051] The triplet feature sequence determination unit is used to pass the feature sequence through a bottleneck network to obtain a triplet feature sequence with a reduced number of channels.

[0052] A feature sequence determination unit with a set dimension and a set length is used to expand and cascade the triplet feature sequence after reducing the number of channels along the spatial dimension to generate a feature sequence with a set dimension and a set length.

[0053] The search region enhancement feature sequence determination unit is used to input the feature sequence of the set dimension and set length into an encoder constructed by n encoder layers to obtain the search region enhancement feature sequence.

[0054] Optionally, the bounding box determination module specifically includes:

[0055] The similarity vector determination unit is used to calculate the similarity vector by using a dot product based on the enhanced feature sequence of the search region and the output embedding;

[0056] The predictive feature sequence determination unit is used to multiply the similarity vector and the enhanced feature sequence of the search region element by element to obtain the predictive feature sequence;

[0057] A bounding box determination unit is used to determine the bounding box of the tracked object based on the predicted feature sequence.

[0058] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0059] This invention provides a multimodal target tracking method based on template update. The method involves inputting a triplet image (comprising an initial target template image, a dynamic template image, and a search image) into a backbone feature extraction network to obtain color image feature vectors and thermal infrared image feature vectors. A cross-modal feature difference complementarity enhancement module is used to fuse the color image feature vectors and thermal infrared image feature vectors to obtain fused features. These fused features are then input into the backbone feature extraction network to obtain deep semantic features corresponding to the triplet image. Based on these deep semantic features, a feature sequence is determined. This feature sequence is input into an encoder to obtain a search region enhancement feature sequence. A decoder can learn and query to obtain the output embeddings for all positions on the initial template, dynamic template, and search features. The bounding box of the tracked object is determined based on the search region enhancement feature sequence and the output embeddings. The output embeddings are input into a prediction head, and the prediction score determines whether to crop and fuse the target template from the RGB search image and thermal infrared image respectively, setting it as a cross-modal dynamic template. This cross-modal dynamic template is unaffected by template information degradation, improving tracking performance. Attached Figure Description

[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0061] Figure 1 This is a flowchart of the multimodal target tracking method based on template update of the present invention;

[0062] Figure 2 This is a structural diagram of the multimodal target tracking system based on template updating according to the present invention. Detailed Implementation

[0063] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] The purpose of this invention is to provide a multimodal target tracking method and system based on template updates, which can improve tracking performance without being affected by template information degradation.

[0065] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0066] Figure 1 This is a flowchart of the multimodal target tracking method based on template updating according to the present invention. Figure 1 As shown, a multimodal target tracking method based on template update includes:

[0067] Step 101: Obtain the triplet image, which includes the initial target template image, the dynamic template image, and the search image.

[0068] Step 102: Input the triplet image into the backbone feature extraction network to obtain color image feature vectors and thermal infrared image feature vectors. The backbone feature extraction network adopts a ResNet50 network structure with the last stage and fully connected layers removed. The input to the backbone consists of triplets composed of initial target template images, dynamic template images, and search images of two modalities. Inputting the triplets into the backbone feature extraction network outputs feature vectors of two modalities.

[0069] Step 103: The color image feature vector and the thermal infrared image feature vector are fused using a cross-modal feature difference complementary enhancement module to obtain fused features, specifically including:

[0070] Based on the color image feature vector and the thermal infrared image feature vector, the difference features are obtained;

[0071] Based on the aforementioned differential features, channel attention method is used to obtain differential complementary features;

[0072] Based on the color image feature vector and the thermal infrared image feature vector, channel complementary features are obtained through the cross-modal feature complementarity enhancement module;

[0073] Based on the difference complementary features and the channel complementary features, the fusion features are obtained.

[0074] In the third stage of ResNet50, a cross-modal feature difference complementation enhancement network (FDCE) was designed to effectively complement the information of the two modalities, with RGB features as input. and thermal infrared characteristics First, the modal difference features were obtained separately using Eq.(1)

[0075]

[0076] Here Represents the unique information contained in RGB images, and similarly... It contains information unique to thermal infrared radiation.

[0077] Then, complementary weights are obtained from the two modal differences using channel attention to integrate complementary information. Therefore, it can be defined as Eq(2).

[0078]

[0079] Here This indicates multiplication by channel, GAP represents global average pooling, and sigmoid(*) represents the sigmoid function. The differential features of the two modalities are respectively processed by global average pooling to compress complementary features into a single vector. The channel weights are then obtained through the sigmoid function. The differential features are multiplied by the channel weights to adaptively supplement the required differential modal information.

[0080] At the same time, an FDE module was also designed, featuring two modalities. and First, appearance information of the modality is extracted through 1*1 convolution, ReLU activation function, and 3*3 convolution. Then, the sigmoid function is used to generate channel weights to control the propagation of complementary features of the modality. These complementary features are used to enhance the feature representation of the target in another modality. This can be expressed as Eq(3).

[0081]

[0082] Here, conv_t and conv_rgb are composed of 1*1 convolution, ReLU activation function and 3*3 convolution, used to extract appearance features of two modalities.

[0083] Finally, the modality fusion output is obtained by adding the cross-modal feature complementarity enhancement module (FDE) to the differential complementarity feature module. This can be expressed as Eq(4).

[0084]

[0085] Here This represents summation of elements. It is the final output of the cross-modal feature difference complementary enhancement network.

[0086] Step 104: Input the fused features into the backbone feature extraction network to obtain the deep semantic features corresponding to the triplet image. The deep semantic features corresponding to the triplet image include the deep semantic features of the initial target template image, the deep semantic features of the dynamic template image, and the deep semantic features of the search image.

[0087] After triplet feature fusion, the fourth stage of ResNet50 yields three final feature maps. That is, the deep semantic features corresponding to the triplet image.

[0088] Step 105: Determine the feature sequence based on the deep semantic features corresponding to the triplet image;

[0089] Step 106: Input the feature sequence into the encoder to obtain the search region enhancement feature sequence, specifically including:

[0090] The feature sequence is passed through a bottleneck network to obtain a triplet feature sequence with a reduced number of channels;

[0091] The triplet feature sequence after reducing the number of channels is expanded and concatenated along the spatial dimension to generate a feature sequence of a set dimension and a set length.

[0092] The feature sequence with the set dimension and length is input into an encoder constructed from n encoder layers to obtain the enhanced feature sequence of the search region.

[0093] Specifically, the triples after passing through the backbone feature extraction network and the cross-modal feature difference complementarity enhancement network... The bottleneck network needs to be used to reduce the number of channels from C to D. Then, the triplet features are expanded and concatenated along the spatial dimension to generate a feature of dimension d and length D. The characteristic sequence.

[0094] Step 107: The decoder can learn the query to obtain the initial template, dynamic template and output embeddings of all positions on the search features.

[0095] The decoder consists of m decoder layers. Since it only predicts the bounding box of one object, a random object query vector is used. The object query vector is fed into the multi-head attention and feedforward network (FFN) along with the enhanced feature sequence from the encoder through a multi-head self-attention mechanism. As a result, the object query vector is transformed by the decoder into an output embedding that can focus on all positions on the initial template, dynamic template, and search features, and is used to predict the final bounding box.

[0096] Step 108: Determine the bounding box of the tracked object based on the enhanced feature sequence of the search region and the output embedding, specifically including:

[0097] A similarity vector is obtained by multiplying the enhanced feature sequence of the search region and the output embedding.

[0098] The predicted feature sequence is obtained by multiplying the similarity vector and the enhanced feature sequence of the search region element by element.

[0099] The predicted feature sequence is input into the fully convolutional network (FCN) module to determine the bounding box of the tracked object.

[0100] The encoder outputs a search region enhancement feature sequence, and the decoder output embedding is compared using a dot product to calculate similarity. Then, the similarity vector is multiplied element-wise with the encoder's output search features to enhance regions relevant to the output embedding and suppress unimportant regions. The resulting new feature sequence is then reshaped into a feature map. The input is fed into the Fully Convolutional Network (FCN) module to obtain the predicted top-left and bottom-right corner points of the bounding box. Finally, the bounding box of the tracked object is obtained by calculating the distribution of the corner point probabilities.

[0101] Step 109: Based on the output, embed the input into the prediction head and determine whether the prediction score is greater than the prediction threshold;

[0102] Step 110: If yes, crop the target template from the visible light image and the thermal infrared image of the tracked object with bounding boxes and fuse them, then set the fused template as a cross-modal dynamic template;

[0103] Step 111: If not, the cross-modal dynamic template will not be updated.

[0104] As tracking progresses over extended periods, changes in the target's appearance can cause initial template information to decay, thus impacting the performance of the tracking model. To address this, a cross-modal dynamic template update network is designed for multimodal tracking algorithms. Specifically, considering the template degradation problem, a high-confidence dynamic template is selected from intermediate frames as additional input. High confidence refers to situations where the target is completely occluded, disappears from view, or the tracker drifts during tracking, at which point the cropped target template is considered unreliable. To solve this problem, a simple score prediction head is designed, consisting of only three layers of perception and activated by a sigmoid function. If the predicted score is greater than a threshold s, the target template is considered reliable. Then, the target template is cropped from both the visible light search image and the thermal infrared image and fused to form the cross-modal dynamic template. Finally, the newly generated dynamic template replaces the original dynamic template for tracking in the next frame.

[0105] This invention designs a robust multimodal target tracking method with template updating. Specifically, it addresses the template information degradation problem by designing a cross-modal dynamic template updating network and a cross-modal feature difference complementarity enhancement network to fuse RGB image and thermal infrared image information. Furthermore, an encoder models the global spatiotemporal feature dependencies of the fused template frame, dynamic template frame, and search frame, while a decoder predicts the spatial location of the target object. The entire method is end-to-end.

[0106] Figure 2 This is a structural diagram of the template-updated multimodal target tracking system of the present invention. Figure 2 As shown, a template-updated multimodal target tracking system includes:

[0107] The triplet image acquisition module 201 is used to acquire triplet images, wherein the triplet images include an initial target template image, a dynamic template image, and a search image;

[0108] The modal feature vector determination module 202 is used to input the triplet image into the backbone feature extraction network to obtain the color image feature vector and the thermal infrared image feature vector. The backbone feature extraction network adopts a ResNet50 network structure with the last stage and the fully connected layer removed.

[0109] The fusion feature determination module 203 is used to fuse the color image feature vector and the thermal infrared image feature vector using the cross-modal feature difference complementary enhancement module to obtain fused features;

[0110] The deep semantic feature determination module 204 is used to input the fused features into the backbone feature extraction network to obtain the deep semantic features corresponding to the triplet image. The deep semantic features corresponding to the triplet image include the deep semantic features of the initial target template image, the deep semantic features of the dynamic template image, and the deep semantic features of the search image.

[0111] The feature sequence determination module 205 is used to determine the feature sequence based on the deep semantic features corresponding to the triplet image;

[0112] The search region enhancement feature sequence determination module 206 is used to input the feature sequence into the encoder to obtain the search region enhancement feature sequence.

[0113] The output embedding determination module 207 is used to obtain the output embeddings of the initial template, dynamic template and all positions on the search features through a learnable query by the decoder.

[0114] The bounding box determination module 208 is used to determine the bounding box of the tracked object based on the enhanced feature sequence of the search region and the output embedding.

[0115] The prediction score determination module 209 is used to determine whether the prediction score is greater than the prediction threshold based on the output embedded into the prediction head.

[0116] The dynamic template update module 210 is used to, when the prediction score is greater than the prediction threshold, crop the target template with bounding boxes in the visible light image and thermal infrared image of the tracked object respectively and fuse them, and set the fused template as a cross-modal dynamic template; when the prediction score is less than or equal to the prediction threshold, the cross-modal dynamic template is not updated.

[0117] The fusion feature determination module 203 specifically includes:

[0118] The difference feature determination unit is used to obtain difference features based on the color image feature vector and the thermal infrared image feature vector;

[0119] The difference-complementary feature determination unit is used to obtain the difference-complementary features based on the difference features using the channel attention method;

[0120] The channel complementary feature determination unit is used to obtain channel complementary features based on the color image feature vector and the thermal infrared image feature vector through the cross-modal feature complementary enhancement module;

[0121] The fusion feature determination unit is used to obtain fusion features based on the difference complementary features and the channel complementary features.

[0122] The search region enhanced feature sequence determination module 206 specifically includes:

[0123] The triplet feature sequence determination unit is used to pass the feature sequence through a bottleneck network to obtain a triplet feature sequence with a reduced number of channels.

[0124] A feature sequence determination unit with a set dimension and a set length is used to expand and cascade the triplet feature sequence after reducing the number of channels along the spatial dimension to generate a feature sequence with a set dimension and a set length.

[0125] The search region enhancement feature sequence determination unit is used to input the feature sequence of the set dimension and set length into an encoder constructed by n encoder layers to obtain the search region enhancement feature sequence.

[0126] The bounding box determination module 208 specifically includes:

[0127] The similarity vector determination unit is used to calculate the similarity vector by using a dot product based on the enhanced feature sequence of the search region and the output embedding;

[0128] The predictive feature sequence determination unit is used to multiply the similarity vector and the enhanced feature sequence of the search region element by element to obtain the predictive feature sequence;

[0129] A bounding box determination unit is used to determine the bounding box of the tracked object based on the predicted feature sequence.

[0130] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0131] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A multimodal target tracking method based on template update, characterized in that, include: Obtain a triplet image, wherein the triplet image includes an initial target template image, a dynamic template image, and a search image; The triplet image is input into the backbone feature extraction network to obtain the color image feature vector and the thermal infrared image feature vector; The color image feature vector and the thermal infrared image feature vector are fused using a cross-modal feature difference complementary enhancement module to obtain fused features; The fused features are input into the backbone feature extraction network to obtain the deep semantic features corresponding to the triplet image. The deep semantic features corresponding to the triplet image include the deep semantic features of the initial target template image, the deep semantic features of the dynamic template image, and the deep semantic features of the search image. Based on the deep semantic features corresponding to the triplet image, a feature sequence is determined; The feature sequence is input into the encoder to obtain the enhanced feature sequence of the search region; The decoder learns the query to obtain the initial template, dynamic template, and output embeddings at all positions on the search features; The bounding box of the tracked object is determined based on the enhanced feature sequence of the search region and the output embedding; the output embedding is input to the prediction head to determine whether the prediction score is greater than the prediction threshold. If so, the visible light image and thermal infrared image of the tracked object are respectively cropped with bounding boxes and fused, and the fused template is set as a cross-modal dynamic template; If not, the cross-modal dynamic template will not be updated.

2. The multimodal target tracking method based on template update according to claim 1, characterized in that, The backbone feature extraction network adopts a ResNet50 network structure with the last stage and fully connected layers removed.

3. The multimodal target tracking method based on template update according to claim 1, characterized in that, The method employs a cross-modal feature difference complementary enhancement module to fuse the color image feature vector and the thermal infrared image feature vector to obtain fused features, specifically including: Based on the color image feature vector and the thermal infrared image feature vector, the difference features are obtained; Based on the aforementioned differential features, channel attention method is used to obtain differential complementary features; Based on the color image feature vector and the thermal infrared image feature vector, channel complementary features are obtained through the cross-modal feature complementarity enhancement module; Based on the difference complementary features and the channel complementary features, the fusion features are obtained.

4. The multimodal target tracking method based on template update according to claim 1, characterized in that, The step of inputting the feature sequence into the encoder to obtain the enhanced feature sequence of the search region specifically includes: The feature sequence is passed through a bottleneck network to obtain a triplet feature sequence with a reduced number of channels; The triplet feature sequence after reducing the number of channels is expanded and concatenated along the spatial dimension to generate a feature sequence of a set dimension and a set length. The feature sequence with the set dimension and length is input into an encoder constructed from n encoder layers to obtain the enhanced feature sequence of the search region.

5. The multimodal target tracking method based on template update according to claim 1, characterized in that, The step of determining the bounding box of the tracked object based on the enhanced feature sequence of the search region and the output embedding specifically includes: A similarity vector is obtained by multiplying the enhanced feature sequence of the search region and the output embedding. The predicted feature sequence is obtained by multiplying the similarity vector and the enhanced feature sequence of the search region element by element. The bounding box of the tracked object is determined based on the predicted feature sequence.

6. A multimodal target tracking system based on template update, characterized in that, include: The triplet image acquisition module is used to acquire triplet images, which include an initial target template image, a dynamic template image, and a search image. The modal feature vector determination module is used to input the triplet image into the backbone feature extraction network to obtain the color image feature vector and the thermal infrared image feature vector; The fusion feature determination module is used to fuse the color image feature vector and the thermal infrared image feature vector using the cross-modal feature difference complementary enhancement module to obtain fused features; The deep semantic feature determination module is used to input the fused features into the backbone feature extraction network to obtain the deep semantic features corresponding to the triplet image. The deep semantic features corresponding to the triplet image include the deep semantic features of the initial target template image, the deep semantic features of the dynamic template image, and the deep semantic features of the search image. The feature sequence determination module is used to determine the feature sequence based on the deep semantic features corresponding to the triplet image; The search region enhancement feature sequence determination module is used to input the feature sequence into the encoder to obtain the search region enhancement feature sequence; The output embedding determination module is used to learn the query through the decoder and obtain the output embeddings of the initial template, dynamic template and all positions on the search features; The bounding box determination module is used to determine the bounding box of the tracked object based on the enhanced feature sequence of the search region and the output embedding; The prediction score determination module is used to determine whether the prediction score is greater than the prediction threshold based on the output embedded into the prediction head. The dynamic template update module is used to crop the target template with bounding boxes and fuse them when the prediction score is greater than the prediction threshold in the visible light image and thermal infrared image of the tracked object, and set the fused template as a cross-modal dynamic template. If the predicted score is less than or equal to the prediction threshold, the cross-modal dynamic template will not be updated.

7. The multimodal target tracking system based on template update according to claim 6, characterized in that, The backbone feature extraction network adopts a ResNet50 network structure with the last stage and fully connected layers removed.

8. The multimodal target tracking system based on template update according to claim 6, characterized in that, The fusion feature determination module specifically includes: The difference feature determination unit is used to obtain difference features based on the color image feature vector and the thermal infrared image feature vector; The difference-complementary feature determination unit is used to obtain the difference-complementary features based on the difference features using the channel attention method; The channel complementary feature determination unit is used to obtain channel complementary features based on the color image feature vector and the thermal infrared image feature vector through the cross-modal feature complementary enhancement module; The fusion feature determination unit is used to obtain fusion features based on the difference complementary features and the channel complementary features.

9. The multimodal target tracking system based on template update according to claim 6, characterized in that, The enhanced feature sequence determination module for the search region specifically includes: The triplet feature sequence determination unit is used to pass the feature sequence through a bottleneck network to obtain a triplet feature sequence with a reduced number of channels. A feature sequence determination unit with a set dimension and a set length is used to expand and cascade the triplet feature sequence after reducing the number of channels along the spatial dimension to generate a feature sequence with a set dimension and a set length. The search region enhancement feature sequence determination unit is used to input the feature sequence of the set dimension and set length into an encoder constructed by n encoder layers to obtain the search region enhancement feature sequence.

10. The multimodal target tracking system based on template update according to claim 6, characterized in that, The bounding box determination module specifically includes: The similarity vector determination unit is used to calculate the similarity vector by using a dot product based on the enhanced feature sequence of the search region and the output embedding; The predictive feature sequence determination unit is used to multiply the similarity vector and the enhanced feature sequence of the search region element by element to obtain the predictive feature sequence; A bounding box determination unit is used to determine the bounding box of the tracked object based on the predicted feature sequence.

Images