RGB-D target tracking method, system and medium based on time-adaptive bidirectional bridging

Abstract

The present invention discloses an RGB-D target tracking method, system and medium based on time-adaptive bidirectional bridging, which belongs to the field of computer vision technology. The visual Transformer ViT is used to cut RGB modality and depth modality images into image blocks, and the image blocks are embedded in the latent space, and feature extraction is performed; the feature fusion between the RGB modality and the depth modality is performed through the time-adaptive bidirectional bridging module; the time-adaptive bidirectional bridging module includes a pre-fusion module, a cross-attention module and a temporal adaptive branch; the classification result and bounding box position of the target are output through the prediction head, and the fused features are used for classification and bounding box regression to determine the position of the target in the current frame; the prediction head performs model training through a loss function. The present invention realizes the effective fusion of depth and RGB features through the time-adaptive bidirectional bridging module, and dynamically updates the online template image features through the temporal adaptive branch, thereby improving the robustness of the tracking model.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an RGB-D target tracking method, system and medium based on time-adaptive bidirectional bridging. Background Art

[0002] The single target visual tracking task is to accurately track the position of the target in the subsequent frames given the target information in the first frame. Single RGB trackers that use only RGB images as input information can complete some general scene tracking tasks, but they are difficult to handle in special situations such as poor lighting conditions and small differences between foreground and background.

[0003] In order to improve the above problems, researchers proposed an RGB-D multimodal tracker, which can alleviate the unstable performance of traditional single RGB trackers in special scenarios by fusing depth data. The integration of depth data can effectively alleviate the unstable performance of traditional single RGB trackers in special scenarios (background interference, lighting changes, severe occlusion, etc.). Therefore, RGB-D multimodal trackers have broad application scenarios in intelligent robots, autonomous driving, smart security, etc.

[0004] However, existing RGB-D multimodal trackers mostly adopt the method of extracting spatial information from the target template and the search area for appearance feature matching. This method cannot capture the target appearance changes in complex multimodal datasets, resulting in poor tracking results.

[0005] Based on this, the present invention designs an RGB-D target tracking method, system and medium based on time-adaptive bidirectional bridging to solve the above problems. Summary of the invention

[0006] In view of the above shortcomings of the prior art, the present invention provides an RGB-D target tracking method, system and medium based on time-adaptive bidirectional bridging. The time-adaptive bidirectional bridging module effectively integrates the depth modality and RGB modality features, improves the performance of the tracker, and can be used for RGB-D target tracking model tasks in general scenarios.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0008] The RGB-D target tracking method based on time-adaptive bidirectional bridging includes the following steps:

[0009] Step 1: Use the visual Transformer ViT to split the RGB modality and depth modality images into image patches, embed the image patches into the latent space, and perform feature extraction;

[0010] Step 2: feature fusion between RGB modality and depth modality is performed through a time-adaptive bidirectional bridging module; the time-adaptive bidirectional bridging module includes a pre-fusion module, a cross attention module and a temporal adaptive branch; the specific steps are as follows:

[0011] Step 2.1: The pre-fusion module pre-fuses the initial features of the RGB and depth modalities through a linear layer to generate template fusion features;

[0012] Step 2.2: The cross attention module uses the cross attention mechanism to perform feature matching and fusion on the RGB modality and the depth modality to obtain the fusion feature;

[0013] Step 2.3: The temporal adaptation branch dynamically updates the online template image features of the RGB modality and the online template image features of the depth modality according to the changes of the target in the temporal dimension, and dynamically adjusts and optimizes the initial template image features of the RGB modality and the initial template image features of the depth modality to capture the appearance changes of the target;

[0014] Step 3: Prediction head design: output the classification result and bounding box position of the target through the prediction head, and use the fusion features for classification and bounding box regression to determine the position of the target in the current frame; the prediction head uses the cross entropy loss function, L1 loss function and GIoU loss function to train the model and optimize the classification result and bounding box regression result.

[0015] Furthermore, the specific steps of step one are:

[0016] Step 1.1. Input bimodal data: Input the search area image, initial template image and online template image of RGB modality and depth modality into the visual Transformer ViT; the visual Transformer ViT divides each image into P×P image blocks and embeds the image blocks into the latent space through a linear layer to form an initial feature vector;

[0017] Step 1.2, feature extraction: Use the Transformer block with shared parameters to extract features from the initial feature vector to obtain initial image features; the initial features include search area image features of the RGB modality, initial template image features of the RGB modality, online template image features of the RGB modality, search area image features of the depth modality, initial template image features of the depth modality, and online template image features of the depth modality.

[0018] Furthermore, step 2.2 is specifically as follows: first, the template area image features of the RGB modality are used as queries, and the template fusion features of the RGB modality and the depth modality are used as keys and values ​​to perform cross-attention calculations to obtain further fusion features; then, the search area image features of the depth modality are used as queries, and the online template image features of the RGB modality and the depth modality are used as keys and values ​​to perform cross-attention calculations to further enhance the feature interaction of the two modalities; then, the respective queries and keys and values ​​are exchanged to perform symmetrical modal feature interaction.

[0019] Furthermore, time-adaptive bidirectional bridge modules are inserted into the 4th, 7th and 10th layers of the Transformer block, respectively.

[0020] Furthermore, the prediction head design includes the following: The fused features are passed through three parallel prediction sub-networks (each prediction sub-network consists of four convolutional layers) to extract task-specific feature representations, which are used to generate classification score maps, bounding box sizes, and offsets, respectively, to accurately predict the location and category of the target.

[0021] Furthermore, the temporal adaptation branch in step 2.3 updates the online template image features of the RGB modality and the online template image features of the depth modality through dynamic threshold selection. The specific steps are as follows:

[0022] (1) Set a dynamic threshold to determine whether the online template image features of the RGB modality and the online template image features of the depth modality need to be updated based on the classification score and bounding box regression results of the current frame;

[0023] (2) If the classification score of the current frame exceeds the set threshold, the target features of the current frame are used as the new online template image features of the RGB modality and the online template image features of the depth modality.

[0024] In order to better achieve the purpose of the present invention, the present invention also provides an RGB-D target tracking system based on time adaptive bidirectional bridging, comprising:

[0025] Visual Transformer ViT is used to divide RGB modality and depth modality images into image blocks, embed the image blocks into the latent space, and extract features to obtain initial features; the initial features include search area image features of RGB modality, initial template image features of RGB modality, online template image features of RGB modality, search area image features of depth modality, initial template image features of depth modality, and online template image features of depth modality;

[0026] A time-adaptive bidirectional bridging module, used for feature fusion between RGB modality and depth modality; the time-adaptive bidirectional bridging module includes a pre-fusion module, a cross-attention module and a temporal adaptation branch;

[0027] The pre-fusion module is used to pre-fuse the initial features of the RGB and depth modalities through a linear layer to generate template fusion features; the cross-attention module uses a cross-attention mechanism to match and fuse the features of the RGB modality and the depth modality to obtain fusion features; the temporal adaptation branch dynamically updates the online template image features of the RGB modality and the online template image features of the depth modality according to the changes of the target in the time dimension, and dynamically adjusts and optimizes the initial template image features of the RGB modality and the initial template image features of the depth modality to capture the appearance changes of the target;

[0028] The prediction head is used to output the classification results and bounding box position of the target, and uses the fused features for classification and bounding box regression to determine the position of the target in the current frame; the prediction head uses the cross entropy loss function, L1 loss function and GIoU loss function to train the model and optimize the classification results and bounding box regression results.

[0029] Furthermore, the search area images, initial template images and online template images of the RGB modality and depth modality are input into the visual Transformer ViT; the visual Transformer ViT divides each image into P×P image blocks, and embeds the image blocks into the latent space through a linear layer to form an initial feature vector; the initial feature vector is extracted using the Transformer block with shared parameters to obtain the initial image features.

[0030] Furthermore, the temporal adaptation branch updates the online template image features of the RGB modality and the online template image features of the depth modality through dynamic threshold selection. The specific steps are as follows:

[0031] (1) Set a dynamic threshold to determine whether the online template image features of the RGB modality and the online template image features of the depth modality need to be updated based on the classification score and bounding box regression results of the current frame;

[0032] (2) If the classification score of the current frame exceeds the set threshold, the target features of the current frame are used as the new online template image features of the RGB modality and the online template image features of the depth modality.

[0033] In order to better achieve the purpose of the present invention, the present invention also provides a computer storage medium, which is used to store program data. When the program data is executed by a computer, it is used to implement the RGB-D target tracking method based on time adaptive bidirectional bridging.

[0034] Compared with the prior art, the present invention has the following beneficial effects: the present invention sets a time-series adaptive branch to dynamically update the online template image features, and by setting a dynamic threshold, determines whether the online template image features need to be updated according to the classification score and bounding box regression result of the current frame, thereby ensuring the timeliness and accuracy of the template information.

[0035] The present invention realizes the effective fusion of depth and RGB features through a time-adaptive bidirectional bridging module, and dynamically updates the online template image features through a temporal adaptive branch, thereby improving the robustness of the tracking model.

[0036] Experimental results show that the method of the present invention performs well in multiple benchmark tests and is superior to the existing technology in terms of performance, speed and generalization ability, meets the requirements of real-time tracking, and has good versatility and transferability. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 It is a flow chart of the RGB-D target tracking method based on time adaptive bidirectional bridging of the present invention;

[0039] Figure 2 It is the principle diagram of the time adaptive bidirectional bridge module;

[0040] Figure 3 The following are visualization results of RGB target tracking; (a) is the visualization result of target tracking of the charger, (b) is the visualization result of target tracking of the red backpack, and (c) is the visualization result of target tracking of the cup.

[0041] Figure 4 It is a heat map after the interaction of bimodal features; (a) represents the search area of ​​RGB image, (b) represents the corresponding heat map of RGB image, (c) represents the search area of ​​depth image, and (d) represents the corresponding heat map of depth image;

[0042] Figure 5 Schematic diagram of updating the online template image features through dynamic threshold selection for the temporal adaptive branch;

[0043] Figure 6 Schematic diagram of the attention changes of the time-adaptive bidirectional bridge module layer by layer. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] Embodiment 1: In some embodiments, please refer to the drawings of the specification Figure 1 ,The RGB-D object tracking method based on temporal adaptive bidirectional bridging, which aims to improve the performance of RGB-D object trackers, is named TABBTrack and includes the following steps:

[0046] Step 1: Use the visual backbone network - Visual Transformer (ViT, an attention-based encoder-decoder model, originally used in the field of natural language processing, and recently extended to computer vision tasks with good results) to split the RGB modality and depth modality images into image patches, embed the image patches into the latent space, and perform feature extraction;

[0047] The specific steps are:

[0048] Step 1.1. Input bimodal data: Input the search region image (SearchRegion), initial template image (Initial Template) and online template image (Online Template) of RGB modality and depth modality into the visual transformer (ViT); the visual transformer (ViT) divides each image into P×P image patches and embeds the image patches into the latent space through a linear layer to form an initial feature vector (Patch Embedding);

[0049] Step 1.2, feature extraction: Use the Transformer Block with shared parameters to extract features from the initial feature vector to obtain the initial image features; each Transformer block includes a multi-head self-attention mechanism and a feedforward neural network, and maintains gradient and numerical stability through residual connections and layer normalization;

[0050] The initial features include search area image features of RGB modality, initial template image features of RGB modality, online template image features of RGB modality, search area image features of depth modality, initial template image features of depth modality, and online template image features of depth modality;

[0051] Step 2: Perform cross-modal feature interaction through the feature mixing module, the time-adaptive bidirectional bridging module; the time-adaptive bidirectional bridging module includes a pre-fusion module, a cross-attention module, and a temporal adaptive branch, which can achieve efficient information fusion between the RGB modality and the depth modality. The specific steps are as follows:

[0052] Step 2.1: The pre-fusion module pre-fuses the initial features of the RGB and depth modalities through a simple linear layer to generate template fusion features;

[0053] Specifically, the initial template image features of the RGB modality in the initial features are pre-fused with the initial template image features of the depth modality through a linear layer; and the online template image features of the RGB modality in the initial features are pre-fused with the online template image features of the depth modality through a linear layer;

[0054] Step 2.2, Cross-Attention Module, see Figure 2 It uses a cross-attention mechanism to match and fuse the RGB modality and the depth modality to obtain fused features. The Transformer module uses the most basic structure. The image will go through the first layer normalization (LayerNorm), the multi-head self attention layer (Multi-Head Self Attention), the second layer normalization and the multi-layer perceptron (Multi-Layer Perceptron) in sequence.

[0055] Specifically, first, the template area image features of the RGB modality are used as queries, and the template fusion features of the RGB modality and the depth modality are used as keys and values, and cross-attention calculations are performed to obtain further fusion features; then, the search area image features of the depth modality are used as queries, and the online template image features of the RGB modality and the depth modality are used as keys and values, and cross-attention calculations are performed to further enhance the feature interaction of the two modalities; then, the respective queries and keys and values ​​are exchanged to perform symmetrical modal feature interaction.

[0056] Step 2.3, the temporal adaptation branch dynamically updates the online template image features of the RGB modality and the online template image features of the depth modality according to the changes of the target in the temporal dimension, and dynamically adjusts and optimizes the initial template image features of the RGB modality and the initial template image features of the depth modality to capture the appearance changes of the target.

[0057] A time-adaptive bidirectional bridge module is inserted after each Transformer block to achieve cross-modal feature interaction at multiple levels.

[0058] Specifically, time-adaptive bidirectional bridging modules are inserted into the 4th, 7th and 10th layers of the Transformer block, respectively, to ensure effective cross-modal information fusion in feature representations of different depths.

[0059] Figure 4 This is the heat map after the interaction of dual-modal features. It can be seen that even in difficult scenes with similar backgrounds or low light, the attention heat map generated by the RGB modality and the attention heat map generated by the depth modality are both very concentrated around the tracked target, which is enough to show the effectiveness of this design.

[0060] Step 3: Prediction head design: The prediction head outputs the classification result and bounding box position of the target, and uses the fusion features for classification and bounding box regression to determine the position of the target in the current frame. The prediction head uses the cross entropy loss function, L1 loss function, and GIoU loss function to train the model, optimize the classification result and bounding box regression result, and improve the accuracy and robustness of target positioning.

[0061] Among them, the prediction head design: the fusion features are extracted through three parallel prediction sub-networks (each prediction sub-network consists of four convolutional layers) to extract task-specific feature representations, which are used to generate classification score maps, bounding box sizes and offsets respectively to accurately predict the location and category of the target.

[0062] Among them, the cross entropy loss function is used to process classification tasks, and the classification results are optimized by calculating the cross entropy between the predicted category and the true category.

[0063] The L1 loss function is used to process the bounding box regression task, which optimizes the bounding box regression result by calculating the absolute difference between the predicted bounding box and the true bounding box.

[0064] The GIoU loss function is used to further optimize the bounding box regression results by calculating the generalized intersection over union (GIoU) ​​to improve the accuracy and robustness of bounding box positioning.

[0065] Preferably, the temporal adaptive branch in step 2.3 updates the online template image features (the online template image features of the RGB modality and the online template image features of the depth modality) by dynamic threshold selection, specifically including:

[0066] ① Set a dynamic threshold to determine whether the online template image features need to be updated based on the classification score and bounding box regression results of the current frame;

[0067] ② If the classification score of the current frame exceeds the set threshold, the image features of the predicted area of ​​the current frame are used as new online template image features (as new online template image features of the RGB modality and online template image features of the depth modality) to replace the old online template image features, thereby ensuring the timeliness and accuracy of the online template information.

[0068] Figure 5 The schematic diagram of updating the online template image features by the temporal adaptive branch through dynamic threshold selection shows that the low-complexity online template selection method of the present invention can select online templates with positive feature guidance for the subsequent tracking process in complex scenes. The features of online templates of different modalities can enhance the discrimination ability of the model in challenging scenes.

[0069] Figure 6 This is a schematic diagram of the attention changes of the time-adaptive bidirectional bridge module layer by layer. It can be seen that as the number of layers of the image passing through the time-adaptive bidirectional bridge module increases, the attention response value output by the model becomes more significant and the target attention boundary becomes clearer, indicating that the module helps the classification and regression decisions of the tracker.

[0070] The present invention sets a time-series adaptive branch to dynamically update the online template image features. By setting a dynamic threshold, it is determined whether the online template image features need to be updated according to the classification score and bounding box regression result of the current frame, thereby ensuring the timeliness and accuracy of the template information.

[0071] In practical applications, the frequency of updating online template image features is adjusted to adapt to different application scenarios and tracking requirements, including:

[0072] According to the specific scenario, the update interval of the online template image feature is set to every 50 frames to ensure that the template can be adjusted in time when the target appearance changes rapidly, thereby improving the robustness of tracking.

[0073] The present invention realizes the effective fusion of depth and RGB features through a time-adaptive bidirectional bridging module, and dynamically updates the online template image features through a temporal adaptive branch, thereby improving the robustness of the tracking model.

[0074] Embodiment 2: In some embodiments, an RGB-D target tracking system based on time adaptive bidirectional bridging includes:

[0075] The visual backbone network, Visual Transformer (ViT), is used to split RGB modality and depth modality images into image patches, embed the image patches into latent space, and perform feature extraction;

[0076] Specifically, the visual transformer (ViT) inputs the search region image (SearchRegion) of the RGB modality and the depth modality, the initial template image (Initial Template) and the online template image (Online Template) into the visual transformer (ViT); the visual transformer (ViT) divides each image into P×P image blocks, and embeds the image blocks into the latent space through a linear layer to form an initial feature vector (Patch Embedding); the initial feature vector is extracted using a Transformer block (Transformer Block) with shared parameters to obtain the initial image features; each Transformer block includes a multi-head self-attention mechanism and a feedforward neural network, and maintains gradient and numerical stability through residual connections and layer normalization; the initial features include the search region image features of the RGB modality, the initial template image features of the RGB modality, the online template image features of the RGB modality, the search region image features of the depth modality, the initial template image features of the depth modality, and the online template image features of the depth modality;

[0077] The feature mixing module - the time-adaptive bidirectional bridging module is used for cross-modal feature interaction; the time-adaptive bidirectional bridging module includes a pre-fusion module, a cross-attention module and a temporal adaptive branch, which can realize efficient information fusion between RGB modality and depth modality.

[0078] Among them, the pre-fusion module is used to pre-fuse the initial features of the RGB and depth modalities through a linear layer to generate a template fusion feature; specifically, the initial template image features of the RGB modality in the initial features are pre-fused with the initial template image features of the depth modality through a linear layer; and the online template image features of the RGB modality in the initial features are pre-fused with the online template image features of the depth modality through a linear layer;

[0079] Among them, the cross attention module adopts the cross attention mechanism to match and fuse the features of RGB modality and depth modality to obtain fusion features;

[0080] Specifically, first, the template area image features of the RGB modality are used as queries, and the template fusion features of the RGB modality and the depth modality are used as keys and values, and cross-attention calculations (matrix multiplication, elementwise addition) are performed to obtain further fusion features; then, the search area image features of the depth modality are used as queries, and the online template image features of the RGB modality and the depth modality are used as keys and values, and cross-attention calculations are performed to further enhance the feature interaction of the two modalities; then, the respective queries and keys and values ​​are exchanged to perform symmetric modality feature interaction.

[0081] Among them, the temporal adaptation branch dynamically updates the online template image features of the RGB modality and the online template image features of the depth modality according to the changes of the target in the temporal dimension, and dynamically adjusts and optimizes the initial template image features of the RGB modality and the initial template image features of the depth modality to capture the appearance changes of the target.

[0082] The prediction head is used to output the classification results and bounding box position of the target, and uses the fusion features for classification and bounding box regression to determine the position of the target in the current frame. The prediction head uses the cross entropy loss function, L1 loss function and GIoU loss function to train the model, optimize the classification results and bounding box regression results, and improve the accuracy and robustness of target positioning.

[0083] Among them, the prediction head design: the fusion features are extracted through three parallel prediction sub-networks (each prediction sub-network consists of four convolutional layers) to extract task-specific feature representations, which are used to generate classification score maps, bounding box sizes and offsets respectively to accurately predict the location and category of the target.

[0084] Among them, the cross entropy loss function is used to process classification tasks, and optimizes the classification results by calculating the cross entropy between the predicted category and the true category. The L1 loss function is used to process the bounding box regression task, and optimizes the bounding box regression results by calculating the absolute difference between the predicted bounding box and the true bounding box. The GIoU loss function is used to further optimize the bounding box regression results, and improves the accuracy and robustness of bounding box positioning by calculating the generalized intersection over union.

[0085] Preferably, the temporal adaptive branch updates the online template image features (the online template image features of the RGB modality and the online template image features of the depth modality) by dynamic threshold selection, specifically including:

[0086] ① Set a dynamic threshold to determine whether the online template image features need to be updated based on the classification score and bounding box regression results of the current frame;

[0087] ② If the classification score of the current frame exceeds the set threshold, the image features of the predicted area of ​​the current frame are used as new online template image features to replace the old online template image features, thereby ensuring the timeliness and accuracy of the online template information.

[0088] The present invention sets a time-series adaptive branch to dynamically update the online template image features. By setting a dynamic threshold, it is determined whether the online template image features need to be updated according to the classification score and bounding box regression result of the current frame, thereby ensuring the timeliness and accuracy of the template information.

[0089] The present invention realizes the effective fusion of depth and RGB features through a time-adaptive bidirectional bridging module, and dynamically updates the online template image features through a temporal adaptive branch, thereby improving the robustness of the tracking model.

[0090] Embodiment 3: In some embodiments, a computer storage medium is used to store program data, and when the program data is executed by a computer, it is used to implement the RGB-D target tracking method based on time adaptive bidirectional bridging.

[0091] Experimental Example: The performance of the proposed method (TABBTrack) is compared with other RGB object tracking methods. The results are shown in Table 1.

[0092] Table 1 Performance comparison results of the proposed method and other RGB object tracking methods on different test sets

[0093]

[0094] It can be seen from Table 1 that the proposed method is significantly better than other RGB object tracking methods, indicating that it is more robust for long-term tracking in complex scenes.

[0095] Figure 3 This is the RGB-D target tracking visualization result diagram of the present invention. It can be seen that Figure 3 In (a), the color of the tracked object is similar to the background, and most trackers have difficulty estimating its size correctly after the target reappears. However, the tracker of the present invention has better size estimation ability. Figure 3 In (b), the search area contains multiple similar objects that cause interference, which poses a significant challenge to tracking when the target is occluded. The temporal adaptive branch of the present invention can use the online updated template to distinguish the correct target object, and can effectively correct the tracking result once the target reappears.

[0096] Meanwhile, under low-light conditions, the traditional RGB tracker exhibits poor robustness, while the tracker of the present invention successfully completes tracking under extreme conditions through effective cross-modal feature interaction.

[0097] Experimental results show that the method of the present invention performs well in multiple benchmark tests and is superior to the existing technology in terms of performance, speed and generalization ability, meets the requirements of real-time tracking, and has good versatility and transferability.

[0098] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An RGB-D target tracking method based on time-adaptive bidirectional bridging, characterized in that: The following steps are involved: Step 1: Use the visual Transformer ViT to split the RGB modality and depth modality images into image patches, embed the image patches into the latent space, and perform feature extraction; Step 2: Perform feature fusion between RGB modality and depth modality through a time-adaptive bidirectional bridging module; the time-adaptive bidirectional bridging module includes a pre-fusion module, a cross-attention module and a temporal adaptive branch; The specific steps are as follows: Step 2.1: The pre-fusion module pre-fuses the initial features of the RGB and depth modalities through a linear layer to generate template fusion features; Step 2.2: The cross attention module uses the cross attention mechanism to perform feature matching and fusion on the RGB modality and the depth modality to obtain fusion features; Step 2.3: The temporal adaptation branch dynamically updates the online template image features of the RGB modality and the online template image features of the depth modality according to the changes of the target in the temporal dimension, and dynamically adjusts and optimizes the initial template image features of the RGB modality and the initial template image features of the depth modality to capture the appearance changes of the target; Step 3: Prediction head design: output the classification result and bounding box position of the target through the prediction head, and use the fusion features for classification and bounding box regression to determine the position of the target in the current frame; the prediction head uses the cross entropy loss function, L1 loss function and GIoU loss function to train the model and optimize the classification result and bounding box regression result.

2. The RGB-D target tracking method based on time adaptive bidirectional bridging according to claim 1 is characterized in that: The specific steps of step one are: Step 1.1: Input bimodal data: Input the search area image, initial template image and online template image of RGB modality and depth modality into the visual Transformer ViT; The visual transformer ViT divides each image into P×P image blocks and embeds the image blocks into the latent space through a linear layer to form an initial feature vector; Step 1.2, feature extraction: Use the Transformer block with shared parameters to extract features from the initial feature vector to obtain initial image features; the initial features include search area image features of the RGB modality, initial template image features of the RGB modality, online template image features of the RGB modality, search area image features of the depth modality, initial template image features of the depth modality, and online template image features of the depth modality.

3. The RGB-D target tracking method based on time adaptive bidirectional bridging according to claim 2 is characterized in that: Step 2.2 is as follows: first, the image features of the template area in the RGB modality are used as the query, and the template fusion features of the RGB modality and the depth modality are used as the key and value to perform cross-attention calculation to obtain further fusion features; Then, the search area image features of the depth modality are used as queries, and the online template image features of the RGB modality and the depth modality are used as keys and values ​​to perform cross-attention calculations to further enhance the feature interaction of the two modalities. Then, the respective queries, keys, and values ​​are swapped to perform symmetrical modality feature interaction.

4. The RGB-D target tracking method based on time adaptive bidirectional bridging according to claim 3 is characterized in that: Temporal adaptive bidirectional bridge modules are inserted into the 4th, 7th and 10th layers of the Transformer block, respectively.

5. The RGB-D target tracking method based on time adaptive bidirectional bridging according to claim 4 is characterized in that: The prediction head design includes the following: The fused features are passed through three parallel prediction sub-networks to extract task-specific feature representations, which are used to generate classification score maps, bounding box sizes, and offsets, respectively.

6. The RGB-D target tracking method based on time adaptive bidirectional bridging according to claim 5 is characterized in that: The temporal adaptive branch in step 2.3 updates the online template image features of the RGB modality and the online template image features of the depth modality through dynamic threshold selection. The specific steps are as follows: (1) Set a dynamic threshold to determine whether the online template image features of the RGB modality and the online template image features of the depth modality need to be updated based on the classification score and bounding box regression results of the current frame; (2) If the classification score of the current frame exceeds the set threshold, the target features of the current frame are used as the new online template image features of the RGB modality and the online template image features of the depth modality.

7. An RGB-D target tracking system based on time-adaptive bidirectional bridging, characterized in that: include: Visual Transformer ViT, which is used to split RGB modality and depth modality images into image patches, embed the image patches into the latent space, and perform feature extraction to obtain initial features; The initial features include search area image features of RGB modality, initial template image features of RGB modality, online template image features of RGB modality, search area image features of depth modality, initial template image features of depth modality, and online template image features of depth modality; A time-adaptive bidirectional bridging module, used for feature fusion between RGB modality and depth modality; the time-adaptive bidirectional bridging module includes a pre-fusion module, a cross-attention module and a temporal adaptation branch; The pre-fusion module is used to pre-fuse the initial features of the two modalities, RGB and depth, through a linear layer to generate template fusion features; The cross attention module uses the cross attention mechanism to match and fuse the features of the RGB modality and the depth modality to obtain the fusion features; The temporal adaptation branch dynamically updates the online template image features of the RGB modality and the online template image features of the depth modality according to the changes of the target in the temporal dimension, and dynamically adjusts and optimizes the initial template image features of the RGB modality and the initial template image features of the depth modality to capture the appearance changes of the target; The prediction head is used to output the classification results and bounding box position of the target, and uses the fused features for classification and bounding box regression to determine the position of the target in the current frame; the prediction head uses the cross entropy loss function, L1 loss function and GIoU loss function to train the model and optimize the classification results and bounding box regression results.

8. The RGB-D target tracking system based on time adaptive bidirectional bridging according to claim 7, characterized in that: Input the search area image, initial template image and online template image of RGB modality and depth modality into the visual Transformer ViT; The visual Transformer ViT divides each image into P×P image blocks and embeds the image blocks into the latent space through a linear layer to form an initial feature vector; the initial feature vector is extracted using the Transformer block with shared parameters to obtain the initial image features.

9. The RGB-D target tracking system based on time adaptive bidirectional bridging according to claim 8, characterized in that: The temporal adaptive branch updates the online template image features of the RGB modality and the online template image features of the depth modality through dynamic threshold selection. The specific steps are as follows: (1) Set a dynamic threshold to determine whether the online template image features of the RGB modality and the online template image features of the depth modality need to be updated based on the classification score and bounding box regression results of the current frame; (2) If the classification score of the current frame exceeds the set threshold, the target features of the current frame are used as the new online template image features of the RGB modality and the online template image features of the depth modality.

10. A computer storage medium, characterized in that: The computer storage medium is used to store program data, and when the program data is executed by a computer, it is used to implement the RGB-D target tracking method based on time adaptive bidirectional bridging according to any one of claims 1 to 6.

Images