Collaborative tracking method of large and small models based on temporal-visual fusion

Abstract

The present invention provides a large and small model collaborative tracking method based on time-vision fusion, which belongs to the fields of machine learning, computer vision, and target tracking. By designing a plug-and-play lightweight small time series model and embedding it into the existing Transformer-based tracking method, the temporal motion information of the encoded target is adaptively interacted with the visual appearance features to achieve robust target tracking in complex scenes. Starting from the temporal coherence unique to video data, the present invention adaptively fuses the timing and visual cues, effectively making up for the inherent limitations of a single visual cue, so that the model can effectively combat background interference, occlusion and other challenges in complex real scenes, and obtain robust tracking performance. The core content of the present invention includes a temporal motion coding module, a visual and motion feature adaptive fusion module, and a prompt learning training strategy.

Description

Technical Field

[0001] The present invention belongs to the fields of machine learning, computer vision, and target tracking, and relates to a motion trajectory modeling algorithm, an adaptive feature fusion algorithm, and a large and small model collaborative prompt learning algorithm; specifically, a large and small model collaborative tracking method based on temporal-visual fusion. Background Art

[0002] Given a video sequence and a target box annotation of a specific target in the initial frame, the target tracking algorithm estimates the position and scale of the target in each subsequent frame by modeling the visual and temporal information of the target and the background. As one of the basic tasks of computer vision, target tracking is the application basis of many computer vision-related problems, such as action recognition, event detection, behavior understanding, and video object detection. In addition, visual tracking has extremely broad practical application prospects in many fields such as medicine, transportation, and entertainment.

[0003] With the development of deep learning technology, target tracking algorithms have gradually transitioned from traditional methods, correlation filtering methods, etc. to methods based on deep networks. In the past two years, thanks to the globality and powerful encoding capabilities of the Transformer network, most of the current mainstream target tracking algorithms are based on the Transformer model. For example, the TransT method uses the attention mechanism to fuse the features of the template and the search area, replacing the cross-correlation operation in the previous method. The STARK method proposes a spatiotemporal tracking framework based on an encoder-decoder Transformer network. The template and search area features extracted by the convolutional network are first concatenated and input into the Transformer encoder as a whole, and then the fusion of the template and search area features is achieved through attention calculation. The above method uses a two-stage tracking framework and relies on the convolutional network to extract image features. Due to the shallow feature fusion process, this hybrid method limits the global advantages of the Transformer network. With the rapid development of large visual models, several methods have successively proposed one-stage tracking algorithms based on pure Transformer networks. For example, the OSTrack method, the Mixformer method, etc. These methods merge the image feature extraction and feature fusion processes through the large visual model ViT to achieve deep fusion between the template and search area features. At the same time, the initialization of model parameters uses a pre-trained large model, such as MAE, which enables this type of tracking method to achieve a leap in performance in large-scale tracking benchmarks. The latest methods attempt to treat target box prediction in target tracking as a sequence generation problem. For example, the SeqTrack method and the ARTrack method both propose sequence autoregressive tracking models, which use the ViT model encoder and the Transformer autoregressive decoder to achieve value-by-value sequential prediction of the sequence (i.e., the target box value). Although these cutting-edge methods have shown relatively accurate tracking performance in existing tracking datasets, due to the inherent limitations of visual information, tracking drift is still prone to occur when faced with complex real-world scenes, and the robustness is relatively weak.

[0004] Since the temporal motion information of the target is usually mixed with observation noise and contains both the target's own motion and the camera's motion, it is difficult for the model to capture and model the target's motion pattern. Therefore, the mainstream tracking method relies solely on the visual model to achieve tracking. However, with the rapid development of visual models, the performance improvement brought by optimizing visual models for tracking problems has gradually reached saturation. Recently, a small number of methods have tried to explore how to effectively use the temporal clues of video data in the tracking framework to further improve the robustness of the algorithm. For example, the STARK method uses a dynamic template to store the multi-frame appearance changes of the target in the inference stage; the SLT method proposes a sequence-level tracking model training strategy based on reinforcement learning, thereby keeping the task objectives and data distribution consistent in the training and inference stages. These two methods target the inference and training processes of the model respectively, and implicitly use temporal clues by additionally encoding multi-frame visual appearance features. The above research work verifies the importance of temporal information to the target tracking problem, but due to the limited computing resources, these methods can only introduce a small number of additional frame images. For example, the SLT method samples 7-10 frames of images in training. This results in very limited target motion clues that the model can refer to. The ARTrack algorithm combines the proposed sequence autoregressive tracking model and regards tracking as a problem of interpreting a coordinate sequence. It realizes trajectory estimation by directly encoding the historical trajectory coordinates, providing a new idea for time series tracking. However, due to the constraints of the coordinate vocabulary capacity and the sequence-level training strategy similar to the SLT method, the amount of time series information that can be encoded by this method is still limited (7 frames of historical trajectory are used in the inference stage). At the same time, the slow reasoning speed and weak generalization of the time series model are also limitations of this method. In view of the limitations of the above-mentioned method, the present invention constructs a lightweight small time series model that can be plug-and-play, and efficiently combines it with a large visual model to realize a time series-visual fusion tracking framework that cooperates with large and small models. Summary of the invention

[0005] The present invention aims to provide a large and small model collaborative tracking method based on temporal-visual fusion. By designing a plug-and-play lightweight small temporal model and embedding it into the existing Transformer-based tracking method, the temporal motion information of the encoded target and the visual appearance features are adaptively interacted to achieve robust target tracking in complex scenes. Starting from the temporal coherence unique to video data, the present invention adaptively fuses temporal and visual cues, effectively making up for the inherent limitations of a single visual cue, so that the model can effectively combat background interference, occlusion and other challenges in complex real scenes, and obtain robust tracking performance. The core content of the present invention includes a temporal motion coding module, a visual and motion feature adaptive fusion module, and a prompt learning training strategy.

[0006] The technical solution of the present invention is:

[0007] The steps of the large and small model collaborative tracking method based on temporal-visual fusion are as follows:

[0008] Step 1: Construct a small temporal model, namely the temporal motion encoding module. Encode the historical motion trajectory information of the target to extract the current temporal motion features of the target. The specific process is: for the historical target frames of the input continuous T frames, each historical target frame is regarded as two different types of corner points, namely the upper left corner point and the lower right corner point. The temporal motion encoding module converts the historical motion trajectory information into 2T motion word vectors with the same dimensions as the visual features through three different types of position encoding mechanisms. The three different types of position encoding mechanisms include corner point encoding, spatial position encoding, and temporal position encoding. The temporal motion encoding module can capture the historical motion trajectory data of the target (i.e., several continuous target frame data) through these three different types of position encoding mechanisms, and subsequently adapt to various motion modes of the target through prompt training and learning, thereby generating temporal motion features containing spatiotemporal correlation information;

[0009] Furthermore, the corner point encoding is used to identify two different types of corner point word vectors; the spatial position encoding enables the motion word vector to correspond one-to-one with the spatial pixels in the visual features; and the temporal position encoding enables the temporal-visual fusion tracking model to understand the temporal relationship of the T-frame historical trajectory.

[0010] In addition, three learnable representative tokens (RT) need to be set, including , and . The first two represent word vectors , They are used to summarize the motion states of the upper left corner and the lower right corner, representing the word vector It is used to learn the confidence of historical motion trajectories and then assign adaptive dynamic weights of visual features and temporal motion features in subsequent stages.

[0011] Furthermore, a nonlinear initialization method is used to initialize the temporal position encoding:

[0012]

[0013] in, and Represent the feature dimension variables and the total dimension respectively, t ∈[0, T -1] Indicates the temporal position of the point in the historical motion trajectory. It means that the trajectory of this frame is closer to the current frame. is set to a constant, and is a hyperparameter used to control the frequency range. Represents the position encoding function.

[0014] Furthermore, the same coordinate mapping mechanism as the spatial position encoding is used to generate a dense spatial position encoding map for the visual feature map. In this way, when the visual features and the temporal motion features are cross-attentionally calculated, their corresponding spatial positions can be matched pixel by pixel.

[0015] Step 2: Construct a fusion decoder to achieve adaptive feature interaction between the small temporal model and the large visual model, and fully integrate the temporal motion features and visual features of the target. The specific process is: the temporal motion features extracted in step 1 and the visual features extracted by the large visual model are input into the fusion decoder together, and the temporal motion features and visual features are fully integrated through multi-layer self-attention and cross-attention interactions. The fused visual features output by the fusion decoder will contain historical motion clues of the target to be tracked, thereby assisting the temporal-visual fusion tracking model to locate the target more accurately, strengthen the target to be tracked itself, and implicitly weaken the influence of similar interferences around it. In addition, an adaptive dynamic weight mechanism is used to make the fused visual features more robust. The adaptive dynamic weight mechanism calculates the weighted average of the above-mentioned output fused visual features and the initial visual features as the final fused features, and uses it as the input of the tracking head in the subsequent tracking process to obtain the final tracking decision.

[0016] Furthermore, the fusion decoder is used to realize visual features and time-series motion characteristics Where b represents the batch size, H, W and C represent the height, width and number of channels of the visual feature map respectively, and the temporal motion feature is represented by the motion word vectors extracted in step 1 and the three representative word vectors, so its number dimension is .

[0017] Furthermore, the fusion decoder includes two Transformer-based model blocks, each of which includes a motion self-attention module, two cross-attention modules with different execution directions, and a fully connected perceptron (MLP). Among them, the motion self-attention module will first mine important motion information from the input motion word vector; the first-layer cross-attention module will then fuse the visual features into the temporal motion features, that is, the temporal motion features Set to the query value (query, q); after being updated by a fully connected perceptron, the three representative word vectors are set as the key value (key, k) and value (value, v) of the second cross-attention module. The second cross-attention module is based on the representative word vectors updated by the previous modules. , and To update the visual features as the query value q ,Through multiple rounds of self-attention and cross-attention learning, fused visual features are obtained.

[0018] Furthermore, the dynamic weight of the weighted average Through a weighted network The weight network is implemented by a fully connected perceptron MLP, and the input is the updated representative word vector output by the fully connected perceptron in the last layer of fusion decoder In addition, a motion network is introduced , which is also implemented by a fully connected perceptron MLP. The task of the motion network is to predict the tracking result of the current frame based on the input motion information. , where the input motion information is the updated representative word vector output by the fully connected perceptron in the last layer of fusion decoder and During the offline training of the temporal-vision fusion tracking model, the tracking result of the current frame The overlap rate IoU will be calculated with the true target box, and the IoU value will be used as the dynamic weight automatically output by the temporal-visual fusion tracking model The training labels of , thus helping the temporal-visual fusion tracking model to provide explicit supervision guidance.

[0019] Step 3: The small temporal model, the large visual model and the fusion decoder constitute a temporal-visual fusion tracking model. During the training of the temporal-visual fusion tracking model, a generalized temporal training sample sampling strategy, a model loss function and a lightweight prompt learning training mechanism are used to achieve efficient model training.

[0020] Among them, for the sampling of time series training samples, the existing tracking method is first used to generate the tracking results of all frames in the tracking training set offline, and then motion trajectory samples of different qualities are generated by designing data enhancement strategies, including sparse sampling, bidirectional sampling, and "copy-paste" image hybrid enhancement strategy.

[0021] For the model loss function, three different loss function terms are used, including baseline tracking loss, motion regression loss, and adaptive weight loss.

[0022] For the prompt learning training mechanism, the existing large vision-based model is used as the visual model and tracking head module, and the parameters of this part are frozen. The other prompt modules in the temporal-visual fusion tracking model: temporal motion encoding module, fusion decoder, weight network and motion network are fine-tuned offline using the existing tracking dataset so that they can quickly adapt to the pre-trained visual model.

[0023] Beneficial effects of the present invention:

[0024] (1) The large and small model collaborative tracking algorithm based on temporal-vision fusion proposed in this invention can be flexibly embedded into the existing vision-based tracking framework. Through efficient prompt learning, a collaborative tracking framework based on temporal-vision adaptive fusion can be quickly realized.

[0025] (2) Since the target's visual and historical motion clues are fully combined in the tracking process, the present invention makes the existing tracking methods more in line with the human tracking paradigm and breaks through the inherent limitations of a single visual clue. It can thus effectively cope with various challenges that may arise in real tracking scenarios, such as interference from similar objects and severe occlusion of objects, thereby greatly improving the robustness and accuracy of tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 1 is a comparison diagram between the existing visual tracking framework and the temporal-visual fusion tracking model framework of the present invention, wherein (a) is the existing visual tracking framework, and (b) is the temporal-visual fusion tracking model of the present invention.

[0027] Figure 2 This is a framework diagram of the large and small model collaborative tracking model based on temporal-visual fusion. DETAILED DESCRIPTION

[0028] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0029] like Figure 1 As shown in the previous vision-based tracking framework ( Figure 1 (a) in the figure only encodes and matches the visual information of the current frame. When there is occlusion or background interference of similar objects, the visual information often fails, resulting in tracking drift failure. The present invention refers to the tracking paradigm of humans combining vision and motion clues and proposes a tracking framework of temporal-visual fusion ( Figure 1 In (b) above, the historical motion trajectory of the target is encoded through a small temporal model, and the visual features are adaptively fused with the temporal motion features through a fusion decoder to obtain comprehensive tracking clues and achieve robust tracking.

[0030] The specific timing-vision fusion tracking model framework is as follows: Figure 2 First, through Figure 2 The temporal motion encoding module shown in (a) encodes the historical motion trajectory information of the target, thereby extracting the current temporal motion features of the target. For the historical target frames of the input continuous T frames, the present invention regards each historical target frame as two different types of corner points, namely the upper left corner point and the lower right corner point. The constructed temporal motion encoding module converts the historical motion trajectory information into 2T motion word vectors with the same dimensions as the visual features through three different types of position encoding mechanisms. Among them, corner point encoding is used to identify two different types of corner point word vectors; spatial position encoding enables the motion word vector to correspond one-to-one with the spatial pixels in the visual features; and temporal position encoding enables the temporal-visual fusion tracking model to understand the temporal relationship of the historical trajectory of T frames. In addition, three additional learnable representative word vectors (Representative Tokens, RT) are designed, including , and The first two representative word vectors are used to summarize the motion status of the upper left corner and the lower right corner respectively, while It is used to learn the confidence of historical motion trajectories and then assign adaptive dynamic weights of visual and temporal motion features in subsequent stages.

[0031] Considering that in a long historical motion trajectory, the trajectory motion information closer to the current frame is more important for predicting the trajectory positioning of the next frame, the present invention designs a nonlinear initialization method to initialize the temporal position encoding, and the nonlinear temporal position encoding can be expressed as follows:

[0032]

[0033] in, and Represent the feature dimension variables and the total dimension respectively, t ∈[0, T -1] Indicates the temporal position of the point in the historical motion trajectory. It means that the trajectory of this frame is closer to the current frame. is set to a constant, and is a hyperparameter used to control the frequency range. Represents the position encoding function. In order to maximize the spatial resolution without causing aliasing effects, according to the Nyquist frequency theorem, we set this hyperparameter It is set to 7.23. In addition, the same coordinate mapping mechanism as the spatial position encoding is used to generate a dense spatial position encoding map for the visual feature map. In this way, when the visual features and the temporal motion features are cross-attentionally calculated, their corresponding spatial positions can be matched pixel by pixel.

[0034] After that, a fusion decoder based on the Transformer network is constructed ( Figure 2 (b) in Figure 2) combines multiple layers of self-attention and cross-attention modules to achieve visual features. and time-series motion characteristics Where b represents the batch size, H, W and C represent the height, width and number of channels of the visual feature map respectively, and the temporal motion feature will be jointly represented by the extracted motion word vectors and the three representative word vectors, so its number dimension is . In order to meet the needs of practical applications and ensure the lightweight of the temporal-visual fusion tracking model, the fusion decoder contains two Transformer-based model blocks. Each block contains a motion self-attention module, two cross-attention modules with different execution directions, and a fully connected perceptron (MLP). Among them, the motion self-attention module will first mine important motion information from the input motion word vector; the first-layer cross-attention module will then fuse the visual features into the temporal motion features, that is, the temporal motion features Set to the query value (query, q); after being updated by a fully connected perceptron, the three representative word vectors RT are set as the key (key, k) and value (value, v) of the second cross-attention module. The second cross-attention module updates the visual features of the query value q based on these representative motion cues. These representative motion prompts include the word vectors updated by the previous modules to summarize the motion status of the upper left corner and the lower right corner. and , and word vectors for learning historical motion trajectory confidence . Through the above-mentioned multiple rounds of self-attention and cross-attention learning, fused visual features are obtained. In addition, considering that the obtained historical motion trajectory of the target is easily mixed with noise generated by tracking failure and irregular camera movement, in order to make the constructed temporal-visual fusion tracking model more robust, the present invention designs an adaptive dynamic weight mechanism to make the fused visual features more robust. The adaptive dynamic weight mechanism calculates the weighted average of the above-mentioned output fused visual features and the initial visual features as the final fused features, and uses it as the input of the tracking head in the subsequent tracking process to obtain the final tracking decision. The adaptive fusion module of the present invention can make a comprehensive judgment on the complementarity and confidence of the visual features and temporal motion features in different scenes, so as to output accurate fused visual features. Dynamic weight of weighted average Through a weighted network The weight network is implemented by a fully connected perceptron MLP, and the input is the updated representative word vector output by the fully connected perceptron in the last layer of fusion decoder In order to explicitly guide the learning of the dynamic weights, an additional motion network is introduced , which is also implemented by a fully connected perceptron MLP. The main task of the motion network is to predict the tracking result of the current frame based on the input motion information. , where the input motion information is the updated representative word vector output by the fully connected perceptron in the last layer of fusion decoder and Indicates. During the offline training of the temporal-vision fusion tracking model, the tracking result of the current frame The overlap rate IoU will be calculated with the true target box, and the IoU value will be used as the dynamic weight automatically output by the temporal-visual fusion tracking model The training labels of , thus helping the temporal-visual fusion tracking model to provide explicit supervision guidance.

[0035] For the training of the temporal-visual fusion tracking model, the present invention adopts an efficient training strategy of prompt learning, and customizes and optimizes the sampling of training samples and the design of loss functions. For the sampling of temporal training samples, the present invention uses the existing tracking method to generate the tracking results of all frames in the tracking training set offline, and generates motion trajectory samples of different qualities by designing data enhancement strategies, including sparse sampling, bidirectional sampling, etc. In order to further enhance the tracking difficulty of the training samples, the sampled search image adopts a "copy-and-paste" image hybrid enhancement strategy (Copy-and-Paste), that is, the targets of different appearances sampled in other frames are copied and pasted to any position of the current search image. This will enable the temporal-visual fusion tracking model to pay more attention to the mining and extraction of target motion clues during the training process, so as to effectively integrate them into the visual features and achieve the research purpose.

[0036] In the design of the model loss function, the present invention designs the following three different loss function terms, so that the model can fully learn to capture the temporal motion characteristics of the target and integrate them with the current visual information. The three different loss function terms include:

[0037] (1) The benchmark tracking loss is expressed as ,in Indicates the tracking header The final output is It represents the true value annotation of the target box to be tracked;

[0038] (2) Motion regression loss: In order to make the representative word vector and The ability to extract important motion patterns is fully learned through the motion network. Set a subtask, namely the target box regression task. By directly predicting the current frame target box and true value supervision, calculate the motion tracking loss, in which the GIoU loss is set to The loss is used to supervise this subtask, which is expressed as follows:

[0039]

[0040] in, represents the motion regression loss, represents the corresponding weight of the overlap loss, represents the overlap loss, represents the squared absolute error loss, represents the corresponding weight of the squared absolute error loss. According to tracking experience, Set to 2, Set to 5;

[0041] (3) Adaptive weight loss: According to the prior empirical assumption, when the motion network Predicted motion tracking results When the quality is low, it usually means that the current motion cue has low confidence in quality and may contain more motion noise. Therefore, in order to make the designed adaptive dynamic weight mechanism more reliable and robust, and the true target box The overlap rate IoU between them is used as the prediction weight The true value label of is explicitly supervised by the mean squared error loss (MSE). The adaptive weight loss is defined as follows:

[0042]

[0043] in, represents the adaptive weight loss, represents the mean square error loss, . Represents the predicted target box and the true target box The overlap rate between .

[0044] Therefore, the overall loss function of the model will contain the above three loss function terms, expressed as follows:

[0045]

[0046] in, Represents the weight value that controls the motion-related loss term.

[0047] Through the efficient parameter fine-tuning mechanism, trajectory sampling strategy and loss function designed in the present invention, the trained tracking model can robustly cope with many interferences that may occur in real complex tracking environments.

[0048] Table 1: Tracking performance of the present invention in various data sets and comparison with other cutting-edge tracking methods

[0049]

[0050] To verify the effectiveness of the present invention, the tracking performance of the present invention is tested in four challenging tracking datasets, including LaSOT, LaSOT EXT , TNL2K and TrackingNet. The evaluation index is the tracking success rate. Table 1 shows the tracking performance of the present invention in each data set and the comparison with other cutting-edge tracking methods. It can be seen that the present invention surpasses other cutting-edge methods in all four data sets and achieves the best performance.

Claims

1. A large and small model collaborative tracking method based on temporal-visual fusion, characterized in that: Here are the steps: Step 1: Build a small temporal model, namely, a temporal motion encoding module; encode the historical motion trajectory information of the target to extract the current temporal motion characteristics of the target; The specific process is as follows: for the historical target frames of the input continuous T frames, each historical target frame is regarded as two different types of corner points, namely the upper left corner point and the lower right corner point; the temporal motion encoding module converts the historical motion trajectory information into 2T motion word vectors with the same dimension as the visual features through three different types of position encoding mechanisms. The three different types of position encoding mechanisms include corner point encoding, spatial position encoding and temporal position encoding; the temporal motion encoding module can capture the historical motion trajectory data of the target through these three different types of position encoding mechanisms, and subsequently adapt to various motion modes of the target through prompt training and learning, thereby generating temporal motion features containing spatiotemporal correlation information; In addition, three learnable representative word vectors need to be set, including , and ; The first two represent word vectors , They are used to summarize the motion states of the upper left corner and the lower right corner, representing the word vector It is used to learn the confidence of historical motion trajectories and then allocate adaptive dynamic weights of visual features and temporal motion features in subsequent stages; Step 2: Build a fusion decoder to achieve adaptive feature interaction between the small temporal model and the large visual model, and fully integrate the temporal motion features and visual features of the target; The specific process is as follows: the temporal motion features extracted in step 1 and the visual features extracted by the large visual model are input into the fusion decoder together. Through multi-layer self-attention and cross-attention interactions, the temporal motion features and visual features are fully integrated. The fused visual features output by the fusion decoder will contain the historical motion clues of the target to be tracked, thereby assisting the temporal-visual fusion tracking model to locate the target more accurately, strengthen the target to be tracked itself, and implicitly weaken the influence of similar interference objects around it. In addition, an adaptive dynamic weight mechanism is used to make the fused visual features more robust. The adaptive dynamic weight mechanism calculates the weighted average of the above-mentioned output fusion visual features and the initial visual features as the final fusion feature, and uses it as the input of the tracking head in the subsequent tracking process to obtain the final tracking decision; Step 3: The small temporal model, the large visual model and the fusion decoder constitute a temporal-visual fusion tracking model. During the training of the temporal-visual fusion tracking model, a generalized temporal training sample sampling strategy, a model loss function and a lightweight prompt learning training mechanism are used to achieve efficient model training.

2. The method for collaborative tracking of large and small models based on temporal-visual fusion according to claim 1 is characterized in that: In step 1, the corner point encoding is used to identify two different types of corner point word vectors; the spatial position encoding enables the motion word vector to correspond one-to-one with the spatial pixels in the visual features; and the temporal position encoding enables the temporal-visual fusion tracking model to understand the temporal relationship of the T-frame historical trajectory.

3. The method for collaborative tracking of large and small models based on temporal-visual fusion according to claim 1 is characterized in that: In step 1, a nonlinear initialization method is used to initialize the temporal position encoding: in, and Represent the feature dimension variables and the total dimension respectively, Indicates the temporal position of the point in the historical motion trajectory; larger This means that the trajectory of this frame is closer to the current frame; is set to a constant, and is a hyperparameter used to control the frequency range. Represents the position encoding function.

4. The method for collaborative tracking of large and small models based on temporal-visual fusion according to claim 1, characterized in that: In the step 1, the same coordinate mapping mechanism as the spatial position encoding is used to generate a dense spatial position encoding map for the visual feature map; in this way, when the visual features and the temporal motion features are cross-attention calculated, their corresponding spatial positions can be matched pixel by pixel.

5. The method for collaborative tracking of large and small models based on temporal-visual fusion according to claim 1, characterized in that: In step 2, the fusion decoder is used to realize visual features and time-series motion characteristics fusion and interaction; where b represents the batch size, H, W and C represent the height, width and number of channels of the visual feature map respectively, and the temporal motion feature is represented by the joint representation of several motion word vectors extracted in step 1 and the three representative word vectors, so its quantity dimension is ; The fusion decoder includes two Transformer-based model unit blocks, each of which includes a motion self-attention module, two cross-attention modules with different execution directions, and a fully connected perceptron; among them, the motion self-attention module will first mine important motion information from the input motion word vector; the first-layer cross-attention module will then fuse the visual features into the temporal motion features, that is, the temporal motion features Set to the query value q; after being updated by a fully connected perceptron, the three representative word vectors are set as the key value k and value v of the second cross-attention module. The second cross-attention module is based on the representative word vectors updated by the previous modules. , and To update the visual features as the query value q ,Through multiple rounds of self-attention and cross-attention learning, fused visual features are obtained.

6. The method for collaborative tracking of large and small models based on temporal-visual fusion according to claim 1, characterized in that: In step 2, the dynamic weight of the weighted average is Through a weighted network Output is obtained; the weight network is implemented by a fully connected perceptron MLP, and the input is the updated representative word vector output by the fully connected perceptron in the last layer of fusion decoder ; In addition, a motion network is introduced , which is also implemented by a fully connected perceptron MLP. The task of the motion network is to predict the tracking result of the current frame based on the input motion information. , where the input motion information is the updated representative word vector output by the fully connected perceptron in the last layer of fusion decoder and ; During the offline training of the temporal-vision fusion tracking model, the tracking result of the current frame The overlap rate IoU will be calculated with the true target box, and the IoU value will be used as the dynamic weight automatically output by the temporal-visual fusion tracking model The training labels of , thus helping the temporal-visual fusion tracking model to provide explicit supervision guidance.

7. The method for collaborative tracking of large and small models based on temporal-visual fusion according to claim 1, characterized in that: In the step 3, for the sampling of time series training samples, the existing tracking method is first used to generate the tracking results of all frames in the tracking training set offline, and motion trajectory samples of different qualities are generated by designing data enhancement strategies, including sparse sampling, bidirectional sampling, and "copy-paste" image hybrid enhancement strategy.

8. The method for collaborative tracking of large and small models based on temporal-visual fusion according to claim 1, characterized in that: In the step 3, three different loss function terms are used for the model loss function, including baseline tracking loss, motion regression loss, and adaptive weight loss.

9. The method for collaborative tracking of large and small models based on temporal-visual fusion according to claim 1, characterized in that: In the step 3, for the prompt learning training mechanism, the existing large vision-based model is used as the visual model and tracking head module, and the parameters of this part are frozen; while the other prompt modules in the temporal-visual fusion tracking model: temporal motion encoding module, fusion decoder, weight network and motion network, use the existing tracking data set for offline fine-tuning, so that it can quickly adapt to the pre-trained visual model.

10. The method for collaborative tracking of large and small models based on temporal-visual fusion according to claim 8, characterized in that: In the model loss function: The baseline tracking loss is expressed as ,in Indicates the tracking header The final output is It represents the true value annotation of the target box to be tracked; The motion regression loss is expressed as follows: in, represents the motion regression loss, represents the corresponding weight of the overlap loss, represents the overlap loss, represents the squared absolute error loss, Represents the corresponding weight of the squared absolute error loss; The adaptive weight loss is defined as follows: in, represents the adaptive weight loss, represents the mean square error loss, . Represents the predicted target box and the true target box The overlap ratio between The model loss function is expressed as follows: in, Represents the weight value that controls the motion-related loss term.

Images