A spatiotemporal consistency data generation method for visual target tracking

Abstract

The present application relates to the technical field of computer vision, and particularly relates to a spatiotemporal consistency data generation method for visual target tracking. Firstly, the present application trains a path generator on a target tracking training set, learns the motion law of the target in a time sequence by using optical flow estimation and conditional variational encoding technology, and generates a target motion trajectory conforming to physical constraints. Then, based on the generated target trajectory, a spatiotemporal consistency attention mechanism is introduced to guide a text-video generation model, and under the condition of keeping the parameters of the basic model frozen, the position, scale and continuity of the target in the generated frame are constrained by an attention network, so as to synthesize a video frame sequence with real motion characteristics. The present application generates target tracking video data with real motion characteristics and high temporal consistency, and can improve the robustness of the model to complex motion, light changes and occlusion conditions in different scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a spatiotemporal consistency data generation method for visual target tracking. BACKGROUND

[0002] Existing visual target tracking algorithms, especially twin networks and Transformer tracking networks based on deep learning, rely on large-scale labeled video samples for training. However, the acquisition cost of real labeled data is high, and the data distribution often cannot cover complex scenarios such as rapid motion, long-term occlusion, dramatic changes in perspective, low illumination, and similar backgrounds, which limits the generalization ability of the model. Although traditional data enhancement methods (such as translation, scaling, flipping, and color perturbation) can expand the data to some extent, they cannot maintain the continuity and physical consistency of video sequences in the temporal dimension, which can easily cause the loss of temporal information and affect the modeling ability of the tracking model for dynamic changes. Therefore, there is an urgent need for a technical solution that can automatically generate diverse training data while ensuring temporal consistency and the authenticity of spatial features to improve the performance and stability of the target tracking algorithm.

[0003] In the target generation process, although existing text-to-video generation models can synthesize video content based on text descriptions, the generated targets often have problems such as jitter, discontinuity, and unstable scale in the temporal dimension, which cannot meet the demand for high temporal consistency and accurate target positioning in the target tracking task. Especially when generating data for training the tracking model, if the inter-frame motion trajectory does not conform to the real physical law, it will seriously affect the reliability and accuracy of the downstream tracking algorithm.

[0004] Therefore, how to design a target tracking data generation method that can ensure both the physical reasonableness of target motion and the temporal consistency without relying on real shooting videos has become a key problem that needs to be solved in the field of target tracking data enhancement and model training. The present application proposes a spatiotemporal consistency data generation method for target tracking, which effectively improves the physical consistency and temporal continuity of the generated data by combining path generation and spatiotemporal consistency guidance mechanisms, and provides high-quality training sample support for target tracking models. SUMMARY

[0005] The present application provides a spatiotemporal consistency data generation method for visual target tracking to overcome the shortcomings of the prior art.

[0006] The present application is implemented by the following technical solutions:

[0007] A spatiotemporal consistency data generation method for visual target tracking, comprising the following steps:

[0008] Step one, target path generation

[0009] Generate target motion trajectory by using conditional variational autoencoder VAE combined with optical flow consistency constraint;

[0010] Step 2, video frame generation based on spatio-temporal consistency constraint

[0011] Under the condition of freezing the parameters of the text-video generation model, introduce the target motion attention module to modify the spatio-temporal mixed attention; in space, construct a Gaussian mask according to the target box to highlight the foreground area; in time, adjust the weight combined with the inter-frame distance to obtain the spatio-temporal attention, and use 2D Gaussian to form a spatial mask with the box center and size; then modify the original attention based on the spatio-temporal attention, and ensure the inter-frame visual and motion continuity in training through content consistency loss and motion consistency loss;

[0012] Step 3, generate target motion video

[0013] In the inference stage, input the motion trajectory and category text generated in step 1 into the text-video generation model introduced in step 2 based on attention spatio-temporal consistency image generation guide mechanism, output target motion video with high spatial precision and time sequence coherence, and realize controllable and realistic target tracking data generation.

[0014] Further, in step 1, the process of generating target motion trajectory by using conditional variational autoencoder VAE combined with optical flow consistency constraint is as follows:

[0015] In the training stage, input the sequence of continuous video frames and the corresponding target position , wherein represents the t-th image with image size , and the sequence of tracking target positions in the image is ;

[0016] Extract the dense optical flow field between adjacent frames by the optical flow estimation network RAFT: , wherein represents the optical flow estimation network, and are adjacent image frames, and the output is a two-dimensional displacement vector of each pixel between adjacent frames;

[0017] Subsequently, input the tracking position trajectory and the optical flow field output to the conditional variational encoder for path analysis: path encoder Project the tracking position trajectory and the optical flow field into the latent space to generate the motion distribution parameters of the latent variable z: , and represent the mean and covariance of latent variable z respectively, represent the encoder, decoder generate the next frame prediction trajectory under the optical flow condition , then the path decoder with latent variable and optical flow / environmental conditions as input, reconstruct or predict the future trajectory ;

[0018] In training, the trajectory reconstruction loss, optical flow consistency loss and scale normalization loss are used to constrain the spatial accuracy, motion direction and target box size of the generated trajectory; in the inference stage, the initial position of the target and the class description Class are input, combined with the historical optical flow to recursively predict the future trajectory , and the complete motion path is obtained .

[0019] Further, in step 1, the conditional variational encoder is trained using multiple losses, including trajectory reconstruction loss, optical flow consistency loss, and scale normalization loss. The trajectory reconstruction loss is calculated as , which is used to constrain the difference in spatial position between the generated trajectory and the true trajectory , represents the length of the time step involved in trajectory prediction or reconstruction, represents the Euclidean norm; optical flow consistency error , where the forward optical flow is , the backward optical flow is , and the forward-backward consistency loss error is calculated as: , is the pixel value of the image. This loss minimization can constrain the local motion direction of the generated path to be consistent with the optical flow estimation result, so that the trajectory conforms to the dynamic characteristics of the scene. In addition, the size normalization loss prevents too large or too small trajectory target boxes from affecting trajectory calculation. The calculation method is as follows: , where and are the width and height of the target position box, and are the width and height of the target position box in the generated trajectory , and the overall optimization objective of the model is represented as: , where , and are the weight coefficients of each loss, is the content consistency loss, for motion consistency loss.

[0020] Further, in step 1, in the reasoning phase, given a frame containing the initial image of the tracking target , the target recognition model or manual annotation method is used to mark the category of the tracking target in the image and the initial position , where , are the horizontal and vertical coordinates of the center of the target bounding box in the pixel coordinate system, respectively represent the width and height of the target bounding box, and the initial position is used as the starting state of the path generator for subsequent recursive prediction of the target motion trajectory. Then, the optical flow estimation network is used to calculate the optical flow information between adjacent frames, and to capture the optical flow sequence in the historical time interval : where represents the dense optical flow field from frame t to frame t+1, and based on the optical flow sequence, the optical flow change in the time range is regarded as the time sequence representation of the target and its local neighborhood motion pattern, which is used to characterize the consistency and evolution trend of pixel-level motion in the time dimension; the input sequence of the path generator is constructed by fusing the target bounding box and the optical flow information: where represents the known or observed target position in the historical stage, and the first frame is the given initial position and the other frames are the predicted target positions; the input sequence is input into the path encoder to extract the joint representation of the time sequence motion and spatial position information, and the path decoder generates the future target position step by step in the autoregressive framework, specifically, in the prediction phase, the target position at time t+k is obtained by the following calculation method:

[0021] ;

[0022] where is the future target position predicted by the model, and through the autoregressive method, the model can utilize the historical observation information while feeding back the prediction result of the previous time step to the next prediction process, thereby effectively describing the continuity and dependency of the target motion trajectory in the time dimension. Finally, the target motion sequence is constructed based on the predicted target position.

[0023] Further, in step 2, the spatio-temporal attention is: ;

[0024] wherein is the distance of the time frame from the current frame, is the position of the tracking target in the image according to the tracking target position generated spatial mask, is the motion attention of the target.

[0025] Further, a spatio-temporal consistency image generation guiding mechanism based on attention is designed in step 2. Without modifying the parameters of the text-video generation model, a target motion attention network is trained using the target tracking training set to learn the motion characteristics of the tracking target in the spatial-single frame image and the temporal-cross video frame. The attention network guides the generation model to generate the target object in the specified box, and the attention network is trained by minimizing the spatial positioning loss, the temporal consistency loss, etc.

[0026] Further, the construction process of the spatio-temporal consistency image generation guiding mechanism based on attention is as follows:

[0027] First, a text-video model for target tracking data generation is constructed, and ModelScopeT2V is used as the basic text-to-video model for generating tracking image data. The basic video generation model is denoted as wherein includes the following structures: a text encoder for mapping input text to semantic representation, a VQGAN for mapping latent space to image / video frame, and a denoising UNet for performing diffusion / restoration operation on the latent space with time dimension; the UNet contains spatio-temporal mixed attention and cross-attention of text condition; the spatio-temporal consistency image generation guiding mechanism based on attention in step 2 is used to correct the deficiency of the spatio-temporal mixed attention in the target tracking data generation, so that the new target motion attention corrects the spatio-temporal mixed attention, so that the target appearance, movement, and scale change are kept in higher temporal consistency and position / size accuracy during generation;

[0028] This step is divided into two stages: training stage and inference stage.

[0029] The specific steps of the training stage are as follows: first, the input of the training data is constructed wherein represents the position coordinates of the tracking target in the t-th frame obtained from the target tracking data set and the size-width and height , pic is the image, and text is the text description of the tracking target; then the motion attention is constructed, and the size of the motion attention is the same as that of the existing spatio-temporal mixed attention, i.e. , B is the Batch number in batch processing, C is the channel number, T is the time frame number, H and W are the feature map size; in the spatial scale, first, according to the target position in the image Use 2D Gaussian to form a spatial mask with the box center ) and size , the calculation method is as follows:

[0030] ; wherein and are hyperparameters for adjusting the scale effect. The spatial mask gives greater weight to the position of the foreground tracking area, guiding the model to focus more attention on the target area and suppress the interference of the background area in the feature extraction and temporal modeling process; then extended to the time frame level, constructed into T spatial masks , denoted as spatial mask ; in the time scale, the attention of the target to the foreground and background is adjusted by the distance between the time frame and the current frame, the background attention is higher on the frames far apart in time; while the frames are closer in time in the video sequence, the foreground attention is higher, the specific formula is as follows:

[0031] ;

[0032] is the complement of , when →0 (near frame), the first term dominates, emphasizing the cross-frame correlation of the spatial mask area (foreground); when →1 (far frame), the second term dominates, emphasizing the consistency of the background (the mask area is weakened or the background is emphasized); finally, the target motion attention is used to correct the spatio-temporal mixed attention to generate a new mixed attention , the calculation method is as follows:

[0033] ;

[0034] Finally, the content and motion consistency loss is calculated to train , wherein the content consistency calculation method is as follows:

[0035] ;

[0036] wherein, is the perceptual feature extraction function, is the generated image of the t-th frame, is the target position in the generated image, are the image of the reference frame and the tracking target position respectively, and The difference between the foreground target and the background information in the generated image can be reduced;

[0037] The motion consistency loss calculation method is as follows:

[0038] ;

[0039] Wherein is the acceleration of the generated target, avoiding unreasonable large displacement of the target in a short time and enhancing the continuity of the generated moving target; the motion attention is trained through gradient descent content and motion consistency loss.

[0040] Further, in the inference stage, the target motion trajectory generated in step 1 is used , wherein the tracking target bounding box is , and the tracking target class text description Class is used as the image generation condition; the image generation condition is input into the text-to-video model introducing the attention-based spatiotemporal consistency image generation guiding mechanism to generate the target motion video, so as to use the tracking target bounding box as the labeled data.

[0041] Compared with the prior art, the present application has the following beneficial effects:

[0042] The present application provides a data generation method for target tracking task, which realizes the real motion law and high temporal consistency of the generated data through the cooperation of "target motion path generation" and "spatiotemporal consistency constraint image frame synthesis". The conditional variational autoencoder is introduced in the path generation stage, combined with the optical flow consistency and scale normalization loss, to ensure that the generated trajectory is smooth, reasonable in size and consistent with the scene dynamic characteristics. The target motion attention mechanism is designed in the image frame synthesis stage, which guides the model to pay attention to the target area in space and keep the motion coherent in time without changing the structure of the basic text-video generation model. Through the joint optimization of content consistency and motion consistency loss, the generated video is more stable and realistic in target position, size and motion direction. Compared with the prior art, the target tracking data generated by the present application has high precision, high coherence and controllability, which can effectively improve the robustness and generalization performance of the target tracking model in complex scenes. BRIEF DESCRIPTION OF DRAWINGS

[0043] The present application will be further described below in conjunction with the drawings.

[0044] Figure 1 It is a schematic diagram of the overall network structure of the present application;

[0045] Figure 2 It is a target tracking image frame synthesis method based on spatiotemporal consistency constraint of the present application. DETAILED DESCRIPTION

[0046] The invention will now be further described with reference to the accompanying drawings.

[0047] Example 1

[0048] This invention constructs a target tracking data generation system that balances motion law learning and high-quality video frame generation through two stages: path generation and spatiotemporal consistency-guided generation. First, a path generator is trained on a target tracking training set, using optical flow estimation and conditional variational coding techniques to learn the target's motion patterns in a time series, generating target motion trajectories that conform to physical constraints. Then, based on the generated target trajectories, a spatiotemporal consistency attention mechanism is introduced to guide the text-video generation model. While keeping the basic model parameters frozen, the attention network constrains the target's position, scale, and continuity in the generated frames, thereby synthesizing a video frame sequence with realistic motion characteristics. This invention generates target tracking video data with realistic motion characteristics and high temporal consistency, improving the model's robustness to complex motion, lighting changes, and occlusion conditions in different scenarios.

[0049] Its overall network structure is as follows Figure 1 As shown, the invention consists of two parts: tracking path generation and target tracking image frame synthesis based on spatiotemporal consistency constraints. The invention will be further described below with reference to the figures and specific implementation schemes:

[0050] I. Tracking Path Generation

[0051] This invention first learns a path generator to generate a tracking path for the target, comprising two steps: training and inference of the path generator, as described below. Figure 1 As shown.

[0052] During the training phase, supervised learning is performed using the existing target tracking training dataset. Specifically, training samples are constructed by combining the target position sequence from consecutive video frames in the existing tracking dataset with the corresponding inter-frame optical flow information. Specifically, the target position sequence from consecutive video frames in the training data is as follows: in Let t be the image of the t-th frame in the target location sequence, and correspondingly, let t be the position trajectory of the target in the video sequence. ,in Indicates the target is in the first place. t The positional state within the frame (such as the bounding box center coordinates and scale parameters). Then, to characterize the temporal motion information of the target and environment, an optical flow estimation network (such as RAFT) is used to extract the dense optical flow field between adjacent frames: ,in This represents the optical flow estimation network. Each pixel value is a two-dimensional displacement vector between two adjacent frames; subsequently, the position trajectory will be tracked. and optical flow field output combined into a conditional variational encoder for path analysis, the conditional variational encoder comprising a path encoder and a path decoder : the path encoder projects the tracking location trajectory and the optical flow field into the latent space, generating motion distribution parameters of latent variables z: wherein is an encoder network, and respectively represent the mean and covariance of the latent variables , and the latent variables are sampled from a Gaussian distribution: . Subsequently, the path decoder reconstructs the future trajectory with the latent variables and the optical flow condition as inputs, wherein represents the reconstructed target trajectory.

[0053] To effectively train the conditional variational autoencoder, the method employs a multi-loss function for joint optimization, mainly including trajectory reconstruction loss, optical flow consistency loss, and scale normalization loss. The trajectory reconstruction loss is used to constrain the consistency of the target trajectory generated by the model and the real trajectory in spatial position, and is defined as: The trajectory reconstruction loss is used to constrain the difference in spatial position between the reconstructed target trajectory and the real trajectory , represents the length of the time step participating in trajectory prediction or reconstruction, represents the Euclidean norm. To further constrain the local motion direction of the trajectory to be consistent with the real dynamic information of the scene, a constraint loss based on forward-backward optical flow consistency is introduced: given two adjacent images and , wherein is the t-th image in the target position sequence, the forward optical flow (from t to t+1 ) is calculated as . The backward optical flow (from t+1 to t) is denoted as , and the forward-backward consistency loss error is calculated as: , is the pixel value of the image. This loss minimization can constrain the local motion direction of the generated path to be consistent with the optical flow estimation result, so that the trajectory conforms to the dynamic characteristics of the scene. In addition, to prevent the target box size from being too large or too small to adversely affect trajectory modeling, a scale normalization loss is introduced to constrain the size change of the target box in adjacent frames: the size normalization loss is calculated as follows: wherein and are the width and height of the target position box, and are the width and height of the target position box in the predicted trajectory; the overall optimization objective of the model can be expressed as a combination of the above loss terms: wherein , and are the weight coefficients of each loss.

[0054] In the inference phase, given an initial image containing a tracking target I 1 , the target recognition model or manual labeling method is used to mark the class of the tracking target in the image and the initial position , wherein are the horizontal and vertical coordinates of the center of the target bounding box in the pixel coordinate system, respectively represent the width and height of the target bounding box, and the initial position is used as the starting state of the path generator for subsequent recursive prediction of the target motion trajectory. Then, the optical flow estimation network is used to calculate the optical flow information between adjacent frames, in order to capture more accurate and continuous optical flow change information, the present application captures the optical flow sequence in the historical time interval : wherein, represents the dense optical flow field from frame t to frame t+1, based on the optical flow sequence, the present application regards the optical flow change in the time range as the time sequence representation of the target and its local neighborhood motion pattern, in order to characterize the consistency and evolution trend of pixel-level motion in the time dimension. The input sequence of the path generator is constructed by fusing the target bounding box and the optical flow information: wherein represents the known or observed target position in the historical stage (wherein the first frame is the given initial position , and the other frames are the predicted target positions). The input sequence is input into the path encoder to extract the joint representation of the time sequence motion and spatial position information, and the path decoder generates the future target position step by step in the autoregressive framework. Specifically, in the prediction phase, the target position at time t+k is obtained by the following calculation method: wherein are the historical images from the first frame to the t+k- 1 frame, is the target position estimate predicted by the model at the previous time step t+k- 1, is the target position predicted by the model.t+k target position at time t, is set. By autoregressive manner, the model can utilize historical observation information and feedback the prediction result at previous time to the next step prediction process, so as to effectively depict the continuity and dependence of target motion trajectory in time dimension. Finally, the target motion sequence is constructed based on the predicted target position .

[0055] II. Image frame synthesis for target tracking based on spatio-temporal consistency constraint

[0056] In the training stage, step 2 designs a spatio-temporal consistency image generation guiding mechanism based on attention. Without modifying the parameters of the text-video generation model, a target motion attention network is trained using the target tracking training set to learn the tracking target motion characteristics in space (within a single frame image) and time (across video frames). The attention network guides the generation of target objects within the specified frame, and the attention network is trained by minimizing the spatial positioning loss, temporal consistency loss, etc. The method is shown as Figure 2 .

[0057] First, a text-video model for target tracking data generation is constructed. In this method, ModelScope T2V is used as the basic text-to-video model for generating tracking image data. The basic video generation model is denoted as , where includes the following structures: a text encoder for mapping input text to semantic representation, a VQGAN for mapping latent space to image / video frame, and a denoising UNet for performing diffusion / restoration operation on latent space with time dimension. The UNet contains spatio-temporal mixed attention and cross-attention of text condition. This step designs a spatio-temporal consistency attention guiding mechanism for target tracking task to correct the deficiency of spatio-temporal mixed attention in target tracking data generation, so that the new target motion attention corrects the spatio-temporal mixed attention, so that the target appearance, movement, and scale are kept in higher temporal consistency and position / size accuracy during generation. This step is divided into two stages: training stage and inference stage. The specific steps of the training stage are as follows: first, the input of the training data is constructed , where This represents the position and size of the target obtained from the target tracking dataset in frame t, where pic is the image and text is a text description of the tracked target category, such as "an image of <category name>". Then, motion attention is constructed, with the same size as the existing spatiotemporal hybrid attention, i.e. B represents the number of batches in batch processing, C represents the number of channels, T represents the number of time frames, and H and W represent the feature map dimensions. Spatially, the feature map is first determined based on the target location in the image. Use a 2D Gaussian to form a spatial mask with the center and size of the box. The calculation method is as follows:

[0058] ;

[0059] in and The x and y coordinates represent the image location. and These are hyperparameters that adjust for the scale effect. Spatial masks impose greater weight on the location of the foreground tracking region, thus guiding the model to focus more attention on the target region and suppress interference from the background region during feature extraction and temporal modeling. This is then extended to the temporal frame level, constructing T spatial masks. , denoted as space mask On a temporal scale, the degree of attention given to the foreground and background is adjusted by the distance between the current frame and the time frame. Background attention is higher on frames that are further apart in time; while foreground attention is higher when frames are closer in time within the video sequence. The specific formula is shown below:

[0060] ;

[0061] for The supplement, Time frames in a frequency sequence i With the current frame j The time interval is calculated as follows: , For the maximum time span. When →0 (near frame), first term dominates, emphasizing cross-frame correlation of spatial mask region (foreground); when →1 (far frame), the second dominant term, emphasizing background consistency (masked areas are weakened or the background is emphasized). Finally, target motion attention is used. Spatiotemporal hybrid attention The correction is made, and the calculation method is as follows:

[0062] ;

[0063] in is a Hadamard product.

[0064] To train , the present application jointly introduces content and motion consistency loss in the training stage, where the content consistency calculation method is as follows:

[0065] ;

[0066] wherein, is a perceptual feature extraction function, is the generated image of the t-th frame, is the target position in the generated image, are the image of the reference frame and the tracking target position, respectively. Minimizing can reduce the difference between the foreground target and the background information in the generated image. In addition, in order to prevent the generated target from appearing "jumping" and "shaking" between adjacent time frames, the motion consistency loss is set to punish unreasonable speed mutation or acceleration mutation, and the motion consistency loss The calculation method is as follows: ;

[0067] wherein is the acceleration of the generated target, wherein are the spatial positions of the target in t+1 , t and t-1 frame, to avoid unreasonable large displacement of the target in a short time and enhance the continuity of the generated moving target.

[0068] In the inference stage, the target motion trajectory generated in step 1 is used , wherein the tracking target bounding box is , and the tracking target class text description Class is used as the image generation condition. The image generation condition is input into the text-to-video model to generate the target motion video, and the tracking target bounding box is used as the annotation data.

[0069] Embodiment 2

[0070] 1 Dataset

[0071] The mainstream public target tracking dataset includes OTB-100, TempleColor-128 (containing 128 target tracking training sequences), UAV123 (containing 123 target tracking training sequences), etc. These sequences cover 11 typical visual tracking difficulties, including motion blur (MB), fast motion (FM), out-of-plane rotation (OP) and out-of-view (OV), occlusion (OCC), deformation (DEF), background clutter (BC), scale variation (SV), low resolution (LR), in-plane rotation (RIP) and illumination variation (IV), etc.

[0072] 2 Evaluation metrics

[0073] This embodiment uses the One-Pass Evaluation (OPE) method to conduct experimental verification on the OTB, UAV123 and Temple-Color datasets, mainly including precision curve graphs and overlap rate curve graphs. The precision curve graph is the percentage of the number of frames whose tracking results and target position center points have an error less than an error threshold to the total number of video frames. The value is usually taken as 20 pixels as a representative distance precision (DP). The overlap rate curve graph shows the percentage of image frames whose Intersection-over-Union (IoU) of the target prediction rectangular box and the real labeled rectangular box exceeds a given threshold (the threshold is between 0 and 1). In the overlap rate curve graph, this section uses the area under the curve (AUC) to compare the tracking methods.

[0074] 3 Hyperparameter settings

[0075] (1) Track path generation

[0076] In the model implementation and training process, the path encoder of the conditional variational autoencoder and the path decoder are both implemented using a deep network structure based on time series modeling. Specifically, the path encoder is composed of 2 layers of time series Transformer encoding modules, each layer containing multi-head self-attention (Multi-Head Self-Attention) and a feedforward network, which are used to jointly model the correlation of target position sequences and optical flow fields in the time dimension. The number of attention heads is set to 8, the hidden feature dimension is 256, and the feedforward layer dimension is 512. To enhance the model's ability to describe motion uncertainty, the dimension of the latent variable z is set to 64. The path decoder The decoding module is composed of two layers of Transformer with residual connection. The optical flow estimation network adopts the pre-trained RAFT model on a large-scale optical flow dataset, and the parameters are kept frozen in this work, only used to provide stable pixel-level motion priors. The input image is uniformly scaled to 256x256 to balance the computational overhead and optical flow accuracy. The model is trained end-to-end using the Adam optimizer, with an initial learning rate of 1x10 -4 , a batch size of 16, and 100 training rounds.

[0077] (2) Target tracking image frame synthesis based on spatiotemporal consistency constraint

[0078] In the implementation of the target tracking image frame synthesis method based on the spatiotemporal consistency constraint, the present application only trains the additional introduced target motion attention module Ws(x,y) under the premise of keeping the structure and parameters of the basic text-to-video generation model G(·) completely frozen. The basic generation model adopts ModelScope T2V, and the text encoder, VQGAN and denoising UNet remain unchanged in the original configuration, wherein the UNet internally contains spatiotemporal mixed attention and text condition cross attention modules. In the spatial scale, the scale adjustment coefficient of the 2D Gaussian mask is set to . The model training uses the Adam optimizer, with an initial learning rate of 2x10 -4 , a batch size of 4, and 50 training rounds. During the training process, only the parameters related to the target motion attention are involved in the back propagation, and all the parameters of the basic text-to-video generation model are kept frozen to ensure that the original semantic generation ability of the generation model is not affected.

[0079] (3) Training of target tracking model based on generated data

[0080] In the training phase of the target tracking model, the present application uses the generated synthetic tracking data as the main source of training data. The generated data is organized in the form of video sequences, and each video contains continuous TThe frame image is provided while providing the corresponding target bounding box label. To improve the generalization ability of the model in real scenes, the generated data and a small amount of real tracking data are mixed in proportion during the training process, and the proportion of generated data and real data is set to 1:2. The target tracking model adopts an end-to-end tracking network with a Siamese architecture, wherein the backbone network is used to extract the feature representation of the template frame and the search region, and the cross-frame association module is used to model the motion relationship of the target in the time dimension. The input image is uniformly scaled to 256x256, and the first frame or the key frame is used as the template frame, and the remaining frames are used as the search frame to input the network. In the training strategy, the model adopts sequence-level supervision training, and for each training sample, a continuous frame sequence with a length of 8 is randomly sampled from the generated video, and the first frame is used as the initialization frame, and the subsequent frames are used for supervision prediction. The network outputs the target position prediction at each time step, and the target tracking loss function is calculated with the corresponding synthesized label.

[0081] 4 Performance comparison with the state-of-the-art method

[0082] To evaluate the effectiveness of the proposed method in the target tracking task, it is compared with advanced methods (TFCR, TADT, GradNet, HCFTs, SiamRPN++, DeepSTRCF, ACT, VITAL, MemTrack, HDT, MCPF, BACF, CFNet, SRDCFdecon, SiamFC and DeepSRDCF). The following briefly introduces the comparison methods

[0083] Siamese network-based tracking methods: SiamFC is a siamese network-based tracking method that uses a fully convolutional network to find the most similar region to the initial template in the search region, achieving end-to-end deep learning tracking. SiamRPN++: a siamese network-based tracking method that addresses the drift problem of deep networks (such as ResNet) in tracking by using spatial perception criteria, and introduces multi-level feature fusion and depth separable correlation to improve tracking. CFNet: integrates a correlation filtering layer as a sub-module into the siamese network, enabling the network to learn online target changes while being trained offline. GradNet: introduces gradient information to update the template. It calculates the gradient through backpropagation, allowing the template to adaptively capture the appearance changes of the target in the video sequence. TADT (Target-Aware Deep Tracking): uses regression and ranking losses to select the most discriminative features for the current target in the pre-trained model, rather than using generic features. MemTrack: introduces a dynamic memory network to store the historical appearance information of the target and dynamically updates it through a read-write mechanism, effectively handling target occlusion and large-scale deformation. VITAL: uses adversarial learning (GAN) to extract robust features and uses adversarial loss to identify the most distinctive parts of the target, suppressing background interference.

[0084] Correlation Filter (CF) based tracking methods: BACF (Background-Aware CF): addresses the edge effect problem of traditional CF by using the circular shift property and training samples cropped from a larger background area. SRDCFdecon: introduces a deconvolution mechanism to handle sample pollution and improve the purity of the model based on the spatial regularization correlation filter (SRDCF). DeepSRDCF: introduces deep features (CNN) into the SRDCF framework. DeepSTRCF: combines deep features based on STRCF to robustly handle drastic changes in target appearance while ensuring real-time performance. HCFTs: uses the hierarchical deep semantic information (anti-drift) and shallow spatial information (accurate positioning) of CNN to work collaboratively through multiple correlation filters. HDT (Hedge Deep Tracking): uses the Hedge algorithm to integrate multiple correlation filters extracted from different layers of CNN. Dynamically allocate weights based on the performance of each layer filter to achieve adaptive fusion. MCPF (Multi-task Correlation Particle Filter): uses correlation filtering to improve positioning accuracy and particle filtering to address non-linear search problems caused by drastic deformation and fast motion.

[0085] Other methods: TFCR: usually refers to a tracker that combines temporal and feature context regularization. It uses the spatio-temporal correlation between historical frames to constrain the search of the current frame, preventing tracking drift. ACT (Action-Decision Network): adopts the idea of reinforcement learning. It treats tracking as a decision-making process, actively searching for the target through a series of predefined actions (such as left shift, zoom in, etc.), rather than passive matching.

[0086] Table 1 shows the comparison results of the proposed method and other advanced tracking methods on the OTB-2015 benchmark video sequence, and the comparison results are evaluated using distance precision (DP) under 20 pixels and area under curve (AUC) score (the proposed method is in bold). Among all the comparison methods, SA-Siam achieves 86.5% distance precision on OTB-2015 by jointly modeling the semantic branch and the appearance branch and training on large-scale datasets. In contrast, the proposed method further improves the distance precision to 90.4% without relying on additional artificial annotation data expansion, which is 3.9% higher than SA-Siam. VITAL and SiamRPN++ perform outstandingly in precision and success rate by introducing an adversarial learning mechanism and a more powerful deep Siamese architecture, respectively. Among them, SiamRPN++ achieves higher scores in both DP and AUC indicators. However, SiamRPN++ mainly relies on large-scale real data annotation, and its performance improvement is accompanied by high data acquisition cost. In contrast, the proposed method introduces a training strategy based on spatio-temporal consistency constraint generated data, which realizes more effective performance than SiamRPN++ under the premise of keeping the model structure unchanged. Compared with the related filter method HCFTs and the Siamese-based method TADT in recent years, the proposed method improves the distance precision by 4.5% and 5%, respectively, while maintaining a stable advantage in the AUC indicator. This shows that the training method based on generated data can effectively improve the overall tracking stability of the model in long sequences and complex motion scenarios. Overall, the experimental results show that training the model with high-quality, spatio-temporally consistent generated tracking data can significantly improve the comprehensive performance of the target tracking method in terms of positioning accuracy and success rate, especially in reducing tracking drift and enhancing robustness in complex scenarios.

[0087] Table 1 Comparison results of various tracking methods on the UAV123 test dataset, including distance precision (DP) and area under curve (AUC)

[0088] 。

Claims

1. A spatio-temporal consistency data generation method for visual target tracking, characterized in that, Comprising the following steps: Step 1, target path generation Generate target motion trajectory by using conditional variational autoencoder VAE combined with optical flow consistency constraint; the process of generating target motion trajectory by using conditional variational autoencoder VAE combined with optical flow consistency constraint is as follows: In the training stage, input the sequence of continuous video frames and the corresponding target position, and extract the dense optical flow field between adjacent frames by the optical flow estimation network RAFT, output the two-dimensional displacement vector of each pixel between adjacent frames; Then, the tracking position trajectory and the optical flow field output are combined and input into the conditional variational encoder for path analysis: the path encoder projects the tracking position trajectory and the optical flow field into the latent space to generate the motion distribution parameters of the latent variables, and the decoder generates the next frame prediction trajectory under the optical flow condition, then the path decoder takes the latent variables and the optical flow / environmental conditions as inputs to reconstruct or predict the future trajectory; In the training, the trajectory reconstruction loss, the optical flow consistency loss and the scale normalization loss are used to constrain the spatial accuracy, motion direction and target frame size of the generated trajectory; In the inference stage, input the initial position and category description of the target, combine the historical optical flow to recursively predict the future trajectory, and obtain the complete motion path; Step 2, video frame generation based on spatiotemporal consistency constraint Under the condition of freezing the parameters of the text-video generation model, introduce the target motion attention module to modify the spatiotemporal mixed attention; In space, construct a Gaussian mask according to the target frame to highlight the foreground area; in time, adjust the weight combined with the inter-frame distance to obtain spatiotemporal attention, and use 2D Gaussian to form a spatial mask with the box center and size; Then, based on the spatiotemporal attention, modify the original attention, and in the training, ensure the visual and motion continuity between generated frames through content consistency loss and motion consistency loss; Step 3, generate target motion video In the inference stage, input the motion trajectory generated in step 1 and the category text into the text-video generation model in step 2 which introduces the attention-based spatiotemporal consistency image generation guiding mechanism, output the target motion video with high spatial accuracy and temporal coherence, and realize the generation of target tracking data; The construction process of the attention-based spatiotemporal consistency image generation guiding mechanism is as follows: First, construct a text-video model for target tracking data generation, use ModelScopeT2V as the basic text-video model to generate tracking image data, use the text encoder to map the input text to semantic representation, use VQGAN to map the latent space to image / video frame, and use the denoising UNet to perform diffusion / reduction operation with time dimension in the latent space; the UNet contains spatiotemporal mixed attention and cross attention of text condition; The attention-based spatiotemporal consistency image generation guide mechanism is used for correcting the deficiency of spatiotemporal mixed attention in target tracking data generation, so that the new target motion attention The spatiotemporal mixed attention is corrected, so that the target appearance, movement and scale change are kept in higher temporal consistency and position / size accuracy during generation.

2. The spatiotemporal consistency data generation method for visual target tracking according to claim 1, wherein, In step 1, the conditional variational encoder is trained using multiple losses, including a trajectory reconstruction loss and an optical flow consistency loss. The trajectory reconstruction loss is used to constrain the difference between the generated trajectory and the real trajectory in spatial position; the optical flow consistency error where the forward optical flow is , the backward optical flow is , and is the adjacent image frame, and outputs a two-dimensional displacement vector of each pixel between adjacent frames; the forward-backward consistency loss error calculation method is: , is the pixel value of the image, and this loss minimization can constrain the local motion direction of the generated path to be consistent with the optical flow estimation result, so that the trajectory conforms to the dynamic characteristics of the scene; in addition, the size normalization loss prevents too large or too small trajectory target boxes from affecting trajectory calculation, and the size normalization loss is calculated.

3. The spatiotemporal consistency data generation method for visual target tracking according to claim 2, wherein, The size normalization loss is calculated as follows: wherein and are the width and height of the target position box, and are the width and height of the target position box in the generated trajectory The overall optimization objective of the model is represented as: wherein , and are the weight coefficients of each loss, is the content consistency loss, is the motion consistency loss.

4. The spatiotemporal consistency data generation method for visual target tracking according to claim 1, wherein, In step 1, in the reasoning stage, given a frame containing an initial image of a tracking target, a target recognition model or a manual labeling method is used to mark the category of the tracking target in the image With the initial position as the starting state of the path generator, the recursive prediction of the subsequent target motion trajectory is generated; then, the optical flow estimation network is used to calculate the optical flow information between adjacent frames, capture the optical flow sequence in the historical time interval, and based on the optical flow sequence, the change of the optical flow in the time range is regarded as the time sequence representation of the target and its local neighborhood motion pattern, so as to characterize the consistency and evolution trend of the pixel-level motion in the time dimension. Fuse the target bounding box and optical flow information to construct the input sequence of the path generator, input the input sequence into the path encoder to extract the joint representation of temporal motion and spatial position information, and generate future target position step by step by the path decoder under the autoregressive framework.

5. The spatiotemporal consistency data generation method for visual target tracking according to claim 1, wherein, In step 3, an attention-based spatio-temporal consistency image generation guide mechanism is designed. Without modifying the parameters of the text-video generation model, a target motion attention network is trained using the target tracking training set to learn the motion characteristics of the tracking target within the space-single frame image and in the time-cross-video frame. The attention network guides the generation model to generate the target object within the specified box. The attention network is trained by minimizing the spatial positioning loss and the temporal consistency loss.

6. The spatio-temporal consistency data generation method for visual target tracking according to claim 1, characterized in that, Step 2 constructs a text-video model for target tracking data generation, which is divided into two stages: training stage and inference stage. The specific steps of the training stage are as follows: first, construct the input of the training data, then construct the motion attention, the size of the motion attention is the same as the existing spatio-temporal hybrid attention; on the spatial scale, first use the 2D Gaussian to form a spatial mask according to the target position in the image with the box center and size; The spatial mask gives greater weight to the position of the foreground tracking area, thereby guiding the model to focus more attention on the target area and suppress the interference of the background area during feature extraction and temporal modeling; then extend to the time frame level and construct T spatial masks; on the time scale, adjust the attention of the target to the foreground and background by the distance between the time frame and the current frame, and the background attention is higher when the frames are farther apart in time; When the frames are closer in time in the video sequence, the foreground attention is higher; then use the target motion attention to modify the spatio-temporal hybrid attention to generate new hybrid attention; Finally, calculate the content and motion consistency loss to train, and realize the training of the motion attention through gradient descent of the content and motion consistency loss; In the inference stage, the target motion trajectory generated in step 1 is used, which contains the tracking target bounding box and the tracking target class text description Class as the image generation condition; Input the image generation condition into the text-video model with the introduction of the attention-based spatio-temporal consistency image generation guide mechanism to generate the target motion video, and use the tracking target bounding box as the labeled data.

Images