Space-time consistency data generation method for visual target tracking

Abstract

The invention relates to the technical field of computer vision, in particular to a space-time consistency data generation method for visual target tracking. The method comprises the following steps: firstly, training a path generator on a target tracking training set, learning a motion law of a target in a time sequence by using optical flow estimation and conditional variation coding technologies, and generating a target motion track conforming to physical constraints; and then, based on the generated target trajectory, introducing a space-time consistency attention mechanism to guide a text-video generation model, and under the condition of keeping basic model parameter freezing, constraining the position, scale and continuity of a target in a generation frame through an attention network, thereby synthesizing a video frame sequence with real motion features. According to the method, target tracking video data with real motion characteristics and high time sequence consistency is generated, and the robustness of the model to complex motion, illumination change and shielding conditions can be improved in different scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a spatiotemporal consistency data generation method for visual target tracking. BACKGROUND

[0002] Existing visual target tracking algorithms, especially twin networks and Transformer tracking networks based on deep learning, rely on large-scale labeled video samples for training. However, real labeled data acquisition costs are high, and the data distribution often cannot cover complex scenarios such as rapid motion, long-time occlusion, dramatic changes in perspective, low illumination, and similar backgrounds, which limits the generalization ability of the model. Although traditional data enhancement methods (such as translation, scaling, flipping, and color disturbance) can expand the data to some extent, they cannot maintain the continuity and physical consistency of video sequences in the time dimension, which easily causes the loss of temporal information and affects the modeling ability of the tracking model for dynamic changes. Therefore, there is an urgent need for a technical solution that can automatically generate diversified training data while ensuring temporal consistency and the authenticity of spatial features to improve the performance and stability of the target tracking algorithm.

[0003] In the target generation process, although existing text-to-video generation models can synthesize video content based on text descriptions, the generated targets often have problems such as jitter, discontinuity, and unstable scale in the time dimension, which cannot meet the demand for high temporal consistency and accurate target positioning in the target tracking task. Especially when generating data for training the tracking model, if the inter-frame motion trajectory does not conform to the real physical law, it will seriously affect the reliability and accuracy of the downstream tracking algorithm.

[0004] Therefore, how to design a target tracking data generation method that can ensure both the physical rationality of target motion and temporal consistency without relying on real shooting videos has become a key problem that needs to be solved in the field of target tracking data enhancement and model training. The present application proposes a spatiotemporal consistency data generation method for target tracking, which effectively improves the physical consistency and temporal continuity of the generated data by combining path generation and spatiotemporal consistency guidance mechanisms, and provides high-quality training sample support for target tracking models. SUMMARY

[0005] The present application provides a spatiotemporal consistency data generation method for visual target tracking to overcome the shortcomings of the prior art.

[0006] The present application is implemented by the following technical solutions: A spatiotemporal consistency data generation method for visual target tracking, comprising the following steps: Step one, target path generation The target motion trajectory is generated by combining a conditional variational autoencoder (VAE) with optical flow consistency constraints. Step 2: Video frame generation based on spatiotemporal consistency constraints Under the condition of freezing the parameters of the text-video generation model, a target motion attention module is introduced to correct the spatiotemporal hybrid attention. Spatially, a Gaussian mask is constructed based on the target box to highlight the foreground region. Temporally, the weights are adjusted by combining the inter-frame distance to obtain spatiotemporal attention. A spatial mask is formed by using a 2D Gaussian mask with the center and size of the box. Then, the original attention is corrected based on the spatiotemporal attention. During training, content consistency loss and motion consistency loss are used to ensure the visual and motion continuity between generated frames. Step 3: Generate target motion video In the inference phase, the motion trajectory and category text generated in step 1 are input into the text-video generation model based on the attention-based spatiotemporal consistency image generation guidance mechanism in step 2. The model outputs a target motion video with high spatial accuracy and temporal coherence, thereby achieving controllable and realistic target tracking data generation.

[0007] Furthermore, in step 1, the process of generating the target motion trajectory using a conditional variational autoencoder (VAE) combined with optical flow consistency constraints is as follows: During the training phase, the input is a continuous sequence of video frames. and corresponding target location ,in Indicates image size is The t-th frame image, image The tracking target location sequence is ; Dense optical flow fields between adjacent frames are extracted using the RAFT optical flow estimation network. ,in This represents the optical flow estimation network. and For adjacent image frames, output the two-dimensional displacement vector of each pixel between adjacent frames; Subsequently, the location trajectory will be tracked. Harmony light flow field output The combined data is then input into a conditional variational encoder for path analysis: Path Encoder Projecting the tracking position trajectory and optical flow field into the latent space generates the motion distribution parameters of the latent variable z: , and Let represent the mean and covariance of the latent variable z, respectively. Represents encoder and decoder Generate the predicted trajectory for the next frame under optical flow conditions. Subsequently, the path decoder With latent variables With optical flow / environmental conditions As input, reconstruct or predict future trajectories ; During training, trajectory reconstruction loss, optical flow consistency loss, and scale normalization loss are used to constrain the spatial accuracy, motion direction, and target box size of the generated trajectory; during the inference phase, the initial position of the target is input. Combined with the category description (Class), historical optical flow is used to recursively predict future trajectories. To obtain the complete motion path .

[0008] Furthermore, in step 1, the conditional variational encoder is trained using multiple losses, including trajectory reconstruction loss and optical flow consistency loss. The trajectory reconstruction loss is calculated as follows: Trajectory reconstruction loss is used to constrain the generated trajectory. With the actual trajectory Differences in spatial location, Indicates the length of the time step involved in trajectory prediction or reconstruction. Represents the Euclidean norm; optical flow uniformity error Among them, the forward optical flow is Reverse optical flow is The forward-backward consistency loss error is calculated as follows: , The loss, defined as the pixel value of the image, constrains the local motion direction of the generated path to align with the optical flow estimation result, thus ensuring the trajectory conforms to the dynamic characteristics of the scene. Furthermore, the size normalization loss prevents the trajectory bounding boxes from being too large or too small from affecting the trajectory calculation. The calculation method is shown below: ,in and The width and height of the target location bounding box. and To generate trajectory Given the bounding box and height of the target location, the overall optimization objective of the model is expressed as: in , and These are the weighting coefficients for each loss. For loss of content consistency, This is a loss of motion consistency.

[0009] Furthermore, in step 1, during the inference phase, an initial image containing the target to be tracked is given. Use object recognition models or manual annotation methods to label the categories of tracked objects in images. relative to the initial position ,in , The x and y coordinates of the center of the target bounding box in the pixel coordinate system. These represent the width and height of the target bounding box, respectively. This initial position serves as the starting state for the path generator, used for recursive prediction of the target's motion trajectory. Subsequently, an optical flow estimation network is used... Calculate optical flow information between adjacent frames to capture more accurate and continuous optical flow variation information, and capture historical time intervals. Optical flow sequence within: ,in, This represents the dense optical flow field from frame t to frame t+1. Based on this optical flow sequence, The changes in optical flow over a time span are considered a temporal representation of the motion patterns of the target and its local neighborhood, used to characterize the consistency and evolution trend of pixel-level motion in the temporal dimension; the input sequence of the path generator is constructed by fusing the target bounding box and optical flow information. ,in This represents the known or observed target location during a historical period, where the first frame represents the given initial location. The other frames represent the predicted target locations; the input sequence... Input to path encoder To extract a joint representation of temporal motion and spatial location information, and to use a path decoder The future target position is generated progressively within an autoregressive framework. Specifically, in the prediction phase, the first... t+k The target position at any given time is obtained by the following calculation method: ; in To represent the model's predicted future target position, an autoregressive approach is used. This allows the model to utilize historical observation information while feeding back predictions from previous moments into the next prediction step, effectively characterizing the continuity and dependencies of the target's trajectory over time. Finally, a target motion sequence is constructed based on the predicted target position. .

[0010] Furthermore, in step 2, the spatiotemporal attention is: ; in The distance between the current frame and the previous frame. To track the target location based on the image The generated spatial mask, Motor attention with a target.

[0011] Furthermore, in step 2, an attention-based spatiotemporal consistency image generation guidance mechanism is designed. Without modifying the parameters of the text-video generation model, a target motion attention network is trained using the target tracking training set to learn the motion characteristics of the tracked target in space-single frame image and in time-across video frames. The attention network guides the generation model to generate the target object within the specified box. The attention network is trained by minimizing spatial localization loss, temporal consistency loss, etc.

[0012] Furthermore, the construction process of the attention-based spatiotemporal consistency image generation guidance mechanism is as follows: First, a text-to-video model for generating target tracking data is constructed. ModelScopeT2V is used as the basic text-to-video model to generate tracking image data. The basic video generation model is denoted as... ,in The structure includes the following components: a text encoder maps the input text to a semantic representation; VQGAN maps the latent space to image / video frames; and a denoising UNet performs temporal diffusion / reconstruction operations on the latent space. The UNet includes spatiotemporal hybrid attention and text-conditional cross-attention. In step 2, an attention-based spatiotemporal consistency image generation guidance mechanism corrects the shortcomings of spatiotemporal hybrid attention in target tracking data generation, enabling the creation of new target motion attention while maintaining the original text-to-video model structure and parameter freezing. The spatiotemporal hybrid attention is modified to maintain higher temporal consistency and position / size accuracy for target appearance, movement, and scale changes during generation; This process is divided into two phases: the training phase and the reasoning phase. The specific steps in the training phase are as follows: First, construct the input of the training data. ,in This represents the position coordinates of the tracked target in frame t, obtained from the target tracking dataset. ) and size - width and height ( (pic) represents the image, and text represents the text description of the target being tracked; then, motion attention is constructed, with the same size as the existing spatiotemporal hybrid attention, i.e. B represents the number of batches in batch processing, C represents the number of channels, T represents the number of time frames, and H and W represent the feature map dimensions; in terms of spatial scale, it is first determined based on the target location in the image. Use 2D Gaussian to center the box ( ) and size form a spatial mask The calculation method is as follows: ;in and These are hyperparameters that adjust for the scale effect. Spatial masks impose greater weight on the location of the foreground tracking region, thus guiding the model to focus more attention on the target region and suppress interference from the background region during feature extraction and temporal modeling. This is then extended to the temporal frame level, constructing T spatial masks. , denoted as space mask In terms of time scale, the degree of attention given to the foreground and background is adjusted by the distance between the current frame and the time frame. Background attention is higher on frames that are farther apart in time; while foreground attention is higher when frames are closer in time in the video sequence. The specific formula is as follows: ; for The complement, for when →0 (near frame), first term dominates, emphasizing cross-frame correlation of spatial mask region (foreground); when →1 (far frame), the second dominant factor, emphasizing background consistency (masked areas are weakened or the background is emphasized); finally, target motion attention is used. Spatiotemporal hybrid attention Make corrections to generate new hybrid attention The calculation method is as follows: ; Finally, the calculation of content and motion consistency loss is crucial. The training process is performed using the following method for calculating content consistency: ; in, For perceptual feature extraction function, For the generated image of frame t, To generate the target location in the image, The image of the reference frame and the position of the tracked target are respectively minimized. This can reduce the difference between foreground objects and background information in the generated image; The method for calculating motion consistency loss is as follows: ; in To generate target acceleration and avoid unreasonable large displacements of the target in a short period of time, the continuity of generated moving targets is enhanced; training of motion attention is achieved through gradient descent content and motion consistency loss.

[0013] Furthermore, in the reasoning phase, the target motion trajectory generated in step 1 is used. This includes the bounding box of the tracking target as The text description of the target category (Class) is used as the image generation condition. The image generation condition is input into a text-to-video model that introduces an attention-based spatiotemporal consistency image generation guidance mechanism to generate target motion videos, with the target bounding box as the annotation data.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention provides a data generation method for target tracking tasks. By coordinating two parts—"target motion path generation" and "spatiotemporal consistency-constrained image frame synthesis"—it achieves realistic motion patterns and high temporal consistency in the generated data. In the path generation stage, a conditional variational autoencoder is introduced, combined with optical flow consistency and scale normalization loss, to ensure smooth, appropriately sized, and scene-dynamic trajectories. In the image frame synthesis stage, a target motion attention mechanism is designed to guide the model to focus on the target region spatially and maintain temporal motion continuity without altering the basic text-video generation model structure. Through joint optimization of content consistency and motion consistency losses, the generated video is more stable and realistic in terms of target position, scale, and motion direction. Compared with existing technologies, the target tracking data generated by this invention possesses high precision, high consistency, and controllability, effectively improving the robustness and generalization performance of target tracking models in complex scenes. Attached Figure Description

[0015] The invention will now be further described with reference to the accompanying drawings.

[0016] Figure 1 This is a schematic diagram of the overall network structure of the present invention; Figure 2 This invention relates to a target tracking image frame synthesis method based on spatiotemporal consistency constraints. Detailed Implementation

[0017] The invention will now be further described with reference to the accompanying drawings.

[0018] Example 1 This invention constructs a target tracking data generation system that balances motion law learning and high-quality video frame generation through two stages: path generation and spatiotemporal consistency-guided generation. First, a path generator is trained on a target tracking training set, using optical flow estimation and conditional variational coding techniques to learn the target's motion patterns in a time series, generating target motion trajectories that conform to physical constraints. Then, based on the generated target trajectories, a spatiotemporal consistency attention mechanism is introduced to guide the text-video generation model. While keeping the basic model parameters frozen, the attention network constrains the target's position, scale, and continuity in the generated frames, thereby synthesizing a video frame sequence with realistic motion characteristics. This invention generates target tracking video data with realistic motion characteristics and high temporal consistency, improving the model's robustness to complex motion, lighting changes, and occlusion conditions in different scenarios.

[0019] Its overall network structure is as follows Figure 1 As shown, the invention consists of two parts: tracking path generation and target tracking image frame synthesis based on spatiotemporal consistency constraints. The invention will be further described below with reference to the figures and specific implementation schemes: I. Tracking Path Generation This invention first learns a path generator to generate a tracking path for the target, comprising two steps: training and inference of the path generator, as described below. Figure 1 As shown.

[0020] During the training phase, supervised learning is performed using the existing target tracking training dataset. Specifically, the target position sequence in consecutive video frames from the existing tracking dataset is combined with the corresponding inter-frame optical flow information to construct training samples. Specifically, the target position sequence in consecutive video frames in the training data is as follows: in Let t be the image of the t-th frame in the target location sequence, and correspondingly, let t be the position trajectory of the target in the video sequence. ,in Indicates the target is in the first place. t The positional state within the frame (such as the bounding box center coordinates and scale parameters). Then, to characterize the temporal motion information of the target and environment, an optical flow estimation network (such as RAFT) is used to extract the dense optical flow field between adjacent frames: ,in This represents the optical flow estimation network. Each pixel value is a two-dimensional displacement vector between two adjacent frames; subsequently, the position trajectory will be tracked. Harmony light flow field output The combined data is then input into a conditional variational encoder for path analysis. The conditional variational encoder includes a path encoder. and path decoder The path encoder projects the tracking position trajectory and optical flow field into the latent space, generating motion distribution parameters for the latent variable z. ,in For encoder networks, and They represent latent variables respectively. The mean and covariance of the variables are obtained, and latent variables are sampled from a Gaussian distribution: Subsequently, the path decoder With latent variables With optical flow conditions Input to reconstruct future trajectories ,in This represents the target trajectory obtained from the reconstruction.

[0021] To effectively train the conditional variational autoencoder, this method employs multiple loss functions for joint optimization, primarily including trajectory reconstruction loss, optical flow consistency loss, and scale normalization loss. The trajectory reconstruction loss, defined as follows, constrains the spatial consistency between the target trajectory generated by the model and the true trajectory: The trajectory reconstruction loss is used to constrain the reconstruction of the target trajectory. With the actual trajectory Differences in spatial location, Indicates the length of the time step involved in trajectory prediction or reconstruction. This represents the Euclidean norm. To further constrain the local motion direction of the trajectory to maintain consistency with the true dynamic information of the scene, a constraint loss based on forward-backward optical flow consistency is introduced: given two adjacent frames... and ,in, For the t-th frame image in the target location sequence, the forward optical flow (from t point to t+1 ) calculated as The reverse optical flow (from t+1 to t) is denoted as... The forward-backward consistency loss error is calculated as follows: , The pixel value of the image is used. Minimizing this loss constrains the local motion direction of the generated path to be consistent with the optical flow estimation result, thus ensuring the trajectory conforms to the dynamic characteristics of the scene. Furthermore, to prevent the target box scale from being too large or too small and adversely affecting trajectory modeling, a scale normalization loss is introduced to constrain the change in target box size between adjacent frames: the scale normalization loss calculation method is shown below: ,in and The width and height of the target location bounding box. and To predict the width and height of the target location box in the trajectory; combining the above loss terms, the overall optimization objective of the model can be expressed as: in , and These are the weighting coefficients for each loss.

[0022] During the inference phase, given an initial image containing the target being tracked, I 1 Use object recognition models or manual annotation methods to label the categories of tracked objects in images. relative to the initial position ,in The x and y coordinates of the center of the target bounding box in the pixel coordinate system. These represent the width and height of the target bounding box, respectively. This initial position serves as the starting state for the path generator, used for recursive prediction of the target's motion trajectory. Subsequently, an optical flow estimation network is used... To calculate optical flow information between adjacent frames and capture more accurate and continuous optical flow variation information, this invention captures historical time intervals. Optical flow sequence within: ,in, This represents the dense optical flow field from frame t to frame t+1. Based on this optical flow sequence, the present invention will... The changes in optical flow over a time span are considered a temporal representation of the motion patterns of the target and its local neighborhood, used to characterize the consistency and evolutionary trend of pixel-level motion in the temporal dimension. The input sequence for the path generator is constructed by fusing target bounding box and optical flow information. ,in This indicates the known or observed target location during a historical phase (where the first frame represents the given initial location). The other frames represent the predicted target locations. The input sequence... Input to path encoder To extract a joint representation of temporal motion and spatial location information, and to use a path decoder The future target position is generated progressively within an autoregressive framework. Specifically, in the prediction phase, the... t+k The target position at any given time is obtained by the following calculation method: ,in For the first frame to the second frame t+k- One frame of historical image, For the model at the previous time step t+k- The target position estimate obtained at time 1. To represent the model's prediction of the first t+k The target location at any given time The set prediction time step. Through autoregression, the model can utilize historical observation information while feeding back predictions from previous moments into the next prediction step, effectively characterizing the continuity and dependencies of the target's trajectory over time. Finally, a target motion sequence is constructed based on the predicted target positions. .

[0023] II. Target Tracking Image Frame Synthesis Based on Spatiotemporal Consistency Constraints In the training phase, step 2 designs an attention-based spatiotemporal consistency image generation guidance mechanism. Without modifying the parameters of the text-video generation model, a target motion attention network is trained using the target tracking training set to learn the motion characteristics of the tracked target in both space (within a single frame) and time (across video frames). The attention network guides the generation model to generate the target object within a specified bounding box. The attention network is trained by minimizing spatial localization loss and temporal consistency loss. The method is as follows: Figure 2 As shown.

[0024] First, a text-to-video model for generating target tracking data is constructed. This method uses ModelScopeT2V as the basic text-to-video model to generate tracking image data. The basic video generation model is denoted as... ,in The structure includes the following components: a text encoder to map input text to semantic representations, VQGAN to map the latent space to image / video frames, and a denoising UNet to perform temporal diffusion / reconstruction operations on the latent space. The UNet includes spatiotemporal hybrid attention and text-conditional cross-attention. This step designs a spatiotemporally consistent attention guidance mechanism for target tracking tasks to correct the shortcomings of spatiotemporal hybrid attention in target tracking data generation. This allows for the creation of a new target motion attention mechanism while maintaining the original text-to-video model structure and parameter freezing. The spatiotemporal hybrid attention mechanism is modified to maintain higher temporal consistency and position / size accuracy for target appearance, movement, and scale changes during generation. This step consists of two phases: a training phase and an inference phase. The specific steps of the training phase are as follows: First, construct the input of the training data. ,in This represents the position and size of the target obtained from the target tracking dataset in frame t, where pic is the image and text is a text description of the tracked target category, such as "an image of <category name>". Then, motion attention is constructed, with the same size as the existing spatiotemporal hybrid attention, i.e. B represents the number of batches in batch processing, C represents the number of channels, T represents the number of time frames, and H and W represent the feature map dimensions. Spatially, the feature map is first determined based on the target location in the image. Use a 2D Gaussian to form a spatial mask with the center and size of the box. The calculation method is as follows: ; in and The x and y coordinates represent the image location. and These are hyperparameters that adjust for the scale effect. Spatial masks impose greater weight on the location of the foreground tracking region, thus guiding the model to focus more attention on the target region and suppress interference from the background region during feature extraction and temporal modeling. This is then extended to the temporal frame level, constructing T spatial masks. , denoted as space mask On a temporal scale, the degree of attention given to the foreground and background is adjusted by the distance between the current frame and the time frame. Background attention is higher on frames that are further apart in time; while foreground attention is higher when frames are closer in time within the video sequence. The specific formula is shown below: ; for The supplement, Time frames in a frequency sequence i With the current frame j The time interval is calculated as follows: , For the maximum time span. When →0 (near frame), first term dominates, emphasizing cross-frame correlation of spatial mask region (foreground); when →1 (far frame), the second dominant term, emphasizing background consistency (masked areas are weakened or the background is emphasized). Finally, target motion attention is used. Spatiotemporal hybrid attention The correction is made, and the calculation method is as follows: ; in It is the Hadamard product.

[0025] For training This invention introduces content and motion consistency loss together during the training phase, wherein the content consistency calculation method is as follows: ; in, For perceptual feature extraction function, For the generated image of frame t, To generate the target location in the image, These represent the image of the reference frame and the position of the tracking target, respectively. Minimize This can reduce the differences between foreground and background information in the generated image. Furthermore, to prevent "jumps" or "jitter" in the generated target between adjacent time frames, a motion consistency loss is applied to penalize unreasonable velocity or acceleration abrupt changes. The calculation method is as follows: ; in To generate the acceleration of the target, where The target is in t+1 , t as well as t-1 The spatial position of the frame is used to avoid unreasonable large displacements of the target in a short period of time, thereby enhancing the continuity of the generated moving targets.

[0026] In the reasoning phase, the target motion trajectory generated in step 1 is used. This includes the bounding box of the tracking target as The text description of the target category (Class) is used as a condition for image generation. This condition is then fed into a text-to-video model to generate a video of the target motion, using the target bounding box as the annotation data.

[0027] Example 2 1. Dataset Mainstream publicly available target tracking datasets include OTB-100, TempleColor-128 (containing 128 target tracking training sequences), and UAV123 (containing 123 target tracking training sequences). These sequences cover 11 typical visual tracking difficulties, including motion blur (MB), fast motion (FM), out-of-plane rotation (OP) and out-of-view (OV), occlusion (OCC), deformation (DEF), background clutter (BC), scaling variation (SV), low resolution (LR), in-plane rotation (RIP), and illumination variation (IV).

[0028] 2. Evaluation Indicators This embodiment uses the One-Pass Evaluation (OPE) method to conduct experimental validation on the OTB, UAV123, and Temple-Color datasets, mainly including accuracy curves and overlap rate curves. The accuracy curve shows the percentage of frames where the error between the tracking result and the target's center point is less than an error threshold out of the total number of video frames. This value is typically taken as 20 pixels to represent the calculated distance accuracy (DP). The overlap rate curve shows the percentage of image frames where the intersection-over-union (IoU) between the predicted target bounding box and the ground truth labeled bounding box exceeds a given threshold (between 0 and 1). In the overlap rate curve, this section uses the area under the curve (AUC) to compare tracking methods.

[0029] 3. Hyperparameter settings (1) Tracking path generation The path encoder of the conditional variational autoencoder during model implementation and training. With path decoder Both implementations employ a deep network structure based on temporal modeling. Specifically, the path encoder consists of two layers of temporal Transformer encoding modules, each containing a multi-head self-attention network and a feedforward network to jointly model the correlation between the target position sequence and the optical flow field in the temporal dimension. The number of attention heads is set to 8, the hidden feature dimension is 256, and the feedforward layer dimension is 512. To enhance the model's ability to characterize motion uncertainties, the dimension of the latent variable z is set to 64. Path decoder... The system consists of two Transformer decoding modules with residual connections. The optical flow estimation network employs a RAFT model pre-trained on a large-scale optical flow dataset, whose parameters are frozen in this work and used only to provide stable pixel-level motion priors. Input images are uniformly scaled to 256×256 to balance computational overhead and optical flow accuracy. The model is trained end-to-end using the Adam optimizer with an initial learning rate of 1×10⁻⁶. -4 The batch size is set to 16, and the number of training rounds is 100.

[0030] (2) Target tracking image frame synthesis based on spatiotemporal consistency constraints In implementing the target tracking image frame synthesis method based on spatiotemporal consistency constraints, this invention maintains the basic text-to-video generation model G(⋅) with its structure and parameters completely frozen, and only trains the additionally introduced target motion attention module Ws(x,y). The basic generation model uses ModelScopeT2V, and its text encoder, VQGAN, and denoising UNet remain unchanged in their original configurations. The UNet internally includes spatiotemporal hybrid attention and text conditional cross-attention modules. Spatially, the scale adjustment coefficient in the 2D Gaussian mask is set to... The model was trained using the Adam optimizer, with an initial learning rate set to 2×10⁻⁶. -4 The batch size was set to 4, and the number of training epochs was 50. During training, only the target motion attention-related parameters participated in backpropagation, and all parameters of the basic text-to-video generation model were kept frozen to ensure that the original semantic generation capability of the generation model was not affected.

[0031] (3) Training of target tracking model based on generated data During the target tracking model training phase, this invention uses generated synthetic tracking data as the primary source of training data. The generated data is organized in the form of video sequences, with each video segment containing continuous... T The generated images are accompanied by corresponding target bounding box annotations. To improve the model's generalization ability in real-world scenes, generated data is mixed with a small amount of real tracking data in a 1:2 ratio during training. The target tracking model adopts an end-to-end tracking network with a Siamese architecture. The backbone network is used to extract feature representations of the template frame and the search region, while the cross-frame association module is used to model the target's motion relationship in the temporal dimension. Input images are uniformly scaled to 256×256, with the first frame or keyframe used as the template frame and the remaining frames used as search frames. For training, the model employs sequence-level supervised training. For each training sample, a continuous frame sequence of length 8 is randomly sampled from the generated video. The first frame is used as the initialization frame, and subsequent frames are used for supervised prediction. At each time step, the network outputs a target position prediction and calculates the target tracking loss function with the corresponding synthetic annotations.

[0032] 4. Performance comparison with cutting-edge methods To evaluate the effectiveness of the proposed method in target tracking tasks, it is compared with state-of-the-art methods (TFCR, TADT, GradNet, HCFTs, ​​SiamRPN++, DeepSTRCF, ACT, VITAL, MemTrack, HDT, MCPF, BACF, CFNet, SRDCFdecon, SiamFC, and DeepSRDCF). The comparison method is briefly described below. Tracking methods based on Siam networks: SiamFC is a Siam network-based tracking method that uses a fully convolutional network to find the region most similar to the initial template in the search area, achieving end-to-end deep learning tracking. SiamRPN++: A Siam network-based tracking method that solves the offset problem in tracking of deep networks (such as ResNet) through spatial awareness criteria, and introduces multi-level feature fusion and deep separable correlation to improve tracking. CFNet: Integrates the correlation filtering layer as a submodule into the Siam network, enabling the network to learn target changes online while training offline. GradNet: Introduces gradient information to update the template. It calculates gradients through backpropagation, allowing the template to adaptively capture the appearance changes of the target in the video sequence. TADT (Target-Aware Deep Tracking): Uses regression and ranking losses to select the most distinctive features for the current target in the pre-trained model, instead of using general features. MemTrack: Introduces a dynamic memory network to store the historical appearance information of the target and updates it dynamically through a read-write mechanism, effectively dealing with target occlusion and large deformations. VITAL: It utilizes adversarial learning (GAN) to extract robust features and uses adversarial loss to identify the most discriminative parts of the target while suppressing background interference.

[0033] Tracking methods based on correlation filtering (CF): BACF (Background-Aware CF): Addresses the edge effect problem of traditional CF by utilizing cyclic shift characteristics and cropping samples from a larger background region for training. SRDCFdecon: Based on Spatial Regularized Correlation Filtering (SRDCF), introduces a deconvolution mechanism to handle sample contamination and improve model purity. DeepSRDCF: Introduces deep features (CNN) into the SRDCF framework. DeepSTRCF: Based on STRCF, combines deep features to robustly handle drastic changes in target appearance while maintaining real-time performance. HCFTs: Utilizes the hierarchical deep semantic information (drift prevention) and shallow spatial information (precise localization) of CNNs, through the collaborative work of multiple correlation filters. HDT (Hedge Deep Tracking): Integrates multiple correlation filters extracted from different layers of CNN using the Hedge algorithm. Weights are dynamically allocated based on the performance of each layer's filters to achieve adaptive fusion. MCPF (Multi-taskCorrelation Particle Filter): Improves localization accuracy using correlation filtering and addresses nonlinear search problems caused by drastic deformation and rapid motion using particle filtering.

[0034] Other methods: TFCR: This typically refers to a tracker that combines temporal and feature context regularization. It utilizes the spatiotemporal correlation between historical frames to constrain the search of the current frame, preventing tracking drift. ACT (Action-Decision Network): This adopts a reinforcement learning approach. It treats tracking as a decision-making process, actively seeking the target through a predefined series of actions (such as left shift, zoom, etc.), rather than passively matching.

[0035] Table 1 presents the comparison results of the proposed method with other state-of-the-art tracking methods on the OTB-2015 benchmark video sequence. The comparison results are evaluated using two aspects: distance accuracy (DP) at 20 pixels and area under the curve (AUC) score (the proposed method is shown in bold). Among all the compared methods, SA-Siam achieved a distance accuracy of 86.5% on OTB-2015 by jointly modeling semantic and appearance branches and training on a large-scale dataset. In contrast, the proposed method further improves the distance accuracy to 90.4% by training on data generated with high temporal consistency without relying on additional manually labeled data expansion, which is 3.9% higher than SA-Siam. VITAL and SiamRPN++ show outstanding performance in terms of accuracy and success rate by introducing adversarial learning mechanisms and a stronger deep Siamese architecture, respectively. Among them, SiamRPN++ achieved high scores in both DP and AUC metrics. However, SiamRPN++ mainly relies on large-scale real data annotation, and its performance improvement comes at the cost of high data acquisition. In contrast, the method of this invention, by introducing a training strategy based on spatiotemporal consistency constraints for generated data, achieves more effective performance than SiamRPN++ while maintaining the model structure. Compared with recent correlation filter methods HCFTs and Siamese-based methods TADT, the proposed method improves distance accuracy by 4.5% and 5%, respectively, while maintaining a stable advantage in AUC. This indicates that training based on generated data can effectively improve the overall tracking stability of the model in long sequences and complex motion scenarios. Overall, experimental results show that training the model with high-quality, spatiotemporally consistent generated tracking data can significantly improve the comprehensive performance of target tracking methods in terms of localization accuracy and success rate, especially showing significant advantages in reducing tracking drift and enhancing robustness in complex scenarios.

[0036] Table 1 compares the tracking methods of this invention on the UAV123 test dataset, including distance accuracy (DP) and area under the curve (AUC). .

Claims

1. A method for generating spatiotemporally consistent data for visual target tracking, characterized in that, Includes the following steps: Step 1: Target Path Generation The target motion trajectory is generated by combining a conditional variational autoencoder (VAE) with optical flow consistency constraints. Step 2: Video frame generation based on spatiotemporal consistency constraints Under the condition of freezing the parameters of the text-video generation model, a target motion attention module is introduced to correct the spatiotemporal hybrid attention; spatial Above, a Gaussian mask is constructed based on the target bounding box to highlight the foreground region; in terms of time, the weights are adjusted by combining the inter-frame distance to obtain spatiotemporal attention, and a spatial mask is formed using a 2D Gaussian mask with the center and size of the box; Then, the original attention is corrected based on spatiotemporal attention, and the visual and motion continuity between generated frames is ensured through content consistency loss and motion consistency loss during training; Step 3: Generate target motion video In the inference phase, the motion trajectory and category text generated in step 1 are input into the text-video generation model based on the attention-based spatiotemporal consistency image generation guidance mechanism in step 2, and the target motion video with high spatial accuracy and temporal coherence is output to realize the generation of target tracking data.

2. The method for generating spatiotemporally consistent data for visual target tracking according to claim 1, characterized in that, In step 1, the process of generating the target motion trajectory using a conditional variational autoencoder (VAE) combined with optical flow consistency constraints is as follows: During the training phase, a continuous video frame sequence and the corresponding target position are input. The dense optical flow field between adjacent frames is extracted by the optical flow estimation network RAFT, and the two-dimensional displacement vector of each pixel between adjacent frames is output. Subsequently, the tracking position trajectory and optical flow field output are combined and input into the conditional variational encoder for path analysis: the path encoder projects the tracking position trajectory and optical flow field into the latent space to generate motion distribution parameters of the latent variables. The decoder generates the predicted trajectory for the next frame under optical flow conditions. Then, the path decoder reconstructs or predicts the future trajectory with the latent variables and optical flow / environmental conditions as input. During training, trajectory reconstruction loss, optical flow consistency loss, and scale normalization loss are used to constrain the spatial accuracy, motion direction, and target box size of the generated trajectory. In the inference stage, the initial position and category description of the target are input, and the future trajectory is recursively predicted by combining historical optical flow to obtain the complete motion path.

3. The method for generating spatiotemporally consistent data for visual target tracking according to claim 2, characterized in that, In step 1, the conditional variational encoder is trained using multiple losses, including trajectory reconstruction loss and optical flow consistency loss. The trajectory reconstruction loss is used to constrain the spatial differences between the generated trajectory and the true trajectory; the optical flow consistency loss... Among them, the forward optical flow is Reverse optical flow is , and For adjacent image frames, output the two-dimensional displacement vector of each pixel between adjacent frames; the forward-backward consistency loss error is calculated as follows: , The pixel value of the image is used to minimize the loss, which constrains the local motion direction of the generated path to be consistent with the optical flow estimation result, so that the trajectory conforms to the dynamic characteristics of the scene. In addition, the size normalization loss prevents the trajectory target box that is too large or too small from affecting the trajectory calculation. The size normalization loss is calculated.

4. The method for generating spatiotemporally consistent data for visual target tracking according to claim 3, characterized in that, The method for calculating size normalization loss is as follows: ,in and The width and height of the target location bounding box. and To generate trajectory Given the bounding box and height of the target location, the overall optimization objective of the model is expressed as: in , and These are the weighting coefficients for each loss. For loss of content consistency, This is a loss of motion consistency.

5. The method for generating spatiotemporally consistent data for visual target tracking according to claim 2, characterized in that, In step 1, during the inference phase, given an initial image frame containing the tracked target, the category of the tracked target in the image is labeled using a target recognition model or a manual annotation method. The initial position serves as the starting state of the path generator and is used for the recursive prediction of the target's motion trajectory. Subsequently, the optical flow estimation network is used to calculate the optical flow information between adjacent frames, capturing the optical flow sequence within the historical time interval. Based on this optical flow sequence, the optical flow changes within the time range are regarded as the temporal representation of the motion pattern of the target and its local neighborhood, in order to characterize the consistency and evolution trend of pixel-level motion in the time dimension. The input sequence of the path generator is constructed by fusing target bounding box and optical flow information. The input sequence is then fed into the path encoder to extract a joint representation of temporal motion and spatial location information. The path decoder then gradually generates the future target location within an autoregressive framework.

6. The method for generating spatiotemporally consistent data for visual target tracking according to claim 1, characterized in that, Step 2 designs an attention-based spatiotemporal consistency image generation guidance mechanism. Without modifying the parameters of the text-video generation model, a target motion attention network is trained using the target tracking training set to learn the motion characteristics of the tracked target in space-single frame image and in time-across video frames. The attention network guides the generation model to generate the target object within the specified box. The attention network is trained by minimizing spatial localization loss, temporal consistency loss, etc.

7. The method for generating spatiotemporally consistent data for visual target tracking according to claim 4, characterized in that, The construction process of the attention-based spatiotemporal consistency image generation guidance mechanism is as follows: First, a text-to-video model for generating target tracking data is constructed. ModelScopeT2V is used as the basic text-to-video model to generate tracking image data. A text encoder is used to map the input text to semantic representation. VQGAN is used to map the latent space to image / video frames. A denoising UNet is used to perform diffusion / reconstruction operations with a time dimension in the latent space. UNet includes spatiotemporal hybrid attention and text-conditional cross attention. Attention-based spatiotemporal consistency image generation guidance mechanisms are used to correct the shortcomings of spatiotemporal hybrid attention in target tracking data generation. This allows for the creation of new target motion attention while maintaining the original text-to-video model structure and parameter freezing. The spatiotemporal hybrid attention is modified to maintain higher temporal consistency and position / size accuracy for target appearance, movement, and scale changes during generation.

8. The method for generating spatiotemporally consistent data for visual target tracking according to claim 4, characterized in that, Step 2, constructing a text-video model for target tracking data generation, consists of two phases: a training phase and an inference phase. The specific steps of the training phase are as follows: First, construct the input of the training data, and then construct the motion attention. The size of the motion attention is the same as the size of the existing spatiotemporal hybrid attention. In terms of spatial scale, first, based on the target position in the image, use a 2D Gaussian to form a spatial mask with the box center and size. Spatial masks impose greater weight on the location of the foreground tracking region, thereby guiding the model to focus more attention on the target region and suppress interference from the background region during feature extraction and temporal modeling. Then, it is extended to the time frame level to construct T spatial masks. At the time scale, the attention of the target to the foreground and background is adjusted by the distance between the time frame and the current frame. The background attention is higher on frames that are farther apart in time. When frames are closer in time within a video sequence, foreground attention is stronger; then, target motion attention is used to refine the spatiotemporal hybrid attention and generate a new hybrid attention. Finally, the content and motion consistency loss are calculated and trained, and motion attention is trained by gradient descent of the content and motion consistency loss.

9. A method for generating spatiotemporally consistent data for visual target tracking according to claim 8, characterized in that, During the inference phase, the target motion trajectory generated in step 1, which includes the target bounding box and the target category text description Class, is used as the image generation condition. The image generation condition is input into the text-to-video model that introduces an attention-based spatiotemporal consistency image generation guidance mechanism to generate a target motion video, with the target bounding box as the labeled data.

Images