Target tracking method, apparatus and device

Abstract

The application provides a target tracking method, device and equipment, the method comprising: obtaining observation information of a current frame; inputting the observation information of the current frame into a time sequence target tracking model based on future scene prediction, and outputting a target trajectory; wherein the time sequence target tracking model based on future scene prediction is used to determine observation information of a future frame according to the observation information of the current frame; and the target trajectory is predicted according to the observation information of the future frame. The method of the application embodiment can effectively overcome the problems of accumulated errors and weakened effectiveness of historical information in the previous time sequence modeling mode.

Description

Technical Field

[0001] This invention relates to the field of target tracking technology, and in particular to a target tracking method, apparatus, and device. Background Technology

[0002] Visual target tracking is a core problem in the field of computer vision, which aims to automatically and continuously locate and analyze the position, size, shape and motion trajectory of a specified target object through video or image sequences.

[0003] In related technologies, temporal target tracking methods based on template updates and those based on target motion estimation are essentially frameworks that use historical prediction information as prior information for current frame model inference. Their reliance on historical frame prediction information is prone to error accumulation, leading to tracking failure or target loss. Summary of the Invention

[0004] This invention provides a target tracking method, apparatus, and device that enhances tracking capabilities by utilizing historical information and predicted future information, effectively overcoming the problems of accumulated errors and reduced effectiveness of historical information in previous time-series modeling methods.

[0005] This invention provides a target tracking method, comprising the following steps.

[0006] Obtain observation information for the current frame;

[0007] The observation information of the current frame is input into a temporal target tracking model based on future scene prediction, and the target trajectory is output. The temporal target tracking model based on future scene prediction is used to determine the observation information of future frames based on the observation information of the current frame, and to predict the target trajectory based on the observation information of future frames.

[0008] According to a target tracking method provided by the present invention, the time-series target tracking model based on future scene prediction includes:

[0009] A future prediction module; the future prediction module is used to determine the observation information of future frames based on the observation information of the current frame and the target vector; the target vector is used to represent the position information of the target;

[0010] Feature encoding module; the feature encoding module is used to obtain the observation information of the future frames of the filtered search area based on the observation information of the future frames;

[0011] A position estimation module; the position estimation module is used to predict the target trajectory based on the observation information of future frames of the filtered search area and the target vector of the future frames.

[0012] According to a target tracking method provided by the present invention, the future prediction module is specifically used for:

[0013] Based on the observation information and target vector of the current frame, determine the target vector of the future frame;

[0014] Based on the target vector of the future frame, the observation information of the future frame is determined.

[0015] According to a target tracking method provided by the present invention, the feature encoding module includes:

[0016] Feature extraction module and self-attention module;

[0017] The feature extraction module is used to extract features from the observation information of the future frame; the self-attention module is used to filter out interference information that is irrelevant to the target after the feature extraction to obtain the observation information of the future frame of the filtered search area.

[0018] According to a target tracking method provided by the present invention, the position estimation module is specifically used for:

[0019] Based on the observation information of future frames of the filtered search area and the target vector of the future frames, an observation information-enhanced target vector is obtained;

[0020] Based on the enhanced target vector from the observed information, the target trajectory is predicted.

[0021] The present invention also provides a target tracking device, comprising the following modules:

[0022] The acquisition module is used to acquire observation information for the current frame;

[0023] The prediction module is used to input the observation information of the current frame into a temporal target tracking model based on future scene prediction, and output the target trajectory; wherein, the temporal target tracking model based on future scene prediction is used to determine the observation information of future frames based on the observation information of the current frame; and predict the target trajectory based on the observation information of future frames.

[0024] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the target tracking method as described above.

[0025] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the target tracking method as described above.

[0026] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the target tracking method as described above.

[0027] The target tracking method, apparatus, and device provided by this invention enhance tracking capabilities by utilizing historical information and predicted future information. This temporal modeling approach is significantly superior to existing methods that are limited to using historical frames for prediction, effectively overcoming the problems of accumulated errors and reduced effectiveness of historical information in previous temporal modeling methods. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0029] Figure 1 This is a schematic diagram of the target tracking framework provided by the present invention.

[0030] Figure 2 This is one of the flowcharts of the target tracking method provided by the present invention.

[0031] Figure 3 This is a schematic diagram comparing the effects of different temporal visual target tracking methods provided by the present invention.

[0032] Figure 4 This is a schematic diagram of the target tracking model provided by the present invention.

[0033] Figure 5 This is a schematic diagram of the future prediction module provided by the present invention.

[0034] Figure 6 This is a schematic diagram of the feature encoding module provided by the present invention.

[0035] Figure 7 This is a schematic diagram of the location estimation module provided by the present invention.

[0036] Figure 8 This is a schematic diagram of the target tracking device provided by the present invention.

[0037] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0039] The following is combined Figures 1-9 The present invention describes the target tracking method, apparatus, and device.

[0040] To facilitate a clearer understanding of the technical solutions of the various embodiments of this application, some technical content related to the various embodiments of this application will be introduced first.

[0041] Visual object tracking is a core problem in computer vision, aiming to automatically and continuously locate and analyze the position, size, shape, and trajectory of a specified target object through video or image sequences. Temporal visual tracking targets a region or object within a video frame or image, without requiring its semantic information (category, etc.). This concept can be described as "tracking everything," meaning that given the state (position, size, etc.) of a target in a particular frame of a video, visual object tracking needs to estimate the target's state in subsequent frames. As a supplement to detection algorithms, it can provide the spatial location of the target in a video or consecutive semantically related images, reducing the complexity of the entire system (e.g., detection is only applied to identify the target in the first frame of the video and to determine the target's position in some subsequent frames, then tracking is applied in the remaining frames to determine the target's position). Classic applications of visual object tracking in computer vision include security (vehicle tracking, license plate recognition, etc.), surveillance (face recognition, gait recognition, etc.), inspection (drone tracking, robot navigation, etc.), and emerging smart living (human-computer interaction, VR / AR, etc.), smart cities (traffic monitoring, etc.), and smart industry (telemedicine, etc.). For example, in autonomous driving systems, by tracking pedestrians, vehicles, and other targets around the vehicle, accurate perception information is provided for downstream trajectory prediction and decision planning, ensuring that the vehicle can successfully avoid collisions with surrounding objects in the future and provide users with a comfortable driving experience.

[0042] Visual target tracking is a long-term, continuous process. The appearance of the target and its surrounding background can change significantly compared to when it was initially specified, affecting the accuracy of target recognition and spatial location estimation. Therefore, temporal learning, which continuously updates the model's understanding of the latest state of the tracked target, is crucial for ensuring accurate and continuous perception in visual target tracking tasks. Most existing temporal learning methods mitigate the impact of long-term target appearance deformation and background shifts by updating the visual template of the visual target tracking framework with predictions from historical frames. However, predictions from historical frames contain some errors, which accumulate over multiple updates, affecting the model's understanding of the tracked target and ultimately leading to target recognition failure and spatial estimation bias. Essentially, such temporal learning frameworks treat visual target tracking as a single-frame detection task, matching the entire image to find the target based on the provided visual template, without considering the spatiotemporal continuity of visual target tracking. Recent techniques treat visual target tracking as a sequence regression task, continuously predicting the target's motion and offset based on the spatial location estimated in the previous frame. This temporal learning method avoids the shortcomings of single-frame matching in terms of its temporal characteristics to some extent, and correlates previous and subsequent frames through motion estimation to achieve more accurate and stable tracking results. However, it still cannot overcome the accumulation of historical errors, and the appearance differences between previous and subsequent frames will also weaken the effectiveness of historical prediction information.

[0043] For example, the temporal target tracking method based on template update and the temporal target tracking method based on target motion estimation are as follows:

[0044] (1) A time-series target tracking method based on template update.

[0045] Template-based temporal target tracking is the most commonly used temporal learning method. For each frame of the input image, the model predicts the spatial location and confidence level of the target. Then, it determines whether to update the visual template based on whether the confidence level exceeds a preset threshold. If the threshold is met, meaning the estimated target state is considered reliable, the prediction result of the current frame is cropped and used as the visual template for the new frame. If the threshold is not met, such as when the target is completely occluded or moves out of sight, or when the tracker's prediction is incorrect, the cropped template is unreliable, and the previous visual template is retained. Some subsequent methods use momentum update strategies to retain information from the initial template; others retain the initial visual template and dynamically maintain the predicted template for each frame, participating in the model's matching process together. By modeling the global relationships between all elements in the spatial and temporal dimensions, distinguishable spatiotemporal features are extracted. However, the accumulation of errors caused by long-term tracking reduces the effectiveness of the new visual template, leading to tracking errors or spatial estimation bias.

[0046] (2) Temporal target tracking method based on target motion estimation.

[0047] Temporal target tracking methods based on target motion estimation treat visual target tracking as a sequence regression task, using the target's spatial coordinates as continuously maintained and updated model variables. These methods continuously predict the target's motion offset based on the spatial position estimated in the previous frame. Given an input image, the spatial features representing the target's location are interacted with semantic features extracted from the image to filter motion-related features and perform position estimation. The target's spatial offset is then added to the global coordinates of the previous frame to obtain the prediction result for the new frame. Compared to template-update-based temporal visual target tracking schemes, these methods treat visual target tracking as a sequence regression task, using the target's spatial coordinates as continuously maintained and updated model variables. Some methods also incorporate template update strategies, inputting the image region corresponding to the prediction result of the previous frame into the model to further enhance spatiotemporal information. This temporal learning method avoids the shortcomings of single-frame matching in terms of its temporal characteristics to some extent, and correlates previous and subsequent frames through motion estimation to achieve more accurate and stable tracking results. However, it still cannot overcome the accumulation of historical errors, and the appearance differences between previous and subsequent frames will also weaken the effectiveness of historical prediction information.

[0048] Temporal target tracking methods based on template updates and those based on target motion estimation, while differing slightly in model design, essentially both use historical prediction information as prior information for the model's inference in the current frame. However, considering that predictions in historical frames cannot be 100% accurate, their errors accumulate during tracking, eventually leading to target loss in a certain frame. Subsequent completely erroneous predictions will continue to affect predictions in subsequent frames, interfering with the model's spatial position estimation rather than correctly re-locking the target. Furthermore, the robustness of visual target tracking relies on the model's understanding of the tracking scene, enabling it to quickly adapt to unexpected situations and distinguish the target from the background. These frameworks rely solely on historical prediction information, rather than learning from changes in the target's state to understand the dynamic characteristics of the tracking scene. This results in insufficient utilization of the temporal characteristics of the continuous tracking process, weakening the effectiveness of historical prediction information.

[0049] This application proposes a temporal target tracking method based on future scene prediction, primarily aimed at overcoming the problems of error accumulation and insufficient environmental understanding caused by inadequate utilization of spatiotemporal information in temporal visual target tracking. It aims to achieve category-independent perception in visual object tracking and enhance the model's adaptability to long-term target appearance changes and background interference. In other words, current trackers typically formulate tracking as a one-time detection problem. Considering the severe environmental shifts and target appearance deformations in long-term tracking, non-temporal architectures relying solely on current frame information struggle to ensure generalization ability in new frames. Outdated templates are unlikely to provide effective target perception knowledge and may even interfere with the tracking process. While many works intuitively attempt to address this issue by maintaining the temporal validity of target information, such as using predictions with high confidence scores from historical frames to update visual templates or passing historical trajectories as prior knowledge, these methods remain limited by temporal architectures that only use historical information, restricting temporal information transmission when transferring previous observations. Therefore, developing a method that can fully learn the spatiotemporal correlation characteristics of frames before and after the tracking sequence, improve understanding of the tracking scene, and accurately perceive targets is a pressing challenge. The following sections of this application provide a more in-depth introduction to existing temporal target tracking algorithms and the world model theories they employ, while also listing experimental schemes most similar to this invention.

[0050] For example, such as Figure 1 As shown, this application will formally define different tracking frameworks to better understand the technical background of temporal visual target tracking, and at the same time provide symbol definitions for the following text.

[0051] First, we introduce the most basic non-temporal target tracking framework. Given a template image of the target in the first frame... The goal of visual object tracking is to track continuously input images. Mid-position target ,in The target state is represented by a bounding box. Mainstream tracking paradigms aim to learn a deep tracking model. The model locates objects based on different detection paradigms (i.e., anchor-box-based architectures, such as Faster R-CNN[1], or anchor-box-free architectures, such as CenterNet[2]) and outputs the results. The tracking process can be described as follows:

[0052]

[0053] To mitigate the impact of target appearance deformation and background changes during long-term tracking, recently popular time-series trackers utilize model outputs from historical frames. As additional prior knowledge, such as updating templates (e.g., TrDiMP[3], STARK[4]) or passing historical target locations (e.g., SeqTrack[5], AR-Track[6][7]). The time-series tracking process can be described as:

[0054]

[0055] In contrast, the method proposed in this application defines temporal visual tracking as a process of predicting the future from the past, based on environmental dynamics learned from historical observations (cached in the tracking model). Target vector In order to predict the future trajectory of the target (including the current frame and the next two frames), this modeling process can be represented as:

[0056]

[0057] Dynamic environmental modeling through parametric modeling ( Instead of explicitly passing historical output This not only enables better generalization in new scenarios of future frames, but also reduces the risk of error accumulation from past experience.

[0058] II. Background of World Model Technology

[0059] Significant progress has been made in the field of artificial intelligence in recent years, particularly in deep learning and reinforcement learning. However, current AI systems still face challenges such as low sample efficiency, poor generalization ability, and difficulty in handling complex real-world situations. To address these issues, constructing "world models" capable of understanding and predicting environmental dynamics has become a highly anticipated research direction. World models, proposed in the field of reinforcement learning, are defined as models capable of predicting the future state of the environment in which an agent exists.

[0060] The core idea of ​​the world model is to construct an internal model of the external world for prediction, planning and control. Its framework consists of three main parts: 1) Visual model (V): using a variational autoencoder (VAE[8]) to compress the original pixel input into a low-dimensional latent vector. The encoder uses convolutional layers to extract image features, and the decoder uses deconvolutional layers to reconstruct the image. The training objectives include reconstruction loss (L2 distance) and KL divergence loss (regularization term) to balance reconstruction quality and latent space regularity. 2) Memory model (M): The latent vector of the next time step is predicted using a recurrent neural network (LSTM[9] / RNN

[10] ). And other environmental information. 3) Control model (C): A simple linear model based on the latent vector at the current time. Hidden states of recurrent neural networks Output Action The model is trained using an adaptive evolution strategy based on the covariance matrix, with the goal of maximizing cumulative reward. Based on the world model theory, a series of algorithms have been proposed, such as Dreamer

[11] , which imagines and learns from the latent space of the learned world model. DreamerV2

[12] uses discrete latent variables to represent the state, which improves the learning efficiency and achieves significant performance improvement on Atari games. DreamerV3

[13] is dedicated to improving the generality and robustness of the algorithm, and has achieved performance exceeding that of domain-specific expert algorithms on more than 150 different tasks.

[0061] Figure 2 This is one of the flowcharts illustrating the target tracking method provided by the present invention, such as... Figure 2 As shown, the method includes the following:

[0062] Step 201: Obtain the observation information of the current frame.

[0063] Specifically, existing temporal target tracking technologies cannot overcome the deformation of target appearance and background shift during long-term tracking. Their reliance on historical frame prediction information is prone to error accumulation, leading to tracking failure or target loss. Furthermore, they cannot form a generalized understanding of the overall tracking scenario and are insufficiently adaptable to unexpected situations. From an application perspective, existing temporal target tracking technologies have limitations in effectiveness and versatility, and cannot be simultaneously applied to long-term tracking as well as scenarios with rapid target movement and rapidly changing backgrounds.

[0064] To address the aforementioned issues, this embodiment first acquires the observation information of the current frame. Optionally, for the input visual template... and search area The image is segmented and flattened, and observation information for the current frame is constructed using linear projection. .

[0065] Step 202: Input the observation information of the current frame into the temporal target tracking model based on future scene prediction, and output the target trajectory; wherein, the temporal target tracking model based on future scene prediction is used to determine the observation information of future frames based on the observation information of the current frame; and to predict the target trajectory based on the observation information of future frames.

[0066] Specifically, in this embodiment, after obtaining the observation information of the current frame, the observation information of the current frame is input into a temporal target tracking model based on future scene prediction, and the target trajectory is output. The temporal target tracking model based on future scene prediction is used to determine the observation information of future frames based on the observation information of the current frame; and to predict the target trajectory based on the observation information of future frames. In other words, this application defines temporal learning in visual target tracking as a process of predicting the future from the past, that is, predicting the future state of the target based on historical observations. It utilizes historical information and predicted future information to enhance tracking capabilities, effectively overcoming the problems of accumulated errors and weakened effectiveness of historical information in previous temporal modeling methods. Its effect is significantly better than previous methods that were limited to using prediction results from historical frames.

[0067] For example, such as Figure 3 As shown, in order to qualitatively analyze the performance of different temporal visual target tracking methods in long-term tracking scenarios, this application selects corresponding test video sequences and initializes them with the same target. Then, it visualizes the prediction structure of different trackers after a certain number of frames of long-term tracking to illustrate the tracking performance in the face of target appearance deformation and background shift. Previous temporal learning methods usually fail to track more than 100 frames due to error accumulation and insufficient scene understanding, i.e., locking the wrong target. However, the method proposed in this application can understand the tracking scene by learning the dynamic changes of historical frames, overcome the appearance changes of the target and background shift, and accurately lock the target. Through verification, the method of this application can achieve a tracking success rate improvement of up to 2.9%, 8.9% and 9.7% respectively on the visual target tracking task OTB100

[14] , LaSOT

[15] and TNL2K

[16] datasets.

[0068] The method described above enhances tracking capabilities by utilizing historical information and predicted future information. This temporal modeling approach is significantly superior to existing methods that are limited to using historical frames for prediction, effectively overcoming the problems of accumulated errors and reduced effectiveness of historical information in previous temporal modeling methods.

[0069] In some embodiments, the time-series target tracking model based on future scenario prediction includes:

[0070] Future prediction module; the future prediction module is used to determine the observation information of future frames based on the observation information of the current frame and the target vector; the target vector is used to represent the position information of the target;

[0071] Feature encoding module; The feature encoding module is used to obtain the observation information of the future frames of the filtered search area based on the observation information of the future frames;

[0072] Position estimation module; The position estimation module is used to predict the target trajectory based on the observation information of future frames of the filtered search area and the target vector of the future frames.

[0073] Specifically, such as Figure 4 As shown, the temporal target tracking model based on future scene prediction in this embodiment includes a future prediction module, a feature encoding module, and a position estimation module. The future prediction module determines the observation information of future frames based on the observation information of the current frame and the target vector; the target vector represents the target's position information, thereby enhancing tracking capabilities based on predicted future information. The feature encoding module obtains the observation information of future frames in the filtered search area based on the observation information of future frames; that is, it refines features through the feature encoding module, finding more noteworthy information and filtering out interference information. The position estimation module predicts the target trajectory based on the observation information of future frames in the filtered search area and the target vector of the future frames. This achieves the goal of enhancing the discriminative power of the learned target features by utilizing historical frame information and predicted future frame information, enhancing the model's understanding of the tracking scene, and improving the accuracy and stability of tracking, avoiding the problems of error accumulation and insufficient temporal information learning in previous temporal learning frameworks.

[0074] In the method described above, the future prediction module determines the observation information of future frames based on the observation information of the current frame and the target vector, thereby enhancing tracking capabilities based on predicted future information. The feature encoding module obtains the observation information of future frames of the filtered search area based on the observation information of future frames, thus achieving feature purification and filtering out interference information. The position estimation module predicts the target trajectory based on the observation information of future frames of the filtered search area and the target vector of the future frames. That is, it uses historical frame information and predicted future frame information to enhance the discriminative power of the learned target features, enhances the model's understanding of the tracking scene, and improves the accuracy and stability of tracking. This avoids the error accumulation and insufficient temporal information learning problems of previous temporal learning frameworks, effectively improving the accuracy of target tracking.

[0075] In some embodiments, the future prediction module is specifically used for:

[0076] Based on the observation information and target vector of the current frame, determine the target vector of the future frame;

[0077] Based on the target vector of the future frame, determine the observation information of the future frame.

[0078] Specifically, such as Figure 5 As shown, in the future prediction module of this application embodiment, three trainable target vectors are initialized. This represents the scene dynamics of the current and the next two frames. For each input visual template... and search area The image is segmented and flattened, and observation information for the current frame is constructed using linear projection. Then, through the gated loop unit (GRU

[17] ) and The cached temporal features are updated autoregressively to obtain the target vector for future frames:

[0079]

[0080] This means redefining the tracking problem as a process of predicting the future from the past, and for the first time passing temporal information into future frames to better adapt to new scenes, target appearance deformation, and background offset in visual target tracking.

[0081] The temporal target tracking method proposed in this application aims to enhance the model's generalization ability in complex scenes within new frames; therefore, predicting the dynamic changes of the future tracking environment is a prerequisite. Updated target vector Used with Obtaining observation information for future frames through a mutual attention mechanism :

[0082]

[0083] It should be noted that, and The interaction aims to enhance the feature quality of the current frame by incorporating historical information derived from before the GRU update. .

[0084] In other words, the future prediction module in this embodiment borrows the idea of ​​using world models to predict future scene distributions, enabling the tracker to predict the future state of the target from the learned historical dynamic changes. This helps the model better distinguish the target from appearance deformation and background interference, providing an effective solution for robust tracking of video sequences facing different types of targets and complex scene changes. Furthermore, while tracking the target, this method can also evolve the video sequence of the next few frames based on the predicted future state, which can be used for video generation and annotation, possessing strong scalability potential.

[0085] Optionally, following the loss design in the world model, the predicted future scene features are respectively... and real-world scene features Mapping to distribution and Then, their KL divergence is minimized. It is worth noting that this application only considers the future prediction process for the supervised search region, because the input visual template always comes from the initial frame. The prior distribution of the mapping is considered a prediction of the dynamic changes in the future scene. In contrast, by The posterior distribution of the mapping represents observations from the real scene. The future dynamics of the KL divergence are measured. KL divergence measures the distance between the predicted future scenario (prior) and the actual future scenario (posterior). This application optimizes the future prediction loss. To enhance the model's ability to predict the future:

[0086] )

[0087] in This represents the KL divergence.

[0088] In addition to supervision from distributional differences, this application... The search region is reconstructed using a decoder to further improve the accuracy of future scene predictions. Predicted future tracking scenes. From the actual future search area By optimizing reconstruction losses To supervise, among which Mean squared error:

[0089]

[0090] The method described in the above embodiments redefines the tracking problem as a process of predicting the future from the past, and for the first time, transmits temporal information to future frames to better adapt to new scenes, target appearance deformation, and background shifts in visual target tracking. Furthermore, unlike traditional temporal learning that only uses historical prediction information, this application predicts the future state of the target by learning from historical dynamic changes and performs multimodal prediction of target motion to cover the uncertainties of different future scenarios. This gives the tracker a better generalization ability for appearance deformation and background shifts, effectively improving the accuracy of target tracking.

[0091] In some embodiments, the feature encoding module includes:

[0092] Feature extraction module and self-attention module;

[0093] The feature extraction module is used to extract features from the observation information of future frames; the self-attention module is used to filter out interference information that is irrelevant to the target after feature extraction, so as to obtain the observation information of future frames of the filtered search area.

[0094] Specifically, this application employs a visual backbone feature extraction network (ViT) as the encoder, jointly modeling search and template images. The encoder extracts features from the search and template images and learns feature-level correspondences to filter out interference information irrelevant to the target. The input future prediction scene features... (Including visual templates and search regions) are concatenated along the spatial dimension and input into the encoder for feature extraction:

[0095]

[0096] in Block represents the Transformer layer in the encoder used for spatial modeling. It refers to the number of layers. Besides spatial modeling between the template and the search region in the Block, the extracted target-aware information can be used to refine the temporal dynamic information along the timeline. Therefore, after each feature extraction stage of the encoder, this application arranges a temporal self-attention module to filter out target-irrelevant interference terms when extracting target information at different levels. For example... Figure 6 As shown, the predicted future scene features The transposed dimension is then fed into the attention module for interaction over time.

[0097]

[0098] This represents a temporal self-attention module. This corresponds to the number of modules in the four stages of the encoder.

[0099] The method described in the above embodiments filters out interference information unrelated to the target through the self-attention module in the feature encoding module, thereby obtaining the filtered observation information of future frames and effectively improving the accuracy of target tracking.

[0100] In some embodiments, the location estimation module is specifically used for:

[0101] Based on the observation information of future frames in the filtered search area and the target vector of the future frames, the target vector with enhanced observation information is obtained.

[0102] The target trajectory is predicted based on the target vector enhanced by observation information.

[0103] Specifically, in the embodiments of this application, based on the updated target vector Temporal target perception observation features of the search area Predict the target trajectory in the current and next two frames. Output from encoder Separation within the template. It should be noted that the template's future prediction features... It was not used as model input because the goal is to locate the target in the search region, and target information was already incorporated after the encoder modeled the model. In the middle. Target vector of the future frame. First, use the temporal mutual attention module to... Extracting location-related feature information yields the target vector enhanced with observation information:

[0104]

[0105] in It is a temporal mutual attention module that outputs a target vector enhanced with observation information. Used to predict the target trajectory in the current and next two frames. That is, in this embodiment of the application, the target vector of the future frame is... Scene information in future frames Through interaction, a target vector integrating future scene information is obtained. In other words, this application first expands from point to surface, from the target vector to the prediction of future scenarios; then it expands from surface to point, encoding the environmental information of the future scenario into the target vector, thus obtaining a target vector that integrates information from the future scenario. Furthermore, based on the target vector that integrates information about future scenarios By performing classification and regression, we can obtain more accurate target tracking results.

[0106] It should be noted that a key challenge in future prediction is the high degree of uncertainty, largely due to the inability to clearly define the target's intent and potential characteristics. For example, targets typically exhibit multimodal future distributions: they can turn, go straight, decelerate, accelerate, etc. This high degree of multimodality also exists in motion prediction tasks for autonomous driving. Accordingly, this application designs multimodal prediction to generate the possible trajectories of the target vehicle, customizing a specific multimodal prediction method for the temporal position estimation proposed in this application. Figure 7 As shown, the target vector for each timestamp uses a stacked linear mapping layer to predict the probabilities of three possible future scenarios. and target state :

[0107]

[0108] After prediction, the target state with the highest probability at each time stamp constitutes the final target trajectory. For training, this application first calculates the target state predicted for each mode. Compared with the true value The prediction with the highest GIoU is assigned a probability label of 1 (the other two are 0), and an L1 loss is applied. And GIoU loss The target state is applied to the output. The probability is determined by 0-1 labels through cross-entropy loss. Supervision.

[0109] For example, the temporal target tracking device based on future scene prediction proposed in this application is trained at the video sequence level in the following manner to achieve end-to-end learning and prediction of the dynamic changes in the environment and trajectory. This application constructs a video segment containing four frames for training (i.e., the first frame is used as a template, and the subsequent three frames are used as the tracking sequence). For each new input video segment, the hidden state vector in the GRU is initialized, which avoids interference from the target in the previous tracked video. Considering the consistent time step between training and inference, three consecutive frames are sampled for the search sequence. The overall optimization objective of the proposed framework includes the uice loss from the future prediction module. and reconstruction losses And the probability loss from multi-model prediction. L1 loss And GIoU loss ,

[0110]

[0111] in The value is the balancing weight coefficient.

[0112] Optionally, during inference, the hidden state vector in the GRU is also initialized for each new input video segment. Notably, the proposed framework does not require sampling video segments for inference; instead, it automatically caches historical dynamics using a defined target vector. Therefore, the designed temporal target tracking method based on future scene prediction only requires frame-level input, consistent with previous non-temporal work, and is easy to deploy. Furthermore, the distribution mapping and image reconstruction in the future prediction module are not needed during the inference phase but can be extended to other tasks, such as video generation.

[0113] The method in the above embodiments uses the target vector of future frames. Scene information in future frames Through interaction, a target vector integrating future scene information is obtained. Furthermore, based on the target vector that integrates information about future scenarios By performing classification and regression, we can obtain more accurate target tracking results.

[0114] In summary, this application proposes a future scene prediction technique for temporal visual target tracking to overcome the problems of accumulated errors and reduced effectiveness of historical information in previous temporal learning methods. This method defines temporal learning in visual target tracking as a process of predicting the future from the past. By learning from the dynamic changes in the past, it predicts the future state of the target, helping the model better distinguish the target from appearance deformation and background interference, thus achieving accurate and robust visual target tracking in various complex scenes.

[0115] (1) This application redefines the tracking problem as a process of predicting the future from the past, and for the first time, passes temporal information to future frames to better adapt to new scenes, target appearance deformation, and background shifts in visual target tracking. This application learns the spatiotemporal dynamic information of the tracking scene based on a future prediction module. This module tracks the dynamic changes of the scene as the current observation results evolve and predicts the future state, thereby giving the tracker the ability to supervise and make future predictions through real future video frames.

[0116] (2) Unlike traditional time-series learning which only uses historical prediction information, this application predicts the future state of the target by learning historical dynamic changes and performs multimodal prediction of the target's motion to cover the uncertainty of different future scenarios, thereby enabling the tracker to have better generalization ability for appearance deformation and background shift.

[0117] (3) This application is aimed at the temporal learning scenario of visual target tracking. It uses historical frame information and predicted future frame information to enhance the discriminativeness of the learned target features, enhance the model's understanding of the tracking scenario and improve the accuracy and stability of tracking, and avoid the problems of error accumulation and insufficient temporal information learning in previous temporal learning frameworks.

[0118] The target tracking device provided by the present invention is described below. The target tracking device described below can be referred to in correspondence with the target tracking method described above. The target tracking device of the embodiments of this application is as follows: Figure 8 As shown, it includes:

[0119] The acquisition module 810 is used to acquire the observation information of the current frame;

[0120] The prediction module 820 is used to input the observation information of the current frame into the temporal target tracking model based on future scene prediction and output the target trajectory; wherein, the temporal target tracking model based on future scene prediction is used to determine the observation information of future frames based on the observation information of the current frame and predict the target trajectory based on the observation information of future frames.

[0121] Optionally, the time-series target tracking model based on future scene prediction includes:

[0122] Future prediction module; the future prediction module is used to determine the observation information of future frames based on the observation information of the current frame and the target vector; the target vector is used to represent the position information of the target;

[0123] Feature encoding module; The feature encoding module is used to obtain the observation information of the future frames of the filtered search area based on the observation information of the future frames;

[0124] Position estimation module; The position estimation module is used to predict the target trajectory based on the observation information of future frames of the filtered search area and the target vector of the future frames.

[0125] Optionally, the future prediction module is specifically used for:

[0126] Based on the observation information and target vector of the current frame, determine the target vector of the future frame;

[0127] Based on the target vector of the future frame, determine the observation information of the future frame.

[0128] Optionally, the feature encoding module includes:

[0129] Feature extraction module and self-attention module;

[0130] The feature extraction module is used to extract features from the observation information of future frames; the self-attention module is used to filter out interference information that is irrelevant to the target after feature extraction, so as to obtain the observation information of future frames of the filtered search area.

[0131] Optionally, the location estimation module is specifically used for:

[0132] Based on the observation information of future frames in the filtered search area and the target vector of the future frames, the target vector with enhanced observation information is obtained.

[0133] The target trajectory is predicted based on the target vector enhanced by observation information.

[0134] Figure 9 A schematic diagram of the physical structure of an electronic device is provided. This electronic device may include a processor 910, a communications interface 920, a memory 930, and a communication bus 940. The processor 910, communications interface 920, and memory 930 communicate with each other via the communication bus 940. The processor 910 can call logical instructions in the memory 930 to execute a target tracking method. This method includes: acquiring observation information of the current frame; inputting the observation information of the current frame into a temporal target tracking model based on future scene prediction, and outputting a target trajectory; wherein the temporal target tracking model based on future scene prediction is used to determine the observation information of future frames based on the observation information of the current frame; and predicting the target trajectory based on the observation information of future frames.

[0135] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0136] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the target tracking method provided by the above methods. The method includes: acquiring observation information of the current frame; inputting the observation information of the current frame into a temporal target tracking model based on future scene prediction, and outputting a target trajectory; wherein, the temporal target tracking model based on future scene prediction is used to determine the observation information of future frames based on the observation information of the current frame; and predicting the target trajectory based on the observation information of future frames.

[0137] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the target tracking method provided by the above methods. The method includes: acquiring observation information of the current frame; inputting the observation information of the current frame into a temporal target tracking model based on future scene prediction, and outputting a target trajectory; wherein the temporal target tracking model based on future scene prediction is used to determine the observation information of future frames based on the observation information of the current frame; and predicting the target trajectory based on the observation information of future frames.

[0138] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0139] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A target tracking method, characterized in that, include: Obtain observation information for the current frame; The observation information of the current frame is input into a temporal target tracking model based on future scene prediction, and the target trajectory is output; wherein, the temporal target tracking model based on future scene prediction is used to determine the observation information of future frames based on the observation information of the current frame; and to predict the target trajectory based on the observation information of future frames. The time-series target tracking model based on future scenario prediction includes: A future prediction module; the future prediction module is used to determine the observation information of future frames based on the observation information of the current frame and the target vector; the target vector is used to represent the position information of the target; Feature encoding module; the feature encoding module is used to obtain the observation information of the future frames of the filtered search area based on the observation information of the future frames; A position estimation module; the position estimation module is used to predict the target trajectory based on the observation information of future frames of the filtered search area and the target vector of the future frames; The future prediction module is specifically used for: Based on the observation information and target vector of the current frame, the target vector of the future frame is determined; the observation information of the current frame is constructed based on the linear projection of the visual template and the search region; the target vector of the future frame is obtained based on the autoregression of the observation information and target vector of the current frame by the gated recurrent unit. The observation information of the future frame is determined based on the mutual attention between the target vector of the future frame and the observation information of the current frame; the future prediction module is trained based on the future prediction loss and the reconstruction loss; the future prediction loss is determined based on the distance between the predicted observation information of the future frame and the actual observation information of the future frame; the reconstruction loss is determined based on the predicted future tracking scene and the actual future search area; the predicted future tracking scene is obtained by the decoder reconstructing the observation information of the future frame.

2. The target tracking method according to claim 1, characterized in that, The feature encoding module includes: Feature extraction module and self-attention module; The feature extraction module is used to extract features from the observation information of the future frame; the self-attention module is used to filter out interference information that is irrelevant to the target after the feature extraction to obtain the observation information of the future frame of the filtered search area.

3. The target tracking method according to claim 1 or 2, characterized in that, The location estimation module is specifically used for: Based on the observation information of future frames in the filtered search region and the target vector of the future frames, an observation-enhanced target vector is obtained. Based on the enhanced target vector from the observed information, the target trajectory is predicted.

4. A target tracking device for implementing the method as claimed in any one of claims 1-3, characterized in that, include: The acquisition module is used to acquire observation information for the current frame; The prediction module is used to input the observation information of the current frame into a temporal target tracking model based on future scene prediction and output the target trajectory; wherein, the temporal target tracking model based on future scene prediction is used to determine the observation information of future frames based on the observation information of the current frame; The target trajectory is predicted based on the observation information of the future frame.

5. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the target tracking method as described in any one of claims 1 to 3.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the target tracking method as described in any one of claims 1 to 3.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the target tracking method as described in any one of claims 1 to 3.

Images