Target tracking method, device and equipment

Abstract

The invention provides a target tracking method, device and equipment. The method comprises the following steps: acquiring observation information of a current frame; inputting the observation information of the current frame into a time sequence target tracking model based on future scene prediction, and outputting a target trajectory; wherein the time sequence target tracking model based on future scene prediction is used for determining observation information of a future frame according to the observation information of the current frame; and predicting a target trajectory according to observation information of a future frame. According to the method provided by the embodiment of the invention, the problems of accumulative errors and weakened historical information effectiveness in a conventional time sequence modeling mode can be effectively solved.

Description

Technical Field

[0001] The present invention relates to the field of target tracking technology, and in particular to a target tracking method, device and equipment. Background Art

[0002] Visual object tracking is a core problem in the field of computer vision, which aims to automatically and continuously locate and analyze the position, size, shape and motion trajectory of a specified target object through video sequences or image sequences.

[0003] In the related technologies, the temporal target tracking method based on template update and the temporal target tracking method based on target motion estimation are essentially frameworks that use historical prediction information as prior information for inferring the current frame model. The schemes that rely on historical frame prediction information are prone to error accumulation, resulting in tracking failure or loss of target. Summary of the Invention

[0004] The present invention provides a target tracking method, apparatus and device, which utilize historical information and predicted future information to enhance tracking capabilities, effectively overcoming the problems of cumulative errors and reduced effectiveness of historical information in previous time series modeling methods.

[0005] The present invention provides a target tracking method, which includes the following steps.

[0006] Get the observation information of the current frame; The observation information of the current frame is input into a time series target tracking model based on future scene prediction to output a target trajectory; wherein the time series target tracking model based on future scene prediction is used to determine the observation information of the future frame based on the observation information of the current frame; and predict the target trajectory based on the observation information of the future frame.

[0007] According to a target tracking method provided by the present invention, the temporal target tracking model based on future scene prediction includes: A future prediction module; the future prediction module is used to determine the observation information of the future frame based on the observation information of the current frame and the target vector; the target vector is used to represent the position information of the target; Feature encoding module; the feature encoding module is used to obtain the observation information of the future frame of the filtered search area based on the observation information of the future frame; Position estimation module; the position estimation module is used to predict the target trajectory based on the observation information of the future frame of the filtered search area and the target vector of the future frame.

[0008] According to a target tracking method provided by the present invention, the future prediction module is specifically used to: Determining a target vector for a future frame based on the observation information and the target vector of the current frame; Observation information of the future frame is determined according to the target vector of the future frame.

[0009] According to a target tracking method provided by the present invention, the feature encoding module includes: Feature extraction module and self-attention module; The feature extraction module is used to extract features from the observation information of the future frame; the self-attention module is used to filter out interference information irrelevant to the target after the feature extraction, and obtain the observation information of the future frame of the filtered search area.

[0010] According to a target tracking method provided by the present invention, the position estimation module is specifically used for: Obtaining an observation information enhanced target vector based on the observation information of the future frame of the filtered search area and the target vector of the future frame; The target trajectory is predicted based on the target vector enhanced by the observation information.

[0011] The present invention also provides a target tracking device, comprising the following modules: Acquisition module, used to obtain the observation information of the current frame; A prediction module is used to input the observation information of the current frame into a time-series target tracking model based on future scene prediction, and output a target trajectory; wherein the time-series target tracking model based on future scene prediction is used to determine the observation information of the future frame based on the observation information of the current frame; and predict the target trajectory based on the observation information of the future frame.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any one of the target tracking methods described above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which implements any of the target tracking methods described above when executed by a processor.

[0014] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the target tracking methods described above.

[0015] The target tracking method, device and equipment provided by the present invention utilize historical information and predicted future information to enhance tracking capabilities. This time series modeling method is significantly superior to existing methods that are limited to using historical frame predictions, and effectively overcomes the problems of cumulative errors and reduced effectiveness of historical information in previous time series modeling methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 It is a schematic diagram of the target tracking framework provided by the present invention.

[0018] Figure 2 This is one of the flow charts of the target tracking method provided by the present invention.

[0019] Figure 3 This is a schematic diagram comparing the effects of different temporal visual target tracking methods provided by the present invention.

[0020] Figure 4 It is a structural diagram of the target tracking model provided by the present invention.

[0021] Figure 5 It is a structural diagram of the future prediction module provided by the present invention.

[0022] Figure 6 It is a structural diagram of the feature coding module provided by the present invention.

[0023] Figure 7 It is a structural diagram of the position estimation module provided by the present invention.

[0024] Figure 8 It is a structural schematic diagram of the target tracking device provided by the present invention.

[0025] Figure 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0026] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0027] The following combination Figures 1-9 The target tracking method, apparatus and device of the present invention are described.

[0028] In order to facilitate a clearer understanding of the technical solutions of the various embodiments of the present application, some technical contents related to the various embodiments of the present application are first introduced.

[0029] Visual object tracking is a core problem in computer vision. It aims to automatically and continuously locate and analyze the position, size, shape, and motion trajectory of a specified target object within a video or image sequence. Temporal visual tracking targets a specific region or object within a video frame or image, without requiring semantic information (such as its category). This concept can be described as "everything tracking." Given the state (position, size, etc.) of an object in one frame of a video, visual object tracking requires estimating its state in subsequent frames. As a complement to detection algorithms, it can provide the spatial location of the object within a video or sequential, semantically related images, reducing overall system complexity. (For example, detection may only be applied to the first frame of a video to identify the object and determine its location within some subsequent frames, with tracking then applying the remaining frames to determine the target's location.) Classic application areas of visual object tracking in computer vision include security (vehicle tracking, license plate recognition, etc.), surveillance (face recognition, gait recognition, etc.), inspection (drone tracking, robot navigation, etc.), as well as emerging applications in smart living (human-computer interaction, VR / AR, etc.), smart cities (traffic monitoring, etc.), and smart industry (telemedicine, etc.). For example, in an autonomous driving system, by tracking pedestrians, vehicles and other targets around the vehicle, accurate perception information is provided for downstream trajectory prediction and decision planning, ensuring that the vehicle can successfully avoid collisions with surrounding objects in the future and providing users with a comfortable driving experience.

[0030] Visual object tracking is a long-term, continuous process. The appearance of the target and the surrounding background may change significantly from the initial designation, thus affecting the accuracy of target recognition and spatial position estimation. Therefore, temporal learning—continuously updating the model's understanding of the latest state of the tracked target—is key to ensuring accurate and continuous perception in visual object tracking tasks. Existing temporal learning methods mostly use historical frame predictions to update the visual template of the visual object tracking framework to mitigate the appearance deformation and background shift of the target during long-term tracking. However, the predictions of historical frames often have some errors. After continuous updating of multiple frames, these errors gradually accumulate, affecting the model's understanding of the tracked target, ultimately leading to target recognition failure and spatial estimation bias. These temporal learning frameworks essentially treat visual object tracking as a single-frame detection task, namely, finding the target by matching the provided visual template across the entire image. They do not consider the spatiotemporal continuity of visual object tracking. Recent techniques treat visual object tracking as a sequential regression task, continuously predicting the motion offset of the tracked target based on the spatial position estimated in the previous frame. This temporal learning method avoids the underutilization of the temporal characteristics of single-frame matching to a certain extent, and associates the previous and next frames through motion estimation to achieve more accurate and stable tracking results. However, it still cannot overcome the accumulation of historical errors. At the same time, the appearance difference between the previous and next frames will also weaken the effectiveness of historical prediction information.

[0031] Exemplary, the temporal target tracking method based on template updating and the temporal target tracking method based on target motion estimation are as follows: (1) Temporal target tracking method based on template updating.

[0032] Temporal object tracking methods based on template updating are the most commonly used temporal learning methods. For each input image frame, the model predicts the spatial position and confidence of the tracked target. The visual template is then updated based on whether the confidence exceeds a preset threshold. If the threshold is met, the current estimated target state is considered reliable, and the prediction result of the current frame is cropped and used as the visual template for the next frame. If the threshold is not met, such as when the target is completely occluded or out of sight, or when the tracker's prediction result is incorrect, the cropped template is unreliable, so the previous visual template is retained. Some subsequent methods use momentum updating strategies to retain the information of the initial template. Other methods retain the initial visual template and dynamically maintain the predicted template for each frame, which also participates in the model matching process. They extract distinguishable spatiotemporal features by modeling the global relationship between all elements in the spatial and temporal dimensions. However, the accumulated errors caused by long-term tracking can reduce the effectiveness of the new visual template, leading to tracking errors or spatial estimation bias.

[0033] (2) Temporal target tracking method based on target motion estimation.

[0034] Temporal object tracking methods based on target motion estimation treat visual object tracking as a sequential regression task, using the target's spatial position coordinates as continuously maintained and updated model variables. These methods continuously predict the target's motion offset based on the spatial position estimated in the previous frame. Given an input image, the spatial position features representing the target are combined with semantic features extracted from the image to filter motion-related feature information and estimate its position. The prediction for the new frame is obtained by adding the target's spatial position offset to the global coordinates of the previous frame. Compared to temporal visual object tracking methods based on template updating, these methods treat visual object tracking as a sequential regression task, using the target's spatial position coordinates as continuously maintained and updated model variables. These methods continuously predict the target's motion offset based on the spatial position estimated in the previous frame. Some methods also incorporate template updating strategies by inputting the image region corresponding to the previous frame's prediction into the model to further enhance spatiotemporal information. This temporal learning method avoids the underutilization of the temporal characteristics of single-frame matching to a certain extent, and associates the previous and next frames through motion estimation to achieve more accurate and stable tracking results. However, it still cannot overcome the accumulation of historical errors. At the same time, the appearance difference between the previous and next frames will also weaken the effectiveness of historical prediction information.

[0035] Temporal target tracking methods based on template updating and target motion estimation. Although the model designs differ slightly, both frameworks essentially use historical prediction information as prior information for model inference in the current frame. Considering that historical frame predictions cannot be 100% accurate, their errors will accumulate over the tracking process, eventually leading to target loss in a certain frame. Subsequently, completely incorrect prediction results will continue to affect the predictions of subsequent frames, interfering with the model's spatial position estimation, rather than correctly re-locking the target. At the same time, the robustness of visual target tracking relies on the model's understanding of the tracking scene, enabling it to quickly adapt and distinguish the target from the background in the event of an emergency. These frameworks only use historical prediction information, rather than learning and understanding the dynamic changes in the tracking scene from changes in the target state. This leads to insufficient utilization of the temporal characteristics of the continuous tracking process and weakens the effectiveness of historical prediction information.

[0036] This application proposes a temporal target tracking method based on future scene prediction, primarily designed to overcome the problems of error accumulation and insufficient environmental understanding caused by insufficient utilization of spatiotemporal information in temporal visual target tracking. It aims to achieve category-independent perception in visual object tracking and enhance the model's adaptability to long-term target appearance changes and background interference. Current trackers typically formulate tracking as a one-shot detection problem. Given the significant environmental drift and target appearance deformation associated with long-term tracking, non-temporal architectures that rely solely on information from the current frame struggle to ensure generalization to new frames. Outdated templates struggle to provide effective target perception knowledge and, in turn, interfere with the tracking process. While many works have intuitively attempted to address this issue by maintaining the temporal validity of target information, such as using predictions with high confidence scores from historical frames to update visual templates or transferring historical trajectories as prior knowledge, these methods remain limited by their temporal architectures that exclusively utilize historical information, limiting temporal information when transmitting previous observations. Therefore, developing a method that can fully learn the spatiotemporal correlations between previous and subsequent frames in a tracking sequence, improve scene understanding, and accurately perceive the target, remains an unmet challenge. The following content of this application provides a more in-depth introduction to the existing time-series target tracking algorithm and the world model theory used, and lists the experimental schemes that are most similar to the present invention.

[0037] For example, Figure 1 As shown, this application will formally give the definitions of different tracking frameworks to better understand the technical background of temporal visual target tracking and provide symbolic definitions for the following text.

[0038] First, we introduce the most basic non-temporal target tracking framework. Given the template image of the target in the first frame , the goal of visual object tracking is to obtain continuous input images Positioning target ,in Represents the target state, i.e. a bounding box. The mainstream tracking paradigm aims to learn a deep tracking model , the model performs localization according to different detection paradigms (i.e., anchor-based architectures such as Faster R-CNN[1], or anchor-free architectures such as CenterNet[2]) and outputs The tracking process can be expressed as: In order to mitigate the impact of target appearance deformation and background changes in long-term tracking, recently popular temporal trackers use the model output of historical frames. As additional prior knowledge, such as updating templates (such as TrDiMP[3], STARK[4]) or passing historical target positions (such as SeqTrack[5], AR-Track[6][7]). The temporal tracking process can be expressed as: In contrast, the method proposed in this application defines temporal visual tracking as a process of predicting the future from the past, which is based on the environment dynamics information learned from historical observations (cached in the tracking model). The target vector ), to predict the future trajectory of the target (including the current frame and the next two frames). This modeling process can be expressed as: By parametrically modeling the environment dynamics ( ) instead of explicitly passing the history output , which not only enables better generalization in new situations in future frames, but also reduces the risk of error accumulation from past experience.

[0039] 2. World Model Technology Background Recent years have seen significant progress in artificial intelligence, particularly in deep learning and reinforcement learning. However, current AI systems still face challenges such as low sample efficiency, poor generalization, and difficulty handling the complex real-world environment. To address these issues, building "world models" that can understand and predict environmental dynamics has become a highly sought-after research area. A world model, developed in the field of reinforcement learning, is defined as a model that predicts the future state of an agent's environment.

[0040] The core idea of ​​the world model is to build an internal model of the external world for prediction, planning, and control. Its framework consists of three main parts: 1) Visual model (V): Use variational autoencoder (VAE[8]) to compress the raw pixel input into a low-dimensional latent vector The encoder uses convolutional layers to extract image features, and the decoder uses deconvolutional layers to reconstruct the image. The training objectives include reconstruction loss (L2 distance) and KL divergence loss (regularization term) to balance the reconstruction quality and the regularity of the latent space. 2) Memory model (M): Use recurrent neural networks (LSTM[9] / RNN

[10] ) to predict the latent vector at the next moment. and other environmental information. 3) Control model (C): A simple linear model that uses the latent vector at the current moment and the hidden state of the recurrent neural network Output Action . The model is trained through a covariance matrix adaptive evolutionary strategy with the goal of maximizing the cumulative reward. Based on the world model theory, a series of algorithms have been proposed, such as Dreamer

[11] , which imagines and learns from the learned world model latent space. DreamerV2

[12] uses discrete latent variables to represent states, improves learning efficiency, and achieves significant performance improvements on Atari games. DreamerV3

[13] is committed to improving the versatility and robustness of the algorithm, and has achieved performance that surpasses domain-specific expert algorithms on more than 150 different tasks.

[0041] Figure 2 This is one of the flow charts of the target tracking method provided by the present invention, such as Figure 2 As shown, the method includes the following: Step 201: Obtain observation information of the current frame.

[0042] Specifically, existing time-series target tracking technologies cannot overcome the target's appearance deformation and background shift during long-term tracking. Solutions that rely on historical frame prediction information are prone to error accumulation, leading to tracking failure or target loss. Furthermore, they cannot form a generalized understanding of the overall tracking scenario and are insufficiently adaptable to unexpected situations. From an application perspective, existing time-series target tracking technologies are limited in effectiveness and versatility, making them incapable of simultaneously applying them to long-term tracking and scenarios with rapidly moving targets and rapidly changing backgrounds.

[0043] In order to solve the above problem, in the embodiment of the present application, the observation information of the current frame is first obtained. Optionally, for the input visual template and search area , the image is segmented and flattened, and the observation information of the current frame is constructed using linear projection .

[0044] Step 202: Input the observation information of the current frame into a time series target tracking model based on future scene prediction, and output the target trajectory; wherein the time series target tracking model based on future scene prediction is used to determine the observation information of the future frame based on the observation information of the current frame; and predict the target trajectory based on the observation information of the future frame.

[0045] Specifically, after obtaining the observation information of the current frame, the embodiment of the present application inputs the observation information of the current frame into a time series target tracking model based on future scene prediction, and outputs the target trajectory. Among them, the time series target tracking model based on future scene prediction is used to determine the observation information of the future frame based on the observation information of the current frame; and predict the target trajectory based on the observation information of the future frame. That is, the present application defines time series learning in visual target tracking as a process of predicting the future from history, that is, predicting the future state of the target based on historical observations, and using historical information and predicted future information to enhance tracking capabilities, effectively overcoming the problems of cumulative errors and weakened effectiveness of historical information in previous time series modeling methods, and its effect is significantly better than the previous method that was limited to using historical frame prediction results.

[0046] For example, Figure 3 As shown in the figure, in order to qualitatively analyze the performance of different temporal visual target tracking methods in long-term tracking scenarios, this application selects corresponding test video sequences and uses the same target for initialization. Then, the prediction structures of different trackers after a certain number of frames of long-term tracking are visualized to illustrate the tracking performance in the face of target appearance deformation and background offset. Previous temporal learning methods usually fail to track when tracking more than 100 frames due to error accumulation and insufficient scene understanding, that is, they lock the wrong target. The method proposed in this application can understand the tracking scene by learning the dynamic changes of historical frames, overcome the target appearance changes and background offset, and accurately lock the target. Through verification, compared with the existing technology, the method of this application can achieve an improvement of tracking success rate of up to 2.9%, 8.9% and 9.7% on the visual target tracking task OTB100

[14] , LaSOT

[15] and TNL2K

[16] datasets respectively.

[0047] The method of the above embodiment uses historical information and predicted future information to enhance tracking capabilities. This time series modeling method is significantly superior to the existing method that is limited to using historical frame predictions, and effectively overcomes the problems of cumulative errors and weakened historical information effectiveness in previous time series modeling methods.

[0048] In some embodiments, a temporal target tracking model based on future scene prediction includes: Future prediction module; the future prediction module is used to determine the observation information of the future frame based on the observation information of the current frame and the target vector; the target vector is used to represent the position information of the target; Feature encoding module; the feature encoding module is used to obtain the observation information of the future frame of the filtered search area based on the observation information of the future frame; Position estimation module: The position estimation module is used to predict the target trajectory based on the observation information of the future frame of the filtered search area and the target vector of the future frame.

[0049] Specifically, if Figure 4 As shown, the temporal target tracking model based on future scene prediction in the embodiment of the present application includes a future prediction module, a feature encoding module and a position estimation module. Among them, the future prediction module is used to determine the observation information of the future frame based on the observation information and target vector of the current frame; the target vector is used to represent the position information of the target, thereby achieving the effect of enhancing the tracking capability based on the predicted future information. The feature encoding module is used to obtain the observation information of the future frame of the filtered search area based on the observation information of the future frame; that is, the feature encoding module is used to perform feature purification, find more noteworthy information, and filter out interference information. The position estimation module is used to predict the target trajectory based on the observation information of the future frame of the filtered search area and the target vector of the future frame, thereby realizing the use of historical frame information and predicted future frame information to enhance the discriminability of the learned target features, enhance the model's understanding of the tracking scene and improve the accuracy and stability of tracking, and avoid the error accumulation and insufficient temporal information learning problems in the previous temporal learning framework.

[0050] In the method of the above embodiment, the future prediction module determines the observation information of the future frame based on the observation information and target vector of the current frame, thereby achieving the effect of enhancing the tracking capability based on the predicted future information. The feature encoding module obtains the observation information of the future frame of the filtered search area based on the observation information of the future frame, thereby achieving feature purification and filtering of interference information. The position estimation module predicts the target trajectory based on the observation information of the future frame of the filtered search area and the target vector of the future frame, that is, it uses historical frame information and predicted future frame information to enhance the discriminability of the learned target features, enhance the model's understanding of the tracking scene and improve the accuracy and stability of tracking, avoid the error accumulation and insufficient learning of temporal information problems of previous temporal learning frameworks, and effectively improve the accuracy of target tracking.

[0051] In some embodiments, the future prediction module is specifically configured to: Determine the target vector of the future frame based on the observation information and target vector of the current frame; According to the target vector of the future frame, the observation information of the future frame is determined.

[0052] Specifically, if Figure 5 As shown, three trainable target vectors are initialized in the future prediction module in the embodiment of the present application. , represents the scene dynamic characteristics of the current and next two frames. For each input visual template and search area , the image is segmented and flattened, and the observation information of the current frame is constructed using linear projection Then, through the gated recurrent unit (GRU

[17] ) and Autoregressively update the cached temporal features to obtain the target vector for the future frame: That is, the tracking problem is redefined as a process of predicting the future from history, and for the first time, the temporal information is passed to future frames to better adapt to new scenes, target appearance deformation and background offset in visual target tracking.

[0053] The goal of the temporal target tracking method proposed in this application is to enhance the generalization ability of the model in a new frame of complex scenes, so predicting the dynamic changes of the future tracking environment is a prerequisite. The updated target vector Used with Obtain observation information of future frames through mutual attention mechanism : It should be noted that and The interaction between them aims to enhance the feature quality of the current frame by introducing historical information, which comes from the previous GRU update. .

[0054] That is, the future prediction module in the embodiment of this application draws on the idea of ​​predicting future scene distributions using a world model, enabling the tracker to predict the future state of the target from the learned historical dynamic changes, helping the model to better distinguish targets from appearance deformation and background interference, and providing an effective solution for robust tracking of video sequences facing different categories of targets and complex scene changes. In addition, while tracking the target, this method can also evolve the next few frames of video sequences through the predicted future state, which can be used for video generation and annotation, and has great expansion potential.

[0055] Optionally, following the loss design in the world model, the predicted future scene features and real scene characteristics Mapping to distribution and , and then minimize their KL divergence. It is worth noting that this application only considers the future prediction process of the supervised search area, because the input visual template is always from the initial frame. The prior distribution of the mapping is regarded as a prediction of the future scene dynamics. The posterior distribution of the mapping represents the observed The KL divergence measures the distance between the predicted future scene (prior) and the real future scene (posterior). This application optimizes the future prediction loss. To enhance the model's ability to make future predictions: ) in represents the KL divergence.

[0056] In addition to supervision from distribution differences, this application The decoder is used to reconstruct the search area to further improve the accuracy of future scene prediction. Predicted future tracking scene By the actual future search area By optimizing the reconstruction loss To supervise, including is the mean square error: The method of the above embodiment redefines the tracking problem as a process of predicting the future from the past, and for the first time passes the timing information to the future frame, so as to better adapt to new scenes, target appearance deformation and background offset in visual target tracking. Moreover, unlike traditional timing learning that only uses historical prediction information, this application predicts the future state of the target by learning historical dynamic changes and performs multimodal prediction of the target motion to cover the uncertainty of different future scenes, so that the tracker has better generalization ability for appearance deformation and background offset, effectively improving the accuracy of target tracking.

[0057] In some embodiments, the feature encoding module includes: Feature extraction module and self-attention module; The feature extraction module is used to extract features from the observation information of future frames; the self-attention module is used to filter out interference information irrelevant to the target after feature extraction, and obtain the observation information of future frames in the filtered search area.

[0058] Specifically, this application uses the visual backbone feature extraction network ViT as an encoder and jointly models the search and template images. The encoder extracts the features of the search and template images and learns feature-level correspondences to filter out interference information that is irrelevant to the target. The input future prediction scene features (including visual templates and search areas) are concatenated along the spatial dimension and input to the encoder for feature extraction: in , Block represents the Transformer layer for spatial modeling in the encoder, is the number of layers. In addition to modeling the space between the template and the search area in the Block, the extracted target perception information can also be used to refine the temporal dynamic information along the timeline. Therefore, after each feature extraction stage of the encoder, this application arranges a temporal self-attention module to filter out interference items not related to the target when extracting target information at different levels. Figure 6 As shown, the predicted future scene characteristics Input to the attention module after transposing the dimensions to allow for interaction in the time dimension: represents the temporal self-attention module, is the number of modules corresponding to the four stages of the encoder.

[0059] The method of the above embodiment filters interference information irrelevant to the target through the self-attention module in the feature encoding module to obtain filtered observation information of the future frame, thereby effectively improving the accuracy of target tracking.

[0060] In some embodiments, the location estimation module is specifically configured to: Obtaining a target vector enhanced with observation information based on the observation information of the future frame of the filtered search area and the target vector of the future frame; The target trajectory is predicted based on the target vector enhanced by the observation information.

[0061] Specifically, in the embodiment of the present application, according to the updated target vector and temporal target perception observation characteristics of the search area Predict the target trajectory in the current and next two frames, Output from encoder It should be noted that the future prediction features of the template is not used as model input because the goal is to locate the object in the search region and the target information is introduced after the encoder modeling The target vector for the future frame First, we use the temporal mutual attention module to Extract the positioning-related feature information and obtain the target vector with enhanced observation information: in It is a temporal mutual attention module that outputs a target vector enhanced by observation information It is used to predict the target trajectory of the current and next two frames. That is, in the embodiment of the present application, the target vector of the future frame is and scene information of future frames Interact to obtain a target vector that integrates future scene information That is, this application first expands from point to surface, from target vector to prediction of future scenes; then from surface to point, the environmental information of the future scene is encoded into the target vector, and the target vector that integrates the future scene information is obtained. , and then based on the target vector that integrates future scene information By performing classification regression, more accurate target tracking results can be obtained.

[0062] It should be noted that the key challenge of future prediction is the high degree of uncertainty, which is largely due to the inability to clearly track the intentions and potential characteristics of the target. For example, the target usually has a multi-modal future distribution: it can turn, go straight, slow down, speed up, etc. High multimodality also exists in the motion prediction task of autonomous driving. This application accordingly designs a multi-modal prediction to generate the possible trajectory of the target vehicle, and customizes a special multi-modal prediction for the time series position estimation proposed in this application. Figure 7 As shown, the target vector for each timestamp predicts the probabilities of three possible future situations using stacked linear mapping layers and target state : After prediction, the target state with the highest probability at each timestamp constitutes the final target trajectory. For training, this application first calculates the target state predicted by each mode and the true value The GIoU between

[19] . The prediction with the largest GIoU is assigned a probability label of 1 (the other two are 0), and the L1 loss and GIoU loss Applied to the output target state. The probability is calculated by 0-1 label through cross entropy loss. Supervision.

[0063] Exemplarily, the temporal target tracking device based on future scene prediction proposed in this application is based on the following method, which is trained at the level of video sequences to achieve end-to-end learning and prediction of the dynamic changes of the environment and trajectory. This application constructs a video clip containing 4 frames for training (i.e., the first frame is used as a template and the subsequent three frames are used as tracking sequences). For each new input of the video clip, the hidden state vector in the GRU will be initialized, which avoids interference from the previous tracked video target. Taking into account the consistent time step between training and inference, three consecutive frames are sampled for the search sequence. The overall optimization goal of the proposed framework includes the uice loss from the future prediction module and reconstruction losses , and the probability loss from multimodal prediction , L1 loss and GIoU loss , in The value is the balance weight coefficient.

[0064] Optionally, during inference, the hidden state vector in the GRU is also initialized for each new input of a video clip. Notably, the proposed framework does not require sampling of video clips for inference, but instead automatically caches historical dynamics using a defined target vector. Therefore, the designed temporal object tracking method based on future scene prediction only requires frame-level input, which is consistent with previous non-temporal work and is easy to deploy. In addition, the distribution mapping and image reconstruction in the future prediction module are not required during the inference phase, but can be extended to other tasks such as video generation.

[0065] The method of the above embodiment is to set the target vector of the future frame and scene information of future frames Interact to obtain a target vector that integrates future scene information , and then based on the target vector that integrates future scene information By performing classification regression, we can obtain more accurate target tracking results.

[0066] In summary, this application proposes a future scene prediction technology for temporal visual target tracking, designed to overcome the cumulative errors and reduced effectiveness of historical information in previous temporal learning. This method defines temporal learning in visual target tracking as the process of predicting the future from the past. By learning the dynamic changes in the historical state, the target's future state is predicted, helping the model better distinguish targets from appearance deformation and background interference, achieving accurate and robust visual target tracking in a variety of complex scenarios.

[0067] (1) This application redefines the tracking problem as a process of predicting the future from the past and, for the first time, transfers temporal information to future frames to better adapt to new scenes, target appearance deformations, and background shifts in visual target tracking. This application learns the spatiotemporal dynamic information of the tracking scene based on a future prediction module. This module tracks the dynamic changes of the scene and predicts the future state as the current observation results evolve, thereby giving the tracker the ability to supervise and make future predictions through real future video frames.

[0068] (2) Unlike traditional temporal learning that only uses historical prediction information, this application predicts the future state of the target by learning historical dynamic changes and performs multimodal prediction of the target's motion to cover the uncertainty of different future scenes, thereby enabling the tracker to have better generalization capabilities for appearance deformation and background offset.

[0069] (3) This application is aimed at the temporal learning scenario of visual target tracking, which uses historical frame information and predicted future frame information to enhance the discriminability of the learned target features, enhance the model's understanding of the tracking scenario and improve the accuracy and stability of tracking, avoiding the error accumulation and insufficient temporal information learning problems of previous temporal learning frameworks.

[0070] The target tracking device provided by the present invention is described below. The target tracking device described below and the target tracking method described above can be referred to in correspondence with each other. Figure 8 As shown, including: An acquisition module 810 is used to obtain observation information of the current frame; The prediction module 820 is used to input the observation information of the current frame into the time series target tracking model based on future scene prediction and output the target trajectory; wherein the time series target tracking model based on future scene prediction is used to determine the observation information of the future frame based on the observation information of the current frame; and predict the target trajectory based on the observation information of the future frame.

[0071] Optionally, the time series target tracking model based on future scene prediction includes: Future prediction module; the future prediction module is used to determine the observation information of the future frame based on the observation information of the current frame and the target vector; the target vector is used to represent the position information of the target; Feature encoding module; the feature encoding module is used to obtain the observation information of the future frame of the filtered search area based on the observation information of the future frame; Position estimation module: The position estimation module is used to predict the target trajectory based on the observation information of the future frame of the filtered search area and the target vector of the future frame.

[0072] Optionally, the future prediction module is specifically used to: Determine the target vector of the future frame based on the observation information and target vector of the current frame; According to the target vector of the future frame, the observation information of the future frame is determined.

[0073] Optionally, the feature encoding module includes: Feature extraction module and self-attention module; The feature extraction module is used to extract features from the observation information of future frames; the self-attention module is used to filter out interference information irrelevant to the target after feature extraction, and obtain the observation information of future frames in the filtered search area.

[0074] Optionally, the position estimation module is specifically configured to: Obtaining a target vector enhanced with observation information based on the observation information of the future frame of the filtered search area and the target vector of the future frame; The target trajectory is predicted based on the target vector enhanced by the observation information.

[0075] Figure 9The present invention provides an example of a physical structure diagram of an electronic device, which may include: a processor 910, a communication interface 920, a memory 930, and a communication bus 940, wherein the processor 910, the communication interface 920, and the memory 930 communicate with each other via the communication bus 940. The processor 910 may call logic instructions in the memory 930 to execute a target tracking method, which includes: obtaining observation information of a current frame; inputting the observation information of the current frame into a time-series target tracking model based on future scene prediction, and outputting a target trajectory; wherein the time-series target tracking model based on future scene prediction is used to determine observation information of future frames based on the observation information of the current frame; and predicting the target trajectory based on the observation information of the future frame.

[0076] Furthermore, the logic instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0077] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the target tracking method provided by the above methods, which includes: obtaining observation information of the current frame; inputting the observation information of the current frame into a time series target tracking model based on future scene prediction, and outputting the target trajectory; wherein the time series target tracking model based on future scene prediction is used to determine the observation information of the future frame based on the observation information of the current frame; and predicting the target trajectory based on the observation information of the future frame.

[0078] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the target tracking method provided by the above-mentioned methods, the method comprising: obtaining observation information of the current frame; inputting the observation information of the current frame into a time series target tracking model based on future scene prediction, and outputting a target trajectory; wherein the time series target tracking model based on future scene prediction is used to determine the observation information of the future frame based on the observation information of the current frame; and predicting the target trajectory based on the observation information of the future frame.

[0079] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0080] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A target tracking method, characterized in that: include: Get the observation information of the current frame; Inputting the observation information of the current frame into a time series target tracking model based on future scene prediction to output a target trajectory; wherein the time series target tracking model based on future scene prediction is used to determine the observation information of the future frame based on the observation information of the current frame; The target trajectory is predicted based on the observation information of the future frame.

2. The target tracking method according to claim 1, characterized in that The temporal target tracking model based on future scene prediction includes: A future prediction module; the future prediction module is used to determine the observation information of the future frame based on the observation information of the current frame and the target vector; the target vector is used to represent the position information of the target; Feature encoding module; the feature encoding module is used to obtain the observation information of the future frame of the filtered search area based on the observation information of the future frame; Position estimation module; the position estimation module is used to predict the target trajectory based on the observation information of the future frame of the filtered search area and the target vector of the future frame.

3. The target tracking method according to claim 2, characterized in that The future prediction module is specifically used to: Determining a target vector for a future frame based on the observation information and the target vector of the current frame; Observation information of the future frame is determined according to the target vector of the future frame.

4. The target tracking method according to claim 2, characterized in that The feature encoding module includes: Feature extraction module and self-attention module; The feature extraction module is used to extract features from the observation information of the future frame; the self-attention module is used to filter out interference information irrelevant to the target after the feature extraction, and obtain the observation information of the future frame of the filtered search area.

5. The target tracking method according to any one of claims 2 to 4, characterized in that: The position estimation module is specifically used for: According to the observation information of the future frame of the filtered search area and the target vector of the future frame, the target vector enhanced by the observation information is obtained. The target trajectory is predicted based on the target vector enhanced by the observation information.

6. A target tracking device, characterized in that: include: Acquisition module, used to obtain observation information of the current frame; A prediction module, configured to input the observation information of the current frame into a time series target tracking model based on future scene prediction, and output a target trajectory; wherein the time series target tracking model based on future scene prediction is configured to determine the observation information of the future frame based on the observation information of the current frame; The target trajectory is predicted based on the observation information of the future frame.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the target tracking method according to any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the target tracking method according to any one of claims 1 to 5 is implemented.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the target tracking method according to any one of claims 1 to 5 is implemented.

Images