Video full-frame-to-dense trajectory estimation method, system, terminal and medium

Abstract

The application provides a video full-frame pair dense trajectory estimation method, system, terminal and medium. The full-frame pair trajectory estimation problem is defined, and the mathematical form of the output result is determined; the space-time patch features of each video frame are extracted based on a video sequence as input; based on the space-time patch features, a correlation representation is constructed for each ordered frame pair, and a frame pair motion label is generated; the motion label is context aggregated and broadcast corrected along the trajectory direction, and is written back to the corresponding frame pair motion label to obtain an optimized frame pair motion feature; based on the space-time patch feature and the frame pair motion feature, a trajectory field decoder is used to output a displacement field, a visibility field and a confidence field, and a final video full-frame pair dense trajectory estimation result is obtained. The application can establish a differentiated matching representation for each ordered frame pair, and can also aggregate the global context along the trajectory direction, and feed back the context to the frame pair prediction.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, video understanding, image processing and artificial intelligence, and specifically to a method, system, terminal and medium for dense trajectory estimation of video full frame pairs for arbitrary source frames and arbitrary target frames in a video. Background Technology

[0002] In video analytics, the goal is typically to know where a pixel or image point in the current frame will move to at any subsequent moment, whether it will still be visible, and whether this correspondence is reliable. Existing technologies for this purpose can be broadly categorized into two types. The first type is optical flow estimation methods, which primarily handle dense displacement estimation between two frames, especially between adjacent frames. The second type is point tracking methods (or Tracking AnyPoint), which primarily handle cross-frame tracking of a user-given query point in the video.

[0003] Optical flow methods offer the advantage of dense output, theoretically providing displacement results for every pixel in an image. However, their common setup involves estimating the motion field between two given frames. When the video is long, the time interval is large, objects are occluded, the viewpoint changes significantly, or there is a lot of texture repetition, relying solely on matching adjacent frames or single frame pairs often fails to guarantee trajectory consistency across the entire sequence. Many existing methods explicitly construct large-volume correlation maps (CostVolume or Correlation Volume) and then refine them through local propagation or recursive updates. This approach is effective in two-frame scenarios, but once extended directly to an entire video, the number of frame pairs increases quadratically, rapidly increasing computational, storage, and consistency constraints.

[0004] Point tracking methods have the advantage of handling long time spans and focusing on the continuous correspondence of a physical point throughout the entire video. However, these methods typically require a query point to be provided first, and the output is inherently sparse. In other words, they answer where these few points went, rather than where all pixels in the entire frame went. To obtain results close to the entire image, it is often necessary to sample a large number of points and repeatedly run the tracking process, which is not only computationally expensive but also prone to inconsistencies between different query points and different starting frames.

[0005] Furthermore, the real challenge in performing two-frame matching lies not only in calculating the match of a single frame pair, but also in organizing the matches of multiple frame pairs into a stable and consistent representation. If relying entirely on independent estimation of individual frame pairs, results from different time intervals and different source frames can easily contradict each other; if relying entirely on local propagation, long-term information is difficult to transmit stably; and if the explicit cost volumes of all frame pairs are directly stacked, the computational and storage burden becomes excessive. Therefore, there is an urgent need in this field for a novel dense trajectory estimation technique for video frame pairs. Currently, no descriptions or reports of techniques similar to this invention have been found, and no similar domestic or international materials have been collected. Summary of the Invention

[0006] To address the aforementioned shortcomings in the prior art, this invention provides a method, system, terminal, and medium for estimating dense trajectories across the entire video frame.

[0007] According to a first aspect of the present invention, a method for estimating dense trajectories across a full frame of a video is provided, comprising: Define the full-frame trajectory estimation problem and determine the mathematical form of the output results, and then proceed with the following steps based on this; Using a video sequence as input, spatiotemporal patch features of each video frame are extracted based on the video sequence; Based on the spatiotemporal patch features, a relevant representation is constructed for each ordered frame pair and a frame pair motion tag is generated. The motion markers are aggregated and broadcast corrected along the trajectory direction, and then written back to the corresponding frame pair motion markers to obtain optimized frame pair motion features; Based on the spatiotemporal patch features and frame pair motion features, the trajectory field decoder outputs the displacement field, visibility field, and confidence field to obtain the final dense trajectory estimation results for the entire video frame pair.

[0008] Preferably, the definition of the full-frame trajectory estimation problem and the determination of the mathematical form of the output result include: Let the input video sequence be In the formula, T Indicates the number of video frames. Represents the image of frame t. t This represents the target frame; each frame is defined on the two-dimensional image domain Ω. For any source frame s and any target frame t, estimate the dense forward correspondence from source frame s to target frame t to establish a full-frame pair trajectory estimation problem; where "dense" means that for each pixel x in the source frame, its corresponding position in the target frame is given. : In the formula, Represents the two-dimensional displacement vector of pixel x from source frame s to target frame t; This indicates the predicted position of pixel x in the target frame t; Simultaneously predict a visibility field. This indicates whether the pixel is still visible in the target frame: Predicted confidence field This represents the reliability of the current position prediction; thus, for source frame s and pixel x, a trajectory set covering the entire sequence is generated. ; The corresponding position Two-dimensional displacement vector Visibility field and confidence field Determine the mathematical form of the output result.

[0009] Preferably, the step of extracting the spatiotemporal patch features of each video frame based on the video sequence includes: Each frame of the input video sequence is divided into regular small blocks, and each small block is called a patch. The spatiotemporal patch encoder encodes each patch of each frame to obtain patch-level feature representations, i.e., spatiotemporal patch features; for source frame s and target frame t, their spatiotemporal patch features are denoted as follows: In the formula, Indicates source frame features, Indicates the features of the target frame. R Represents the set of real numbers. P This indicates the total number of patches per frame. D Indicates the feature dimension.

[0010] Preferably, the step of constructing a relevant representation and generating frame pair motion tags for each ordered frame pair based on the spatiotemporal patch features includes: Based on the spatiotemporal patch features, residual information is constructed for any ordered frame pair, and a frame pair related representation is generated through a related library. Frame pair motion tags are generated based on the source frame features, target frame features, and the frame pair correlation representation. Preferably, the step of constructing residual information for any ordered frame pair based on the spatiotemporal patch features and generating a frame pair correlation representation through a correlation library includes: For any ordered frame pair (s,t), residual information is constructed based on the difference between the target frame features and the source frame features. In the formula,h (·) denotes a convolutional network used to organize the raw differences into a representation suitable for subsequent related queries; Design a set of learnable relevance tags shared across frame pairs, and organize these learnable relevance tags into a relevance library. A : In the formula, K Indicates the number of related tags; Use residual information For related libraries A Perform cross-attention query to obtain the relevant representation specific to the current frame pair. : In the formula, Attn(·) represents attention calculation; Q, K, and V represent query, key, and value, respectively. The step of generating frame pair motion markers based on source frame features, target frame features, and the frame pair correlation representation includes: After obtaining the frame pair correlation representation Then, based on the source frame features As a query term, based on target frame features As a key, the obtained frame pairs are related to the representation. As a value item, it participates in the matching process, performs an enhanced matching, and generates frame pair motion tags for the current ordered frame pairs. : Preferably, the step of performing context aggregation and broadcast correction on the motion markers along the trajectory direction and writing them back to the corresponding frame pair motion markers includes: For a source frame s and a patch position p, collect motion markers for patch position p on all target frames t to form a temporal motion marker sequence that unfolds along the trajectory direction: In the formula, Let τ represent the motion marker sequence of patch position p in source frame s across all target frames t, where τ represents the target frame index, T represents the total number of video frames, and p represents the patch position index in source frame s. D Indicates feature dimension; Introduce another set of learnable context labels U, and combine this context label U with the above motion label sequence. splicing: In the formula, The input sequence represents the patch position p in the source frame s; For the input sequence Perform self-attention processing and retain the first K output positions as trajectory context markers. The remaining positions serve as updated temporal motion markers. ; Trajectory context marking at different spatial locations within the same source frame s By mixing, we obtain the optimized context. ; The optimized context As a broadcast context, it is re-applied to the corresponding motion tag sequence. The optimized frame pair motion features are obtained as follows: In the formula, This represents the broadcast-corrected temporal motion marker sequence of the source frame s and patch position p across all target frames. This represents the updated temporal motion marker sequence of the source frame s and patch position p across all target frames.

[0011] Preferably, the step of using a trajectory field decoder to output a displacement field, a visibility field, and a confidence field based on the spatiotemporal patch features and frame pair motion features includes: The trajectory field decoder first predicts coarse-grained two-dimensional displacement vectors, visibility fields, and confidence fields from the source frame s to the target frame t based on spatiotemporal patch features. Then, by iteratively updating the frame pair motion features and reading the current motion markers and local source-target frame evidence, residual-based refinement is performed on the predicted two-dimensional displacement vectors, visibility fields, and confidence fields obtained at the current stage. Finally, the results are restored to full resolution through learned upsampling, outputting an explicit full-frame pair trajectory field. In the formula, This represents the two-dimensional displacement field from the source frame s to the target frame t; Represents the visibility field; Represents the confidence field; Based on the full-frame trajectory field, for any pixel x in any source frame s, read the two-dimensional displacement field on all target frames t. Based on the output position information The two-dimensional displacement vector of pixel x in the entire video is recovered. That is, to obtain the displacement field; combined with the visibility field Obtain the visibility field of pixel x throughout the video. Combined with confidence field Obtain the confidence field of pixel x in the entire video. The final video full-frame dense trajectory estimation results are obtained.

[0012] According to a second aspect of the present invention, a video full-frame pair dense trajectory estimation system is provided, comprising: The target estimation design module defines the full-frame pair trajectory estimation problem and determines the mathematical form of the output, which serves as the basis for the work of the following modules. The patch feature extraction module is used to extract spatiotemporal patch features for each video frame based on the video sequence as input. A motion tag generation module, which constructs a relevant representation and generates frame pair motion tags for each ordered frame pair based on the spatiotemporal patch features; A broadcast motion blending module is used to perform context aggregation and broadcast correction on the motion markers along the trajectory direction, and write back to the corresponding frame pair motion markers to obtain optimized frame pair motion features; The trajectory field decoding module, based on the spatiotemporal patch features and frame pair motion features, uses the trajectory field decoder to output the displacement field, visibility field, and confidence field to obtain the final dense trajectory estimation result for the entire video frame pair.

[0013] According to a third aspect of the present invention, a computer terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, can be used to perform the method described in any one of the above inventions.

[0014] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, can be used to perform the method described in any one of the preceding inventions.

[0015] By adopting the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art: This invention employs All-Pairs Tracking (APT) as its output format, enabling the direct establishment of an explicit correspondence between any source frame, any target frame, and any pixel. This solves the problem that existing technologies typically only output two frames of optical flow or a small number of query point trajectories. Subsequently, the trajectory of any pixel can be read on demand from the same trajectory field, avoiding inconsistencies caused by repeatedly running a single-source tracker.

[0016] This invention employs a spatio-temporal patch encoder, which first converts the original video frames into patch-level feature representations with a unified structure and easy computation. This significantly reduces the complexity of directly matching all frames at the original pixel level while ensuring the ability to express spatio-temporal information, laying the foundation for subsequent large-scale frame pair modeling.

[0017] This invention employs a Correlation Bank (CorrBank) to compress and represent matching patterns in different frame pairs using shared learnable correlation labels, and conditionally invokes these shared labels with the help of residual information. Compared with explicitly constructing large-scale four-dimensional cost volumes, this invention maintains expressive power while facilitating the reuse of matching priors in long videos and multi-frame scenes, and can better handle repetitive textures, large displacements, and complex frame pair relationships.

[0018] This invention employs a pairwise motion token as an intermediate representation, integrating the source frame, target frame, and frame pair related representations into a unified token space. This allows subsequent modules to perform global modeling around a more compact and semantic motion representation without directly relying on the original image, thereby improving the scalability and expressive flexibility of frame pair estimation technology.

[0019] This invention employs a Broadcast Motion Mixer (BMM), which first aggregates motion markers from the same source location across different target frames at the trajectory level, then performs contextual mixing within the same source frame, and finally broadcasts the aggregated information back to each frame pair for correction. In this way, the results for each frame pair are no longer determined locally and independently, but simultaneously incorporate auxiliary evidence from the entire trajectory and the spatial region within the same frame, thereby significantly improving trajectory consistency, occlusion handling capabilities, and structural stability over long time spans.

[0020] This invention employs a trajectory field decoder and a decoding method of coarse prediction-iterative refinement-upsampling, which can stably recover the full-resolution output from the internal representation at the patch level, making the results more reliable in boundary positions, fine structure regions, and large displacement regions. It solves the problem of boundary blurring and local distortion that easily occur when relying solely on coarse-grained prediction.

[0021] This invention employs a joint output of displacement, visibility, and confidence fields, which not only answers where a pixel went, but also simultaneously answers whether the pixel is still visible and whether the current prediction is reliable. This multi-information joint output method significantly improves the interpretability and usability of the results, facilitating subsequent system operations such as trajectory filtering, occlusion removal, reliability weighting, and editing control.

[0022] This invention supports long video sliding window inference extension, which can extend processing capabilities to longer videos without changing the core structure, taking into account both engineering feasibility and sequence-level consistency, and is suitable for practical long video analysis, video editing and robot vision scenarios.

[0023] This invention combines full-frame explicit representation, CorrBank conditional correlation modeling, BMM trajectory-level context aggregation and broadcast correction, and trajectory field decoding to achieve more unified, stable, and reusable dense trajectory estimation in complex video scenes, overcoming the shortcomings of existing technologies in long-range consistency, occlusion handling, repetitive texture disambiguation, and global representation. Attached Figure Description

[0024] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating the overall workflow of a video full-frame pair dense trajectory estimation method in a preferred embodiment of the present invention.

[0025] Figure 2 This is a schematic diagram of the components of a video full-frame pair dense trajectory estimation system in a preferred embodiment of the present invention.

[0026] Figure 3 This is a schematic diagram illustrating the operation of a dense trajectory estimation method for a full-frame video frame in an example of an implementable method and approach of the present invention.

[0027] Figure 4 This is a flowchart illustrating the workflow of a related library in one of the possible implementation methods and approaches of the present invention.

[0028] Figure 5 This is a flowchart illustrating the broadcast motion hybrid process in one of the possible implementation methods and approaches of the present invention.

[0029] Figure 6 This is a flowchart illustrating the trajectory field decoding and result reading process in an example of an implementable method and approach of the present invention. Detailed Implementation

[0030] The embodiments of the present invention are described in detail below: These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

[0031] In the existing technology, there is a lack of a new video frame pair dense trajectory estimation technology that can not only establish a discriminative matching representation for each ordered frame pair, but also aggregate the global context along the trajectory direction and feed this context back into the prediction of each frame pair, so as to obtain more consistent and reliable results under long time sequence, complex occlusion and large displacement conditions.

[0032] To address the aforementioned shortcomings, one embodiment of the present invention provides a method for dense trajectory estimation across full-frame pairs in video. This method takes a video sequence as input, first extracts spatio-temporal patch features, then constructs a relevant representation for each ordered frame pair and generates pairwise motion tokens. Subsequently, these motion tokens are aggregated and broadcast-corrected along the trajectory direction. Finally, a trajectory field decoder outputs a displacement field, a visibility field, and a confidence field. This enables the establishment of an explicit dense correspondence between any source frame and any target frame, and allows for unified modeling of ambiguities caused by occlusion, large displacements, and repetitive textures over long time spans. It organizes the trajectory information in the entire video into a reusable and on-demand unified representation, rather than simply providing two frames of optical flow or a small number of query point trajectories. This method can be applied to long-range point tracking, dense trajectory recovery, video editing, multi-keyframe consistency processing, 3D / 4D reconstruction assistance, and robot vision perception scenarios.

[0033] Specifically, such as Figure 1 As shown, the video full-frame pair dense trajectory estimation method provided in this embodiment may include: S1, Define the full-frame trajectory estimation problem and determine the mathematical form of the output result, and continue with the following steps based on this; S2 takes a video sequence as input and extracts the spatiotemporal patch features of each video frame based on the video sequence; S3, based on spatiotemporal patch features, constructs relevant representations for each ordered frame pair and generates frame pair motion tags; S4. Perform context aggregation and broadcast correction on the motion markers along the trajectory direction, and write back to the corresponding frame pair motion markers to obtain the optimized frame pair motion features; S5, based on spatiotemporal patch features and frame pair motion features, uses a trajectory field decoder to output displacement field, visibility field, and confidence field to obtain the final dense trajectory estimation results for the entire video frame pair.

[0034] In some preferred embodiments, S1 above, which defines the full-frame trajectory estimation problem and determines the mathematical form of the output result, may further include: S11, Let the input video sequence be... In the formula, T Indicates the number of video frames. Represents the image of frame t. t This represents the target frame. Each frame of the image is defined on the two-dimensional image domain Ω.

[0035] S12, for any source frame s and any target frame t, estimate the dense forward correspondence from source frame s to target frame t. Dense means that for each pixel x in source frame s, its corresponding position in target frame t must be given. : In the formula, Represents the two-dimensional displacement vector of pixel x from source frame s to target frame t; This indicates the predicted position of pixel x in the target frame t.

[0036] S13, in order to handle situations such as occlusion and out-of-view, simultaneously predicts a visibility field. This indicates whether the pixel is still visible in the target frame: In the formula, The closer to 1, the more likely pixel x is to be visible in the target frame t; the closer to 0, the more likely it is to be occluded or out of view.

[0037] S14, in its complete implementation, further predicts the confidence field. This is used to represent the reliability of the current position prediction. Therefore, for a fixed source frame s and source pixel x, a trajectory set covering the entire sequence is generated. This is the meaning of a full-frame video to a dense trajectory field: it is not a single two-frame result, but a complete set of explicit correspondences that can be directly read from any source point.

[0038] The corresponding position Two-dimensional displacement vector Visibility field and confidence field Determine the mathematical form of the output result.

[0039] In some preferred embodiments, the above-mentioned S2, which extracts the spatiotemporal patch features of each video frame based on the video sequence, may further include: S21, directly performing pairwise matching on all frames and all pixels at the original pixel level would be computationally very expensive. Therefore, this step first models at a lower-dimensional, more regular patch level. Specifically, each frame of the input video sequence is first divided into regular small blocks, each called a patch. A patch can be understood as a small square in an image; for example, after an image is cut into many equal-sized blocks, each block corresponds to a location and a feature vector.

[0040] S22, then, the spatio-temporal patch encoder (ST-Patch Encoder) is used to encode each patch of each frame to obtain the patch-level feature representation, i.e., the spatio-temporal patch feature. For the source frame s and the target frame t, their spatio-temporal patch features are denoted as: In the formula, Indicates source frame features, Let R represent the target frame features, P represent the set of real numbers, P represent the total number of patches in each frame, and D represent the dimension of each patch feature. In other words, a frame of image is represented as P feature vectors of length D. These features not only contain the appearance information of the patch but also carry certain temporal context and spatial location information, preparing for subsequent cross-frame modeling. In a specific application example, this step uses a ConvNeXt-based patch encoder to extract initial mesh features, and then injects temporal embeddings and two-dimensional positional encoding during subsequent frame pair inference.

[0041] In some preferred embodiments, S3 above, which constructs a relevant representation and generates frame pair motion markers for each ordered frame pair based on the spatiotemporal patch features, may further include: S31, based on spatiotemporal patch features, residual information is constructed for any ordered frame pair, and a correlation representation of the frame pair is generated through a correlation library; further: A core design principle of this step is: it does not directly address... and Instead of performing a regular correlation, we first construct a conditional correlation representation for each ordered frame pair. Here, an ordered frame pair means that the order of the source frame s and the target frame t cannot be interchanged, because the displacement from the source frame s to the target frame t is usually different from the displacement from the target frame t to the source frame s.

[0042] S311, First, for any ordered frame pair (s,t), construct residual information based on the difference between the target frame features and the source frame features. : In the formula, h(·) represents a small convolutional network. Here... This represents the characteristic differences between patches, which can be intuitively understood as the changes that occur in the current target frame relative to the source frame. h The purpose of (·) is to further organize these original differences into a representation that is more suitable for subsequent related queries, making local motion clues more obvious.

[0043] S312, Next, design a set of learnable correlation tokens shared across frame pairs, and organize them into a correlation bank A (CorrelationBank, CorrBank): In the formula, K This indicates the number of related tags. Related libraries can be used. A View it as a set of learnable matching prototypes or a shared matching dictionary: different frame pairs can extract the matching patterns they need from this set of shared tags, but the results extracted by different frame pairs are not exactly the same because they are driven by their respective residual information.

[0044] S313, subsequently, using residual information For related libraries A Perform cross-attention query to obtain the relevant representation specific to the current frame pair. : In the formula, Attn(·) represents attention calculation; Q, K, and V represent query, key, and value, respectively. Here, residual information... The first part is responsible for specifying the desired matching pattern, while the shared relevance library A is responsible for providing the learnable patterns to choose from. After this step, the desired result is... It is no longer a representation that is completely universal for all frame pairs, but has been conditionally adjusted for the current frame pair.

[0045] S32, based on the source frame features, target frame features, and frame pair correlation representation, generate frame pair motion markers; further: After obtaining the frame pair correlation representation Next, this step performs another round of enhanced matching (Bank-AugmentedMatching) to generate pairwise motion tokens for the current ordered frame pairs. : This step means: using source frame features As a query term, based on target frame features As a key term, the frame pairs obtained in S21 are then correlated. As a value, it participates in the matching, thereby outputting the motion marker of the current frame pair. Each motion marker here can be understood as a compressed representation of the motion and correspondence of a source patch when facing the current target frame.

[0046] Compared to directly performing two-frame correlation, this method introduces the additional factor generated by CorrBank. The advantage of doing this is that the generated It not only utilizes the direct relationship between the source frame and the target frame, but also inherits the matching patterns learned from the shared related library, thus having a greater chance of making stable judgments in complex scenes such as repeated textures, large displacements, and occlusions.

[0047] In some preferred embodiments, S4, which involves performing context aggregation and broadcast motion mixing on the frame pair motion markers along the trajectory direction and writing back the results to each frame pair, may further include: If only step S3 is performed, each frame pair remains relatively independent. To make the results more consistent across the entire video, this step further introduces a Broadcast Motion Mixer (BMM) to organize the motion information scattered across different frame pairs along the trajectory direction. Specifically: S41, First, for a fixed source frame s and a fixed patch position p, collect its motion markers on all target frames t: In the formula, Let τ represent the motion marker sequence of a fixed patch position p in a fixed source frame s across all target frames t, where τ represents the target frame index, T represents the total number of video frames, and p represents the patch position index in the source frame s. D The above formula represents the feature dimension of the motion markers. It means that for the same position p in the source frame s, the motion markers obtained when facing each target frame t are arranged in chronological order, thus forming a temporal sequence of motion markers unfolding along the trajectory direction. Intuitively, this means that instead of looking at the performance of this position in a single target frame, we consider the performance of this position across all target time points together for unified observation.

[0048] S42, subsequently, another set of learnable context labels U is introduced, and this context label U is compared with the above motion label sequence. splicing: In the formula, This represents the input sequence of a fixed patch position p in a fixed source frame s. This input sequence is formed by concatenating the learnable context label U with the motion label sequence of that position in each target frame, and is used for subsequent self-attention aggregation. S43, then, several self-attention modules are used to process the input sequence. Processing is performed. After processing, the first K output positions are retained as trajectory context markers. The remaining positions serve as updated temporal motion markers. This process can be understood as: compressing and summarizing multi-moment information along the entire trajectory. In this way, it becomes a summary of the trajectory of the source location p throughout the entire video.

[0049] Simply performing temporal aggregation on a single trajectory is insufficient. This step further involves tagging the trajectory context at different spatial locations within the same source frame s. The mixture is then blended to obtain the refined context. The purpose of this step is to allow different regions within the same frame to exchange information. For example, the contextual relationships between the main body, edges, and background regions of an object help improve the stability of judging occlusion boundaries, repeating textures, and fine-grained structures.

[0050] S44, Finally, the refined context. As a broadcast context, it is re-applied to the corresponding motion tag sequence. The optimized frame pair motion features are obtained as follows: In the formula, This represents the broadcast-corrected temporal motion marker sequence of the source frame s and patch position p across all target frames. This represents the updated temporal motion marker sequence of the source frame s and patch position p across all target frames.

[0051] After this step, the motion markers at each moment no longer rely solely on local evidence from a single frame pair, but simultaneously absorb comprehensive information about the trajectory across the entire time dimension, as well as auxiliary information provided by other spatial regions within the same source frame. This is the core value of the broadcast motion mixer: first aggregate, then broadcast, and then refine.

[0052] In some preferred embodiments, the above-mentioned S5, based on spatiotemporal patch features and frame pair motion features, uses a trajectory field decoder to output a displacement field, a visibility field, and a confidence field, and may further include: S51, after obtaining the optimized pairwise motion tokens in S4, this step sends the above features into the trajectory field decoder. The task of this decoder is to restore the patch-level and token-level internal representations into directly usable pixel-level results, obtaining the explicit full-frame pairwise trajectory field.

[0053] In a complete implementation, specifically: the trajectory field decoder first predicts coarse-grained two-dimensional displacement vectors, visibility fields, and confidence fields from the source frame s to the target frame t on the patch grid; it then refines the results by iteratively updating the current motion markers and local source-target frame evidence (i.e., local matching cues or local correspondence information extracted from the source frame s and target frame t near the current position); finally, it restores the results to full resolution through learned upsampling, outputting the trajectory field from the explicit full-frame view: In the formula, This represents the two-dimensional displacement field from the source frame s to the target frame t; Represents the visibility field; This represents the confidence field. Therefore, the output of this invention is not a simple displacement answer, but a complete full-frame trajectory field: it tells the subsequent system where the pixel has moved, whether the pixel is still visible, and whether the current judgment is reliable.

[0054] S52 reads arbitrary pixel trajectories on demand from the explicit full-frame track field; specifically: After S51 has output all ordered frame pairs, an explicit full-frame pair trajectory field is obtained. From the explicit full-frame pair trajectory field, for any pixel x in any source frame s, it is only necessary to read the two-dimensional displacement fields on all target frames t. That is, it can determine the result based on the corresponding position in the output. Recover the two-dimensional position vector of the pixel throughout the video. That is, the displacement field is obtained; combined with the visibility field Recover the visibility field of pixel x throughout the video. This allows us to determine when a pixel is visible and when it is occluded; combined with the confidence field... Obtain the confidence field of pixel x in the entire video. Furthermore, results with high reliability can be further screened.

[0055] In the above embodiments of the present invention, it further includes: When the length of the input video exceeds a preset threshold, the process of extracting spatiotemporal patch features, constructing relevant representations and generating frame pair motion markers, context aggregation and broadcast correction, and trajectory field decoding can be performed within multiple overlapping time windows, and the prediction results of the overlapping areas are used to initialize subsequent time windows.

[0056] In the above embodiments of the present invention, it further includes: During the model training phase in S5, the network parameters involved in each step are jointly optimized. The joint optimization includes at least trajectory loss, confidence loss, visibility loss, and corresponding consistency regularization term.

[0057] Unlike traditional two-frame optical flow or sparse point tracking, the video full-frame pair dense trajectory estimation method provided in the above embodiments of the present invention generates a unified representation that can be repeatedly invoked. Subsequent applications do not need to rerun the tracker for each new starting point, but can directly read the results from the already generated full-frame pair trajectory field. It has important application value for tasks such as multi-keyframe video editing, long-range tracking, and consistency constraint reconstruction.

[0058] The above embodiments of the present invention design a dense trajectory estimation method for the entire video frame. Its goal is not to perform optical flow on only two adjacent frames, nor to track only a small number of query points, but to directly establish an explicit dense correspondence between "any source frame - any target frame - any pixel".

[0059] The first technical problem to be solved by the method provided in the above embodiments of the present invention is: how to output a uniform full-frame dense trajectory field for the entire video without relying on manually specifying query points, so that the trajectory of any pixel in the entire video can be directly read.

[0060] The second technical problem to be solved by the method provided in the above embodiments of the present invention is: how to avoid the high computational and storage requirements of traditional explicit large-scale correlation banks in long video scenarios, while still maintaining the ability to express complex frame pair relationships. To this end, the present invention designs a correlation bank (CorrBank), which uses shared learnable correlation tags to compress and conditionally call the matching patterns of different frame pairs.

[0061] The third technical problem to be solved by the method provided in the above embodiments of the present invention is: how to uniformly model motion information scattered in multiple frame pairs along the trajectory direction, rather than letting each frame pair independently determine the result. To this end, the present invention designs a Broadcast Motion Mixer (BMM), which first performs trajectory-level aggregation on motion markers at the same source position on different target frames, and then broadcasts the aggregated context back to each frame pair for correction.

[0062] The fourth technical problem to be solved by the method provided in the above embodiments of the present invention is: how to simultaneously provide displacement, visibility and reliability information so that the subsequent system not only knows where the pixel went, but also knows whether the pixel is still visible and whether the current prediction is reliable, thereby improving the practicality of the method in video editing, reconstruction and perception systems.

[0063] Based on the same inventive concept, an embodiment of the present invention also provides a video full-frame pair dense trajectory estimation system.

[0064] Specifically, such as Figure 2 As shown, the video full-frame pair dense trajectory estimation system provided in this embodiment may include: The target estimation design module defines the full-frame pair trajectory estimation problem and determines the mathematical form of the output, which serves as the basis for the work of the following modules. The patch feature extraction module is used to extract the spatiotemporal patch features of each video frame based on the video sequence as input. The motion tag generation module constructs a relevant representation and generates motion tags for each ordered frame pair based on spatiotemporal patch features. The broadcast motion blending module is used to perform context aggregation and broadcast correction on motion tags along the trajectory direction, and write back to the corresponding frame pair motion tags to obtain optimized frame pair motion features; The trajectory field decoding module, based on spatiotemporal patch features and frame pair motion features, uses a trajectory field decoder to output displacement field, visibility field, and confidence field to obtain the final dense trajectory estimation results for the entire video frame.

[0065] It should be noted that the steps in the method provided by the present invention can be implemented using corresponding modules, devices, units, etc. in the system. Those skilled in the art can refer to the technical solution of the method to realize the composition of the system. That is, the embodiments in the method can be understood as preferred examples for building the system, and will not be elaborated here.

[0066] The following is combined with Figures 3 to 6 The following specific implementation examples further illustrate the feasible methods and approaches of the technical solutions provided in the above embodiments of the present invention. The following implementation examples are used to help understand the method composition, data flow direction, and key parameter settings of the present invention, and do not constitute a limitation on the scope of protection of the present invention. As long as the core idea of ​​combining frame pair correlation modeling, trajectory-level broadcast correction, and trajectory field decoding is not deviated from, the relevant modules, parameters, and implementation details can be adjusted according to application requirements.

[0067] Implementation Example 1: Standard Video Full Frame Pair Dense Trajectory Estimation Implementation Example likeFigure 3 As shown, this specific application example corresponds to the standard inference flow of this invention and is suitable for scenarios with moderate video length that can complete the entire sequence processing in one go. The input is a video sequence. First, size and pixel normalization are performed on each frame, and it is then segmented into a regular patch grid. Subsequently, a spatiotemporal patch encoder is used to extract patch-level features from each frame, resulting in... .

[0068] like Figure 4 As shown, for any ordered frame pair (s,t), the feature differences are first calculated and residual information is generated through a small convolutional network: Then, use the residual information to query the shared correlation library A to generate the correlation representation of the current frame pair: Then, frame pair motion tags are generated using source frame features, target frame features, and frame pair correlation representations: like Figure 5 As shown, for a fixed source frame s and a fixed patch position p, the motion tags corresponding to all target frames are collected to form a time sequence. Concatenate a set of learnable context labels U with the sequence: Subsequently, the trajectory context is extracted by the self-attention module, and the trajectory contexts of different patch positions are mixed within the same source frame. The mixed result is then broadcast back to the corresponding temporal motion tokens to obtain refinedpairwisemotiontokens. Finally, as... Figure 6 As shown, the output from the trajectory field decoder Here, refined pairwise motion tokens represent the optimized frame-pair motion features obtained after broadcast motion mixing.

[0069] When it is necessary to read the trajectory of any pixel x, simply follow the formula Simply read the position in each target frame one by one. If A lower value indicates that the pixel may have been occluded or removed from view at that time; if If the value is low, it indicates that the current estimate is not reliable enough, and the system can choose to reduce the weight or remove it in subsequent iterations.

[0070] Implementation Example 2: Long Video Sliding Window Inference Implementation Example When the input video is very long, processing all frames at once would place a heavy burden on GPU memory and computation. Therefore, this specific application example employs a long video sliding window inference method. This method does not change the core network structure, only the inference organization.

[0071] Specifically, a time window length L is pre-defined. For a fixed source frame s, the window of length L that is closest to it in time is processed first to obtain the displacement field, visibility field, and confidence field within this local range. Subsequently, windows with a longer time range are processed, and the new windows are allowed to overlap with the old windows in time.

[0072] For predictions already obtained within overlapping areas, these can be used as initialization information for subsequent windows. For time locations not yet covered in a new window, the state at the boundary time can be copied as an initial value, and then corrected by subsequent refinement modules. The purpose of this is to reuse existing results from adjacent windows in long video scenarios, reduce the instability caused by estimating from scratch, and enhance the continuity and consistency between the outputs of different windows.

[0073] By repeating the above process on all source frames, the full-frame trajectory field of the entire long video can be obtained. This implementation method is particularly suitable for scenarios such as long video editing, surveillance video analysis, and long-term robot perception.

[0074] Implementation Example 3: Model Training and Optimization Example This specific application example illustrates a concrete implementation of the invention during the training phase. Since this invention pertains to a learning-based video trajectory estimation method, it requires training the network parameters using data with ground truth displacements and visibility annotations. In one specific implementation of this application example, the training data consists of a 1:1 mixture of π-R10K and Kubric data, with 30-60 frames of video footage sampled during training. The input resolution is 384×512. To improve generalization ability, enhancement methods such as random cropping, random scaling, horizontal flipping, color perturbation, and synthetic occluders can be employed during training.

[0075] Let the true displacement field be The truth visibility field is First, define the pixel-by-pixel displacement residual: The total loss consists of trajectory loss, confidence loss, visibility loss, and correspondence consistency regularization term: Where L represents the total loss; λ_traj, λ_conf, λ_vis, and λ_corr represent the trajectory loss, confidence loss, visibility loss, and the weight coefficients corresponding to the consistency regularization term, respectively.

[0076] Among them, trajectory loss directly constrains displacement prediction: If the network simultaneously outputs a confidence field Then, a single-pixel confidence penalty can be defined: This design encourages higher confidence levels for locations with smaller displacement errors, while using a logarithmic prevention network to simply push the confidence level close to zero.

[0077] Visibility loss can be achieved using binary cross-entropy: Where BCE(·) represents the binary cross-entropy function.

[0078] To ensure consistency across the entire sequence for different starting points belonging to the same real physical trajectory, this work also introduces a corresponding consistency regularization term. If ((s,x),(t,x′)) represents a pair of pixels located on the same real trajectory, then the following definition is provided: where, This represents the set of pixel pairs that belong to the same real trajectory; This represents the trajectory description of the entire sequence obtained starting from pixel x in the source frame s; This represents the complete sequence trajectory description obtained starting from pixel x′ in the source frame t.

[0079] The purpose of this constraint is to ensure that even if two pixels come from different source frames, their trajectory descriptions over the entire sequence should be close to each other, as long as they essentially belong to the same real physical point. This constraint helps improve long-term consistency and reduce minor deviations that occur during trajectory round trips or occlusion recovery.

[0080] In a specific training configuration, the AdamW optimizer can be used with a weight decay of 0.01 and a learning rate of 2×10^-4, combined with cosine decay and a 1000-step warm-up. Training hardware can use eight H100-80G GPUs, with 50,000 training iterations. The loss term weights can be set to ( = (1, 1, 1, 0.3). After training, the obtained network parameters are used in the inference process of Example 1 and Example 2.

[0081] An embodiment of the present invention also provides a computer terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can be used to perform any of the methods described above in the present invention.

[0082] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules that implement the above methods), computer instructions, etc., and the aforementioned computer programs and computer instructions can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0083] A processor is used to execute computer programs stored in memory to implement the various steps of the methods or various modules of the systems involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method and system embodiments.

[0084] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0085] An embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to perform the methods described above in the present invention.

[0086] Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of computer programs from one place to another. Storage media can be any available medium accessible to a general-purpose or special-purpose computer. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a user device. Of course, the processor and storage medium can also exist as discrete components in a communication device.

[0087] The video full-frame pair dense trajectory estimation method, system, terminal and medium provided in the above embodiments of the present invention are for full-frame pair dense trajectory estimation from any source frame to any target frame. They adopt the shared correlation library CorrBank and the broadcast motion mixer BMM, and output an explicit full-frame pair trajectory field that can read the trajectory of any pixel on demand.

[0088] The video full-frame pair dense trajectory estimation method, system, terminal and medium provided in the above embodiments of the present invention do not rely on external query points, but directly output dense trajectory fields for all source frames and all target frames. The core objects are patch features, frame pair correlation representation, frame pair motion markers and temporal motion marker sequences organized along the trajectory.

[0089] The video full-frame pair dense trajectory estimation method, system, terminal and medium provided in the above embodiments of the present invention directly define the task as full-frame pair tracking, and introduce a combination of CorrBank and BMM in the structure: first, the residual information is used to query the shared correlation library to generate the frame pair correlation representation, then the frame pair motion label is generated, and then context aggregation and broadcast correction are performed along the trajectory direction.

[0090] Any matters not covered in the above embodiments of the present invention are well-known in the art.

[0091] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A method for estimating dense trajectories across a full frame of a video, characterized in that, include: Define the full-frame trajectory estimation problem and determine the mathematical form of the output results, and then proceed with the following steps based on this; Using a video sequence as input, spatiotemporal patch features of each video frame are extracted based on the video sequence; Based on the spatiotemporal patch features, a relevant representation is constructed for each ordered frame pair and a frame pair motion tag is generated. The motion markers are aggregated and broadcast corrected along the trajectory direction, and then written back to the corresponding frame pair motion markers to obtain optimized frame pair motion features; Based on the spatiotemporal patch features and frame pair motion features, the trajectory field decoder outputs the displacement field, visibility field, and confidence field to obtain the final dense trajectory estimation results for the entire video frame pair.

2. The video full-frame pair dense trajectory estimation method according to claim 1, characterized in that, The definition of the full-frame trajectory estimation problem and the mathematical form for determining the output include: Let the input video sequence be In the formula, T Indicates the number of video frames. Represents the image of frame t. t This represents the target frame; each frame is defined on the two-dimensional image domain Ω. For any source frame s and any target frame t, estimate the dense forward correspondence from source frame s to target frame t, and establish a full-frame pair trajectory estimation problem; wherein, for each pixel x in source frame s, its corresponding position in target frame t is given. : ； In the formula, Represents the two-dimensional displacement vector of pixel x from source frame s to target frame t; This indicates the predicted position of pixel x in the target frame t; Simultaneously predict a visibility field. This indicates whether the pixel is still visible in the target frame: ； Predicted confidence field This represents the reliability of the current position prediction; thus, for source frame s and pixel x, a trajectory set covering the entire sequence is generated. ; The corresponding position Two-dimensional displacement vector Visibility field and confidence field Determine the mathematical form of the output result.

3. The video full-frame pair dense trajectory estimation method according to claim 1, characterized in that, The extraction of spatiotemporal patch features for each video frame based on the video sequence includes: Each frame of the input video sequence is divided into regular small blocks, and each small block is called a patch. The spatiotemporal patch encoder encodes each patch of each frame to obtain patch-level feature representations, i.e., spatiotemporal patch features; for source frame s and target frame t, their spatiotemporal patch features are denoted as follows: ； In the formula, Indicates source frame features, Indicates the features of the target frame. R Represents the set of real numbers. P This indicates the total number of patches per frame. D Indicates the feature dimension.

4. The video full-frame pair dense trajectory estimation method according to claim 1, characterized in that, The step of constructing a relevant representation and generating frame pair motion tags for each ordered frame pair based on the spatiotemporal patch features includes: Based on the spatiotemporal patch features, residual information is constructed for any ordered frame pair, and a frame pair related representation is generated through a related library. Frame pair motion tags are generated based on the source frame features, target frame features, and the frame pair correlation representation.

5. The video full-frame pair dense trajectory estimation method according to claim 4, characterized in that, The process of constructing residual information for any ordered frame pair based on the spatiotemporal patch features, and generating a frame pair correlation representation using a correlation library, includes: For any ordered frame pair (s,t), residual information is constructed based on the difference between the target frame features and the source frame features. : ； In the formula, h (·) denotes a convolutional network used to organize the raw differences into a representation suitable for subsequent related queries; Design a set of learnable relevance tags shared across frame pairs, and organize these learnable relevance tags into a relevance library. A : ； In the formula, K Indicates the number of related tags; Use residual information For related libraries A Perform cross-attention query to obtain the relevant representation specific to the current frame pair. : ； In the formula, Attn(·) represents attention calculation; Q, K, and V represent query, key, and value, respectively. The step of generating frame pair motion markers based on source frame features, target frame features, and the frame pair correlation representation includes: After obtaining the frame pair correlation representation Then, based on the source frame features As a query term, based on target frame features As a key, the obtained frame pairs are related to the representation. As a value item, it participates in the matching process, performs an enhanced matching, and generates frame pair motion tags for the current ordered frame pairs. : 。 6. The video full-frame pair dense trajectory estimation method according to claim 1, characterized in that, The step of performing context aggregation and broadcast correction on the motion markers along the trajectory direction and writing them back to the corresponding frame pair motion markers includes: For a source frame s and a patch position p, collect motion markers for patch position p on all target frames t to form a temporal motion marker sequence that unfolds along the trajectory direction: ； In the formula, Let τ represent the motion marker sequence of patch position p in source frame s across all target frames t, where τ represents the target frame index, T represents the total number of video frames, and p represents the patch position index in source frame s. D Indicates feature dimension; Introduce another set of learnable context labels U, and combine this context label U with the above motion label sequence. splicing: ； ； In the formula, The input sequence represents the patch position p in the source frame s; For the input sequence Perform self-attention processing and retain the first K output positions as trajectory context markers. The remaining positions serve as updated temporal motion markers. ; Trajectory context marking at different spatial locations within the same source frame s By mixing, we obtain the optimized context. ; The optimized context As a broadcast context, it is re-applied to the corresponding motion tag sequence. The optimized frame pair motion features are obtained as follows: ； In the formula, This represents the broadcast-corrected temporal motion marker sequence of the source frame s and patch position p across all target frames. This represents the updated temporal motion marker sequence of the source frame s and patch position p across all target frames.

7. The video full-frame pair dense trajectory estimation method according to claim 1, characterized in that, The process of using a trajectory field decoder to output a displacement field, a visibility field, and a confidence field based on the spatiotemporal patch features and frame pair motion features includes: The trajectory field decoder first predicts coarse-grained two-dimensional displacement vectors, visibility fields, and confidence fields from the source frame s to the target frame t based on spatiotemporal patch features. Then, by iteratively updating the frame pair motion features and reading the current motion markers and local source-target frame evidence, residual-based refinement is performed on the predicted two-dimensional displacement vectors, visibility fields, and confidence fields obtained at the current stage. Finally, the results are restored to full resolution through learned upsampling, outputting an explicit full-frame pair trajectory field. ； In the formula, This represents the two-dimensional displacement field from the source frame s to the target frame t; Represents the visibility field; Represents the confidence field; Based on the full-frame trajectory field, for any pixel x in any source frame s, read the two-dimensional displacement field on all target frames t. Based on the output position information The two-dimensional displacement vector of pixel x in the entire video is recovered. That is, to obtain the displacement field; combined with the visibility field Obtain the visibility field of pixel x throughout the video. Combined with confidence field Obtain the confidence field of pixel x in the entire video. The final video full-frame dense trajectory estimation results are obtained.

8. A video full-frame pair dense trajectory estimation system, characterized in that, include: The target estimation design module defines the full-frame pair trajectory estimation problem and determines the mathematical form of the output, which serves as the basis for the work of the following modules. The patch feature extraction module is used to extract spatiotemporal patch features for each video frame based on the video sequence as input. A motion tag generation module, which constructs a relevant representation and generates frame pair motion tags for each ordered frame pair based on the spatiotemporal patch features; A broadcast motion blending module is used to perform context aggregation and broadcast correction on the motion markers along the trajectory direction, and write back to the corresponding frame pair motion markers to obtain optimized frame pair motion features; The trajectory field decoding module, based on the spatiotemporal patch features and frame pair motion features, uses the trajectory field decoder to output the displacement field, visibility field, and confidence field to obtain the final dense trajectory estimation result for the entire video frame pair.

9. A computer terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it can be used to perform the method of any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program can be used to perform the method of any one of claims 1-7.

Images