Point tracking method in large displacement scenarios based on time pyramid

Abstract

This invention belongs to the field of computer vision technology and relates to a point tracking method for large displacement scenes based on a time-dimensional pyramid. It adaptively solves various point tracking situations by utilizing time windows of different sizes. The point tracking model architecture includes a data processing module and an iterative update module with shared parameters. The data processing module includes the following steps: (1) selection of the point to be tracked; (2) image feature extraction; (3) point trajectory initialization. The iterative update module consists of two parts: a related layer and a time-dimensional information fusion network. In the time-dimensional information fusion network part, the designed network has different receptive fields of different sizes at different levels to form a time-dimensional feature pyramid to comprehensively solve different displacement situations. The specific steps are: (4) similarity feature calculation; (5) time-dimensional information fusion; (6) point trajectory and point feature update; (7) final point trajectory output. Repeat the above steps (4) to (6) until the set number of iterations is reached. At this time, point tracking of the target point in the input video sequence is completed, and the point trajectory update result output by the model is the final point tracking result output by the method. Point tracking is completed.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and specifically relates to a point tracking method for large displacement scenes based on a time pyramid. Background Technology

[0002] Motion is crucial for scene understanding; perceiving how objects move and deform helps in understanding their physical properties and inferring their intentions. Current research on motion and tracking problems largely focuses on three types of issues: fine-grained short-range tracking (optical flow), coarse-grained long-range tracking (target tracking, segmentation, etc.), and sparse feature tracking (tracking features across images). Research on the intersection of fine-grained and long-range tracking (point tracking) is relatively limited.

[0003] Data-driven deep neural network architectures represent a technology that offers superior performance compared to traditional methods. This approach has achieved significant breakthroughs in many vision-related problems, providing new solutions for these fields. Deep learning-based methods have achieved better performance than traditional methods in predicting long-distance pixel-level trajectories using a feedforward approach. However, most of these studies focus on long-distance point tracking on smooth, continuous videos with continuous, natural motion.

[0004] Currently, the most advanced point tracking algorithms are trained on trajectories within continuous short video sequences, focusing on video sequences with continuous natural motion, without specifically considering the handling of large displacements. However, real-life scenarios often involve edited video scenes containing large displacements caused by discontinuous motion. Summary of the Invention

[0005] To address the shortcomings of existing point tracking algorithms in large displacement scenarios and improve their robustness to such scenarios, this invention proposes a point tracking method based on a temporal pyramid. This invention introduces a multi-layered post-processing network with receptive fields of different temporal dimensions, which effectively combines the advantages of different time window sizes to enhance robustness to large displacements. This invention argues that smaller temporal receptive field features help point tracking algorithms capture large displacement abrupt changes, while larger temporal receptive field features enable appearance updates, smooth point trajectories, and suppress jitter.

[0006] The method proposed in this invention fuses features of different time window sizes, uses multiple prediction heads to output multiple suggested positions at different levels, and takes the position of the point with the highest confidence as the final point tracking result. Moreover, the constructed model has no restrictions on the length of the input video sequence or the number of tracking points, and can complete the tracking of all points in all frames in parallel through only one forward propagation process.

[0007] To achieve the above objectives, this application adopts the following technical solution:

[0008] A point tracking method for large displacement scenarios based on a time pyramid includes:

[0009] The target tracking point selection is used to collect the coordinates of the target tracking points in the user-specified input video sequence;

[0010] Image feature extraction is used to extract image features at the frame level from the input video sequence and extract the features of the target tracking point at the corresponding position from the feature map sequence based on the target tracking point coordinates;

[0011] Point trajectory initialization involves tiling the target tracking point coordinates and corresponding point features along the time dimension as the initial trajectory values.

[0012] Similarity feature calculation involves calculating the feature similarity between the features of the point to be tracked at each time step and the features of all points in the feature map at the corresponding time step.

[0013] The temporal dimension information fusion network concatenates the extracted point features and the similarity features calculated by the relevant layers as input features, and outputs the coordinate update value and point feature update value for the current point trajectory.

[0014] The point trajectory and point feature update are obtained by fusing the time dimension information into the point trajectory coordinate update value and point feature update value output by prediction heads at multiple different levels, as well as the corresponding update confidence. A max operation is performed on the output of multiple levels based on the confidence to obtain the final point trajectory coordinate update value and point feature update value.

[0015] Finally, the point trajectory is output. The above similarity feature calculation steps and point trajectory and point feature update steps are repeated until the preset number of iterations K. The point trajectory output in the Kth update is the point tracking result.

[0016] As one aspect of the present invention, the step of selecting the tracking point is performed as follows:

[0017] For a user-selected input video sequence, any number of arbitrary pixels in any frame are selected as tracking points, where the coordinates of each point are represented as (t, y, x), where t represents the frame number of the pixel to be tracked, and y and x represent the vertical and horizontal coordinates of the point in the pixel coordinate system, respectively; the coordinates of N selected pixels in B input videos are integrated into a tensor form of (B, N, 3).

[0018] As one aspect of the present invention, the step of selecting the tracking point is performed as follows:

[0019] The input video sequence length is a multiple of 24. For video sequences that do not meet this requirement, the last frame is copied multiple times to make it a multiple of 24. The size of the processed video frame is adjusted to 256×256, and the RGB pixel values ​​are scaled to between -1 and 1. The resulting image data tensor with shape (B,S,H,W,3) is shaped into (B*S,H,W,3). The number of input videos is B, the length of the video sequence is S, the width of the video image is W, and the height of the video image is H.

[0020] As one aspect of the present invention, the image feature extraction step is performed as follows:

[0021] The input is a 2D ResNet network to perform frame-level image feature extraction, and the feature extraction results are obtained. Based on the coordinates of the point to be tracked, the corresponding features of the point to be tracked are extracted from the feature map of the corresponding frame using bilinear interpolation.

[0022] For the extracted features, in order to calculate the similarity features between point pairs, the feature vector is normalized in the feature dimension (D represents the length of the feature dimension, which is 256 in this invention) using the following formula:

[0023]

[0024] Where i, t, h, and w represent the video index, frame index, y-value of the feature in the frame coordinate system, and x-value, respectively, and d represents the length of the feature at a specific location uniquely determined by i, t, h, and w.

[0025] As one aspect of the present invention, the point trajectory initialization step is performed as follows:

[0026] For a point p0 = (t, y, x) to be tracked, where the frame number t and position coordinates (x, y) are decomposed, the point feature vector representing the appearance of the target is obtained by bilinear sampling of the target position (x, y) on the feature map of the frame t containing the point to be tracked. The coordinates of the point to be tracked and its feature vector are tiled along the time dimension, resulting in the following initial values ​​for the trajectory and features:

[0027] {(p t ,f t )}={(p0,f0)},t∈{1,2,…,S};

[0028] Where t represents the frame index of the point to be tracked, x and y represent the horizontal and vertical coordinates of the point in the frame coordinate system, respectively, and p t f represents the position information of the point to be tracked in frame t. tThis represents the feature vector representation of the point to be tracked in frame t.

[0029] As one aspect of the present invention, the similarity feature calculation step is performed as follows:

[0030] In the relevance layer, the query features of a trackable point in a video at a given time are... and the corresponding feature map The correlation cost volume C is formed by taking the dot product of all point pairs. For the case where the number of input videos is B, the video sequence length is S, and the number of points to be tracked is N, the correlation cost volume C is calculated as a single matrix multiplication.

[0031]

[0032] C i,t,n,h,w =∑ d Q i,t,n,d ·g θ (I i,t,h,w,d );

[0033] in Table B contains the frame feature tensors of the input videos over the length S of the complete video sequence. Let C(g) represent the point feature tensors corresponding to the N query points in the B input videos over the length S of the complete video sequence, and let C(g) represent the associated cost volume. θ (I),Q) is the frame feature tensor g mentioned above. θ (I) and the dot product of the feature tensor Q of the point to be tracked on the feature dimension D, where i, t, n, h, and w represent the video index, frame index, query point index, query point x-coordinate value, and query point y-coordinate value, respectively, and d represents the index on the specifically determined one-dimensional point feature vector, Q i,t,n,d That is, the value of the one-dimensional feature vector of the nth query point in the tth frame of the i-th video at index d, g θ (I i,t,h,w,d C is the value of the one-dimensional feature vector of the pixel with coordinates (w,h) in the t-th frame of the i-th video at index d. i,t,n,h,w That is, the dot product of the nth query point in the tth frame of the i-th video and the pixel with coordinates (w,h) in frame t.

[0034] As one aspect of the present invention, the similarity feature calculation step further includes:

[0035] By using pooling operations with kernel sizes of 1, 2, 4, and 8, and a constant step size of 2, the last two dimensions of the relevant cost body C are pooled to construct a 4-layer relevance pyramid {C}. 1 C 2 C 3 C4}, where cost body C k It has dimensions B×S×N×H / 2 k ×W / 2 k .

[0036] As one aspect of the present invention, the time dimension information fusion step is performed as follows:

[0037] The tiled point feature sequence obtained from the point trajectory initialization step Similarity features obtained from similarity feature calculation steps Where L=4 represents the number of layers in the similarity feature pyramid, and P=7 represents the length of the sampling block;

[0038] The features obtained from these two steps are concatenated in the last dimension, and the resulting feature sequence is shaped as follows: This makes it conform to the requirements of the 1D Swin Transformer network for input feature tensors.

[0039] As one aspect of the present invention, the time dimension information fusion step is performed as follows:

[0040] Multi-head self-attention and MLP operations are performed within different time windows based on the input features, and the feature dimension is continuously increased while downsampling sequence length is achieved through Patch Merging operations between the two levels, thereby forming a time-dimensional feature pyramid across 4 stages.

[0041] As one aspect of the present invention, the point trajectory and point feature update step is performed as follows:

[0042] The update operation is performed according to the following formula, and the updated point trajectory and point features are used as the initial values ​​for the next iteration:

[0043] X k+1 =X k +ΔX;

[0044] F k+1 =F k +ΔF;

[0045] where X k F represents the coordinate information of all points to be tracked in frame k. k ΔX represents the feature vector information of all points to be tracked in frame k, ΔX represents the update amount of the position information output by the network, and ΔF represents the update amount of the feature vector output by the network.

[0046] Beneficial effects:

[0047] (1) The point tracking method based on time pyramid in large displacement scenarios provided by the present invention can complete point tracking in large displacement scenarios, fully combining the sensitivity of small time window to sudden motion changes and the smoothness of large time window to long motion, and has strong robustness to large displacement.

[0048] (2) The point tracking method based on time pyramid in large displacement scenarios provided by this invention tracks all points independently and without correlation, and all can be completed in parallel;

[0049] (3) The network of the point tracking method for large displacement scenes based on time pyramid provided in this invention can complete the tracking of all points to be tracked on all frames in one forward propagation process, and there is no limit to the length of the input video sequence, without having to divide a long video into fixed-size blocks and track each block sequentially. Therefore, our points to be tracked can be selected on any frame, and the algorithm will predict their trajectories forward and backward at the same time;

[0050] (4) The point tracking method based on time pyramid in large displacement scenarios provided by this invention uses window-based self-attention in the iterative update network, which is scalable. When the window size is fixed (3 in this method), the complexity of the algorithm is linear with the sequence length.

[0051] (5) The network of the point tracking method based on time pyramid in large displacement scene provided by the present invention can be trained directly on continuous video sequences and achieve good performance in non-continuous large displacement scene.

[0052] Specific embodiments of the invention are disclosed in detail with reference to the following description and accompanying drawings, indicating how the principles of the invention can be employed. It should be understood that the embodiments of the invention are not therefore limited in scope. Within the spirit and scope of the appended claims, embodiments of the invention include many changes, modifications, and equivalents.

[0053] Features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, combined with features in other embodiments, or substituted for features in other embodiments.

[0054] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, whole, step, or component, but does not exclude the presence or addition of one or more other features, wholes, steps, or components. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is an overall architecture diagram of a point tracking method based on a time pyramid in a large displacement scenario according to an embodiment of the present invention;

[0057] Figure 2 yes Figure 1 Feature extraction network structure diagram;

[0058] Figure 3 yes Figure 1 A multi-level post-processing network structure diagram;

[0059] Figure 4 yes Figure 1 A schematic diagram of window shifting and patch merging. Detailed Implementation

[0060] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0061] This invention provides a point tracking method for large displacement scenarios based on a time pyramid. The overall model architecture is divided into a data processing part and an iterative update part.

[0062] The data processing section provides input features for the iterative network update by extracting and processing features from the input video sequence and the coordinates of the points to be tracked. The specific steps are as follows:

[0063] Step S1: Select the tracking point, used to collect the coordinates of the target tracking point in the user-specified input video sequence;

[0064] Step S2: Image feature extraction, used to extract frame-level image features from the input video sequence;

[0065] Step S3: Point trajectory initialization, used to set the initial trajectory value as the starting point for iterative updates.

[0066] Further, step S1: Selection of tracking points. The user can select any number of arbitrary pixels on any frame of the input video sequence as the target for point tracking. Based on the collected set of user-selected point tracking coordinates, the present invention can simultaneously complete the position tracking of all selected points on all video frames through a single forward propagation process.

[0067] Further, step S2: Image feature extraction. This consists of two parts: data preprocessing and a 2D feature extraction network. Frame-level image feature extraction is performed on the input video sequence, and the features of the tracked point at the corresponding position are extracted from the feature map sequence based on the coordinates of the tracked point. The data preprocessing involves adjusting the RGB image size of each frame of the input video sequence to 256×256, scaling the pixel values ​​to between -1 and 1, and reshaping the resulting image data tensor (B,S,C,H,W) into (B*S,C,H,W). Then, a 2D ResNet network with 4 stages is used to complete frame-level multi-level image feature extraction, and the corresponding tracked point features are extracted from the feature map of the corresponding frame using bilinear interpolation based on the coordinates of the tracked point.

[0068] Further, step S3: Point trajectory initialization. The goal of the point tracking algorithm is to use a deep neural network model to model an optimization algorithm. Therefore, it is necessary to provide an initial value for the optimization of the point trajectory, and the trained point tracking network iterates step by step to the final result based on the initial value. This method simply flattens the coordinates and features of the point to be tracked in the time dimension as the initial value of the trajectory, that is, assuming no movement, and the tracking algorithm provides us with updated values ​​for iteration.

[0069] The iterative update network is responsible for progressively updating and correcting the initialized point trajectory. The iterative update module consists of two parts: a correlation layer and a temporal information fusion network. The correlation layer calculates the feature similarity between the tiled point feature sequence and all point feature pairs in the corresponding feature map. The information fusion network concatenates the image features provided by the data processing part with the similarity features generated by the correlation layer, using this as input and outputting the updated trajectory value. The specific steps are as follows:

[0070] Step S4: Calculate similarity features;

[0071] Step S5: Fusion of time-dimension information;

[0072] Step S6: Update point trajectory and point features;

[0073] Step S7: Output the trajectory of the final point.

[0074] Further, step S4: Similarity feature calculation. A feature similarity is calculated between the features of the point to be tracked at each time step and the features of all points in the corresponding time step feature map. Three consecutive pooling operations with a kernel size of 2 and a stride of 2 are applied to the image feature map sequence output by the 2D ResNet network, resulting in a 4-layer feature map pyramid. The feature map sizes of each layer are 32×32, 16×16, 8×8, and 4×4, respectively. Within each layer, a 7×7 region is extracted centered on the currently estimated point trajectory. A similarity is calculated between the features of all points in this region and the features of the point to be tracked extracted in step 2. The similarity features from the four layers are then concatenated to form a final similarity feature.

[0075] Further, step S5: Temporal information fusion. Temporal information fusion is based on a 1D Swin Transformer with 4 stages, modified from 2D, to fuse information in the temporal dimension of the video sequence. The tiled point feature sequence obtained in step S3 is then used... Similarity features obtained in step S4 Concatenate along the last dimension, and further reshape the resulting feature sequence. As input to the 1D Swin Transformer network, the network outputs four fusion features with different temporal receptive fields in its four stages. These four fusion features at different levels are further processed by a prediction head to output the updated value ΔX of the point position coordinates and the updated value ΔF of the point features, as well as the corresponding confidence scores. The output with the highest confidence score will be used as the updated value of the point features and point positions for the current round.

[0076] Further, step S6: Update the point trajectory and point features. Based on the point coordinate update value ΔX and point feature update value ΔF output in step S5, update the point trajectory and point features according to formula (1) and formula (2) respectively. The updated point trajectory and point features are used as the initial values ​​for the next iteration.

[0077] X k+1 =X k +ΔX

[0078] F k+1 =F k +ΔF

[0079] Further, step S7: Output the final point trajectory. Repeat steps S4 to S6 up to K times. The network updates iteratively share parameters, where K can take any value. The final point trajectory output after the Kth update is the point tracking result of this method.

[0080] The invention will be better understood by describing one embodiment of the invention in detail below.

[0081] like Figures 1 to 4 As shown in the figure, the point tracking method for large displacement scenarios based on time pyramids in this embodiment has an overall architecture including a data processing module and an iterative update module. The specific steps involved in the data processing module are as follows:

[0082] S1: Selection of tracking points, used to collect the coordinates of target tracking points in the user-specified input video sequence. For the user-selected input video sequence, the user can select any number of arbitrary pixels in any frame as tracking points, where the coordinates of each point are represented as (t, y, x), where t represents the frame number of the tracking pixel, and y and x represent the vertical and horizontal coordinates of the point in the pixel coordinate system, respectively. The coordinates of N selected pixels in B input videos will be integrated into a tensor form of (B, N, 3).

[0083] S2: Image Feature Extraction, used to extract image features at the frame level from the input video sequence. It consists of two parts: data preprocessing and a 2D feature extraction network. First, the input video sequence is preprocessed, and the processing result is input into the feature extraction network in the form of an image tensor for frame-level image feature extraction. Then, based on the coordinates of the point to be tracked, the features of the corresponding position of the point to be tracked are extracted from the feature map sequence.

[0084] The data preprocessing section requires processing the video data input to the network to conform to the network's input requirements. Since the window size in the self-attention module of the 1D Swin Transformer in the iterative update module is set to 3, and the network contains 4 layers, a patch merging operation is performed between every two layers to halve the sequence length and double the feature dimensions. To ensure that the sequence length is still divisible by the window size of 3 in the last layer, the length of the input video sequence must be a multiple of 24. For video sequences that do not meet this requirement, the last frame needs to be copied multiple times to make it a multiple of 24. The processed video frame size is adjusted to 256×256, the RGB pixel values ​​are scaled to between -1 and 1, and the resulting image data tensor of shape (B,S,H,W,3) is shaped to (B*S,H,W,3).

[0085] The input is a 2D ResNet network to perform frame-level image feature extraction, and the feature extraction results are obtained. Based on the coordinates of the point to be tracked, the corresponding features of the point to be tracked are extracted from the feature map of the corresponding frame using bilinear interpolation.

[0086] For the extracted features, in order to perform subsequent similarity feature calculations between point pairs, the feature vectors need to be further normalized:

[0087]

[0088] Where i represents the video index; t represents the frame index; h represents the y-value of the feature in the frame coordinate system; and w represents the x-value of the feature in the frame coordinate system.

[0089] S3: Point trajectory initialization. The coordinates of the point to be tracked and its corresponding point features are tiled along the time dimension as the initial trajectory values. That is, assuming no motion, the initialized point trajectory will serve as the starting point for iteratively updating the network. For a point to be tracked, p0 = (t, y, x), the initial trajectory is the point feature vector representing the appearance of the target obtained by bilinear sampling of the target position (x, y) from the feature map of the frame t containing the point to be tracked. The coordinates of the point to be tracked and its feature vector are tiled along the time dimension, resulting in the following initial values ​​for the trajectory and features:

[0090]

[0091] The iterative update network is responsible for progressively updating and correcting the initialized point trajectory. Based on a set number of iterations K, it gradually updates the point trajectory tracking results starting from the initial value, with the result of the previous iteration serving as the initial value for the next iteration. The iterative update of the point trajectory consists of two parts: a correlation layer and a time-dimensional information fusion network. The specific steps are as follows:

[0092] S4: Similarity feature calculation: This calculates the feature similarity between the features of the point to be tracked at each time step and the features of all points in the feature map at the corresponding time step. In the relevance layer, given the query features of a point to be tracked in a video at a certain time step... and the corresponding feature map The correlation cost volume C is formed by taking the dot product of all point pairs. For a given number of input videos of length B, a video sequence length of length S, and N points to be tracked, the correlation cost volume C can be efficiently calculated as a single matrix multiplication:

[0093]

[0094] in Table B contains the frame feature tensors of the input videos over the length S of the complete video sequence. Let C(g) represent the point feature tensors corresponding to the N query points in the B input videos over the length S of the complete video sequence, and let C(g) represent the associated cost volume. θ (I),Q) is the frame feature tensor g mentioned above. θ(I) and the dot product of the feature tensor Q of the point to be tracked on the feature dimension D, where i, t, n, h, and w represent the video index, frame index, query point index, query point x-coordinate value, and query point y-coordinate value, respectively, and d represents the index on the specifically determined one-dimensional point feature vector, Q i,t,n,d That is, the value of the one-dimensional feature vector of the nth query point in the tth frame of the i-th video at index d, g θ (I i,t,h,w,d C is the value of the one-dimensional feature vector of the pixel with coordinates (w,h) in the t-th frame of the i-th video at index d. i,t,n,h,w That is, the dot product of the nth query point in the tth frame of the i-th video and the pixel with coordinates (w,h) in frame t.

[0095] To ensure that the relevant cost body provides information on displacements from largest to smallest, a pooling operation with kernel sizes of 1, 2, 4, and 8, and a constant step size of 2, is used to pool the last two dimensions of the relevant cost body to construct a 4-layer relevant pyramid {C}. 1 C 2 C 3 C 4}, where cost body C k It has dimensions B×S×N×H / 2 k ×W / 2 k .

[0096] S5: Temporal Information Fusion. The temporal information fusion network concatenates the point features extracted by the data processing module and the similarity features calculated by the relevant layers as input features, and outputs the updated value for the current point trajectory. It is based on a 1DSwin Transformer with 4 stages, which is modified from a 2D Swin Transformer and is suitable for 1D time series, to perform information fusion in the temporal dimension of video sequences.

[0097] The tiled point feature sequence obtained from step S3 Similarity features obtained in step S4 Where L=4 represents the number of layers in the similarity feature pyramid, and P=7 represents the length of the sampling block. The features obtained in these two steps are concatenated in the last dimension, and the resulting feature sequence is further shaped into... This makes it conform to the requirements of the 1D Swin Transformer network for input feature tensors.

[0098] The temporal information fusion network performs multi-head self-attention and MLP operations in different time windows based on the input features, and continuously increases the feature dimension while downsampling the sequence length based on the Patch Merging operation between the two levels, thus forming a temporal feature pyramid in 4 stages.

[0099] S6: Update of point trajectory and point features. Based on the point coordinate update value and point feature update value output by the prediction head at the four levels in step S5, and the corresponding update confidence, perform a max operation on the output of the four levels according to the confidence to obtain the final point position coordinate update value and point feature update value. Perform update operation according to formula (5) and formula (6) respectively. The updated point trajectory and point features are used as the initial values ​​for the next iteration.

[0100] X k+1 =X k +ΔX (5)

[0101] F k+1 =F k +ΔF (6)

[0102] S7: Output of the final point trajectory. Repeat steps S4 to S7 until the preset number of iterations K. The final point trajectory output after the Kth update is the point tracking result output by this method.

[0103] This invention utilizes the characteristic that the designed temporal information fusion network has receptive fields of different sizes at different network levels to construct a temporal pyramid, generating a multi-level, multi-predictor iterative update network. Small time windows at lower levels are used to perceive motion abrupt changes, while large time windows at higher levels smooth motion and reduce jitter. Any modifications, fine-tuning, combinations, simplifications, and substitutions of the network structure made by those skilled in the art within the scope of the technology disclosed in this invention, without departing from the spirit and conceptual design of this invention, are included within the protection scope of this invention.

[0104] Multiple elements, components, parts, or steps can be provided by a single integrated element, component, part, or step. Alternatively, a single integrated element, component, part, or step can be divided into multiple separate elements, components, parts, or steps. The use of "a" or "an" to describe an element, component, part, or step does not imply the exclusion of other elements, components, parts, or steps.

[0105] It should be understood that the above description is for illustrative purposes and not for limitation. Many embodiments and applications beyond the provided examples will be apparent to those skilled in the art upon reading the above description. Therefore, the scope of this teaching should not be determined by reference to the above description, but rather by reference to the appended claims and the full scope of their equivalents. For purposes of completeness, all articles and references, including patent applications and publications, are incorporated herein by reference. The omission of any aspect of the subject matter disclosed herein in the preceding claims is not intended as a waiver of that subject matter, nor should it be construed as an indication that the inventors have not considered that subject matter as part of the disclosed inventive subject matter.

Claims

1. A point tracking method for large displacement scenarios based on a time pyramid, characterized in that, include: The target tracking point selection is used to collect the coordinates of the target tracking points in the user-specified input video sequence; Image feature extraction is used to extract image features at the frame level from the input video sequence and extract the features of the target tracking point at the corresponding position from the feature map sequence based on the target tracking point coordinates; Point trajectory initialization involves tiling the target tracking point coordinates and corresponding point features along the time dimension as the initial trajectory values. Similarity feature calculation involves calculating the feature similarity between the features of the point to be tracked at each time step and the features of all points in the feature map at the corresponding time step. Temporal information fusion is employed in this network. The extracted point features and similarity features calculated by related layers are concatenated and used as input to a 1D Swin Transformer network. Multi-head self-attention and MLP operations are performed within different time windows based on the input features. Patch merging between two layers continuously increases the feature dimension while downsampling the sequence length, forming a temporal feature pyramid across four stages. Four different fused features are output from each stage. These four fused features are further processed by a prediction head to output updated point coordinates, updated point features, and corresponding confidence scores. The output with the highest confidence score is used as the updated point feature and point position for the current iteration. The point trajectory and point features are updated based on the output point coordinate update values ​​and point feature update values. The updated point trajectory and point features are used as the initial values ​​for the next iteration. Finally, the point trajectory is output. Repeat the above similarity feature calculation steps and point trajectory and point feature update steps until the preset number of iterations K. The point trajectory output in the Kth update is the point tracking result.

2. The point tracking method for large displacement scenarios based on a time pyramid as described in claim 1, characterized in that, The step of selecting the point to be tracked is performed as follows: For a user-selected input video sequence, any number of arbitrary pixels in any frame are chosen as the points to be tracked, where the coordinates of each point are represented as follows: ,in This indicates the frame number of the pixel to be tracked. and These represent the ordinate and abscissa of a point in the pixel coordinate system, respectively; for The selected videos from each input video The coordinates of each pixel were integrated into The tensor form.

3. The point tracking method for large displacement scenarios based on a time pyramid as described in claim 2, characterized in that, The step of selecting the point to be tracked is performed as follows: The input video sequence length must be a multiple of 24. For video sequences that do not meet this requirement, the last frame is copied multiple times to make it a multiple of 24, and the size of the processed video frame is adjusted accordingly. The RGB pixel values ​​are scaled to between -1 and 1, and the resulting shape is... Image data tensor shaping The input video sequence has a length of B, a width of W, and a height of H.

4. The point tracking method for large displacement scenarios based on a time pyramid as described in claim 1, characterized in that, The image feature extraction step is performed as follows: The input is a 2D ResNet network to perform frame-level image feature extraction, and the feature extraction results are obtained. And based on the coordinates of the point to be tracked, extract the corresponding features of the point to be tracked from the feature map of the corresponding frame using bilinear interpolation; For the extracted features, in order to calculate the similarity features between point pairs, the feature vector is normalized in the feature dimension using the following formula: ； in, , , , These represent the video index, frame index, and features in the frame coordinate system, respectively. value and value, Representative by , , , The length of a feature that uniquely identifies a specific location; S represents the length of the video sequence.

5. The point tracking method for large displacement scenarios based on a time pyramid as described in claim 1, characterized in that, The point trajectory initialization step is performed as follows: For a point to be tracked The frame sequence number was extracted from it. and location coordinates The frame where the tracking point is located Target location on feature map The point feature vector representing the appearance of the target obtained by bilinear sampling The coordinates of the point to be tracked and its feature vectors are tiled along the time dimension, resulting in the following initial values ​​for the trajectory and features: ； in This indicates the frame index of the point to be tracked. , These represent the x and y coordinates of the point in the frame coordinate system, respectively. Indicates the point to be tracked is at the 1st dt. Position information in the frame, Indicates the point to be tracked is at the 1st dt. Feature vector representation in a frame.

6. The point tracking method for large displacement scenarios based on a time pyramid as described in claim 1, characterized in that, The similarity feature calculation step is performed as follows: In the relevance layer, the query features of a trackable point in a video at a given time are... and the corresponding feature map Take the dot product of all pairs of points to form the relevant cost volume. For the case where the number of input videos is B, the video sequence length is S, and the number of tracking points is N, the relevant cost body Calculated as a single matrix multiplication: ； ； in surface The length of each input video in the complete video sequence The frame feature tensor on, express In the input video Each query point is within the length of the complete video sequence. The corresponding point feature tensors and related cost volumes are shown above. That is, the frame feature tensor mentioned above. and the feature tensor of the point to be tracked In feature dimension The dot product obtained on, , , , , These represent the video index, frame index, query point index, and query point, respectively. Coordinates, query point Coordinate values This represents the index on the feature vector of a specifically determined one-dimensional point. That is, the first The first video Frame number The one-dimensional feature vector of each query point is The value at the index, That is, the first The first video The coordinates on the frame are The one-dimensional feature vector of the pixel in The value at the index, That is, the first The first video Frame number Query points and frames The upper coordinate is The dot product of the pixels.

7. The point tracking method for large displacement scenarios based on a time pyramid as described in claim 6, characterized in that, The similarity feature calculation step further includes: Pooling operations with kernel sizes of 1, 2, 4, and 8, and a constant step size of 2, are used to group the relevant cost components. The last two dimensions are pooled to construct a 4-layer correlation pyramid. Among them, cost body Having dimensions .

8. The point tracking method for large displacement scenarios based on a time pyramid as described in claim 1, characterized in that, The time dimension information fusion step is performed as follows: The tiled point feature sequence obtained from the point trajectory initialization step The similarity features obtained from the similarity feature calculation steps ;in This represents the number of layers in the similarity feature pyramid. Indicates the length of the sampling block; The features obtained from these two steps are concatenated in the last dimension, and the resulting feature sequence is shaped as follows: This makes it conform to the requirements of the 1D Swin Transformer network for input feature tensors.

9. The point tracking method for large displacement scenarios based on a time pyramid as described in claim 1, characterized in that, The point trajectory and point feature update steps are performed as follows: The update operation is performed according to the following formula, and the updated point trajectory and point features are used as the initial values ​​for the next iteration: ； ； in This indicates that all points to be tracked are in the frame. Coordinate information in the middle, This indicates that all points to be tracked are in the frame. Feature vector information in This indicates the amount of location information updated by the network output. This represents the update amount of the feature vector output by the network.

Images