Time pyramid-based point tracking method in large-displacement scene

Abstract

The invention belongs to the technical field of computer vision, and relates to a point tracking method in a large displacement scene based on a time dimension pyramid, which utilizes time windows with different sizes to adaptively solve various point tracking conditions. The point tracking model architecture comprises a data processing module and a shared parameter iterative updating module, wherein the data processing module comprises the following steps: (1) selecting a point to be tracked; (2) extracting image features; and (3) point trajectory initialization. The iteration updating module is composed of a correlation layer and a time dimension information fusion network, in the time dimension information fusion network part, a characteristic pyramid of a time dimension is formed to comprehensively solve different displacement conditions by utilizing the characteristic that the designed network has receptive fields of different sizes at different layers, and the time dimension information fusion network is obtained. The method comprises the following specific steps: (4) similarity feature calculation; (5) time dimension information fusion; (6) updating point tracks and point features; and (7) outputting a final point track, repeating the steps (4)-(6) until a set number of iteration updating times is reached, at the moment, finishing point tracking of the target point in the input video sequence, taking a point track updating result output by the model as a final point tracking result output by the method, and finishing point tracking.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision, and in particular relates to a point tracking method in a large displacement scene based on a time pyramid. Background Art

[0002] Motion is crucial for scene understanding. Perceiving how objects move and deform helps us understand the physical properties of objects and infer their intentions. Most of the current research on motion and tracking problems focuses on three types of problems: fine-grained short-range tracking (i.e., optical flow), coarse-grained long-range tracking (i.e., object tracking, segmentation, etc.), and sparse feature tracking (i.e., tracking features across images). There is less research at the intersection of fine-grained and long-range tracking (i.e., point tracking).

[0003] Data-driven deep neural network architectures represent a technology that performs better than traditional methods and have achieved significant breakthroughs in many vision problems, providing new solutions for these areas. Deep learning-based methods have achieved better performance than traditional methods by predicting long-distance pixel-level trajectories in a feed-forward manner. However, most of these studies focus on long-distance point tracking on smooth continuous videos with continuous natural motion.

[0004] The most advanced point tracking algorithms are trained based on trajectories in continuous short video sequences, focusing on video sequences with continuous natural motion, without considering the processing of large displacements. However, most scenes in real life contain cut video scenes with large displacements caused by discontinuous motion. Summary of the invention

[0005] In order to make up for the shortcomings of existing point tracking algorithms in large displacement scenarios and improve the robustness of point tracking algorithms to large displacements, the present invention proposes a point tracking method in large displacement scenarios based on a time pyramid. The present invention introduces a multi-level post-processing network with different time dimension receptive fields, which can fully combine the advantages of time windows of different sizes and effectively improve the robustness to large displacements. The present invention believes that the small time dimension receptive field feature can help the point tracking algorithm capture large displacement mutations, while the large time dimension receptive field feature can achieve appearance updates and smooth point trajectories and suppress jitter phenomena.

[0006] The method proposed in the present invention fuses features of different time window sizes, uses multiple prediction heads to output multiple suggested positions at different levels, and takes the point position with the highest confidence as the final point tracking result. The constructed model has no restrictions on the length of the input video sequence and the number of tracking points, and can complete the tracking of all points in all frames in parallel through only one forward propagation process.

[0007] In order to achieve the above objectives, this application adopts the following technical solutions:

[0008] A point tracking method in a large displacement scene based on a time pyramid, comprising:

[0009] The selection of points to be tracked is used to collect the coordinates of target tracking points in the input video sequence specified by the user;

[0010] Image feature extraction, which is used to extract frame-level image features from the input video sequence and extract the features of the points to be tracked at the corresponding positions from the feature map sequence according to the coordinates of the target tracking points;

[0011] Initialize point trajectory, flatten the target tracking point coordinates and corresponding point features in the time dimension as the initial value of the trajectory;

[0012] Similarity feature calculation, calculating the feature similarity between the features of the point to be tracked at each moment and the features of all points in the feature graph at the corresponding moment;

[0013] Time dimension information fusion, the time dimension information fusion network concatenates the extracted point features and the similarity features calculated by the relevant layer as input features, and outputs the coordinate update value and point feature update value for the current point trajectory;

[0014] Point trajectory and point feature update: according to the time dimension information, the point trajectory coordinate update value and the point feature update value are outputted in the prediction heads at multiple different levels respectively, and the corresponding updated confidences are performed on the outputs of multiple levels according to the confidences to obtain the final point trajectory coordinate update value and point feature update value;

[0015] The final point trajectory is output. After repeating the above similarity feature calculation steps ~ point trajectory and point feature update steps to a preset number of iterations K, the point trajectory output by the Kth update is the point tracking result.

[0016] As one aspect of the present invention, the step of selecting points to be tracked is performed as follows:

[0017] For the input video sequence selected by the user, any number of arbitrary pixel points on any frame are selected as the points to be tracked, where the coordinate representation of each point is (t, y, x), where t represents the frame number of the pixel point to be tracked, and y and x represent the ordinate and abscissa of the point in the pixel coordinate system respectively; the coordinates of the N pixel points selected from the B input videos are integrated into a tensor form of (B, N, 3).

[0018] As one aspect of the present invention, the step of selecting points to be tracked is performed as follows:

[0019] The length of the video sequence input to the network is a multiple of 24. For video sequences that do not meet the requirement, the last frame is copied multiple times at the end of the video to make up for the multiple of 24 frames, and the processed video frame size is adjusted to 256×256, the RGB pixel value is scaled to between -1 and 1, and the resulting image data tensor with a shape of (B, S, H, W, 3) is reshaped to (B*S, H, W, 3); the number of input videos is B, the length of the video sequence is S, the width of the video image is W, and the height of the video image is H.

[0020] As one aspect of the present invention, the image feature extraction step is performed as follows:

[0021] Input 2D ResNet network to complete frame-level image feature extraction and obtain feature extraction results And according to the coordinates of the point to be tracked, the corresponding features of the point to be tracked are extracted from the feature map of the corresponding frame by using bilinear interpolation;

[0022] For the extracted features, in order to calculate the similarity features between the point pairs later, the feature vector is normalized using the following formula in the feature dimension (D represents the length of the feature dimension, which is 256 in the present invention):

[0023]

[0024] Among them, i, t, h, and w represent the video index, frame index, y value, and x value of the feature in the frame coordinate system, respectively, and d represents the length of the feature at a specific position uniquely determined by i, t, h, and w.

[0025] As one aspect of the present invention, the point trajectory initialization step is performed as follows:

[0026] For a point to be tracked p0 = (t, y, x), the frame number t and the position coordinates (x, y) are decomposed, and the point feature vector representing the target appearance is obtained by bilinear sampling of the target position (x, y) on the feature map of the frame t where the tracking point is located. The coordinates of the points to be tracked and the feature vectors of the points are flattened in the time dimension, and the initial values ​​of the trajectory and features are obtained:

[0027] {(p t ,f t )}={(p0,f0)},t∈{1,2,…,S};

[0028] Where t represents the frame index of the point to be tracked, x and y represent the horizontal and vertical coordinates of the point in the frame coordinate system, respectively. t represents the position information of the point to be tracked in the tth frame, f tRepresents the feature vector representation of the point to be tracked in the tth frame.

[0029] As one aspect of the present invention, the similarity feature calculation step is performed as follows:

[0030] In the relevant layer, given a query feature of a point to be tracked in a video at a certain moment And the characteristic graph at the corresponding time The dot product between all point pairs is taken to form the relevant cost volume C; for the case where the number of input videos is B, the length of the video sequence is S, and the number of points to be tracked is N, the relevant cost volume C is calculated as a single matrix multiplication:

[0031]

[0032] C i,t,n,h,w =∑ d Q i,t,n,d ·g θ (I i,t,h,w,d );

[0033] in Table B The frame feature tensor of the input video in the complete video sequence length S, represents the point feature tensor corresponding to each of the N query points in the B input videos on the complete video sequence length S, and the related cost volume C(g θ (I),Q) is the above frame feature tensor g θ (I) and the dot product of the feature tensor Q of the point to be tracked on the feature dimension D, i, t, n, h, w represent the video index, frame index, query point index, query point x coordinate value, query point y coordinate value, d represents the index on the specifically determined one-dimensional point feature vector, Q i,t,n,d That is, the value of the one-dimensional feature vector of the n-th query point on the t-th frame in the i-th video at index d, g θ (I i,t,h,w,d ) is the value of the one-dimensional feature vector of the pixel with coordinates (w, h) on the t-th frame in the i-th video at index d, C i,t,n,h,w That is, it is the dot product of the nth query point on the tth frame in the i-th video and the pixel with coordinates (w, h) on frame t.

[0034] As one aspect of the present invention, the similarity feature calculation step further includes:

[0035] Through the pooling operation with kernel sizes of 1, 2, 4, and 8 and a constant step size of 2, the last two dimensions of the relevant cost volume C are pooled to construct a 4-layer relevance pyramid {C 1 ,C 2 ,C 3 ,C4}, where the cost body C k With dimensions B×S×N×H / 2 k ×W / 2 k .

[0036] As one aspect of the present invention, the time dimension information fusion step is performed as follows:

[0037] The flattened point feature sequence obtained by the point trajectory initialization step Similarity features obtained in the similarity feature calculation step Where L = 4 represents the number of layers of the similarity feature pyramid, and P = 7 represents the length of the sampling block;

[0038] The features obtained in the two steps are concatenated in the last dimension, and the resulting feature sequence is reshaped as Make it meet the requirements of the 1D Swin Transformer network for input feature tensors.

[0039] As one aspect of the present invention, the time dimension information fusion step is performed as follows:

[0040] According to the input features, multi-head self-attention and MLP operations are performed in different time windows, and the feature dimension is continuously increased while downsampling the sequence length according to the Patch Merging operation between the two levels, thus forming a feature pyramid of time dimension in 4 stages.

[0041] As one aspect of the present invention, the point trajectory and point feature updating step is performed as follows:

[0042] The update operation is performed according to the following formula, and the updated point trajectory and point features are used as the initial values ​​of the next iteration:

[0043] X k+1 =X k +ΔX;

[0044] F k+1 =F k +ΔF;

[0045] Where X k represents the coordinate information of all points to be tracked in frame k, F k represents the feature vector information of all points to be tracked in frame k, ΔX represents the update amount of the position information output by the network, and ΔF represents the update amount of the feature vector output by the network.

[0046] Beneficial effects:

[0047] (1) The point tracking method in a large displacement scene based on the time pyramid provided by the present invention can complete point tracking in a large displacement scene, fully combining the sensitivity of a small time window to motion mutations and the smoothness of a large time window to long motions, and has strong robustness to large displacements;

[0048] (2) In the point tracking method for large displacement scenarios based on the time pyramid provided by the present invention, all the points to be tracked are independent and unrelated, and all can be completed in parallel;

[0049] (3) The network of the point tracking method in large displacement scenarios based on the time pyramid provided by the present invention will complete the tracking of all points to be tracked in all frames in one forward propagation process, and there is no restriction on the length of the input video sequence, and there is no need to divide a long video into blocks of fixed size and track each block in turn. Therefore, our points to be tracked can be selected on any frame, and the algorithm will predict its trajectory forward and backward at the same time;

[0050] (4) The window-based self-attention used in the iterative update network of the point tracking method based on the time pyramid in large displacement scenarios provided by the present invention is scalable. When the window size is fixed (fixed to 3 in this method), the complexity of the algorithm is linear with the sequence length.

[0051] (5) The network of the point tracking method in large displacement scenes based on the time pyramid provided by the present invention can be directly trained on continuous video sequences and directly obtain good performance in non-continuous large displacement scenes.

[0052] With reference to the following description and accompanying drawings, specific embodiments of the present invention are disclosed in detail, indicating the manner in which the principles of the present invention can be adopted. It should be understood that the embodiments of the present invention are not limited in scope. Within the spirit and scope of the appended claims, the embodiments of the present invention include many changes, modifications and equivalents.

[0053] Features described and / or illustrated with respect to one embodiment may be used in the same or similar manner in one or more other embodiments, combined with features in other embodiments, or substituted for features in other embodiments.

[0054] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, integers, steps or components, but does not exclude the presence or addition of one or more other features, integers, steps or components. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying creative labor.

[0056] Figure 1 It is a general architecture diagram of a point tracking method in a large displacement scenario based on a time pyramid in an embodiment of the present invention;

[0057] Figure 2 yes Figure 1 The feature extraction network structure diagram in ;

[0058] Figure 3 yes Figure 1 Multi-level post-processing network structure diagram;

[0059] Figure 4 yes Figure 1 Schematic diagram of window shifting and patch merging. DETAILED DESCRIPTION

[0060] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0061] An embodiment of the present invention provides a point tracking method in a large displacement scenario based on a time pyramid, and the overall model architecture is divided into a data processing part and an iterative update part.

[0062] The data processing part extracts and processes the features of the input video sequence and the coordinates of the points to be tracked to provide input features for the iterative update network. The specific steps are as follows:

[0063] Step S1: Selection of points to be tracked, used to collect the coordinates of target tracking points in the input video sequence specified by the user;

[0064] Step S2: Image feature extraction, used to extract frame-level image features from the input video sequence;

[0065] Step S3: point trajectory initialization, used to set the initial value of the trajectory as the starting point for iterative update.

[0066] Furthermore, step S1: selection of points to be tracked. The user can select any number of arbitrary pixel points on any frame of the input video sequence as the target of point tracking. According to the collected set of point tracking coordinates selected by the user, the present invention can simultaneously complete the position tracking of all selected points on all video frames through a forward propagation process.

[0067] Furthermore, step S2: image feature extraction. It consists of two parts: data preprocessing and 2D feature extraction network. It performs frame-level image feature extraction on the input video sequence, and extracts the features of the points to be tracked at the corresponding positions from the feature map sequence according to the coordinates of the points to be tracked. The data preprocessing is to adjust the size of the RGB image of each frame of the input video sequence to 256×256, scale the pixel value to between -1 and 1, and reshape the resulting image data tensor with a shape of (B, S, C, H, W) to (B*S, C, H, W). Then, a 2D ResNet network with 4 stages is used to complete the multi-level image feature extraction at the frame level, and extract the corresponding features of the points to be tracked from the feature map of the corresponding frame according to the coordinates of the points to be tracked using bilinear interpolation.

[0068] Further, step S3: point trajectory initialization. The goal of the point tracking algorithm is to use a deep neural network model to model an optimization algorithm, so it is necessary to provide an initial value for the optimization of the point trajectory, and the trained point tracking network iterates step by step to the final result based on the initial value. This method simply flattens the coordinates and point features of the points to be tracked in the time dimension as the initial value of the trajectory, that is, it assumes that there is no movement, and the tracking algorithm provides us with updated values ​​for iteration.

[0069] The iterative update network is responsible for gradually updating and correcting the initialized point trajectory. The iterative update module of the point trajectory consists of two parts: the correlation layer and the time dimension information fusion network. The correlation layer is responsible for calculating the feature similarity between the flattened point feature sequence and all point feature pairs in the corresponding feature map, while the information fusion network splices the image features provided by the data processing part with the similarity features generated by the correlation layer as the input part of the network and outputs the updated value of the trajectory. The specific steps are:

[0070] Step S4: similarity feature calculation;

[0071] Step S5: time dimension information fusion;

[0072] Step S6: updating point trajectory and point features;

[0073] Step S7: Output of final point trajectory.

[0074] Further, step S4: similarity feature calculation. A feature similarity is calculated between the features of the point to be tracked at each moment and the features of all points in the feature map at the corresponding moment. The image feature map sequence output by the 2D ResNet network is continuously applied 3 times with a pooling operation of a kernel size of 2 and a step size of 2 to obtain a 4-layer feature map pyramid, in which the sizes of the feature maps of each layer are 32×32, 16×16, 8×8, and 4×4, respectively. And in each layer, a 7×7 area is extracted with the current estimated point trajectory as the center, and a similarity is calculated between all point features in the area and the features of the point to be tracked extracted in step 2, and the similarity features of the 4 levels are spliced ​​to form a final similarity feature.

[0075] Further, step S5: time dimension information fusion. The fusion of time dimension information is based on a 1D Swin Transformer with 4 stages transformed from 2D to fuse the video sequence in the time dimension. The flattened point feature sequence obtained in step S3 is Similarity features obtained in step S4 Do concatenation on the last dimension and further reshape the resulting feature sequence into As the input of the 1D Swin Transformer network. The network outputs four fusion features with different temporal receptive fields in its four stages, and further outputs the update value ΔX of the point position coordinates and the update value ΔF of the point feature and the corresponding confidence through a prediction head. The output with the highest confidence will be used as the point feature and point position update value of the current round.

[0076] Further, step S6: updating the point trajectory and point features. According to the point coordinate update value ΔX and the point feature update value ΔF output in step S5, the point trajectory and point features are updated according to formula (1) and formula (2) respectively, and the updated point trajectory and point features are used as the initial values ​​of the next iteration.

[0077] X k+1 =X k +ΔX

[0078] F k+1 =F k +ΔF

[0079] Further, step S7: output the final point trajectory. Repeat the above steps S4 to S6 to K times, the iterative update network is a shared parameter, and K here can take any value. The point trajectory output by the final Kth update is the point tracking result of this method.

[0080] The present invention will be better understood by describing one embodiment of the present invention in detail below.

[0081] like Figures 1 to 4 As shown, the overall architecture of the point tracking method in a large displacement scenario based on a time pyramid of this embodiment includes a data processing module and an iterative update module, wherein the specific steps involved in the data processing module are as follows:

[0082] S1: Selection of points to be tracked, used to collect the coordinates of target tracking points in the input video sequence specified by the user. For the input video sequence selected by the user, the user can select any number of arbitrary pixel points on any frame as the points to be tracked, where the coordinate representation of each point is (t, y, x), where t represents the frame number of the pixel to be tracked, and y and x represent the ordinate and abscissa of the point in the pixel coordinate system, respectively. The coordinates of the N pixel points selected from the B input videos will be integrated into a tensor form of (B, N, 3).

[0083] S2: Image feature extraction, used to extract frame-level image features from the input video sequence. It consists of two parts: data preprocessing and 2D feature extraction network. First, the input video sequence is preprocessed, and the processing results are input into the feature extraction network in the form of image tensors for frame-level image feature extraction. The features of the points to be tracked at the corresponding positions are extracted from the feature map sequence according to the coordinates of the points to be tracked.

[0084] The data preprocessing part needs to process the video data input to the network to make it consistent with the network input. Since the window size in the window self-attention module of the 1D Swin Transformer designed by the iterative update module is set to 3, and the network contains 4 levels, a Patch Merging operation is performed between every two levels, which halves the sequence length and doubles the number of feature dimensions. In order to still meet the requirement that the sequence length can be divided by the window size of 3 at the last level, the length of the video sequence input to the network must be a multiple of 24. For video sequences that do not meet the requirement, the last frame must be copied multiple times at the end of the video to make up for the multiple of 24 frames. The processed video frame size is adjusted to 256×256, the RGB pixel value is scaled to between -1 and 1, and the resulting image data tensor with a shape of (B, S, H, W, 3) is reshaped to (B*S, H, W, 3).

[0085] Input 2D ResNet network to complete frame-level image feature extraction and obtain feature extraction results And according to the coordinates of the point to be tracked, the corresponding features of the point to be tracked are extracted from the feature map of the corresponding frame using bilinear interpolation.

[0086] For the extracted features, in order to calculate the similarity features between the subsequent point pairs, the feature vector needs to be further normalized:

[0087]

[0088] Among them, i represents the video index; t represents the frame index; h represents the y value of the feature in the frame coordinate system; and w represents the x value of the feature in the frame coordinate system.

[0089] S3: Initialize the point trajectory. The coordinates of the point to be tracked and the corresponding point features are tiled in the time dimension as the initial value of the trajectory. That is, assuming that there is no movement, the initialized point trajectory will be used as the starting point for iterative updating of the network. For a point to be tracked p0 = (t, y, x), and the point feature vector representing the target appearance obtained by bilinear sampling of the target position (x, y) from the feature map of the frame t where the point to be tracked is located The coordinates of the points to be tracked and the feature vectors of the points are flattened in the time dimension, and the initial values ​​of the trajectory and features are obtained:

[0090]

[0091] The iterative update network is responsible for gradually updating and correcting the initialized point trajectory. According to the set number of iterations K, the point trajectory tracking results are gradually updated from the initial value. The result of the previous iterative update will be used as the initial value of the next iterative update. The iterative update of the point trajectory consists of two parts: the correlation layer and the time dimension information fusion network. The specific steps are as follows:

[0092] S4: Similarity feature calculation, calculate a feature similarity between the features of the point to be tracked at each moment and the features of all points in the feature graph at the corresponding moment. In the relevant layer, given a query feature of a point to be tracked in a video at a certain moment And the characteristic graph at the corresponding time The dot product between all point pairs is taken to form the relevant cost volume C. For the case where the number of input videos is B, the length of the video sequence is S, and the number of points to be tracked is N, the relevant cost volume C can be effectively calculated as a single matrix multiplication:

[0093]

[0094] in Table B The frame feature tensor of the input video in the complete video sequence length S, represents the point feature tensor corresponding to each of the N query points in the B input videos on the complete video sequence length S, and the related cost volume C(g θ (I),Q) is the above frame feature tensor g θ(I) and the dot product of the feature tensor Q of the point to be tracked on the feature dimension D, i, t, n, h, w represent the video index, frame index, query point index, query point x coordinate value, query point y coordinate value, d represents the index on the specifically determined one-dimensional point feature vector, Q i,t,n,d That is, the value of the one-dimensional feature vector of the n-th query point on the t-th frame in the i-th video at index d, g θ (I i,t,h,w,d ) is the value of the one-dimensional feature vector of the pixel with coordinates (w, h) on the t-th frame in the i-th video at index d, C i,t,n,h,w That is, it is the dot product of the nth query point on the tth frame in the i-th video and the pixel with coordinates (w, h) on frame t.

[0095] In order to make the relevant cost volume give the information of displacement from large to small, the last two dimensions of the relevant cost volume are pooled through the pooling operation with kernel sizes of 1, 2, 4, and 8 and a constant step size of 2 to construct a 4-layer correlation pyramid {C 1 ,C 2 ,C 3 ,C 4}, where the cost body C k With dimensions B×S×N×H / 2 k ×W / 2 k .

[0096] S5: Time dimension information fusion, the time dimension information fusion network splices the point features extracted by the data processing module and the similarity features calculated by the relevant layer as input features, and outputs the updated value for the current point trajectory. It is based on a 1DSwin Transformer with 4 stages adapted to 1D time series, which is modified from a 2D Swin Transformer, to perform information fusion of video sequences in the time dimension.

[0097] The tiled point feature sequence obtained in step S3 Similarity features obtained in step S4 Where L = 4 represents the number of layers of the similarity feature pyramid, and P = 7 represents the length of the sampling block. The features obtained in these two steps are concatenated in the last dimension, and the resulting feature sequence is further reshaped as Make it meet the requirements of the 1D Swin Transformer network for input feature tensors.

[0098] The time dimension information fusion network will perform multi-head self-attention and MLP operations in different time windows according to the input features, and continuously increase the feature dimension while downsampling the sequence length according to the Patch Merging operation between the two levels, thereby forming a time dimension feature pyramid in the four stages.

[0099] S6: Update of point trajectory and point features. According to the point coordinate update values ​​and point feature update values ​​outputted from the prediction heads at the four levels in step S5 and the corresponding updated confidences, a max operation is performed on the outputs of the four levels according to the confidences to obtain the final point position coordinate update values ​​and point feature update values. The update operation is performed according to formula (5) and formula (6) respectively. The updated point trajectory and point features are used as the initial values ​​for the next iteration.

[0100] X k+1 =X k +ΔX (5)

[0101] F k+1 =F k +ΔF (6)

[0102] S7: Output of the final point trajectory. After repeating the above steps S4 to S7 to a preset number of iterations K, the point trajectory output by the Kth update is the point tracking result output by this method.

[0103] The embodiment of the present invention builds a pyramid of time dimension by utilizing the characteristics of the designed time dimension information fusion network having time window receptive fields of different sizes at different levels of the network, and generates an iterative update network with multiple levels and multiple prediction heads. Small time windows at low levels are used to perceive motion mutations, and large time windows at high levels are used to smooth motion and reduce jitter. Any modification, fine-tuning, combination, simplification and replacement of the network structure made by any technician familiar with the technical field within the technical scope disclosed by the present invention without departing from the spirit and principle of the present invention are included in the protection scope of the present invention.

[0104] Multiple elements, ingredients, parts or steps can be provided by a single integrated element, ingredient, part or step. Alternatively, a single integrated element, ingredient, part or step can be divided into separate multiple elements, ingredients, parts or steps. The disclosure "one" or "an" used to describe an element, ingredient, part or step is not intended to exclude other elements, ingredients, parts or steps.

[0105] It should be understood that the above description is for illustration and not for limitation. Many embodiments and many applications beyond the examples provided will be apparent to those skilled in the art upon reading the above description. Therefore, the scope of the present teachings should not be determined with reference to the above description, but should be determined with reference to the appended claims and the full scope of equivalents possessed by such claims. For the purpose of comprehensiveness, all articles and references, including disclosures of patent applications and publications, are incorporated herein by reference. The omission of any aspect of the subject matter disclosed herein in the preceding claims is not intended to be a waiver of the subject matter, nor should it be considered that the inventors did not consider the subject matter to be part of the disclosed inventive subject matter.

Claims

1. A point tracking method in a large displacement scene based on a time pyramid, characterized in that: include: The selection of points to be tracked is used to collect the coordinates of target tracking points in the input video sequence specified by the user; Image feature extraction, which is used to extract frame-level image features from the input video sequence and extract the features of the points to be tracked at the corresponding positions from the feature map sequence according to the coordinates of the target tracking points; Initialize point trajectory, flatten the target tracking point coordinates and corresponding point features in the time dimension as the initial value of the trajectory; Similarity feature calculation, calculating the feature similarity between the features of the point to be tracked at each moment and the features of all points in the feature graph at the corresponding moment; Time dimension information fusion, the time dimension information fusion network concatenates the extracted point features and the similarity features calculated by the relevant layer as input features, and outputs the coordinate update value and point feature update value for the current point trajectory; Point trajectory and point feature update: according to the time dimension information, the point trajectory coordinate update value and the point feature update value are outputted in the prediction heads at multiple different levels respectively, and the corresponding updated confidences are performed on the outputs of multiple levels according to the confidences to obtain the final point trajectory coordinate update value and point feature update value; The final point trajectory is output. After repeating the above similarity feature calculation steps ~ point trajectory and point feature update steps to a preset number of iterations K, the point trajectory output by the Kth update is the point tracking result.

2. The point tracking method in a large displacement scene based on a time pyramid as claimed in claim 1, characterized in that: The step of selecting points to be tracked is performed as follows: For the input video sequence selected by the user, any number of arbitrary pixel points on any frame are selected as the points to be tracked, where the coordinate representation of each point is (t, y, x), where t represents the frame number of the pixel point to be tracked, and y and x represent the ordinate and abscissa of the point in the pixel coordinate system respectively; the coordinates of the N pixel points selected from the B input videos are integrated into a tensor form of (B, N, 3).

3. The point tracking method in a large displacement scene based on a time pyramid as claimed in claim 2, characterized in that: The step of selecting points to be tracked is performed as follows: The length of the video sequence input to the network is a multiple of 24. For video sequences that do not meet the requirement, the last frame is copied multiple times at the end of the video to make up for the multiple of 24 frames, and the processed video frame size is adjusted to 256×256, the RGB pixel value is scaled to between -1 and 1, and the resulting image data tensor with a shape of (B, S, H, W, 3) is reshaped to (B*S, H, W, 3); the number of input videos is B, the length of the video sequence is S, the width of the video image is W, and the height of the video image is H.

4. The point tracking method in a large displacement scene based on a time pyramid as claimed in claim 1, characterized in that: The image feature extraction step is performed as follows: Input 2D ResNet network to complete frame-level image feature extraction and obtain feature extraction results And according to the coordinates of the point to be tracked, the corresponding features of the point to be tracked are extracted from the feature map of the corresponding frame by using bilinear interpolation; For the extracted features, in order to calculate the similarity features between the subsequent point pairs, the following formula is used to normalize the feature vector in the feature dimension: Among them, i, t, h, and w represent the video index, frame index, y value, and x value of the feature in the frame coordinate system respectively; d represents the length of the feature at a specific position uniquely determined by i, t, h, and w; and S represents the length of the video sequence.

5. The point tracking method in a large displacement scene based on a time pyramid as claimed in claim 1, characterized in that: The point trajectory initialization step is performed as follows: For a point to be tracked p0 = (t, y, x), the frame number t and the position coordinates (x, y) are decomposed, and the point feature vector representing the target appearance is obtained by bilinear sampling of the target position (x, y) on the feature map of the frame t where the tracking point is located. The coordinates of the points to be tracked and the feature vectors of the points are flattened in the time dimension, and the initial values ​​of the trajectory and features are obtained: {(p t ,f t )}={(p0,f0)},t∈{1,2,…,S}; Where t represents the frame index of the point to be tracked, x and y represent the horizontal and vertical coordinates of the point in the frame coordinate system, respectively. t represents the position information of the point to be tracked in the tth frame, f t Represents the feature vector representation of the point to be tracked in the tth frame.

6. The point tracking method in a large displacement scene based on a time pyramid as claimed in claim 1, characterized in that: The similarity feature calculation step is performed as follows: In the relevant layer, given a query feature of a point to be tracked in a video at a certain moment And the characteristic graph at the corresponding time The dot product between all point pairs is taken to form the relevant cost volume C; for the case where the number of input videos is B, the length of the video sequence is S, and the number of points to be tracked is N, the relevant cost volume C is calculated as a single matrix multiplication: C i,t,n,h,w =∑ d Q i,t,n,d· g θ (I i,t,h,w,d ); in Table B The frame feature tensor of the input video in the complete video sequence length S, represents the point feature tensor corresponding to each of the N query points in the B input videos on the complete video sequence length S, and the related cost volume C(g θ (I),Q) is the above frame feature tensor g θ (I) and the dot product of the feature tensor Q of the point to be tracked on the feature dimension D, i, t, n, h, w represent the video index, frame index, query point index, query point x coordinate value, query point y coordinate value, d represents the index on the specifically determined one-dimensional point feature vector, Q i,t,n,d That is, the value of the one-dimensional feature vector of the n-th query point on the t-th frame in the i-th video at index d, g θ (I i,t,h,w,d ) is the value of the one-dimensional feature vector of the pixel with coordinates (w, h) on the t-th frame in the i-th video at index d, C i,t,n,h,w That is, it is the dot product of the nth query point on the tth frame in the i-th video and the pixel with coordinates (w, h) on frame t.

7. The point tracking method in a large displacement scene based on a time pyramid as claimed in claim 6, characterized in that: The similarity feature calculation step also includes: Through the pooling operation with kernel sizes of 1, 2, 4, and 8 and a constant step size of 2, the last two dimensions of the relevant cost volume C are pooled to construct a 4-layer relevance pyramid {C 1 ,C 2 ,C 3 ,C 4 }, where the cost body C k With dimensions B×S×N×H / 2 k ×W / 2 k .

8. The point tracking method in a large displacement scene based on a time pyramid as claimed in claim 1, characterized in that: The time dimension information fusion step is performed as follows: The flattened point feature sequence obtained by the point trajectory initialization step Similarity features obtained in the similarity feature calculation step Where L = 4 represents the number of layers of the similarity feature pyramid, and P = 7 represents the length of the sampling block; The features obtained in the two steps are concatenated in the last dimension, and the resulting feature sequence is reshaped as Make it meet the requirements of the 1D Swin Transformer network for input feature tensors.

9. The point tracking method in a large displacement scene based on a time pyramid as claimed in claim 8, characterized in that: The time dimension information fusion step is performed as follows: According to the input features, multi-head self-attention and MLP operations are performed in different time windows, and the feature dimension is continuously increased while downsampling the sequence length according to the Patch Merging operation between the two levels, thus forming a feature pyramid of time dimension in 4 stages.

10. The point tracking method in a large displacement scene based on a time pyramid as claimed in claim 1, characterized in that: The point trajectory and point feature update steps are performed as follows: The update operation is performed according to the following formula, and the updated point trajectory and point features are used as the initial values ​​of the next iteration: X k+1 =X k +ΔX; F k+1 =F k +ΔF; Where X k represents the coordinate information of all points to be tracked in frame k, F k represents the feature vector information of all points to be tracked in frame k, ΔX represents the update amount of the position information output by the network, and ΔF represents the update amount of the feature vector output by the network.

Images