Tracking Query Points in Videos Using Neural Networks

Abstract

A method, system, and apparatus, including a computer program encoded on a computer storage medium, for tracking query points in a video using a point tracking neural network.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application No. 63 / 317,535, filed March 7, 2022. The disclosure of the prior application is considered part of, and is incorporated by reference into, the disclosure of this application.

[0002] This specification relates to using neural networks to process inputs including video frames. [Background technology]

[0003] A neural network is a machine learning model that employs one or more layers of nonlinear units to predict an output with respect to a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the input of the current values ​​of the respective set of parameters. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Adam W Harley, Zhaoyuan Fang, and Katerina Fragkiadaki, Particle video revisited: Tracking through occlusions using point trajectories, Proceedings of European Conference on Computer Vision (ECCV), pp. 59–75, Springer, 2022 [Non-Patent Document 2] Teed, Z., Deng, J., Raft: Recurrent all-pairs field transforms for optical flow, European conference on computer vision, pp. 402-419, Springer (2020) Summary of the Invention [Means for solving the problem]

[0005] This specification describes a system implemented as a computer program on one or more computers at one or more locations that processes input including: (i) a video sequence including a plurality of video frames; and (ii) a set of one or more query points.

[0006] Each query point is a point in a corresponding one of the video frames, i.e. a point that specifies a respective spatial location, i.e. a respective pixel, in a corresponding one of the video frames.

[0007] The system processes a set of one or more query points and a video sequence, i.e., intensity values ​​of pixels of video frames in the video sequence, using a point tracking neural network to generate a network output that includes, for each query point, a predicted spatial location of each of the query points in other video frames in the sequence, i.e., video frames other than the query point's corresponding video frame.

[0008] In one aspect, a method includes obtaining a video sequence including a plurality of video frames; obtaining a set of one or more query points, each query point specifying a respective spatial location in a corresponding video frame of the plurality of video frames; and processing the set of one or more query points and the video sequence using a point tracking neural network to generate a network output including, for each query point, a respective predicted spatial location of the query point in each of the video frames in the sequence other than the corresponding video frame of the query point.

[0009] In some implementations, processing a set of one or more query points and the video sequence using a point tracking neural network to generate a network output including, for each query point, a predicted spatial location of the query point in each of the video frames in the sequence other than the corresponding video frame of the query point includes processing video frames in the video sequence using a visual backbone neural network to generate a feature grid including respective visual features for each of a plurality of spatial locations in each of the video frames; generating, for each query point, extracted features of the query point from the spatial location of the query point in the corresponding video frame and the respective visual features for one or more of the plurality of spatial locations in the corresponding video frame; and generating, from the feature grid and the extracted features of the query point, a predicted spatial location of the query point in each of the video frames other than the corresponding video frame.

[0010] In some implementations, generating an extracted feature of the query point from the spatial position of the query point in the corresponding video frame and respective visual features for each of a plurality of spatial locations in the corresponding video frame includes performing an interpolation of visual features of a set of spatial locations within a local neighborhood of each spatial position of the query point in the corresponding video frame.

[0011] In some implementations, generating a predicted spatial location of each of the query points in each of the video frames other than the corresponding video frame from the feature grid and the extracted features of the query points includes generating a cost volume from the feature grid and the extracted features of the query points, and processing the cost volume using a decoder neural network to generate a predicted location of each of the query points in the video frame for each of the video frames other than the corresponding video frame.

[0012] In some implementations, processing the cost volume using a decoder neural network includes processing the cost volume using a decoder neural network to generate, for each of a plurality of video frames other than the corresponding video frame, a respective score for each spatial location in the video frame, and generating, for each of the plurality of video frames other than the corresponding video frame, a predicted position from the respective scores for the spatial locations in the video frame.

[0013] In some implementations, generating a predicted location from the respective scores includes identifying the spatial location with the highest score, identifying each spatial location that is within a fixed-sized window of the spatial locations with the highest score, and determining the predicted location by calculating a weighted sum of the spatial locations within the fixed-sized window of the spatial locations with the highest score, where the weight of each spatial location is calculated based on the score for the spatial location and the sum of the scores for the spatial locations within the fixed-sized window.

[0014] In some implementations, the network output further includes, for each query point, a respective occlusion score for the query point for each of the video frames other than the corresponding video frame, each occlusion score representing the likelihood that the query point is occluded in the video frame.

[0015] In some implementations, processing the cost volume using a decoder neural network to generate a predicted location for each of the query points comprises: The method includes processing the cost volume using a decoder neural network to generate, for each video frame other than the corresponding video frame, a predicted position of each of the query points in the video frame and an occlusion score for each of the query points in the video frame.

[0016] In another aspect, a method for training a point tracking neural network includes obtaining one or more batches of training video sequences; (i) determining ground truths for query points in each of the video frames in the training video sequences; the first loss term that measures an error between the ground truth spatial location of the query point and the corresponding predicted spatial location of the query point for each training video sequence, and (ii) a ground truth occlusion score for each of the video frames in the training video sequence; processing the one or more sets of query points for the training video sequences and the training video sequence using a point tracking neural network to generate, for each training video sequence, a respective predicted spatial location of the query point in each of the video frames in the training video sequence, and a respective occlusion score for the query point for each of the video frames in the training video sequence; and computing a gradient with respect to parameters of the point tracking neural network of a loss function, wherein the loss function includes (i) a first loss term that measures an error between the ground truth spatial location of the query point and a corresponding predicted spatial location of the query point for each training video sequence, and (ii) a second loss term that measures an error between the ground truth occlusion score of the query point and the corresponding predicted occlusion score of the query point for each training video sequence.

[0017] In some implementations, the first loss term is the Huber loss.

[0018] In some implementations, the second loss term is a cross-entropy loss.

[0019] In some implementations, the loss function also includes (iii) a contrastive learning loss term.

[0020] In some implementations, the method further includes steps of generating, for each of one or more of the training video sequences, ground truth spatial locations of the query points in each of the video frames in the training video sequence and respective ground truth occlusion scores of the query points for each of the video frames, the steps including obtaining another training video sequence that is a variation of the training video sequence, processing the one or more sets of query points and the another training video sequence using a point tracking neural network to generate, for each query point, predicted spatial locations of the query points in each of the video frames in the another training video sequence and respective occlusion scores of the query points for each of the video frames in the another training video sequence, and generating, from data generated by the point tracking neural network while processing the one or more sets of query points and the another training video sequence using the point tracking neural network, ground truth spatial locations of the query points in each of the video frames in the training video sequence and respective ground truth occlusion scores of the query points for each of the video frames.

[0021] In some implementations, generating ground truth spatial locations of the query points in each of the video frames in the training video sequence and ground truth occlusion scores for each of the query points for each of the video frames from data generated by the point tracking neural network while processing the one or more sets of query points and the other training video sequence using the point tracking neural network includes processing the data generated by the point tracking neural network while processing the one or more sets of query points and the other training video sequence using the point tracking neural network using a predictor neural network.

[0022] In some implementations, the predictor neural network is a multi-layer perceptron (MLP).

[0023] In some implementations, the training video sequence and the another training video sequence are generated from the same original training video sequence by applying different data augmentations to the original training video sequence.

[0024] In some implementations, data generated by the point tracking neural network while processing the set of one or more query points and the separate training video sequence using the point tracking neural network includes (i) a predicted spatial location of each of the query points in the video frames in the separate training video sequence, and (ii) an occlusion score for each of the query points for each of the video frames in the separate training video sequence.

[0025] In some implementations, the method further includes training a point tracking neural network using the calculated gradients.

[0026] The subject matter described herein can be implemented in particular embodiments to realize one or more of the following advantages.

[0027] This specification describes a neural network that performs tracking of arbitrary query points in a video. It processes low-level features (such as pixel attributes) in frames of a video. This can provide information, for example, about how object surfaces of objects depicted in the video deform and move during the video. Such information can be useful for various downstream tasks, for example, video understanding or robotics tasks. Furthermore, the neural network can predict when a point is occluded, providing finer-grained occlusion tracking.

[0028] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief description of the drawings]

[0029] [Figure 1] FIG. 1 is a diagram of an example video processing system. [Diagram 2] 1 is a flow diagram of an example process for tracking query points through a video. [Diagram 3] FIG. 13 illustrates an example of generating a cost volume for a query point. [Figure 4] FIG. 1 illustrates an example of predicting position and occlusion from cost volumes for a given video frame. [Diagram 5] 1 is a flow diagram of an exemplary process for training a point tracking neural network. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0030] Like reference numbers and designations in the various drawings indicate like elements.

[0031] 1 is a diagram of an example video processing system 100. Video processing system 100 is an example of a system in which the systems, components, and techniques described below may be implemented as a computer program on one or more computers at one or more locations.

[0032] The system 100 receives input including: (i) a video sequence 102 including a number of video frames; and (ii) a set of one or more query points 104 .

[0033] Each query point 104 is a point in a corresponding one of the video frames, i.e., a point that specifies a respective spatial location, i.e., a respective pixel, in a corresponding one of the video frames. Thus, each query point 104 can be represented as a point (x, y, t), where x, y are the spatial coordinates of the query point 104 and t is the index of the corresponding video frame in the video sequence 102.

[0034] The system 100 processes a set of one or more query points 104 and the video sequence 102, i.e., intensity values ​​of pixels of video frames in the video sequence 102, using a point tracking neural network 110 to generate a network output 112 that includes, for each query point 104, a predicted spatial location 114 of each of the query points 104 in other video frames in the sequence 102, i.e., video frames other than the corresponding video frame t of the query point 104. Due to the architecture of the neural network 110 or due to the design of the processing pipeline for generating the output using the neural network 110, the network output 112 may also include a predicted spatial location for the corresponding video frame. However, in some of these cases, the system 100 may ignore the predicted spatial location for the corresponding video frame (since the actual location in the corresponding video frame is provided as an input to the system).

[0035] That is, given a query point 104 in one of the video frames, the point tracking neural network 110 can generate a prediction of the spatial location of the query point 104 in other video frames in the video sequence.

[0036] The predicted location of a given query point in a given other video frame is a prediction of the location of the portion of the scene that was depicted at the given query point in the corresponding video frame. For example, if at a given query point, the corresponding video frame depicts a particular point on the surface of an object in the scene, then the predicted location of the given query point identifies the predicted location of the same particular point on the surface of the object in the given other video frame. More generally, if the video is a real-world video, then each query point corresponds to a physical point in the world that was depicted at the query point in the corresponding video, and the predicted location of a given query point in a given video frame is a prediction of the location of the physical point in the world in the given video frame, i.e., a prediction of which pixels depict the physical point in the world in the given video frame.

[0037] In some implementations, the point tracking neural network 110 also generates, for each query point 104, a respective occlusion score 116 for the query point 104 with respect to each of the other video frames in the sequence 102. That is, the network output 112 includes both the predicted position 114 and the occlusion score 116.

[0038] The occlusion score 116 of a given query point in a given video frame represents the likelihood that the query point is occluded, e.g., not visible, in the given video frame, i.e., that the portion of the scene depicted at the query point in the corresponding video frame is occluded (e.g., by another object in the scene) in the given video frame. For example, if the video is a real-world video, the occlusion score of a given query point in a given video frame is the predicted likelihood that the corresponding physical point in the world is occluded (e.g., by another object in the real world) in the given video frame.

[0039] Due to the architecture of the neural network 110 or due to the design of the processing pipeline, the network output 112 may also include a predicted occlusion score 116 for the corresponding video frame, however, in some such cases, the system 100 may ignore the occlusion score for the corresponding video frame.

[0040] The predicted position 114 (and optionally the occlusion score 116) may be used for any of a variety of purposes.

[0041] For example, the predicted position 114 can be used to generate a reward signal for training a robot or other agent through reinforcement learning; for example, if a task being performed by the agent requires moving a point in a scene from one location to another, the distance between the predicted position of the point in the last video frame of the sequence and the target location may be used to generate a reward.

[0042] As another example, the predicted position 114 (and optionally the occlusion score 116) may be provided as an additional input to a policy neural network for controlling a robot or other agent interacting with an environment. In this example, the query point may be a point of interest in the last frame of a video sequence, and predictions for earlier frames in the sequence may be provided as inputs to the policy neural network, for example to provide a signal regarding the recent movement of the agent or other object in the environment.

[0043] As another example, the predicted position 114 (and optionally the occlusion score 116) may be provided as an input to a video understanding neural network, along with the video sequence 102, to provide the video understanding neural network, e.g., an action classification neural network or a topic classification neural network, with additional information about motion in the scene.

[0044] As another example, the predicted position 114 (and optionally the occlusion score 116) may be used for imitation learning, ie, to enable imitation of motion rather than appearance.

[0045] 2 is a flow diagram of an exemplary process 200 for tracking query points through a video sequence. For convenience, process 200 is described as being performed by one or more computer systems located at one or more locations. For example, a video processing system suitably programmed in accordance with this specification, such as the video processing system 100 depicted in FIG. 1, may perform process 200.

[0046] The system obtains a video sequence including multiple video frames (step 202). For example, the video sequence may depict a real-world or synthetic scene over a time window, with each video frame depicting the scene at a corresponding point in the time window. Each video frame may include multiple pixels, each pixel having one or more intensity values, e.g., RGB color values, grayscale values, or one or more intensity values ​​of different color representation schemes. Each pixel may be associated with a corresponding spatial location within the video frame. Each pixel may represent a corresponding portion or point of a scene (e.g., a real-world or synthetic scene) depicted by the video frame at a corresponding time in the video frame.

[0047] The system obtains a set of one or more query points, step 204. As described above, each query point specifies a respective spatial location in a corresponding one of the multiple video frames.

[0048] The system processes one or more sets of query points and the video sequence using a point tracking neural network to generate a network output (step 206).

[0049] The network output includes, for each query point, a respective predicted spatial location of the query point in each of the video frames in the sequence other than the query point's corresponding video frame.

[0050] Optionally, the network output may also include, for each query point, a respective occlusion score for the query point for each of the video frames other than the corresponding video frame, each occlusion score representing the likelihood that the query point is occluded in the video frame.

[0051] The point tracking neural network may generally have any suitable architecture that enables the point tracking neural network to map video sequences and query points to a network output related to the query points.

[0052] As a particular example, the point tracking neural network may include a visual backbone neural network. In general, a visual backbone neural network is a feature extraction neural network, i.e., a neural network configured to extract features from an input (e.g., a video frame).

[0053] In this example, the system can process video frames in a video sequence using a visual backbone neural network to generate a feature grid that includes respective visual features for each of a plurality of spatial locations in each of the video frames.

[0054] Then, for each query point, the system can generate extracted features for the query point from the spatial location of the query point in the corresponding video frame and respective visual features for one or more of the multiple spatial locations in the corresponding video frame, and then generate respective predicted spatial locations of the query point in each of the video frames other than the corresponding video frame and, optionally, an occlusion score from the feature grid and the extracted features of the query point.

[0055] As an example, the point tracking neural network may also include a decoder neural network.

[0056] In this example, the system can generate a cost volume from the feature grid and the extracted features of the query points, and then process the cost volume using a decoder neural network to generate predicted positions of each of the query points in the video frames, and optionally, occlusion scores, for each of the video frames other than the corresponding video frame.

[0057] An example of generating a cost volume is described below with reference to FIG.

[0058] An example of processing the cost volume using a decoder neural network to generate predicted positions of each of the query points in a given video frame and an occlusion score for the given video frame is described below with reference to FIG. 4.

[0059] FIG. 3 illustrates an example 300 of generating a cost volume for a query point.

[0060] In particular, in the example of FIG. 3, the system processes a sequence of w×h video frames and, for video frame t q Query points (i q , j q , t q )

[0061] As described above, the system processes video frames in a video sequence using a visual backbone neural network to generate a feature grid that includes a respective visual feature, i.e., a feature vector, for each of a plurality of spatial locations in each of the video frames. In general, each of the spatial locations corresponds to a different region of the video frame. For example, the feature grid can be a w / 8×h / 8 grid of d-dimensional visual features, where each visual feature corresponds to an 8×8 grid of pixels from a corresponding video frame.

[0062] The visual backbone neural network can have any suitable architecture that enables the neural network to map a video sequence to a feature grid. In the example of FIG. 3, the visual backbone neural network is a 3D convolutional neural network (ConvNet), such as TSM-ResNet-18 or other suitable convolutional neural network. In other examples, the visual backbone neural network can be a different type of neural network, such as a vision transformer neural network.

[0063] Then, for each query point, the system can generate extracted features for the query point from the spatial location of the query point in the corresponding video frame and respective visual features for one or more of the multiple spatial locations in the corresponding video frame.

[0064] As shown in FIG. 3, the system q The spatial location of each query point in (i q , j q ), an extracted feature (a “query feature”) can be generated by performing an interpolation, e.g., bilinear interpolation, of the visual features of a set of spatial locations within a local neighborhood of

[0065] The system then generates a cost volume from the feature grid and the extracted features of the query points.

[0066] In some implementations, to generate a cost volume from the feature grid and extracted features of the query points, the system may divide the visual features and extracted features in the feature grid into n heads, i.e., such that the extracted features include respective d / n dimensional features for each of the n heads, and the feature grid includes respective d / n dimensional visual features for each head.

[0067] The system can then generate a respective initial cost volume for each head and concatenate the initial cost volumes to generate a final cost volume. By using multiple heads, i.e., by computing a multi-headed cost volume, the system can generate substantially richer features for further processing, i.e., for use in predicting position and occlusion scores.

[0068] The initial cost volume for a given head has a respective cost value for each spatial location in each of the video frames, i.e., the initial cost volume for a given head contains an h' x w' x 1 grid of cost values ​​for each video frame in the sequence.

[0069] To calculate a cost value for a given spatial location in a given video frame, the system calculates the dot product between the extracted features of the head and the visual features of the head for the given spatial location in the given video frame.

[0070] In some other implementations, the system directly calculates the final cost volume by calculating dot products between the extracted features and the visual features without dividing the extracted features and the visual features into heads. That is, the system uses n=1 heads when performing the above operations. In these implementations, the system directly calculates a final cost volume having a respective cost value for each spatial location in each of the video frames. That is, the final cost volume includes an h'xw'x1 grid of cost values ​​for each video frame in the sequence. To calculate the cost value for a given spatial location in a given video frame, the system calculates the dot product between the extracted features and the visual features for the given spatial location in the given video frame.

[0071] As described above, the system can then process the cost volume using a decoder neural network to generate predicted positions of each of the query points in the video frames, and optionally, occlusion scores, for each of the video frames other than the corresponding video frame.

[0072] FIG. 4 shows an example 400 of predicting location and occlusion from cost volumes for a given video frame.

[0073] Generally, to generate the predicted location, the system can process the cost volume using a decoder neural network to generate a respective score for each spatial location in the video frame for each of a number of video frames other than the corresponding video frame.

[0074] For each of the multiple video frames other than the corresponding video frame, the system can then generate a predicted position from the respective scores for the spatial locations in the video frame.

[0075] When the system also predicts occlusions, as in the example of FIG. 4, the system can process the cost volume using a decoder neural network to generate, for each video frame other than the corresponding video frame, a predicted position of each of the query points in the video frame and an occlusion score for each of the query points in the video frame.

[0076] In the example of Figure 4, the system performs this process independently for each of the video frames. That is, for a given video frame, the system uses the decoder neural network to process a portion of the cost volume of h'xw'xn for the given video frame to generate a predicted location of the query point in the given video frame and an occlusion score for the query point in the video frame. When the system does not use a "head" when generating the cost volume, n is equal to 1 and the portion of the cost volume is a portion of h'xw'x1.

[0077] In particular, in the example of FIG. 4, the decoder neural network includes a set of shared layers and respective branches for position inference and occlusion inference.

[0078] The decoder neural network processes a portion of the cost volume for a given video frame using a shared layer, for example a convolutional layer followed by a rectified linear unit (ReLU) activation function layer, to generate a shared output.

[0079] For the occlusion inference branch, the decoder neural network processes the shared output using layers of the occlusion inference branch to generate a single occlusion score ("logit") for a given video frame. For example, in the example of FIG. 4, the occlusion inference branch includes a first set of layers that convolve the shared output into a vector, e.g., using spatial average pooling, followed by a second set of layers that regress occlusion logits for the given video frame from the single vector. For example, the second set of layers may include a linear layer, a Leaky ReLU, and another linear layer that produces a single logit. The Leaky ReLU may be a ReLU activation function with a slope for negative values ​​(e.g., a relatively small slope, such as a slope of 0.01) (e.g., as opposed to a flat slope for negative values).

[0080] For the location inference branch, the system can apply a set of layers, for example a convolutional (Conv) layer with a single output followed by a spatial softmax, to generate a respective score for each spatial location in the video frame.

[0081] The system can then compute a "soft argmax" that calculates a position from each score.

[0082] To calculate the soft argmax, the system can identify the highest scoring spatial location, i.e., the "argmax" location, according to the respective scores for the spatial locations. The system can then determine a predicted location by identifying each spatial location that is within a fixed-size window B of the highest scoring spatial location, i.e., the argmax location, and then calculating a weighted average.

[0083] That is, the system determines the predicted location by computing a weighted sum of spatial locations within a fixed-size window of argmax spatial locations, with the weight of each spatial location being calculated, for example, based on the ratio of the score for the spatial location to the sum of the scores for the spatial locations within the fixed-size window, for example, equal to or directly proportional to such ratio.

[0084] Optionally, once the system has calculated the predicted position and occlusion estimates independently for each video frame for a given query point as described above, the system may iteratively refine the initial trajectory of the predicted position and occlusion estimates in the video frames using an iterative refinement technique to generate a final predicted position and occlusion estimate. For example, given an initial setting, the system may apply an iterative refinement technique that searches a local neighborhood and smooths the track, i.e., trajectory, of the position and occlusion estimates over time. For example, the system may use the predicted position and occlusion estimates described above as an initial setting for a Persistent Independent Particle (PIP) search. PIP is described in more detail in Adam W Harley, Zhaoyuan Fang, and Katerina Fragkiadaki, Particle video revisited: Tracking through occlusions using point trajectories, Proceedings of European Conference on Computer Vision (ECCV), pages 59-75, Springer, 2022.

[0085] 5 is a flow diagram of an exemplary process 500 for training a point tracking neural network. For convenience, the process 500 is described as being performed by one or more computer systems located at one or more locations. For example, a video processing system suitably programmed in accordance with this specification, such as the video processing system 100 depicted in FIG. 1, may perform the process 500.

[0086] The system can repeatedly run iterations of process 500 on different batches of training videos to update the parameters of the point tracking neural network.

[0087] That is, in each iteration of process 500, the system obtains a batch of training videos, e.g., by sampling a batch from a larger set of training data, and uses one or more batches of training videos to update the parameters of the neural network.

[0088] The system may continue to perform iterations of process 500 until a termination criterion for training the neural network is met, e.g., until the parameters converge, until a threshold amount of wall clock time has elapsed, or until a threshold number of iterations of process 500 have been performed.

[0089] The system obtains one or more batches of training video sequences (step 502).

[0090] The system obtains (i) a ground truth spatial location of the query point in each of the video frames in the training video sequences, and (ii) a respective ground truth occlusion score of the query point for each of the video frames in the training video sequences, for each query point in each training video sequence and in the set of query points for the training video sequences (step 504).

[0091] For each training video sequence, the system processes the training video sequence and a set of one or more query points using a point tracking neural network to generate, for each query point, a predicted spatial location of each of the query points in each of the video frames in the training video sequence, and a respective occlusion score for each of the query points for each of the video frames in the training video sequence (step 506), e.g., as described above with reference to Figures 1-4.

[0092] The system calculates the gradient of the loss function with respect to the parameters of the point tracking neural network, for example by backpropagation (step 508).

[0093] In general, the loss function includes a set of supervising learning terms that utilize ground truth spatial locations and ground truth occlusion scores.

[0094] In particular, the supervised learning terms include (i) a first loss term that measures, for each training video sequence, the error between the ground truth spatial location of the query point and the corresponding predicted spatial location of the query point, and (ii) a second loss term that measures, for each training video sequence, the error between the ground truth occlusion score of the query point and the corresponding predicted occlusion score of the query point.

[0095] For example, the first loss term can be based on the Huber loss and the second term can be a cross-entropy loss.

[0096] As a specific example, the loss function L for a given query point in a given video sequence is

[0097]

number

[0098] where t is the number of video frames in the video,

[0099]

number

[0100] is the ground truth occupancy of video frame t,

[0101]

number

[0102] is the ground truth location of the query point in video frame t,

[0103]

number

[0104] is the predicted occupancy of video frame t,

[0105]

number

[0106] is the predicted location of the query point in video frame t,

[0107]

number

[0108] teeth,

[0109]

number

[0110] and

[0111]

number

[0112] where λ is the Huber loss between the input and the weights, which can be given as inputs to the system or determined by a hyperparameter sweep.

[0113]

number

[0114] teeth,

[0115]

number

[0116] and

[0117]

number

[0118] The cross entropy loss between

[0119]

number

[0120] Meet the following.

[0121] In some implementations, the loss function also includes a set of auxiliary loss terms.

[0122] For example, the set of auxiliary loss terms may also include a contrastive loss term, e.g., the contrastive term may promote similarity between features that correspond to the same query point and dissimilarity between features that do not correspond to the same query point.

[0123] The system uses the computed gradients to train the neural network (step 510), i.e., the system uses the gradients to update the current values ​​of the network parameters.

[0124] In general, the system updates the current values ​​of the network parameters by applying an optimizer to the current values ​​using gradients to generate updated values ​​of the network parameters. The optimizer can be any suitable neural network training optimizer, e.g., Adam, Adafactor, rmsProp, etc. The optimizer may aim to adjust (optimize) the values ​​of the network parameters to minimize the loss from a loss function.

[0125] In some implementations, the system also uses self-supervised learning to train a point-tracking neural network on unlabeled training video sequences. In these implementations, for some iterations of process 500, rather than obtaining ground truth outputs from an external source, the system may use the neural network to generate ground truth occlusion and predicted point outputs (to provide a form of self-supervision). For example, the system may train using self-supervised learning for the initial few iterations of process 500 before training using supervised learning, may interleave self-supervised iterations among supervised iterations, or may repeatedly perform self-supervised iterations in parallel with supervised iterations and then update the current values ​​of the parameters using the average of the gradients calculated for the two iterations.

[0126] In particular, to generate a ground truth output for a given training video sequence, the system can obtain another training video sequence that is a variation of the training video sequence, e.g., the system can generate both training video sequences by applying different data augmentations to the original unlabeled training video sequence, e.g., such that both training sequences are different "views" of the same original training video sequence.

[0127] The system may then process the set of one or more query points and the another training video sequence using a point tracking neural network to generate, for each query point, a predicted spatial location of each of the query points in each of the video frames in the another training video sequence, and a respective occlusion score for the query point with respect to each of the video frames in the another training video sequence, e.g., as described above.

[0128] The system can then generate ground truth spatial locations of the query points in each of the video frames in the training video sequence and ground truth occlusion scores for each of the query points for each of the video frames from the data generated by the point tracking neural network while processing the set of one or more query points and another training video sequence using the point tracking neural network.

[0129] For example, the system may use a predictor neural network to process data generated by a point tracking neural network while processing one or more sets of query points and another training video sequence to predict ground truth spatial locations and ground truth occlusion scores. The predictor neural network may be, for example, a multi-layer perceptron (MLP) or other suitable neural network architecture.

[0130] During processing of one or more sets of query points and another training video sequence using the point tracking neural network, data generated by the point tracking neural network may include (i) predicted spatial locations of each of the query points in the video frames in the other training video sequence, and (ii) occlusion scores of each of the query points for each of the video frames in the other training video sequence. For example, the data may be a concatenation of (i) and (ii) into a single vector according to a predetermined order. Thus, the predictor neural network "projects" the predictions generated by the point tracking neural network to generate different projections that serve as ground truth for training.

[0131] The system can obtain the training video sequences and the datasets used to obtain the ground truth locations and occlusion scores in any of a variety of ways.

[0132] In some implementations, given a training video sequence, a user may manually label the locations of a given query point in all of the frames of the video.

[0133] In some other implementations, the system may generate the dataset using a hybrid approach where a user provides locations of query points in a subset of frames in a training video sequence and the system uses optical flow estimates to generate locations of query points in other video frames. An optical flow estimate between two frames in a video specifies for each pixel in a first frame the location of that pixel in the second frame. That is, optical flow is a pixel-level estimate of the motion of pixels in the first frame to the second frame. For example, the optical flow estimate may include an optical flow vector for each pixel in the first frame, which is an estimate of the pixel's motion between the first and second frames.

[0134] An example of using optical flow estimates to generate training examples that include a training video sequence and locations of initial points in some or all of the frames in the training video sequence follows.

[0135] The system acquires a video sequence that includes a sequence of video frames.

[0136] The system generates an optical flow estimate for a video sequence. For example, the system may process the video sequence using an optical flow prediction neural network to generate the estimate. As described above, the optical flow estimate may specify for each of a plurality of pairs of video frames in the video sequence a predicted location in a second frame of the pair for each pixel of a first frame of the pair. For example, the optical flow prediction neural network may be trained by unsupervised learning on a large dataset of training video sequences. An example of such a neural network is described in Teed, Z., Deng, J., Raft: Recurrent all-pairs field transforms for optical flow, European conference on computer vision, pp. 402-419, Springer (2020).

[0137] The system receives input specifying an initial point at an initial location in a first video frame of the sequence and a target location in a second, subsequent video frame that represents the spatial location of the initial point in the second, subsequent video frame.

[0138] For example, the system may obtain a first user input specifying an initial point at an initial location in a first video frame of a sequence, and then obtain a second user input specifying a target location in a second, subsequent video frame that represents the spatial location of the initial point in the second, subsequent video frame. That is, the user may provide input that annotates the video sequence to identify the initial and target locations.

[0139] As a particular example, before obtaining the first and second user inputs, the system may obtain a third user input that identifies a bounding box in the first video frame that depicts the object, and then receive a user input that selects one of the points in the bounding box as an initial point.

[0140] The system uses the optical flow estimates to generate respective target positions for the initial point in each video frame between the first and second video frames in the sequence.

[0141] In particular, the system can use the optical flow estimates to generate a predicted path from an initial position in a first video frame to a final position in a second video frame using an optical flow path, the predicted path including respective positions in each of the video frames between the first and second video frames in the sequence. The system can then use a point in the path as a target position.

[0142] For example, to generate a predicted path, the system can generate a path from an initial position in a first video frame to a final position in a second video frame that minimizes the mismatch with the optical flow estimate. For example, the mismatch between a given path and the optical flow estimate can be the squared mismatch, i.e., for each video frame between the first and second video frames in the sequence, the sum of (i) the delta between the position in the path in the video frame and the position in the path in the previous video frame, and (ii) the squared distance between the optical flow vector for the previous video frame.

[0143] In some implementations, to find a path that minimizes the mismatch, the system may identify a shortest path from a first node representing an initial position in a first video frame to a second node representing a final position in a second video frame in a graph that includes (i) a respective node for each of a plurality of pixels in a plurality of frames, including a first frame, a second frame, and each frame between the first frame and the second frame in the video sequence, and (ii) a respective edge between each particular pixel of the plurality of pixels in the frame and each particular pixel of the plurality of pixels in a subsequent frame, for the first frame and each frame between the first frame and the second frame in the video sequence, where each edge is weighted by a mismatch, e.g., a square of the mismatch, between a path from a particular pixel in a frame to a particular pixel in a subsequent frame and an optical flow estimate, (i) a delta between a particular pixel in a subsequent frame and a particular pixel in the frame, and (ii) a square of the distance between the optical flow vector for the particular pixel in the frame. For example, the system may efficiently identify this shortest path using a form of Dijkstra's algorithm.

[0144] Optionally, the user may be given the option to modify the target position generated by the system, for example to "break up" an insufficient path by adding an additional point that will become the new end point for another iteration of generation guided by the optical flow estimate.

[0145] [Table 1]

[0146] Table 1 shows the performance of the described technique (TAP-Net) compared to three baselines: a) COTR: a baseline based on correspondence transformer for image-to-image matching, b) a baseline based on an extension of RAFT, and c) a baseline based on VFS.

[0147] The performance is presented on four video datasets (Kinetics, Kubric, DAVIS, and RGB-Stacking) and three metrics. As can be seen in Table 1, the described technique outperforms all three baselines on all four datasets, both in predicting occlusions and, more generally, in accurately tracking points.

[0148] This specification uses the term "configured" in connection with systems and computer program components. One or more computer systems configured to perform a particular operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof that, during operation, causes the system to perform the operation or action. One or more computer programs configured to perform a particular operation or action means that one or more programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or action. With respect to a particular neural network, the neural network may be configured to perform a particular action by being trained to perform the particular action.

[0149] The embodiments and functional operations of the subject matter described herein can be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware, including the structures disclosed herein and their structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described herein can be implemented as one or more computer programs, e.g., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by or for controlling the operation of a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, generated to encode information for transmission to a receiver device suitable for execution by a data processing device.

[0150] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. The apparatus may also be or further include special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Optionally, in addition to hardware, the apparatus may include code that creates an execution environment for a computer program, such as code that constitutes a processor's firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0151] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may correspond to a file in a file system, but this need not be the case. A program can be stored as part of a file that holds other programs or data, for example one or more scripts stored in a markup language document, a single file dedicated to the program in question, or in multiple organized files, for example files that store one or more modules, subprograms, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one location or distributed at multiple locations and connected together by a data communication network.

[0152] As used herein, the term "database" is used broadly to refer to any collection of data, i.e., the data need not be structured in any particular way, or at all, and may be stored on storage devices in one or more locations. Thus, for example, an index database may contain multiple collections of data, each of which may be organized and accessed in a different way.

[0153] Similarly, the term "engine" is used broadly herein to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers are dedicated to a particular engine, and in other cases, multiple engines can be installed and running on the same computer or computers.

[0154] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by performing operations on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry, e.g., FPGAs or ASICs, or a combination of special purpose logic circuitry and one or more programmed computers.

[0155] A computer suitable for executing a computer program can be based on a general-purpose or dedicated microprocessor or both, or any other type of central processing unit. In general, the central processing unit receives instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for executing or executing instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory can be supplemented by or incorporated in special-purpose logic circuitry. In general, the computer also includes one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively coupled to receive data from or transfer data to the mass storage devices, or both. However, the computer may not have such devices. Additionally, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, such as a Universal Serial Bus (USB) flash drive, to name just a few.

[0156] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD ROM disks and DVD-ROM disks.

[0157] To provide interaction with a user, embodiments of the subject matter described herein can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, as well as a keyboard and a pointing device, e.g., a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user, e.g., the feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and the input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, the computer can interact with the user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to the web browser of the user's device in response to a request received from the web browser. The computer can also interact with the user by sending text messages or other forms of messages to a personal device, e.g., a smartphone, running a messaging application, and receiving a response message from the user in return.

[0158] A data processing apparatus for implementing machine learning models may also include dedicated hardware accelerator units, for example for handling the often computationally intensive parts of the machine learning training or creation, e.g., inference, workload.

[0159] The machine learning model may be implemented and deployed using a machine learning framework, for example, the TensorFlow framework or the Jax framework.

[0160] An embodiment of the subject matter described herein can be implemented in a computing system that includes a back-end component, e.g., as a data server, or includes a middleware component, e.g., an application server, or includes a front-end component, e.g., a client computer having a graphical user interface, web browser, or app through which a user can interact with an implementation of the subject matter described herein, or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), e.g., the Internet.

[0161] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server sends data, e.g., HTML pages, to a user device for the purpose of, e.g., displaying data to a user interacting with the device acting as a client and receiving user input from such a user. Data generated at the user device, e.g., results of user interaction, can be received from the device at the server.

[0162] Although this specification contains many specific implementation details, these should not be considered as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to certain embodiments of a particular invention. Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as working in a particular combination, and may even be initially claimed as such, one or more features from the claimed combination may in some cases be deleted from the combination, and the claimed combination may be directed to a subcombination, or a variation of the subcombination.

[0163] Similarly, although operations are depicted in the figures and claimed in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all of the operations shown be performed to achieve a desired result. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the division of various system modules and components in the above-described embodiments should not be understood as requiring such division in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products.

[0164] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0165] 100 Video Processing System 102 Video Sequences 104 query points 110 Point Tracking Neural Network 112 Network Output 114 Predicted spatial location, predicted location 116 Covert Score 200 processes Example of generating cost volume for 300 query points 400 Example of predicting location and occlusion from cost volumes for a given video frame 500 processes

Claims

1. A method implemented by one or more computers, comprising: obtaining a video sequence comprising a plurality of video frames; obtaining a set of one or more query points, each query point specifying a respective spatial location in a corresponding one of the plurality of video frames; processing the set of one or more query points and the video sequence using a point tracking neural network to generate a network output including, for each query point, a predicted spatial location of each of the query points in each of the video frames in the video sequence other than the corresponding video frame of the query point; A method comprising:

2. processing the set of one or more query points and the video sequence using a point tracking neural network to generate a network output including, for each query point, a predicted spatial location of each of the query points in each of the video frames in the video sequence other than the corresponding video frame of the query point; processing the video frames in the video sequence using a visual backbone neural network to generate a feature grid comprising respective visual features for each of a plurality of spatial locations in each of the video frames; For each query point, generating extracted features of the query point from the spatial location of the query point in the corresponding video frame and the respective visual features for one or more of the plurality of spatial locations in the corresponding video frame; generating, from the feature grid and the extracted features of the query points, respective predicted spatial locations of the query points in each of the video frames other than the corresponding video frame; 2. The method of claim 1, comprising:

3. generating extracted features of the query point from the spatial location of the query point in the corresponding video frame and the respective visual features for one or more of the plurality of spatial locations in the corresponding video frame, The method of claim 2 , comprising performing an interpolation of the visual features of a set of spatial locations that are within a local neighborhood of the respective spatial positions of the query points in the corresponding video frames.

4. generating, from the feature grid and the extracted features of the query points, the respective predicted spatial locations of the query points in each of the video frames other than the corresponding video frame, comprising: generating a cost volume from the feature grid and the extracted features of the query points; processing the cost volume using a decoder neural network to generate the respective predicted positions of the query points in the video frames for each of the video frames other than the corresponding video frame; The method of claim 2 or claim 3, comprising:

5. processing the cost volume using the decoder neural network, processing the cost volume using the decoder neural network to generate, for each of the plurality of video frames other than the corresponding video frame, a respective score for each spatial location in the video frame; generating, for each of the plurality of video frames other than the corresponding video frame, the predicted location from the respective scores for the spatial locations in the video frames; 5. The method of claim 4, comprising:

6. generating the predicted location from the respective scores, identifying the spatial location with the highest score; identifying each spatial location that is within a fixed size window of the spatial location with the highest score; determining the predicted location by calculating a weighted sum of the spatial locations within the fixed-size window of the spatial location with the highest score, where a weight for each spatial location is calculated based on the score for the spatial location and the sum of the scores for the spatial locations within the fixed-size window; 6. The method of claim 5, comprising:

7. the network output further comprises, for each query point, a respective occlusion score for the query point with respect to each of the video frames other than the corresponding video frame; The method of claim 1 , wherein the respective occlusion scores represent the likelihood that the query point is occluded in the video frame.

8. processing the cost volume using a decoder neural network to generate the respective predicted locations of the query points; 8. The method of claim 7 when dependent on claim 5 or 6, comprising processing the cost volume using the decoder neural network to generate, for each video frame other than the corresponding video frame, the respective predicted positions of the query points in the video frames and the respective occlusion scores of the query points in the video frames.

9. A method for training a point tracking neural network according to claim 7 or 8, comprising the steps of: obtaining one or more batches of training video sequences; obtaining, for each query point in each training video sequence and a set of query points for the training video sequence, (i) a ground truth spatial location of the query point in each of the video frames in the training video sequence, and (ii) a respective ground truth occlusion score of the query point for each of the video frames in the training video sequence; processing the set of one or more query points for the training video sequence and the training video sequence using the point tracking neural network to generate, for each training video sequence, a predicted spatial location of each of the query points in each of the video frames in the training video sequence, and a respective occlusion score for the query point for each of the video frames in the training video sequence; Calculating a gradient of a loss function with respect to parameters of the point tracking neural network, the loss function being: (i) a first loss term that measures, for each training video sequence, an error between a ground truth spatial location of the query point and a corresponding predicted spatial location of the query point; and (ii) a second loss term that measures, for each training video sequence, an error between a ground truth occlusion score of the query point and a corresponding predicted occlusion score of the query point; A method comprising:

10. The method of claim 9 , wherein the first loss term is a Huber loss.

11. The method of claim 9 or claim 10, wherein the second loss term is a cross-entropy loss.

12. The loss function is 12. The method of claim 9, further comprising: (iii) a control learning loss term.

13. generating, for each of one or more of the training video sequences, ground truth spatial locations of the query points in each of the video frames in the training video sequences and the respective ground truth occlusion scores of the query points for each of the video frames, obtaining another training video sequence that is a variation of the training video sequence; processing the set of one or more query points and the other training video sequence using the point tracking neural network to generate, for each query point, a predicted spatial location of each of the query points in each of the video frames in the other training video sequence and a respective occlusion score of the query point for each of the video frames in the other training video sequence; generating, from data generated by the point tracking neural network while processing the set of one or more query points and the other training video sequence, ground truth spatial locations of the query points in each of the video frames in the training video sequence and the respective ground truth occlusion scores of the query points for each of the video frames.

13. The method of any one of claims 9 to 12, further comprising the steps of:

14. generating, from data generated by the point tracking neural network while processing the set of one or more query points and the other training video sequence, ground truth spatial locations of the query points in each of the video frames in the training video sequence and the respective ground truth occlusion scores of the query points for each of the video frames; 14. The method of claim 13, comprising using a predictor neural network to process the data generated by the point tracking neural network while processing the set of one or more query points and the other training video sequence using the point tracking neural network.

15. The method of claim 14, wherein the predictor neural network is a multi-layer perceptron (MLP).

16. 16. The method of claim 13, wherein the training video sequence and the further training video sequence are generated from the same original training video sequence by applying different data augmentation to the original training video sequence.

17. While processing the set of one or more query points and the other training video sequence using the point tracking neural network, the data generated by the point tracking neural network is: (i) the respective predicted spatial locations of the query points in the video frames within the separate training video sequence; and (ii) the respective occlusion scores of the query points for each of the video frames in the separate training video sequence.

17. The method of any one of claims 13 to 16, comprising:

18. 18. The method of claim 9, further comprising training the point tracking neural network using the calculated gradients.

19. A method implemented by one or more computers, comprising: obtaining a video sequence comprising a sequence of video frames; generating an optical flow estimate for the video sequence; obtaining input specifying an initial point at an initial location in a first video frame of the video sequence and a target location in the second, subsequent video frame representing a spatial location of the initial point in the second, subsequent video frame; using the optical flow estimates to generate respective target positions for the initial point in each video frame between the first video frame and the second, subsequent video frame in the video sequence; A method comprising:

20. from the video sequence and (i) the second, subsequent video frame, and (ii) the respective target positions of the initial point in each video frame between the first video frame and the second, subsequent video frame in the video sequence; training the point tracking neural network with training data including training examples; 20. The method of claim 19, further comprising:

21. 21. The method of claim 20, wherein the point tracking neural network is a point tracking neural network according to any one of claims 1 to 8.

22. 22. A method according to claim 20 or claim 21, wherein training the point tracking neural network comprises performing the respective operations of any one of claims 9 to 18.

23. 23. The method of claim 19, wherein generating an optical flow estimate for the video sequence comprises processing the video sequence using an optical flow prediction neural network.

24. The method of claim 23 , wherein the optical flow prediction neural network is trained by unsupervised learning.

25. using the optical flow estimates to generate respective target positions for the initial point in each video frame between the first video frame and the second, subsequent video frame in the video sequence, 25. The method of claim 19, comprising: generating a predicted path using the optical flow estimate from the initial position in the first video frame to a final position in the second, subsequent video frame, the predicted path including respective positions in each of the video frames that are between the first video frame and the second, subsequent video frame in the video sequence.

26. generating a predicted path from the initial position in the first video frame to the final position in the second, subsequent video frame using the optical flow estimate, 26. The method of claim 25, comprising generating a path from the initial position in the first video frame to the final position in the second, subsequent video frame that minimizes discrepancy with the optical flow estimate.

27. generating a path from the initial position in the first video frame to the final position in the second, subsequent video frame that minimizes discrepancy with the optical flow estimate; identifying a shortest path in a graph from a first node representing the initial position in the first video frame to a second node representing the final position in the second, subsequent video frame, the graph comprising: (i) a respective node for each of a plurality of pixels in a plurality of frames, including the first video frame, the second, subsequent video frame, and each frame between the first video frame and the second, subsequent video frame in the video sequence; and (ii) for the first video frame in the video sequence and for each frame between the first video frame and the second, subsequent video frame, a respective edge between each particular pixel of the plurality of pixels in the frame and each particular pixel of the plurality of pixels in a subsequent frame, the respective edge being weighted by a disparity between a path from the particular pixel in the frame to the particular pixel in the subsequent frame and the optical flow estimate.

27. The method of claim 26, comprising:

28. obtaining an input specifying an initial point at an initial location in a first video frame of the video sequence and a target location in the second, subsequent video frame representing a spatial location of the initial point in the second, subsequent video frame; obtaining a first user input specifying the initial point at the initial location within the first video frame of the video sequence; obtaining a second user input specifying the target position within the second, subsequent video frame representing the spatial location of the initial point in the second, subsequent video frame; 28. The method of any one of claims 19 to 27, comprising:

29. obtaining an input specifying an initial point at an initial location in a first video frame of the video sequence and a target location in the second, subsequent video frame representing a spatial location of the initial point in the second, subsequent video frame; prior to acquiring the first user input and the second user input, acquiring a third user input identifying a bounding box in the first video frame depicting an object; 30. The method of claim 28, wherein the first user input selects the initial point from a plurality of points that are within the bounding box.

30. One or more computers; one or more storage devices storing instructions that, when executed by said one or more computers, cause said one or more computers to perform the operations of each of the methods recited in any one of claims 1 to 29; A system including:

31. One or more computer storage media having stored thereon instructions that, when executed by one or more computers, cause the one or more computers to perform the respective method operations recited in any one of claims 1 to 29.

Images