Method for view prediction and computer product

Abstract

Systems, methods, and apparatuses are described herein for estimating motion of a plurality of 3D points in a scene and predicting the scene at a future time instant based on the estimated motion. These tasks can be accomplished using one or more machine learning (ML) models. A first ML model can be used to predict motion features of a temporal state of the scene based on motion features of a previous state. A second ML model can be used to predict a motion field characterizing displacement or deformation of the plurality of 3D points from a source time to a target time. Then, a third ML model can be used to predict respective image properties of the 3D points based on updated locations of the 3D points at the target time and / or in a viewing direction. Then, an image of the scene at the target time can be generated based on the predicted image properties of the 3D points.

Description

Technical Field

[0001] This application relates to the field of motion estimation. Background Technology

[0002] To navigate and / or interact with their environment, autonomous systems (e.g., robots) may rely on computer vision to understand the position and structure of objects in a scene and / or predict how these objects may move or deform over time and / or space, for example, to adapt the autonomous system's movement path or trajectory to avoid collisions with objects. However, despite significant advances in dynamic scene representation techniques, motion estimation and view prediction remain challenging tasks, for example, due to image blurring caused by motion that may exist between multiple points, lighting conditions that may vary spatially or temporally, etc. Therefore, systems and methods capable of determining accurate representations of scenes, estimating the motion of objects within the scene, and / or accurately predicting intermediate or future states of the scene are of great importance. Summary of the Invention

[0003] This document describes systems, methods, and apparatuses associated with motion estimation and view (or scene) prediction. An apparatus capable of performing these tasks may include one or more processors configured to receive at least one of a target time or viewing direction associated with a scene, wherein the scene comprises multiple points (e.g., three-dimensional (3D) points) associated with one or more objects. The one or more processors may also be configured to use one or more machine learning (ML) models to predict image properties of the multiple points at the target time or in the viewing direction. The one or more ML models may be trained to learn corresponding motion and voxel representations or volumetric representations of the scene (e.g., color and / or density attributes of the scene in 3D space) of the multiple points based on a training set of images depicting the scene over a time period, and the one or more ML models may predict image properties of the multiple points based on the corresponding motion and voxel representations or volumetric representations of the scene. Based on the image properties of the multiple points predicted by the one or more ML models, the one or more processors of the apparatus may generate an image (e.g., a synthetic image) depicting the scene at a target time, which may be within or outside the time period covered by the training image set. In the former case, the image generated by the device can be considered an interpolation of the training image, while in the latter case, the image generated by the device can be considered an extrapolation of the training image.

[0004] In the example, one or more ML models described herein may include a first ML model, a second ML model, and a third ML model. The first ML model may be trained to determine multiple features associated with a scene, which indicate motion of the scene from a source time (e.g., within a time period) to a target time (e.g., corresponding motion of multiple points in the scene). The second ML model may be trained to determine a motion field indicating the corresponding positions of multiple points in the scene at the target time based on the multiple features determined by the first ML model, while the third ML model may be trained to predict image characteristics of multiple points at the target time or in the viewing direction based on the corresponding positions of the multiple points indicated by the motion field.

[0005] In the example, the corresponding image properties of multiple points predicted by the third ML model may include the corresponding color or density of the points at the target time, and the image at the target time may be generated by sampling multiple points along the viewing direction, determining the corresponding image properties of the multiple points, and aggregating the corresponding colors of the multiple points based on their corresponding densities to obtain the pixel values ​​of the image. In the example, the image may be generated without distinguishing between different viewing directions, in which case it may be assumed that the color of the points (e.g., each of the multiple points described herein) is the same for different viewing directions.

[0006] In the examples, multiple features indicating the motion of the scene from the source time to the target time can be derived based on multiple motion representation vectors, such as motion-embedding basis vectors, which are determined based on a training image set. In the examples, the multiple features can also be derived based on a set of weights associated with the multiple motion representation vectors (e.g., as a dot product of the multiple motion representation vectors and the weight set). In these examples, the first ML model can be trained to predict the weight set, for example, based on motion information extracted from the training image set.

[0007] In the example, the first, second, and third ML models can be jointly trained at least based on an image reconstruction loss (and / or an additional feature prediction loss). For example, joint training of the first, second, and third ML models may include extracting a feature set from the training image using the current parameters of the first ML model, wherein the feature set may indicate motion from a first time point to a second time point. Joint training may also include using the current parameters of the second ML model to determine an estimated motion field that may indicate the displacement of one or more points in the training image, wherein the estimated motion field may be determined at least based on the feature set extracted by the first ML model. Additionally, joint training may include predicting the corresponding positions of one or more points at a second time point based at least on the motion field and the corresponding positions of one or more points at the first time point, and predicting the corresponding image characteristics of one or more points at the second time point using the current parameters of the third ML model based at least on the corresponding positions of one or more points at the second time point. Subsequently, an output image may be generated based on the predicted image characteristics of one or more points, and the corresponding current parameters of the first, second, and third ML models may be adjusted at least based on the difference between the output image and the gold standard image (e.g., based on the image reconstruction loss). Attached Figure Description

[0008] The examples disclosed herein can be understood in more detail from the following description, which is given by way of example in conjunction with the accompanying drawings.

[0009] Figure 1 This is a simplified block diagram illustrating example techniques for motion estimation and / or view prediction according to one or more embodiments of the present disclosure provided herein.

[0010] Figure 2 This is a simplified block diagram illustrating example operations that can be associated with motion estimation and / or view prediction according to one or more embodiments of the present disclosure provided herein.

[0011] Figure 3 This is a simplified block diagram illustrating example techniques for training a motion view predictor according to one or more embodiments of the present disclosure provided herein.

[0012] Figure 4 This is a simplified block diagram illustrating example techniques for optimizing a motion view predictor according to one or more embodiments of the present disclosure provided herein.

[0013] Figure 5 This is a simplified flowchart illustrating example operations that can be performed to train a neural network according to one or more embodiments of the present disclosure provided herein.

[0014] Figure 6This is a simplified block diagram illustrating example components of a device that can be configured to perform the tasks described in one or more embodiments of the present disclosure provided herein. Detailed Implementation

[0015] The present disclosure is illustrated by way of example rather than limitation in the figures.

[0016] Figure 1Example techniques for motion estimation and view prediction according to one or more embodiments of the present disclosure provided herein are illustrated. As shown, a system or device 100 (referred to herein as a motion-view predictor or MVP) capable of performing motion estimation and / or view prediction tasks can acquire or be provided with knowledge about the motion and / or appearance (e.g., color, density, etc.) of a scene, and subsequently, given a target time and / or viewing orientation, use one or more machine learning (ML) models to predict image characteristics of multiple points in the scene at the target time or in the viewing orientation, and generate an image depicting the scene at the target time or in the viewing orientation (e.g., a view representing the scene) based on the image characteristics of the multiple points predicted by the one or more ML models. As will be described in more detail below, the one or more ML models can be trained to learn the corresponding motion of the multiple points and voxel representations of the scene (e.g., color and / or density attributes of the scene in 3D space) based on a set of training images that can depict the scene over a time period. The one or more ML models can also be trained to predict the image characteristics of multiple points at the target time and / or in the viewing orientation based on the corresponding motion of the multiple points and voxel representations of the scene learned from the training images. Multiple points can be associated with one or more objects in a scene (e.g., in the three-dimensional (3D) space of the scene) and may be referred to herein as 3D points. All or a subset of these 3D points may exhibit motion during a time period (e.g., 0 to T) covered by training images, and each training image may depict the state of the scene at a corresponding time (e.g., t0, t1, t2, t3, etc.) within the time period (e.g., each training image may depict the temporal state of the scene). Thus, for example, before the MVP 100 is deployed for motion estimation and / or view prediction, one or more artificial neural networks can be used to learn (e.g., extract from training images) the corresponding motion and / or image characteristics of multiple points from the training images. In the example, the training images may be captured by one or more sensors (e.g., a visual sensor such as a red-green-blue (RGB) sensor or an RGB camera, a depth sensor, a thermal sensor, etc.), which may be part of the MVP 100 or may be communicatively coupled to the MVP 100 to send images to the MVP 100. The sensor can be mounted in different locations (e.g., in a medical environment) so that the images captured by the sensor can depict the scene (e.g., objects in the scene) from different viewpoints (e.g., different viewing directions) over a time period (0, T).

[0017] MVP 100 may include a motion predictor (e.g., Figure 1MVP 104 is configured to receive at least a target time and determine (e.g., predict) multiple features (e.g., salient features) using a first machine learning (ML) model that indicate motion from the source time to the target time associated with the scene (e.g., corresponding motion of multiple 3D points in the scene). The source time may correspond to a training image (e.g., the source time may be within a time period covered by the training image), while the target time may not have a corresponding image (e.g., the target time may be outside the time period covered by the training image, or the target time may be within a time period such as between time slots associated with two existing training images). MVP 100 may also include a motion field predictor (e.g., Figure 1 The second ML model (106) is configured to receive features extracted by the motion predictor 104 and determine a motion field using a second ML model. This motion field indicates the corresponding positions of multiple points in the scene at a target time (e.g., the motion field indicates the displacement of multiple points from the source time to the target time). Such a motion field may be referred to herein as a dense motion field, for example, because it indicates the corresponding displacement of multiple 3D points in the scene, rather than just the displacement of an object considered as a single point. Therefore, using the motion field determined by the motion field predictor 106, the corresponding positions of multiple points in the scene can be determined for a target time, for example, by determining the amount of displacement of each point from the source time to the target time based on the motion field and applying that displacement to the position of the point in the source image.

[0018] The MVP 100 may also include a spatial / temporal field predictor (e.g., Figure 1 108 (in the example) is configured to use a third ML model to predict corresponding image characteristics of multiple points at a target time based on their respective locations at the target time, the target time itself, and / or the viewing direction described herein. The image characteristics predicted by the spatial / temporal field predictor 108 may include, for example, the corresponding color and / or density of multiple points in the scene. As will be described in more detail below, voxel representations of the scene (e.g., indicating color and / or density attributes of the scene in 3D space) can be acquired or learned through machine learning processes, for example, by encoding the scene's color, radiance, and / or density information into one or more neural fields (e.g., the storage location of MVP 100), and subsequently querying the neural fields based on the location of the 3D points, the viewing direction described herein, and / or the target time to obtain the color, radiance, and / or density of the 3D points. By repeating the above operations for a sufficient number of rays (e.g., optical or camera rays), a view of the scene at the target time (e.g., a synthetic image) can be obtained based on the predicted image characteristics (e.g., color and / or density) of multiple 3D points.

[0019] Therefore, use Figure 1The illustrated example techniques can estimate the corresponding motion and image properties of multiple 3D points (e.g., all 3D points) of a scene based on existing images of the scene (e.g., training images), and can predict the view of the scene at a target time and / or in the viewing direction based on the estimated motion and image properties of the multiple 3D points (e.g., by generating synthetic images). If the target time is within the time period covered by the existing images, the predicted image / view can be considered an interpolated image / view of the scene, while if the target time is outside the time period associated with the existing images, the predicted image / view can be considered an extrapolated image / view of the scene. Thus, the example techniques disclosed herein can be used to construct machine vision of an environment by capturing a set of images of an environment (e.g., such as a medical or manufacturing environment) (e.g., using one or more sensors described herein), acquiring knowledge about the environment from the captured images using machine learning, and obtaining an additional (e.g., synthetic) view of the environment using the acquired knowledge. The machine vision can then be used, for example, by medical or manufacturing robots to navigate and / or interact with the environment.

[0020] Figure 2 Example operations that can be associated with motion estimation and / or view prediction according to one or more embodiments of the present disclosure provided herein are illustrated. The example operations may be performed by a motion view predictor (MVP) 200 (e.g., such as...). Figure 1 The MVP100 will be executed, but those skilled in the art will understand that this motion view predictor may not be configured to execute. Figure 2 All the operations and / or motion view predictors shown can also be configured to perform Figure 2The operations are not shown in the figure. As illustrated, MVP 200 can be configured to receive a target time and / or viewing direction associated with a scene, and predict the view of the scene at the target time and / or in the viewing direction (e.g., generate an image). MVP 200 can be configured to implement a first ML model 204 (e.g., a motion prediction model) for determining the corresponding motion of multiple 3D points (e.g., all 3D points) in the scene from the source time to the target time. The first ML model can be learned and / or implemented using an artificial neural network (ANN) (e.g., such as a convolutional neural network), for example, by training the ANN to extract features from the scene and presenting (e.g., encoding) the extracted features in a suitable format (e.g., as feature vectors) to indicate the corresponding motion of the multiple 3D points from the source time to the target time. The ANN can acquire the ability to extract these motion features from the scene based on a set of training images that can depict the scene within a time period (0, T) (e.g., the training images can be captured at corresponding time points t0, t1, t2, t3, etc. within the time period (0, T)). The source time described herein may correspond to a training image (e.g., an image captured at t0, t1, t2, or t3), while the target time may be within a time period (e.g., between t2 and t3) or outside a time period (e.g., a future time t4).

[0021] In the example, the first ML model 204 can be trained to determine one or more motion feature vectors based on training images to encapsulate the dynamic characteristics (e.g., motion) of the scene within a time period covered by the training images, and to infer additional feature vectors indicating the motion of the scene from the source time to the target time based on the determined motion feature vectors. This operation can be illustrated by the following equation 1):

[0022] ω t→t+δt =P(ω) prev )1)

[0023] Where, ω t→t+δt ω can represent the eigenvector indicating the motion between time t and t+δt. prev P can represent the set of motion feature vectors extracted from the training images, and P can represent the function performed by the first ML model (e.g., by optimizing parameters θ). P (Definition). For example, the first ML model can be trained to estimate the motion of a scene (e.g., the corresponding motion of multiple 3D points in the scene) based on training images (e.g., images captured at t0, t1, t2, and t3), and encode the estimated motion into corresponding motion feature vectors (e.g., {ω 0→1 ω 1→2 ω 2→3Then, given a target time (e.g., t4), the first ML model can generate motion feature vectors (e.g., ω) based on the motion feature vectors determined from the training images. 3→4 ), where the new motion feature vector (e.g., ω) 3→4 It can indicate the movement of multiple 3D points in the scene from t3 to t4.

[0024] In the example, the first ML model 204 can be trained based on multiple (e.g., n) motion representation vectors B = [b1,...,b2]. n ] T Determine the motion of the scene from the source time to the target time (e.g., the corresponding motion of multiple 3D points in the scene), where b i ∈R m (For example, Figure 2 (of 220). For example, such as Figure 2 As shown, the first ML model 204 can be trained based on the motion representation vector 220 and a set of weights associated with relevant timestamps (e.g., W). 3-＞4 This generates feature vectors (e.g., ω) that can indicate the motion of the scene between two time stamps (e.g., t3 and t4). 3→4 (For example, the feature vector can be determined as motion representation vector 220 and weight W) 3-＞4 (Dot product). Multiple motion representation vectors can be learned from training images and can be used to estimate motion between arbitrary pairs of images or timestamps (e.g., basis vectors can be shared by multiple temporal states or images of a scene). For example, through training, the first ML model 204 can optimize the first weight set W. 0-＞1 It can be applied to the motion representation vector 220 to derive the feature vector ω. 0→1 This vector indicates the motion between timestamps t0 and t1. The first ML model 204 can also optimize the second weight set W. 1-＞2 It can be applied to the motion representation vector 220 to derive the feature vector ω. 1→2 This vector indicates the motion between timestamps t1 and t2. Similarly, the first ML model 204 can optimize the third weight set W. 2-＞3 It can be applied to the motion representation vector 220 to derive the feature vector ω. 2→3 This vector indicates the motion between timestamps t2 and t3. For example... Figure 2As shown, weights (and thus motion features) associated with a temporal state of the scene (e.g., associated with timestamp t4) can be derived based on weights (and thus motion features) associated with previous temporal states of the scene (e.g., associated with timestamps t0, t1, t2, and / or t3), thereby modulating the motion representation of the scene through predictability. Once trained and given a target time t2, the first ML model 204 can determine the weight set W. 3-＞4 It can be applied to the motion representation vector 220 to derive the feature vector ω. 3→4 This vector indicates the motion between timestamps t3 and t4. This reduces the input and / or output space of the first ML model 204, resulting in faster convergence and / or more consistent and smoother motion estimation (e.g., compared to estimating the frame-by-frame motion vector ω). 0→1 ω 1→2 and ω 2→3 And then use these vectors to derive ω 3→4 compared to).

[0025] The first ML model described herein can be implemented and / or learned using an artificial neural network, such as a convolutional neural network (CNN) comprising multiple convolutional layers, one or more pooling layers, one or more recurrent layers, and / or one or more fully connected layers. Each convolutional layer can include multiple convolutional kernels or filters with corresponding weights (e.g., parameters corresponding to the first ML model), which can be configured to extract features from the input image. Following the convolution operation can be batch normalization and / or an activation function (e.g., a rectified linear unit (ReLU) activation function), and the features extracted by the convolutional layers can be downsampled by one or more pooling layers and / or one or more fully connected layers to obtain a representation of the features, e.g., in the form of a feature map or feature vector.

[0026] The feature representations obtained using the first ML model 204 (e.g., motion feature vectors ω) 3→4 ) can be provided to the second ML model 206 (e.g., by parameter θ) M A defined motion field prediction model is used to decode feature representations and derive a motion field based on the decoded features. This motion field can indicate the displacement and / or deformation of multiple 3D points (e.g., all 3D points) in a scene from a source time to a target time. Since the corresponding positions of multiple 3D points in the scene at the source time can be known (e.g., based on an image captured at the source time), the motion field can be used to determine the corresponding positions of multiple 3D points at the target time based on the displacement and / or deformation indicated by the motion field. For example, the position of a 3D point at the target time can be determined by applying the displacement and / or deformation (e.g., (Δx, Δy, Δz)) of the 3D point to its position at the source time (e.g., (x, y, z)).

[0027] In the example, the second ML model can be implemented and / or learned using an artificial neural network, such as a multilayer perceptron (MLP) neural network or a convolutional neural network as described herein. In the example, the artificial neural network may include multiple fully connected layers configured to take N input values ​​(e.g., N may be equal to the dimension of the features provided by the first ML model 204), map them to a larger dimension through one or more fully connected layers, and then reduce the dimension through one or more final fully connected layers to derive the motion field as described herein.

[0028] The target temporal locations of multiple 3D points determined using the second ML model can be provided to the third ML model 208 (e.g., by parameter θ). F A defined spatial / temporal field prediction model is used to determine the corresponding image characteristics (or image attributes) of multiple 3D points at a target time. In the example, the third ML model 208 may include a coordinate-based neural network trained on a set of scene observation data (e.g., training images of the scene captured from different viewpoints and / or at different times) to encode scene-associated geometry (e.g., occupancy and / or distance), radiosity (e.g., color), and / or density (e.g., opacity) information into multiple neural fields (e.g., storage locations storing the spatial, temporal, and / or image characteristics of the scene), such that the view or image of the scene can subsequently be predicted by querying the neural fields based on the corresponding location of 3D points in the scene, viewing direction, and / or time of interest (e.g., target time t4). This operation can be illustrated by the following equation 2):

[0029] F(v t ,d t ,t;θ F )={c t ,σ t}2)

[0030] Where F can represent the function performed by the third ML model 208, θ F The parameters of the third ML model can be characterized, t can be characterized by the target time of interest, and v t It can represent 3D points in the scene, d t It can characterize the viewing direction (e.g., defined by a vector), c t It can characterize the color at a 3D point (e.g., the observed radiance), and σ t It can characterize the density of 3D points (e.g., opacity).

[0031] Once the scene's image characteristics have been learned through training, these characteristics can be queried to predict the scene's view at a target time (e.g., rendering a composite image), for example, using one or more volumetric rendering techniques (e.g., by...). Figure 2 The C(r) characterization in the text. For example, for each of multiple camera rays r defined by the optical origin o and the direction d intersecting the pixel (e.g., for a virtual camera), it can be represented by sampling points along the camera ray (e.g., for p). i The process involves sampling (using o+id), querying the corresponding image characteristics or attributes (e.g., color and / or density) of sample points from the neural field, and accumulating the image characteristics or attributes of the sample points to obtain pixel values ​​to determine the color of a pixel. For example, if a sample point along the camera ray r has a density value of 0, the point can be considered "transparent," and its color may not affect the aggregate color of the pixel. If the current sample point along the camera ray has a density of 1, the point can be considered to be on a solid surface, and the colors of other points behind it may not affect the aggregate color of the pixel (e.g., because these points may be occluded). If a sample point on the camera ray has a density value between 0 and 1, the color of that point can be mixed with the colors of other points along the camera ray to obtain the color of the pixel. Using these techniques, a composite view or image of a scene can be generated, for example, by repeating the above operations on a sufficient number of rays (e.g., 1024 rays).

[0032] In the example, the third ML model can be implemented and / or learned using an artificial neural network such as an MLP neural network, which includes multiple fully connected layers (e.g., an input layer, one or more hidden layers, and an output layer) and / or one or more convolutional layers. During training such a neural network, a dataset of captured images of the scene (e.g., RGB images), corresponding camera poses and / or intrinsic parameters, and / or scene boundaries can be provided to the neural network. In response, the neural network can sample coordinates (e.g., 5D coordinates representing 3D position and viewing direction) along the camera ray, feed the sampled coordinates into the MLP to produce color and volume density, and synthesize these values ​​into an image using volumetric rendering techniques. Since this rendering function can be differentiable, gradient descent can be used to optimize the neural network parameters, for example, by minimizing the difference between the synthesized image and the gold standard image. In the example, the third ML model can be trained to generate synthesized images based solely on 3D point positions (e.g., without viewing direction). In these cases, the model can assume that the color of the 3D point can be the same for different viewing directions (e.g., with simple refraction / reflection or without refraction / reflection), and this technique can help reduce the noise often encountered in dense motion estimation.

[0033] Therefore, through Figure 2The illustrated example operations can predict (e.g., synthesize) views or images of the scene from different viewpoints (e.g., characterized by the viewing direction as described herein) and / or at different times (e.g., intermediate or future times) based on existing images of the scene. Prediction can be performed without causing a significant increase in storage consumption (e.g., due to the use of neural fields), and predictions can be consistent and smooth (e.g., by utilizing a set of base motion vectors to characterize the scene's motion state). Furthermore, the ambiguity between rigid and non-rigid deformations in the scene (e.g., whether the growth of an element in the scene is due to the element moving closer to the observer or because the element's volume actually expands) can also be reduced due to the regularization provided by dense modeling of motion (e.g., motion estimation of all 3D points in the scene using the first ML model described herein). For example, the distinction between rigid motion and non-rigid expansion of an object may be blurred when viewed from a single 2D image, but may be clear when multiple points of the object are represented in 3D. Moreover, by memorizing the motion features of a scene that can be predicted (e.g., using a first ML model), the techniques disclosed herein may be able to (e.g.) detangle real motion from noise by recognizing and ignoring changes in the image appearance of objects caused by sensor noise, temporary occlusion, etc.

[0034] Figure 3 Example techniques for training the motion view predictor described herein (e.g., a neural network or ML model implemented by the motion view predictor) to estimate the motion of a scene and / or predict views of the scene (e.g., images) are illustrated. As shown, training can be performed using a set of 302 training images depicting the scene over a time period and based on one or more losses (e.g., prediction loss L). pred and / or image reconstruction loss L recon To perform this, predict the loss L. pred It can be based on the predicted motion feature vector (e.g., ω) 3→4 The difference between the original vector and the sampled gold standard vector is used to determine the image reconstruction loss L. recon The difference can be determined based on the difference between the view or image generated by the motion view predictor (e.g., synthetic image 310) and the gold standard image. Training of multiple ML models (e.g., motion prediction model 304, motion field prediction model 306, and spatial / temporal field prediction model 308) can be performed individually or end-to-end. For example, during training, image frames 302 (e.g., images with timestamps t0, t1, t2, t3, t4, etc.) can be sampled from the training dataset, and motion prediction model 304 can infer the motion feature vector ω associated with a future image frame (e.g., with timestamp t4) based on one or more previous image frames (e.g., with timestamps t0, t1, t2, and t3). 3→4 As described in this article, the motion feature vector ω3→4 It can indicate the corresponding motion of multiple 3D points in the scene from t3 to t4, and this vector can be generated based on motion information extracted from previous image frames (e.g., by weights w). 0→1 w 1→2 and / or w 2→3 Characterization) Prediction weight set w 3→4 Weights are then applied to multiple motion representation vectors 312, which can also be learned from the scene, for inference. For example, at the start of training, initial values ​​can be assigned to the motion representation vectors 312 (e.g., based on random sampled values), and throughout training, the weights can be applied based on the aforementioned loss (e.g., prediction loss L). pred and / or image reconstruction loss L recon The gradient descent is used to adjust (e.g., iteratively with weights) the values ​​of the basis vectors 312. Internally, the dot product of the weights and basis vectors can be computed to represent features based on the previous states of the scene (e.g., computed as basis vectors and weights (such as w) corresponding to previous timestamps). 0→1 w 1→2 and / or w 2→3 The corresponding dot product of ) is used to derive the feature representation of the scene's temporal state at the current time point (e.g., ω). 3→4 By splitting the learning of feature representations of time states into two parts (e.g., weights and basis vectors), training can converge faster and produce more accurate results. This is likely because, for example, weights associated with individual time points can represent time-step-specific information, while basis vectors can represent inter-time-step information. Together, they can provide more guidance to the neural network while reducing the overall dimensionality to be optimized (e.g., the number of weights).

[0035] The motion feature vectors predicted by motion prediction model 304 can be used by motion field prediction model 306 to estimate the motion field and determine the corresponding positions of multiple 3D points at t4. The updated positions of the 3D points can then be provided to spatial-temporal field prediction model 308 to determine the corresponding image properties (e.g., color and / or density) of the 3D points and generate a synthetic image (e.g., image 310) depicting the scene at t4. Various predictions made through the above operations can be used, for example, by determining the prediction loss L based on the total error or mean square error between the prediction result and the corresponding gold standard. pred and / or image reconstruction loss L recon The loss (e.g., one or more gradient descent steps associated with the loss) can then be backpropagated through one or more neural networks configured to implement the ML model to tune the parameters of the neural network / ML model with the aim of minimizing the loss.

[0036] In the example, the weights predicted by motion prediction model 304 (e.g., w) t→t+δtIt can be obtained online, and in L pred and L recon Joint optimization. For example, at each training iteration, the current frame weight w can be used to calculate L. pred And accordingly optimize the downstream ML model. Then, it can be compared with L recon Optimize the weights themselves. In the example, L recon It can be applied to reconstructed images with and without motion reparameterization in order to untangle appearance-related information from motion-related information.

[0037] Figure 4 An example of an optimized motion view predictor P (e.g., a neural network or ML model implemented by the motion view predictor) according to one or more embodiments of the present disclosure provided herein is illustrated. As shown, during optimization (e.g., training), the motion predictor P can infer a motion feature vector ω associated with a fourth image frame based on three previous image frames. Prediction loss L pred It can be determined and used to minimize the difference between the predicted vector and the gold standard, while the reconstruction loss L recon The parameters for further optimization of the ML model can be determined based on synthetic image frames generated by the motion view predictor and gold standard image frames. In the example, prediction loss and reconstruction loss can be combined and used for optimization. For example, the loss can be combined with a balancing parameter γ, such as L = L pred +γL recon The value of γ can be adjusted based on the requirements of each specific use case. In the example, the reconstruction loss L... recon This can be applied with and without motion reparameterization to unwrap appearance-related information from motion-related information. For example, image properties (e.g., color / density) of 3D points used for image prediction can be sampled as F(p+M(p,ω)). t→t+δt ),t+δt) and F(p,t).

[0038] Figure 5Example operations that can be associated with training one or more neural networks to perform the motion and view prediction tasks described herein are illustrated. As shown, the training operation may include, at 502, initializing the parameters of the neural network (e.g., weights associated with the individual filters or kernels of the neural network) based, for example, on samples collected from one or more probability distributions or parameter values ​​from another neural network with a similar architecture. The training operation may also include, at 504, feeding training data (e.g., a sequence of images associated with a scene) to the neural network, and at 506 causing the neural network to extract motion features associated with the time of interest (e.g., in the form of feature vectors). At 508, using the extracted features, the neural network can predict the motion field and / or synthetic image associated with the time of interest. Then, at 510, the neural network can compare the results of the foregoing operations with corresponding gold standards to determine various losses associated with the operations (e.g., feature prediction loss and / or image reconstruction loss). At 512, the losses may be evaluated, for example, individually or as a combination of losses (e.g., using a balancing factor), to determine whether one or more training termination criteria have been met. For example, if the aforementioned loss is below a predetermined threshold, or if the change in loss between two training iterations (e.g., between consecutive training iterations) falls below a predetermined threshold, then the training termination criterion can be considered satisfied. If it is determined at 512 that the training termination criterion has been satisfied, then training can end. Otherwise, before training returns to 506, the loss (e.g., alone or as a combined loss) can be backpropagated through the neural network at 614 (e.g., based on the corresponding gradient descent associated with the loss or gradient descent of the combined loss).

[0039] For the sake of simplicity, the training steps are depicted and described in a specific order herein. However, it should be understood that training operations can occur in various orders, simultaneously, and / or with other operations not presented or described herein. Furthermore, it should be noted that not all operations that may be included in the training process are depicted and described herein, and not all exemplified operations need to be performed.

[0040] The systems, methods, and / or apparatuses described herein may be implemented using one or more processors, one or more storage devices, and / or other suitable auxiliary devices (such as display devices, communication devices, input / output devices, etc.). Figure 6This is a block diagram illustrating an example device 600 that can be configured to perform the motion estimation and view prediction tasks described herein. As shown, device 600 may include a processor (e.g., one or more processors) 602, which may be a central processing unit (CPU), graphics processing unit (GPU), microcontroller, reduced instruction set computer (RISC) processor, application-specific integrated circuit (ASIC), application-specific instruction set processor (ASIP), physical processing unit (PPU), digital signal processor (DSP), field-programmable gate array (FPGA), or any other circuitry or processor capable of performing the functions described herein. Device 600 may also include communication circuitry 604, memory 606, mass storage device 608, input device 610, and / or communication link 612 (e.g., communication bus) through which one or more components shown in the figure exchange information.

[0041] Communication circuitry 604 can be configured to send and receive information using one or more communication protocols (e.g., TCP / IP) and one or more communication networks, including local area networks (LANs), wide area networks (WANs), the Internet, and wireless data networks (e.g., Wi-Fi, 3G, 4G / LTE, or 5G networks). Memory 606 may include a storage medium (e.g., a non-transitory storage medium) configured to store machine-readable instructions that, when executed, cause processor 602 to perform one or more functions described herein. Examples of machine-readable media may include volatile or non-volatile memory, including but not limited to semiconductor memory (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)), flash memory, etc.). Mass storage device 608 may include one or more disks, such as one or more internal hard disks, one or more removable disks, one or more magneto-optical disks, one or more CD-ROMs or DVD-ROMs, etc., on which instructions and / or data may be stored for operation of processor 602. Input device 610 may include a keyboard, mouse, voice-controlled input device, touch-sensitive input device (e.g., touch screen), etc., for receiving user input from device 600.

[0042] It should be noted that device 600 can operate as a standalone device or can be connected to other computing devices (e.g., networked or clustered) to perform the functions described herein. And even in Figure 6 Only one example of each component is shown in the figure, and those skilled in the art will understand that device 600 may include multiple instances of one or more components shown in the figure.

[0043] Although this disclosure has been described according to certain embodiments and generally associated methods, changes and variations of the embodiments and methods will be apparent to those skilled in the art. Therefore, the above description of exemplary embodiments does not limit this disclosure. Other changes, substitutions, and modifications are possible without departing from the spirit and scope of this disclosure. Furthermore, unless specifically stated otherwise, discussions using terms such as “analyze,” “determine,” “enable,” “identify,” and “modify” refer to the actions and processes of a computer system or similar electronic computing device that manipulate and transform data characterized as physical (e.g., electronic) quantities within the registers and memories of the computer system into other data characterized as physical quantities within the computer system's memory or other such information storage, transmission, or display devices.

[0044] It should be understood that the above description is intended to be illustrative and not restrictive. Many other embodiments will become apparent to those skilled in the art upon reading and understanding the above description. Therefore, the scope of this disclosure should be determined by reference to the appended claims and the full scope of their equivalents.

Claims

1. A method for view prediction, comprising: Receive at least one of a target time or viewing direction associated with a scene, wherein the scene includes multiple points associated with one or more objects in the scene; One or more machine learning (ML) models are used to predict the corresponding image features of the plurality of points at a target time or in the viewing direction, wherein the one or more ML models are trained to learn the corresponding motions of the plurality of points and the voxel representation of the scene from a set of training images depicting the scene over a time period, and the one or more ML models are further trained to predict the image features of the plurality of points based on the corresponding motions of the plurality of points and the voxel representation of the scene; and An image depicting the scene at the target time or in the viewing direction is generated based on the image characteristics of the plurality of points predicted by the one or more ML models.

2. The method according to claim 1, wherein, The one or more ML models include a first ML model, a second ML model, and a third ML model, and wherein, The first ML model is trained to determine multiple features associated with the scene that indicate the motion of the scene from a source time to a target time, the source time being within the time interval; The second ML model is trained to determine a motion field indicating the corresponding positions of the plurality of points in the scene at the target time, wherein the motion field is determined based on the plurality of features determined by the first ML model; and The third ML model is trained to determine the image characteristics of the plurality of points at the target time or in the viewing direction based on the corresponding positions of the plurality of points indicated by the motion field.

3. The method according to claim 2, wherein, The voxel representation of the scene indicates the color and density attributes of the scene, and the image characteristics of the plurality of points at the target time include the corresponding color and density of the plurality of points at the target time.

4. The method according to claim 3, wherein, Generating the image depicting the scene at the target time includes sampling multiple points along the viewing direction and aggregating the corresponding colors of the multiple points based on their corresponding densities to obtain pixel values ​​for the image.

5. The method according to claim 3, wherein, Generating the image depicting the scene at the target time includes the assumption that the color of the points is the same from different viewing directions.

6. The method according to claim 2, wherein, The plurality of features indicating the motion of the scene from the source time to the target time are derived based on a plurality of motion representation vectors determined from the training image set; or, the plurality of features indicating the motion of the scene from the source time to the target time are also derived based on a set of weights associated with the plurality of motion representation vectors and the target time; or, the plurality of features are represented by feature vectors derived as the dot product of the plurality of motion representation vectors and the set of weights.

7. The method according to claim 6, wherein, The first ML model is trained to predict the set of weights for the target time based on motion information extracted from the training image set, wherein the first ML model, the second ML model, and the third ML model are jointly trained based at least on image reconstruction loss.

8. The method according to claim 1, wherein, The target time is either within the time period associated with the training image set or outside the time period associated with the training image set.

9. A computer program product comprising instructions that, when run on a computer, cause the computer to perform the method as described in any one of claims 1-8.

10. A method for training a machine learning (ML) model, the method comprising: Multiple features are extracted from a set of training images associated with a scene using the current parameters of a first ML model, wherein the scene includes multiple points associated with one or more objects in the scene, and wherein the multiple features indicate the corresponding motion of the multiple points from a source time to a target time. The motion field indicating the corresponding position of the plurality of points in the scene at the target time is determined using the current parameters of the second ML model, wherein the motion field is determined based on the plurality of features extracted by the first ML model; The current parameters of the third ML model are used to predict the corresponding image characteristics of the plurality of points at the target time or in the viewing direction based on the corresponding positions of the plurality of points indicated by the motion field. An output image is generated based at least on the predicted image characteristics of the plurality of points; and The corresponding current parameters of the first ML model, the second ML model, and the third ML model are adjusted based at least on the difference between the output image and the gold standard image.

Images