Three-dimensional motion estimation method, system and computer program product

Abstract

A method for estimating a three-dimensional flow of an object between a first two-dimensional image and a second two-dimensional image, the first two-dimensional image including an image of the object in a first configuration, and the second two-dimensional image including an image of the object in a second configuration, the method comprising: inputting the first two-dimensional image, the second two-dimensional image, and a two-dimensional representation of an estimated three-dimensional shape of the object into a convolutional neural network; and using the convolutional neural network to generate the three-dimensional flow of the object between the first two-dimensional image and the second two-dimensional image from the first two-dimensional image, the second two-dimensional image, and the two-dimensional representation of the estimated three-dimensional shape of the object.

Description

Technical Field

[0001] This specification relates to methods and systems for estimating three-dimensional motion flow between two-dimensional images, and more particularly to methods and systems for estimating three-dimensional motion flow between objects (e.g., human faces) in two-dimensional images. Background Technology

[0002] Optical flow estimation is a challenging computer vision problem in which the 2D motion / flow of an object is estimated between two images. The 3D correspondence of optical flow estimation (i.e., estimating the 3D motion of scene points at different times in 3D space from a 2D image) is an even more challenging problem, but it has many important applications, including 3D face reconstruction, autonomous driving, action recognition, human motion and head pose estimation, and video-to-video conversion.

[0003] Scene flow combines the challenges of 3D shape reconstruction and dense flow estimation, and is a particularly intractable problem due to factors such as depth blur and aperture issues. Current methods typically rely on stereo images or RGB-D images, and to address this issue, they apply prior knowledge to the smoothness of the reconstructed surface and the estimated motion field, or the stiffness of the motion. Summary of the Invention

[0004] According to a first aspect of this specification, a computer-implemented method is described for estimating a three-dimensional flow of an object between a first two-dimensional image and a second two-dimensional image, the first two-dimensional image comprising an image of the object in a first configuration, and the second two-dimensional image comprising an image of the object in a second configuration. The method includes: inputting the first two-dimensional image, the second two-dimensional image, and a two-dimensional representation of an estimated three-dimensional shape of the object into a convolutional neural network; and using the convolutional neural network to generate the three-dimensional flow of the object between the first two-dimensional image and the second two-dimensional image from the first two-dimensional image, the second two-dimensional image, and the two-dimensional representation of the estimated three-dimensional shape of the object.

[0005] The two-dimensional representation of the estimated three-dimensional shape of the object may include a color image, wherein the pixels of the color image encode the three-dimensional coordinates of each point in the estimated three-dimensional shape of the object, the points corresponding to pixels of the object in the first two-dimensional image.

[0006] The two-dimensional representation of the estimated three-dimensional shape of the object includes normalized projective coordinate encoding.

[0007] The three-dimensional flow of the object between the first two-dimensional image and the second two-dimensional image may include a color image, which encodes the three-dimensional flow of pixels between the first two-dimensional image and the second two-dimensional image.

[0008] The convolutional neural network may include: a first input branch for processing the first two-dimensional image through one or more convolutional layers; a second input branch for processing the second two-dimensional image through one or more convolutional layers; a third input branch for processing the two-dimensional representation of the estimated three-dimensional shape of the object through one or more convolutional layers; and a relay network for jointly processing the outputs derived from the first input branch, the second input branch, and the third input branch through multiple convolutional layers. The convolutional neural network may further include: an association layer for associating the outputs of the first branch and the second branch before processing by the relay network. The relay network may include: an encoding section for generating a coarse feature map from the outputs derived from the first input branch, the second input branch, and the third input branch through multiple convolutional layers; and a decoding section for generating the three-dimensional flow of the object between the first two-dimensional image and the second two-dimensional image from the coarse feature map through multiple deconvolutional layers.

[0009] The method may further include: generating the two-dimensional representation of the estimated three-dimensional shape of the object from the first image using another neural network before inputting the two-dimensional representation of the estimated three-dimensional shape of the object into the convolutional neural network. Generating the two-dimensional representation of the estimated three-dimensional shape of the object from the first image may include: generating the estimated three-dimensional shape of the object from the first image using the other neural network; encoding the three-dimensional coordinates of each point in the estimated three-dimensional shape in the two-dimensional image to generate the two-dimensional representation of the estimated three-dimensional shape of the object.

[0010] The object can be a human face.

[0011] According to another aspect of this specification, a computer-implemented method is described for training a convolutional neural network to estimate a three-dimensional flow of an object between a first two-dimensional image and a second two-dimensional image, the first two-dimensional image comprising an image of the object in a first configuration, and the second two-dimensional image comprising an image of the object in a second configuration. The method includes: inputting the first two-dimensional image, the second two-dimensional image, and a two-dimensional representation of an estimated three-dimensional shape of the object into the convolutional neural network; generating the three-dimensional flow of the object between the first two-dimensional image and the second two-dimensional image using the convolutional neural network from the first two-dimensional image, the second two-dimensional image, and the two-dimensional representation of the estimated three-dimensional shape of the object; comparing the generated three-dimensional flow with a ground truth three-dimensional flow between the first two-dimensional image and the second two-dimensional image using an objective function; and updating the parameters of the convolutional neural network based on the comparison.

[0012] The objective function may include an endpoint error term based on the difference between the generated 3D stream and the ground truth 3D stream between the first 2D image and the second 2D image. Generating the 3D stream of the object between the first 2D image and the second 2D image using the convolutional neural network may include generating a series of 3D streams at different resolutions. The endpoint error term may also compare the generated series of 3D streams at different resolutions with a corresponding series of ground truth 3D streams at different resolutions. The series of ground truth 3D streams at different resolutions may be derived from the ground truth 3D stream between the first 2D image and the second 2D image through downsampling or other methods.

[0013] The objective function may include a color consistency error, which compares pixel values ​​sampled from the second 2D image with pixel values ​​in the first 2D image. The pixel values ​​in the second 2D image can be sampled using the generated 3D flow of the object between the first and second 2D images, based on the warping of the 2D representation of the estimated 3D shape of the object.

[0014] The two-dimensional representation of the estimated three-dimensional shape of the object may include a color image. The pixels of the color image may encode the three-dimensional coordinates of points in the estimated three-dimensional shape of the object, each point corresponding to a pixel of the object in the first two-dimensional image.

[0015] The three-dimensional flow of the object between the first two-dimensional image and the second two-dimensional image may include a color image, which encodes the three-dimensional flow of pixels between the first two-dimensional image and the second two-dimensional image.

[0016] The convolutional neural network may include: a first input branch for processing the first two-dimensional image through one or more convolutional layers; a second input branch for processing the second two-dimensional image through one or more convolutional layers; a third input branch for processing the two-dimensional representation of the estimated three-dimensional shape of the object through one or more convolutional layers; and a relay network for jointly processing the outputs derived from the first input branch, the second input branch, and the third input branch through multiple convolutional layers.

[0017] The method may further include: generating the two-dimensional representation of the estimated three-dimensional shape of the object from the first image using another neural network before inputting the two-dimensional representation of the estimated three-dimensional shape of the object into the convolutional neural network.

[0018] The object can be a human face.

[0019] According to another aspect of this specification, a computer-implemented method is described for estimating a three-dimensional flow of an object between a first two-dimensional image and a second two-dimensional image, the first two-dimensional image comprising an image of the object in a first configuration and the second two-dimensional image comprising an image of the object in a second configuration; the method includes using a neural network trained using any of the training methods disclosed herein.

[0020] According to another aspect of this specification, a computer-implemented method is described for generating a training dataset for training a model to estimate a three-dimensional flow of an object between a first two-dimensional image and a second two-dimensional image, the first two-dimensional image including an image of the object in a first configuration, and the second two-dimensional image including an image of the object in a second configuration. The method includes: extracting a plurality of frames from a video data segment, each frame including an image of an object of a predefined type; for each extracted frame, fitting a three-dimensional model of the predefined type of object onto the image of the object in the frame; for multiple pairs of frames, generating the three-dimensional flow from the corresponding fitted three-dimensional model of the object for each frame in the pair; and storing in memory each of the extracted frames and the generated three-dimensional flow, as well as the correlation between the generated three-dimensional flow and the corresponding pair of extracted frames from which it is generated.

[0021] Each of the three-dimensional streams may include a color image, wherein the pixels of the color image encode a point in the first frame of the pair to the three-dimensional stream of the second frame of the pair.

[0022] According to another aspect of this specification, a computer-implemented method is described for training a convolutional neural network to estimate a three-dimensional flow of an object between a first two-dimensional image and a second two-dimensional image, the first two-dimensional image comprising an image of the object in a first configuration and the second two-dimensional image comprising an image of the object in a second configuration; the method includes generating a dataset using any of the dataset creation methods disclosed herein.

[0023] According to another aspect of this specification, a system is described that includes a memory and one or more processors, the memory including computer-readable code that, when executed by the one or more processors, causes the system to perform any of the methods disclosed herein.

[0024] According to another aspect of this specification, a computer program product is described, the computer program product including computer-readable code that, when executed by a computing device, causes the computing device to perform any of the methods disclosed herein. Attached Figure Description

[0025] The embodiments will now be described by way of non-limiting examples with reference to the following figures, wherein:

[0026] Figure 1 An overview diagram showing an example of an estimate for generating a 3D flow between two images;

[0027] Figure 2 A flowchart illustrating an exemplary method for generating an estimate of a 3D flow between two images is shown.

[0028] Figure 3 An overview diagram of another example of an estimate used to generate a 3D flow between two images is shown;

[0029] Figure 4 A flowchart is shown for another exemplary method for generating an estimate of a 3D flow between two images;

[0030] Figure 5 An example of the structure of a neural network for estimating the 3D flow between two images is shown;

[0031] Figure 6 Another example of the structure of a neural network for estimating the 3D flow between two images is shown;

[0032] Figure 7 A schematic overview of a method 700 for training a neural network to estimate a three-dimensional flow between two images is shown.

[0033] Figure 8 An example of a training schedule 800 for training a neural network to estimate a 3D flow between two images is shown;

[0034] Figure 9 A flowchart of an exemplary method 900 for training a neural network to estimate a three-dimensional flow between two images is shown;

[0035] Figure 10 An example of a method 1000 for generating ground truth training data for training a neural network to estimate the 3D flow of an object between two images is shown;

[0036] Figure 11 An example of a method 1100 for training a neural network to generate the 3D shape of an object from a 2D image is shown;

[0037] Figure 12 A schematic diagram of a system / apparatus for performing any of the methods described herein is shown. Detailed Implementation

[0038] The methods and systems described in this paper seek to estimate the three-dimensional (3D) motion field of an object from multiple pairs of monocular two-dimensional (2D) images containing the object, i.e., to perform scene flow estimation. A convolutional neural network is used to estimate the 3D flow of the object from a pair of 2D images, each 2D image including images of the object in different configurations (e.g., pose, position / location, expression, orientation, etc., which may also be collectively referred to as "capture settings") and an estimated 3D shape of the object derived from one of the images.

[0039] In the following description and illustration, exemplary methods and systems are used in conjunction with facial images (i.e., images including one or more human faces). However, it should be understood that the methods and systems can be alternatively applied to other objects in two-dimensional images, such as vehicles, human bodies, buildings, landscapes, animals, and / or natural phenomena.

[0040] Figure 1 An overview diagram of an exemplary method 100 for generating an estimate of a 3D flow of an object between two images is shown. A convolutional neural network 102 receives as input a first 2D image 104, a second 2D image 106, and a 2D representation of the estimated 3D shape of an object 108. The first 2D image 104 includes an image of the object in a first configuration (in this example, a face image with a first expression), and the second 2D image 106 includes an image of the object in a second configuration (in this example, a face image with a second expression). The convolutional neural network 102 processes the input through multiple neural network layers to generate an estimated 3D flow 110 of the object between the first 2D image 104 and the second 2D image 106. In addition to the two 2D images (104, 106), by using the estimated 3D shape of the object 108 as input, the method can produce an accurate 3D flow between images without requiring stereoscopic or RGB-D images that are difficult or impossible to capture in natural environments. By including such images, the CNN can better correlate each pixel value in the first 2D image 104 with the corresponding point in 3D space, thereby providing better initialization for the network in the problem space and establishing a reference 3D mesh that helps to perform warping in 3D space during training / inference.

[0041] Each of the first 2D image (I1, 104) and the second 2D image (I2, 106) includes a set of pixel values ​​in a 2D array. For example, in a color image H is the height of the image (in pixels), W is the width of the image (in pixels), and the image has three color channels (e.g., RGB or CIELAB). Alternatively, the first 2D image (I1, 104) and the second 2D image (I2, 106) can be black and white images or grayscale images. The first 2D image (I1, 104) and the second 2D image (I2, 106) can be the same size.

[0042] Each of the first 2D image (I1, 104) and the second 2D image (I2, 106) contains an image of an object (e.g., a face) moving between the images (i.e., in different configurations). The images (104, 106) may represent images of objects captured at different times, from different locations, and / or from different orientations. For example, the images (104, 106) may be video data frames captured at different times. The first 2D image 104 may precede the second 2D image 106, and vice versa.

[0043] In embodiments where the object is a face, the image of the face in the first image 104 may have a first expression, pose, and / or orientation, etc. In the example shown, the image of the face in the first image 104 is smiling. The image of the face in the second image 106 may have a second expression, pose, and / or orientation, etc. Typically, this may differ from the image of the face in the first image. In the example shown, the image of the face in the second image 106 is laughing.

[0044] The 2D representation of the estimated 3D shape of the object (108, I3) can be represented as a color image. The pixel color values ​​in the image represent the 3D coordinates of the corresponding pixels on the surface of the object visible in the first 2D image 104. An example of such a representation is "Projected Normalized Coordinates Code (PNCC)" (hereinafter referred to as...). Figure 3 (To be described in more detail), however, other representations may be used alternatively. Examples of such representations include a UV-space representation, which flattens the cylindrical or spherical surface of a 3D face manifold into a continuous 2D atlas, where the visible vertices of the face in I1 are set to their x, y, z coordinates in the UV image, and the invisible vertices are set to zero.

[0045] In fact, the 2D representation of the estimated 3D shape of the object 108 is encoded by normalized x, y, z coordinates of the object vertices visible from each corresponding pixel in the first 2D image 104, and can be based on the camera that captures the viewpoint of the first image.

[0046] The convolutional neural network (CNN) 102 includes multiple layers of nodes, each node associated with one or more parameters. The parameters of each node in the neural network 102 may include one or more weights and / or thresholds. Each node takes one or more outputs from nodes in the previous layer as input. The one or more outputs from nodes in the previous layer are used by the node to generate activation values ​​using an activation function and the parameters of the CNN. One or more of the layers in the CNN 102 are convolutional layers, each layer being used to apply one or more convolutional filters. The following describes... Figure 5 and Figure 6 An example describing the CNN 102 architecture is provided. The CNN 102 can use the U-Net architecture with skip connections.

[0047] In some embodiments, the CNN 102 includes an encoder section and a decoder section. The encoder section takes as input the 2D representations of the estimated 3D shapes of the first 2D image 104, the second 2D image 106, and the object 108, and generates their encoded representations. The encoder section may include one or more downsampling layers. The decoder section takes as input the encoded representations of the 2D representations of the estimated 3D shapes of the first 2D image 104, the second 2D image 106, and the object 108, and generates, based on them, the estimated 3D stream 110 of the object between the first 2D image 104 and the second 2D image 106. The decoder section may include one or more upsampling layers.

[0048] The 3D flow 110 between the first 2D image 104 and the second 2D image 106 can be represented as a 2D image. The 3D optical flow between the first 2D image 104 and the second 2D image 106 is encoded. The 3D flow 110 can be encoded pixel-by-pixel, meaning the pixel-by-pixel optical flow between the first 2D image 104 and the second 2D image 106 is encoded as pixel color values ​​in F. In other words, the color value of each pixel in F encodes the corresponding pixel in the 3D flow vector from I1 to I2.

[0049] Considering the 2D representation of the estimated 3D shape of the first 2D image (104, I1), the second 2D image (106, I2), and the object (108, I3), the 3D flow estimation problem of a color image can be represented notally as a mapping. .

[0050] Figure 2A flowchart of an exemplary method 200 for estimating a 3D flow of an object between two images is shown. The method can be implemented on a computer.

[0051] In operation 2.1, a first 2D image, a second 2D image, and a 2D representation of the estimated 3D shape of an object in the first image are input into a CNN. The object may be a human face.

[0052] The 2D representation of the estimated 3D shape of the object may include a color image. The pixels of the color image may encode the 3D coordinates of points in the estimated 3D shape of the object, each point corresponding to a pixel of the object in either the first or second 2D image. The 2D representation of the estimated 3D shape of the object may use PNCC, however, other representations may also be used.

[0053] In operation 2.2, the CNN is used to generate a 3D flow of the object between the first 2D image and the second 2D image from the first 2D image, the second 2D image, and the 2D representation of the estimated 3D shape of the object. The 3D flow of the object between the first 2D image and the second 2D image may include a color image that encodes the 3D flow of pixels between the first 2D image and the second 2D image.

[0054] Figure 3 An overview diagram of another example of a method 300 for estimating a 3D flow between two images is shown. This method 300 is combined with the above. Figure 1 The described method is similar, but includes a preprocessing step 312, in which a 2D representation of the estimated 3D shape of object 308 is generated from a first 2D image 304. Following the preprocessing step 312, the method is as follows: Figure 1 As shown, the convolutional neural network 302 receives as input a 2D representation of the estimated 3D shape of a first 2D image 304, a second 2D image 306, and an object 308. The first 2D image 304 includes an image of the object in a first configuration (in this example, a face image with a first expression), and the second 2D image 306 includes an image of the object in a second configuration (in this example, a face image with a second expression). The convolutional neural network 302 processes the input through multiple neural network layers to generate an estimated 3D flow 310 of the object between the first 2D image 304 and the second 2D image 306.

[0055] The purpose of the preprocessing step 312 is to densely regress the 3D geometry of the object in the first 2D image 104. A 3D shape 314 is generated from the first 2D image 304, wherein the 2D representation of the estimated 3D shape of the object 308 is generated from the first 2D image 304 using a projection model 316 (e.g., PNCC).

[0056] In the illustrated embodiment, the preprocessing step uses a pre-trained neural network 318 to generate the 3D shape (314, 304, 11) of the object in the first 2D image (304, 11). (i.e., an N-vertex mesh). The pre-trained neural network 318 may be a convolutional neural network. The pre-trained neural network 318 regresses a dense 3D mesh (314, S) by estimating the 3D coordinates of each vertex.

[0057] In some embodiments, the neural network is a convolutional neural network with one or more skip connections between layers. The network may include one or more fully connected layers. For example, the last layer may be a fully connected layer. An example of such a network architecture is ResNet-50, where the last layer is replaced by a convolutional layer, followed by a fully connected layer.

[0058] Although the illustrated embodiment uses a neural network 318 to regress the dense 3D mesh (314, S), it should be understood that other methods for fitting the 3D mesh to an image can be used alternatively, such as iterative fitting of a 3D deformable model (3DMM).

[0059] The projection model 316 takes the 3D mesh 314 as input and generates the 2D representation of the estimated 3D shape of the object 308 from the 3D mesh 314. The projection model may also take one or more camera parameters c as input, which encode attributes of the camera capturing the first 2D image 304. For example, the camera parameters may include rotation angle R, translation t, etc. 3D and / or scaling parameter f c In some embodiments, a specific camera model may be assumed, such as scaled orthographic projection.

[0060] The projection model 316 can use Projected Normalized Coordinates Code (PNCC). The PNCC encodes the normalized x, y, z coordinates of the vertices of the 3D mesh 314 visible from each corresponding pixel in the first 2D image 304 according to the camera's viewpoint. Considering the 3D mesh (314, S), the 2D representation of the estimated 3D shape of the object (308, I3) can be determined using the following equation:

[0061]

[0062] in, It is a function that renders a normalized version of S. These are the camera parameters (i.e., rotation angle R, translation t). 3D and scaling f c P is the diagonal with main diagonal (f c / W、f c / H、f c The first 2D image is a 3x3 diagonal matrix (I1, I2), where W is the width, H is the height, and D is the maximum z-value. M is a set of foreground pixels in the first 2D image (I1, I2). In this document, S is represented as a matrix storing the columnar x, y, and z coordinates of the N vertices of the 3D mesh 314. The visible triangle of pixel (x, y) in I1 is represented as... ,in,( This can be detected using a hardware-based renderer. b is the projected triangle t located on the first 2D image (302, I1). j The centroid coordinates of the pixel (x, y) in the image.

[0063] Multiplying by P first uses f in the image space coordinates. c The 3D mesh 314 is scaled and then normalized using the width and height of the rendered image size and the maximum z-value D calculated based on the entire dataset used for training. This gives the image 306 RGB channels that store the normalized (i.e., in the range [0, 1]) xyz coordinates of the corresponding 3D mesh 314.

[0064] Figure 4 A flowchart of another exemplary method 400 for generating an estimate of a 3D flow between two images is shown. This method can be implemented on a computer.

[0065] In operation 4.1, another neural network is used to generate an estimated 3D shape of the object from the first image.

[0066] In operation 4.2, the three-dimensional coordinates of each point in the estimated three-dimensional shape in the two-dimensional image are encoded to generate the two-dimensional representation of the estimated three-dimensional shape of the object.

[0067] Operations 4.3 and 4.4 are respectively combined as follows Figure 2 The operations described in shown in steps 2.1 and 2.2 are carried out.

[0068] Figure 5 An example of the structure of a neural network 500 for estimating a 3D flow between two images is shown. The neural network 500 includes an encoder section 512 and a decoder section 514. The encoder section 512 takes as input 2D representations of the estimated 3D shapes of a first 2D image 504, a second 2D image 506, and an object 508, and generates their encoded representations 502. The decoder section 514 takes the encoded representations 502 (also called embeddings) as input and generates from them an estimated 3D flow 510 of the object.

[0069] In some embodiments, the encoder portion 512 includes a plurality of input branches 516. The plurality of input branches 516 include: a first input branch for processing the first 2D image 504 through one or more convolutional layers; a second input branch for processing the second 2D image 506 through one or more convolutional layers; and a third input branch for processing the 2D representation of the estimated three-dimensional shape of the object 508 through one or more convolutional layers. Each of the plurality of input branches 516 may include one or more downsampling layers for downsampling its input. Each of one or more levels of the input branches 516 may generate an intermediate feature map, which may be stored for use in the decoder portion 514 of the neural network 500.

[0070] In some embodiments, the plurality of input branches 516 include an association layer for associating the outputs of the first branch and the second branch. The association layer can be used to convolve data patches from the output of the first branch with data patches from the output of the second branch. Therefore, the association layer has no learnable weights. To avoid convolving all patches from the output of the first branch with all data patches from the output of the second branch, a maximum distance can be set for the comparison.

[0071] The encoder portion 512 may further include a relay network 518 (also referred to as a “relay line”). The relay network 518 receives input derived from the plurality of input branches 516 and processes the input jointly through a plurality of convolutional layers to generate an encoded representation 502. The relay network 518 may include one or more downsampling layers for downsampling its input. Each of one or more stages of the relay network 518 may generate an intermediate feature map, which may be stored for use in the decoder portion 514 of the neural network 500.

[0072] The decoder section 514 takes the encoded representation 502 as input and processes the input through multiple deconvolutional layers to generate the estimated 3D stream 510 of the object. The stream 510 can be estimated from the coarsest level to full resolution. In the input layer of the decoder section 514, one or more deconvolutional filters can be applied to the encoded representation 502 to generate a set of initial feature maps and a first estimated stream. At each deconvolutional layer after the initial layer, the decoder section 514 can take the current estimated stream (or an upsampled version of the current estimated stream) and the feature map output from the previous deconvolutional layer as input and output a set of feature maps. Furthermore, the current estimated stream can also be generated. Additionally, the corresponding intermediate feature maps from the encoder section 512 can be input to each deconvolutional layer of the decoder section 514 after the initial layer. Deconvolution can be implemented using fractional-step convolutions.

[0073] Figure 6 A detailed example of the structure of a neural network 600 for estimating the 3D flow between two images is shown. Figure 5 As shown, the neural network 600 includes an encoder section 612 and a decoder section 614. The encoder section 612 takes as input the 2D representations of the estimated 3D shape of the first 2D image 604, the second 2D image 606, and the object 608, and generates their encoded representations 602. The decoder section 614 takes the encoded representations 602 as input and generates an estimated 3D stream 610 of the object from it.

[0074] In the following description, specific dimensions of the input image, multiple sets of feature maps, and / or convolutional kernel sizes will be described. However, it should be understood that other examples of these numbers can be used alternatively. In application, downsampling is performed using a stride of length 2, although in other embodiments, downsampling can be performed using other strides greater than 1 (e.g., strides of length 3, 4, etc.). ReLU layers are used to implement non-linearity, but other types of non-linear activations can be used alternatively.

[0075] The encoder portion 612 includes three input branches 516. The first input branch is used to process the first 2D image 504 through one or more convolutional layers; the second input branch is used to process the second 2D image 506 through one or more convolutional layers; and the third input branch is used to process the estimated 2D representation of the three-dimensional shape of the object 508 through one or more convolutional layers.

[0076] In input section 616, each input branch takes its corresponding 2D image (604, 606, 608) with dimensions of 224x224x3 as input and applies a set of initial convolutional filters (denoted as "conv1") to it to generate a corresponding first set of feature maps. The first set of feature maps has dimensions of 224x224x64, and each of the initial convolutional filters (conv1) has a kernel size of 5x5 and a stride of 2. A second set of convolutional filters (denoted as "conv2") with a kernel size of 5x5 and a stride of 2 is applied to each of the first set of feature maps to generate a corresponding second set of feature maps, where each feature map has dimensions of 112x112x128. A third set of convolutional filters (denoted as "conv3") with a kernel size of 5x5 and a stride of 2 is applied to each of the second set of feature maps to generate a corresponding third set of feature maps, where each feature map has dimensions of 56x56x256.

[0077] An association layer (denoted as "corr") is applied between the third set of feature maps in the first input branch and the third set of feature maps in the second input branch. An example of such an association layer is described in "FlowNet: Learning Optical Flow with Convolutional Networks" (P. Fischer et al., Proceedings of the IEEE International Conference on Computer Vision, pp. 2758-2766, 2015), the contents of which are incorporated herein by reference. The association layer produces an output of dimension 56x56x441. A neighborhood search size of 2x(21)+1 pixels can be used as the maximum distance (neighborhood) of the correlation, however, other sizes can be used alternatively.

[0078] Additionally, a set of convolutional filters (denoted as "conv") with a kernel size of 1x1 and a stride of 1 are applied to the third set of feature maps in the first input branch to generate an output with a dimension of 56x56x32. This same set of convolutional filters (denoted as "conv") with a kernel size of 1x1 and a stride of 1 are also applied to the third set of feature maps in the first input branch to generate another output with a dimension of 56x56x32. These outputs are then concatenated with the output of the associated layer to generate a fourth set of feature maps with a dimension of 56x56x(441+32+32).

[0079] The next convolutional layer (denoted as "conv3_1") applies a set of convolutional filters with a kernel size of 3x3 and a stride of 1 to the fourth set of feature maps to generate a fifth set of feature maps with dimensions of 56x56x256. The fifth set of feature maps is then fed into the next convolutional layer (denoted as "conv4") with a kernel size of 3x3 and a stride of 2 to generate a sixth set of feature maps with dimensions of 28x28x512. Another convolutional layer (denoted as "conv4_1") applies a set of convolutional filters with a kernel size of 3x3 and a stride of 1 to the sixth set of feature maps to generate a seventh set of feature maps with dimensions of 28x28x512. The next convolutional layer (denoted as "conv5") applies a set of convolutional filters with a kernel size of 3x3 and a stride of 2 to the sixth set of feature maps to generate an eighth set of feature maps with dimensions of 14x14x512. The next convolutional layer (denoted as "conv5_1") applies a set of convolutional filters with a kernel size of 3x3 and a stride of 1 to the eighth set of feature maps to generate a ninth set of feature maps with a dimension of 14x14xx512.

[0080] The last layer (referred to as "conv6") of the encoder portion 618 applies a set of convolutional filters with a kernel size of 3x3 and a stride of 2 to the ninth set of feature maps to generate the encoded representation 602 with a dimension of 7x7x1024.

[0081] The decoder portion 614 includes multiple convolutional layers, each of which takes as input a set of feature maps from the relay / input branch, a set of deconvolutional feature maps, and (if available) a corresponding concatenation of the current flow estimate. Each set of convolutional layers outputs a deconvolutional feature map and an updated current flow estimate.

[0082] The first layer of the decoder portion 614 of the network applies an initial deconvolution filter (denoted as "deconv5") to the embedded representation 602 to generate a first set of deconvolutional feature maps with dimensions of 14x14x512. In some embodiments, an initial flow estimate with dimensions of 14x14x3 can be estimated from the encoded representation 602. One or more convolutional layers can be applied to the encoded representation 602 to generate the initial flow estimate.

[0083] The first set of deconvolutional feature maps is concatenated with the ninth set of feature maps in the relay branch and (if available) the initial flow estimate to generate a tensor of dimension 14x14x(512+512+3). A deconvolutional layer (denoted as "deconv4") is applied to this tensor to generate a second set of deconvolutional feature maps of dimension 28x28x256. Furthermore, an updated current flow estimate of dimension 14x14x3 (denoted as "flow5") can also be generated. The updated current flow estimate can be generated by applying one or more convolutional layers to the concatenation of the initial flow estimate, the ninth set of feature maps in the relay branch, and the first set of deconvolutional feature maps.

[0084] The second set of deconvolutional feature maps is concatenated with the seventh set of feature maps in the relay branch and (if available) an amplified current flow estimate to generate a tensor of dimension 28x28x(256+512+3). A deconvolutional layer (denoted as "deconv3") is applied to this tensor to generate a third set of deconvolutional feature maps of dimension 56x56x128. Furthermore, an updated current flow estimate of dimension 28x28x3 (denoted as "flow4") can also be generated. The updated current flow estimate can be generated by applying one or more convolutional layers to the concatenation of the amplified current flow estimate, the seventh set of feature maps in the relay branch, and the second set of deconvolutional feature maps.

[0085] The third set of deconvolutional feature maps is concatenated with the fifth set of feature maps in the relay branch and (if available) an amplified current flow estimate to generate a tensor of dimension 56x56x(128+256+3). A deconvolutional layer (denoted as "deconv2") is applied to this tensor to generate a fourth set of deconvolutional feature maps of dimension 112x112x64. Furthermore, an updated current flow estimate of dimension 56x56x3 (denoted as "flow3") can also be generated. The updated current flow estimate can be generated by applying one or more convolutional layers to the concatenation of the amplified current flow estimate, the fifth set of feature maps in the relay branch, and the third set of deconvolutional feature maps.

[0086] The fourth set of deconvolutional feature maps is concatenated with the second set of feature maps in the input branch (i.e., the second set of feature maps in the first input branch, the second set of feature maps in the second input branch, or the second set of feature maps in the third input branch) and (if available) an amplified current flow estimate to generate a tensor of dimension 112x112x(64+128+3). A deconvolutional layer (denoted as "deconv1") is applied to this tensor to generate a fifth set of deconvolutional feature maps of dimension 224x224x64. Additionally, an updated current flow estimate of dimension 112x112x3 (denoted as "flow2") can be generated. The updated current flow estimate can be generated by applying one or more convolutional layers to the concatenation of the amplified current flow estimate, the second set of feature maps in the input branch, and the fourth set of deconvolutional feature maps.

[0087] The fifth set of deconvolutional feature maps is concatenated with the first set of feature maps in the input branch (i.e., the first set of feature maps in the first input branch, the first set of feature maps in the second input branch, or the first set of feature maps in the third input branch) and (if available) the amplified current flow estimate to generate a tensor of dimension 224x224x(64+64+3). A final flow estimate 610 (denoted as "flow1") of dimension 224x224x3 is generated from this tensor. The final flow estimate can be generated by applying one or more convolutional layers to the concatenation of the amplified current flow estimate, the first set of feature maps in the input branch, and the fifth set of deconvolutional feature maps.

[0088] Figure 7A schematic overview of a method 700 for training a neural network to estimate a 3D flow between two images is shown. A convolutional neural network 702 receives as input a first 2D image 704, a second 2D image 706, and an estimated 3D shape 2D representation of an object 708. The first 2D image 704 includes an image of the object in a first configuration (in this example, a face image with a first expression), and the second 2D image 706 includes an image of the object in a second configuration (in this example, a face image with a second expression). The first image 704 and the second image 706 form a training pair 712 obtained from a training set and are associated with a known 3D flow 314 (also referred to herein as a “ground truth 3D flow”) between them. The convolutional neural network 702 processes the input through multiple neural network layers to generate an estimated 3D flow 710 of the object between the first 2D image 704 and the second 2D image 706. The estimated 3D flow 710 is compared to the known 3D flow 714 using an objective function 716 (also referred to herein as a loss function). The objective function 716 is used to determine one or more parameter updates 718 of the neural network 702, for example, using an optimization process. The method can be iterated until a threshold condition is met, thus completing a threshold number of training cycles.

[0089] The training pairs 712 of images can be obtained from a batch of training data. In some embodiments, the images in the pairs can be normalized within the range [0,1]. The batch of training data includes one or more training pairs, typically multiple training pairs, which are subsets of the entire training dataset. For example, the batch size can be in the range [10,100], preferably in the range [10,30]. For example, the batch size can be 16. Multiple batches cover the entire training dataset. The parameter update 718 of the neural network can be determined after each batch has been fully processed, i.e., after the objective function for the values ​​of each training pair 712 in the batch has been determined. The training method 700 can iterate the training dataset multiple times, where each forward and backward pass defines a training cycle.

[0090] An optimization process can be applied to the objective function to determine the parameter updates. Examples of such optimization processes are stochastic gradient descent and the Adam optimization algorithm (e.g., parameter optimization). and However, any suitable optimization process can be used alternatively. This optimization process can be related to the learning rate that controls the size of the parameter updates (718). The learning rate may vary throughout the training process, for example, in the following combination: Figure 8 As mentioned above.

[0091] The objective function 716 may include an endpoint error term. The endpoint error term compares the estimated 3D flow 710 between the first 2D image 704 and the 2D image 706 with the ground truth 3D flow 714 between the first 2D image 704 and the 2D image 706. The endpoint error term may include the Frobenius norm error between the estimated 3D flow 710 and the ground truth 3D flow 714, i.e.

[0092]

[0093] Among them, F GT Here, F(Ψ) is the ground truth 3D flow 714, F(Ψ) is the estimated 3D flow 710, and w is a hyperparameter controlling the relative importance of the endpoint error term 716 in the objective function 716. However, it should be understood that other norms, such as L, can be used alternatively. 2,1 norm or L p,q Norm.

[0094] In an embodiment where the neural network 702 generates a series of resolution estimation streams, for example, by combining... Figure 5 and Figure 6 The described embodiment, the endpoint error term This can include a weighted sum of endpoint errors for each resolution. For example, the endpoint errors can be derived using the following equation:

[0095]

[0096] Among them, F i GT It is the ground truth 3D stream with resolution i, F i (Ψ) is the estimated 3D flow 710 at resolution i, w i These are hyperparameters that control the relative importance of each resolution in the objective function 716. The resolutions are indexed by integer labels i, starting from the lowest resolution 1 (e.g., in...). Figure 6 In the example, it is 14x14x3) to the resolution L of the input image (e.g., in Figure 6 In the example, it is 224x224x3). As mentioned above, it should be understood that other norms can be used instead of the Frobenius norm, such as L. 2,1 norm or L p,q Norm. The ground truth 3D stream 714 at resolution i can be generated by downsampling the ground truth 3D stream 714 at the resolution of the input image.

[0097] For each resolution, the hyperparameter w iThe values ​​of can be the same. Alternatively, the hyperparameter w i One or more of the parameters can be combined with other hyperparameters w. i One or more of the hyperparameters have different values. The value of each hyperparameter can be in the range [0.1, 10], for example, in the range [0.5, 2]. In some embodiments, the hyperparameter w... i All of them can be equal to 1.

[0098] Alternatively or additionally, the objective function 716 may include a color consistency error term. The color consistency error term compares the pixel values ​​sampled from the second 2D image 706 with the pixel values ​​in the first 2D image 704. Using the estimated 3D stream 710, the pixel values ​​in the second 2D image are sampled based on the warping of the 2D representation of the estimated three-dimensional shape of the object 708. For example, using the estimated 3D stream 310, the 2D representation of the estimated three-dimensional shape of the object 708 is warped, and the second 2D image 706 is sampled at the vertices of the obtained 3D shape. These pixel values ​​are then compared with the corresponding pixel values ​​in the first 2D image 704.

[0099] An example of the color consistency error term is given by the following equation:

[0100]

[0101] Where F is the estimated 3D flow, I1 is the first 2D image 704, I2 is the second 2D image 706, and I3 is the 2D representation of the estimated three-dimensional shape of the object 708. α is a hyperparameter controlling the relative importance of the color consistency error term in the objective function 716. α can be in the range [1, 100], preferably in the range [5, 20]. For example, α can be equal to 10. It is a function that warps the 3D shape of I1 encoded within the 2D representation 708 using the estimated flow F, and samples I2 at the vertices of the resulting projected 3D shape. The warping function can be implemented by using a differentiable layer that detects any occlusion through the 3D flow, and sampling the second 2D image 706 in a differentiable manner at the output stage of the neural network 702 (i.e., reverse warping). It should be understood that other norms can be used instead of the Frobenius norm, such as L... 2,1 norm or L p,q Norm.

[0102] Figure 8An example of a training schedule 800 for training a neural network to estimate a 3D flow between two images is shown. The training schedule 800 illustrates an example of how the learning rate changes during training. The y-axis represents the learning rate 802. The x-axis represents time 804, measured in training cycles. In the example shown, training lasts for 40 cycles.

[0103] Within the first number of cycles 806 (the first 10 cycles in this example), at the first learning rate (10x10 in this example). -5 Training is then performed. Then, within a second number of cycles 808 (a further 10 cycles in this example), the learning rate decreases to a second learning rate (5x10 in this example). -5 Then, within the third number of cycles 810 (a further 10 cycles in this example), the learning rate is increased to a third learning rate (in this example, restored to 10x10). -5 In some embodiments, the third learning rate may be the same as the first learning rate. Alternatively, the third learning rate may be different from the first learning rate. Then, within a fourth number of cycles 812 (5 cycles in this example), the learning rate decreases again to a fourth learning rate (1x10 in this example). -5 Then it drops again to the fifth learning rate (0.1x10 in this example). -5 (until training 814 ends).

[0104] Figure 9 A flowchart of an exemplary method 900 for training a neural network to estimate a three-dimensional flow between two images is shown.

[0105] In operation 9.1, a first 2D image, a second 2D image, and a 2D representation of the estimated 3D shape of an object in the first image are input into the CNN. The object can be a face. Operation 9.1 can be combined with the above. Figure 2 The operation described is the same as in 2.1.

[0106] The first 2D image and the second 2D image can be selected from a training dataset, which includes multiple image pairs, each image pair being associated with a ground truth 3D flow between them.

[0107] In operation 9.2, the CNN is used to generate a 3D flow of the object between the first 2D image and the second 2D image from the first 2D image, the second 2D image, and the estimated 3D shape of the object. Operation 9.2 can be combined with the above. Figure 2 The operation described is the same as in 2.2.

[0108] In operation 9.3, the generated 3D stream and the ground truth 3D stream between the first 2D image and the second 2D image are compared using an objective function.

[0109] The objective function may include an endpoint error term based on the difference between the generated 3D stream and the ground truth 3D stream between the first 2D image and the second 2D image. The endpoint error term may also compare the generated series of 3D streams at different resolutions with a corresponding series of ground truth 3D streams at different resolutions. The series of ground truth 3D streams at different resolutions may be derived from the ground truth 3D stream between the first 2D image and the second 2D image through downsampling or similar methods.

[0110] The objective function may include a color consistency error, which compares pixel values ​​sampled from the second two-dimensional image with pixel values ​​in the first two-dimensional image, wherein the pixel values ​​in the second two-dimensional image are sampled based on the warping of the two-dimensional representation of the estimated three-dimensional shape of the object, using the generated three-dimensional flow of the object between the first and second two-dimensional images.

[0111] You can iterate through operations 9.1 to 9.3 on the training batch, and then go to operation 9.4.

[0112] In operation 9.4, the parameters of the neural network can be updated based on the comparison. An optimization process can be used to determine the parameter update. The parameter update can be based on the evaluation of all objective functions performed in the training batch. For example, the update can be determined based on the average / sum of the objective functions sampled on the corresponding training batch.

[0113] Operations 9.1 through 9.4 can be iterated on the training dataset until a threshold condition is met. The threshold condition can be the number of training cycles required.

[0114] Figure 10An example of a method 1000 for generating ground truth training data for training a neural network to estimate a 3D flow of an object between two images is illustrated. A sequence of 2D images 1002a-c is extracted from each of one or more sets of video data. For each image 1002a-c, a 3D morphable model (3DMM) 1004a-c of a given type of object is fitted to the image. Multiple pairs of images 1006 are selected from the dataset, and the corresponding 3DMMs of the multiple pairs of images are compared to generate a ground truth (or “pseudo-ground truth”) 3D flow 1008 between the multiple pairs of images. It should be understood that other methods for generating ground truth 3D flows of objects between images can be used alternatively to generate suitable training sets for the training method described herein.

[0115] The 3DMM can use additive combinations of facial expressions and identity changes. The object shape (e.g., face shape) can be represented as including... Vectorized 3D shape of each vertex . It can be parameterized as:

[0116]

[0117] in, It is the average shape vector. It has n i There are n principal components (where n is a principal component). i <<3N) identity basis, It has n e There are n principal components (where n is a principal component). e <<3N) of the expression base, and These are the identity parameter and the expression parameter. For facial images, the identity component can be derived from a large-scale facial model, but alternatively, other facial models can be used. For example, the expression component can be derived from the model described in "The 3d menpofacial landmark tracking challenge" (S. Zafeiriou et al., Proceedings of the IEEE International Conference on Computer Vision, pp. 2503–2511, 2017).

[0118] The 3DMM can be fitted to multiple object (e.g., face) markers in an image. A batch-based approach can be used to simultaneously consider information from multiple video frames and leverage the large amount of dynamic information typically contained in the object (e.g., face) video. The batch-based approach can be energy minimization to simultaneously fit a combined identity and expression 3DMM model to the facial markers of the multiple frames in the input video.

[0119] For example, to generate a facial training dataset, a collection of 12,000 RGB videos with a total of 19 million frames and 2,500 unique identities can be used. The aforementioned 3D reconstruction method, along with some video trimming steps, is applied to prevent automatic estimation failures.

[0120] To determine the ground truth 3D flow 1008 given a pair of images I1 and I2 with corresponding 3D shapes S1 and S2, the following equation can be used to create the 3D flow graph F:

[0121]

[0122] Where M is a set of foreground pixels in I1, and S 2 R3N is a matrix storing the columnar xyz coordinates of the N vertices of the 3D shape of I1. It is a rotation matrix. It's 3D translation, f c1 and f c2 These are the scaling parameters of the orthogonal cameras for the first image and the second image, respectively. ( ) is the visible triangle in pixel (x, y) of image I1 (e.g., detected using a hardware-based renderer), and T is the set of all triangles in the grid of S. It is the projected triangle t located on the image I1 j The centroid coordinates of the pixels (x; y) in the image.

[0123] For example, a masking loss can be used to set the background pixels to zero and ignore the background pixels during training.

[0124] Figure 11 An example of a method 1100 for training a neural network to generate the 3D shape of an object from a 2D image is shown. The purpose of the training is to train the neural network 1104 to regress a dense 3D mesh through vertex-by-vertex 3D coordinate estimation. For example, after training, the neural network 1104 can be combined with the above... Figure 3 and Figure 4 Used in the described preprocessing steps.

[0125] A 2D image 1102 is input into the neural network 1004. The neural network processes the input image 1102 to generate a desired 3D shape (1006, ...) from it. (i.e., an N-vertex mesh). Using the objective function (1110, The expected 3D shape 1106 is compared with the ground true 3D shape (1108, ) The parameters of the neural network 1104 are compared to determine the parameter updates 1012. Stochastic gradient descent or Adam optimization algorithms (e.g., parameter optimization) can be used. and Optimization functions such as ) can be used to determine the parameter updates, however, any suitable optimization process can be used instead.

[0126] Training is performed using a dataset of 2D images 1102 (each with a corresponding ground truth 3D shape 1108). The dataset can be divided into multiple batches, each batch comprising multiple 2D images 1102 and their corresponding ground truth 3D shapes 1108. The batch size can be in the range [10, 100], preferably in the range [25, 40]. In some embodiments, the batch size is 32.

[0127] The neural network can have a combination of Figure 3 The structure of the neural network.

[0128] The objective function 1110 may include vertex terms. The vertex term compares the positions of vertices in the expected 3D shape 1006 with the positions of vertices in the ground truth 3D shape 1008. The vertex term penalizes the deviation between the 3D coordinates of each vertex in the expected 3D shape 1006 and the corresponding ground truth coordinates. An example of the vertex term is given by the following equation:

[0129]

[0130] Here, i denotes the N vertices of the mesh. It should be understood that other forms of the vertex entries can be used alternatively, such as using different types of norms.

[0131] Alternatively or additionally, the objective function 1010 may include marginal terms. The edge term compares the lengths of the edges in the expected 3D shape 1006 with the lengths of the edges in the ground truth 3D shape 1008. The edge term ensures that the edge lengths between vertices in the expected 3D shape 1006 and the ground truth 3D shape 1108 are similar. An example of the edge term is given by the following equation:

[0132]

[0133] Here, j denotes the O edges of the mesh. It should be understood that other forms of the edge terms can be used alternatively, such as using different types of norms.

[0134] Figure 12 A schematic diagram of a system / apparatus for performing any of the methods described herein is shown. The system / apparatus shown is an example of a computing device. It will be understood by those skilled in the art that other types of computing devices / systems can be used alternatively to implement the methods described herein, such as distributed computing systems.

[0135] The device (or system) 1200 includes one or more processors 1202. The one or more processors control the operation of other components of the system / device 1200. For example, the one or more processors 1202 may include general-purpose processors. The one or more processors 1202 may be single-core devices or multi-core devices. The one or more processors 1202 may include a central processing unit (CPU) or a graphics processing unit (GPU). Alternatively, the one or more processors 1202 may include dedicated processing hardware, such as a RISC processor with embedded firmware or programmable hardware. Multiple processors may be included.

[0136] The system / device includes working memory or volatile memory 1204. The one or more processors can access the volatile memory 1204 to process data and can control the storage of data in the memory. The volatile memory 1204 may include any type of RAM such as Static RAM (SRAM) and Dynamic RAM (DRAM); or, it may include flash memory such as an SD card.

[0137] The system / device includes a non-volatile memory 1206. The non-volatile memory 1206 stores a set of operation instructions 1208 in the form of computer-readable instructions for controlling the operation of the processor 1202. The non-volatile memory 1206 can be any type of memory, such as read-only memory (ROM), flash memory, or magnetically driven memory.

[0138] The one or more processors 1202 are configured to execute the operation instructions 1208 to cause the system / device to perform any of the methods described herein. The operation instructions 1208 may include code related to hardware components of the system / device 1200 (i.e., drivers), and code related to the basic operation of the system / device 1200. Typically, the one or more processors 1202 use the volatile memory 1204 to execute one or more instructions of the operation instructions 1208 that are permanently or semi-permanently stored in the non-volatile memory 1206 to temporarily store data generated during the execution of the operation instructions 1208.

[0139] The methods described herein can be implemented in digital electronic circuits, integrated circuits, application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These may include computer program products (e.g., software stored on disks, optical discs, memory, programmable logic devices, etc.), which include computer-readable instructions, in combination with... Figure 12 When a computer or similar device executes the computer-readable instructions described herein, the computer performs one or more of the methods described herein.

[0140] Any system feature described herein can also be provided as a methodological feature, and vice versa. Alternatively, the apparatus and functional features used herein can be represented according to their respective structures. Specifically, methodological aspects can be applied to system aspects, and vice versa.

[0141] Furthermore, any, some, and / or all features of one aspect may be applied in any suitable combination to any, some, and / or all features of any other aspect. Moreover, it should be understood that specific combinations of the various features described and defined in any aspect of the invention may be implemented and / or provided and / or used independently.

[0142] Although several embodiments have been shown and described, those skilled in the art will understand that changes can be made to these embodiments without departing from the principles of the invention, the scope of which is defined in the claims.

Claims

1. A computer-implemented method, characterized in that, For estimating the three-dimensional flow of an object between a first two-dimensional image and a second two-dimensional image, the first two-dimensional image including an image of the object in a first configuration, and the second two-dimensional image including an image of the object in a second configuration; the method includes: The first two-dimensional image, the second two-dimensional image, and the estimated three-dimensional shape of the object are input into a convolutional neural network; The convolutional neural network is used to generate the three-dimensional flow of the object between the first two-dimensional image and the second two-dimensional image from the first two-dimensional image, the second two-dimensional image, and the two-dimensional representation of the estimated three-dimensional shape of the object.

2. The method according to claim 1, characterized in that, The two-dimensional representation of the estimated three-dimensional shape of the object includes a color image, wherein the pixels of the color image encode the three-dimensional coordinates of points in the estimated three-dimensional shape of the object, the points corresponding to pixels of the object in the first two-dimensional image.

3. The method according to claim 1 or 2, characterized in that, The two-dimensional representation of the estimated three-dimensional shape of the object includes normalized projective coordinate encoding.

4. The method according to claim 1 or 2, characterized in that, The three-dimensional flow of the object between the first two-dimensional image and the second two-dimensional image includes a color image that encodes the three-dimensional flow of pixels between the first two-dimensional image and the second two-dimensional image.

5. The method according to claim 1 or 2, characterized in that, The convolutional neural network includes: The first input branch is used to process the first two-dimensional image through one or more convolutional layers; The second input branch is used to process the second two-dimensional image through one or more convolutional layers; The third input branch is used to process the estimated three-dimensional shape of the object into a two-dimensional representation through one or more convolutional layers; A relay network is used to process the output derived from the first input branch, the second input branch, and the third input branch through multiple convolutional layers.

6. The method according to claim 5, characterized in that, The convolutional neural network further includes an association layer for associating the output of the first input branch with the output of the second input branch before processing by the relay network.

7. The method according to claim 5, characterized in that, The relay network includes: The encoding part is used to generate a coarse feature map from the outputs derived from the first input branch, the second input branch, and the third input branch through multiple convolutional layers; The decoding section is used to generate the three-dimensional flow of the object between the first two-dimensional image and the second two-dimensional image from the coarse feature map through multiple deconvolution layers.

8. The method according to claim 1 or 2, characterized in that, Also includes: Before inputting the two-dimensional representation of the estimated three-dimensional shape of the object into the convolutional neural network, another neural network is used to generate the two-dimensional representation of the estimated three-dimensional shape of the object from the first two-dimensional image.

9. The method according to claim 8, characterized in that, Generating the two-dimensional representation of the estimated three-dimensional shape of the object from the first two-dimensional image includes: The estimated three-dimensional shape of the object is generated from the first two-dimensional image using the other neural network; The three-dimensional coordinates of each point in the estimated three-dimensional shape in the two-dimensional image are encoded to generate the two-dimensional representation of the estimated three-dimensional shape of the object.

10. The method according to claim 1 or 2, characterized in that, The object in question is a human face.

11. A computer-implemented method, characterized in that, The method includes training a convolutional neural network to estimate a 3D flow of an object between a first 2D image and a second 2D image, wherein the first 2D image includes an image of the object in a first configuration, and the second 2D image includes an image of the object in a second configuration; the method includes: The first two-dimensional image, the second two-dimensional image, and the estimated three-dimensional shape of the object are input into a convolutional neural network; The convolutional neural network is used to generate the three-dimensional flow of the object between the first two-dimensional image and the second two-dimensional image from the two-dimensional representation of the estimated three-dimensional shape of the object; Using an objective function, the generated 3D flow and the ground truth 3D flow between the first 2D image and the second 2D image are compared; The parameters of the convolutional neural network are updated based on the comparison.

12. The method according to claim 11, characterized in that, The objective function includes an endpoint error term based on the difference between the generated 3D flow and the ground truth 3D flow between the first 2D image and the second 2D image.

13. The method according to claim 12, characterized in that, Generating the three-dimensional flow of the object between the first two-dimensional image and the second two-dimensional image using the convolutional neural network includes generating a series of three-dimensional flows at different resolutions; The endpoint error term also compares the generated series of three-dimensional streams at different resolutions with the corresponding series of ground truth three-dimensional streams at different resolutions, wherein the series of ground truth three-dimensional streams at different resolutions are derived from the ground truth three-dimensional stream between the first two-dimensional image and the second two-dimensional image.

14. The method according to any one of claims 11 to 13, characterized in that, The objective function includes a color consistency error, which compares pixel values ​​sampled from the second two-dimensional image with pixel values ​​in the first two-dimensional image, wherein the pixel values ​​in the second two-dimensional image are sampled based on the warping of the two-dimensional representation of the estimated three-dimensional shape of the object, using the generated three-dimensional flow of the object between the first and second two-dimensional images.

15. The method according to any one of claims 11 to 13, characterized in that, The two-dimensional representation of the estimated three-dimensional shape of the object includes a color image, wherein the pixels of the color image encode the three-dimensional coordinates of points in the estimated three-dimensional shape of the object, the points corresponding to pixels of the object in the first two-dimensional image.

16. The method according to any one of claims 11 to 13, characterized in that, The three-dimensional flow of the object between the first two-dimensional image and the second two-dimensional image includes a color image that encodes the three-dimensional flow of pixels between the first two-dimensional image and the second two-dimensional image.

17. The method according to any one of claims 11 to 13, characterized in that, The convolutional neural network includes: The first input branch is used to process the first two-dimensional image through one or more convolutional layers; The second input branch is used to process the second two-dimensional image through one or more convolutional layers; The third input branch is used to process the estimated three-dimensional shape of the object into a two-dimensional representation through one or more convolutional layers; A relay network is used to process the output derived from the first input branch, the second input branch, and the third input branch through multiple convolutional layers.

18. The method according to any one of claims 11 to 13, characterized in that, Also includes: Before inputting the two-dimensional representation of the estimated three-dimensional shape of the object into the convolutional neural network, another neural network is used to generate the two-dimensional representation of the estimated three-dimensional shape of the object from the first two-dimensional image.

19. The method according to any one of claims 11 to 13, characterized in that, The object in question is a human face.

20. A computer-implemented method, characterized in that, The method is used to estimate the three-dimensional flow of an object between a first two-dimensional image and a second two-dimensional image, the first two-dimensional image including an image of the object in a first configuration and the second two-dimensional image including an image of the object in a second configuration; the method includes a neural network trained using the method according to any one of claims 11 to 19.

21. A computer-implemented method, characterized in that, The method includes generating a training dataset for training a model to estimate the 3D flow of an object between a first 2D image and a second 2D image, wherein the first 2D image includes an image of the object in a first configuration, and the second 2D image includes an image of the object in a second configuration; the method includes: Extract multiple frames from a video dataset, where each frame includes an image of an object of a predefined type; For each extracted frame, the 3D model of the object of the predefined type is fitted onto the image of the object in the frame; For multiple pairs of frames, the 3D flow is generated from the corresponding fitted 3D model of the object for each frame in the pair; The extracted frames and each of the generated 3D streams, as well as the correlation between the generated 3D streams and the corresponding pair of extracted frames generated therefrom, are stored in memory.

22. The method according to claim 21, characterized in that, Each of the three-dimensional streams includes a color image, wherein the pixels of the color image encode the three-dimensional stream from a point in the first frame of the pair to the second frame of the pair.

23. A computer-implemented method, characterized in that, The method is used to train a convolutional neural network to estimate the three-dimensional flow of an object between a first two-dimensional image and a second two-dimensional image, the first two-dimensional image including an image of the object in a first configuration, and the second two-dimensional image including an image of the object in a second configuration; the method includes using a dataset generated by the method according to claim 21 or 22.

24. A three-dimensional motion estimation system, characterized in that, The system includes a memory and one or more processors, the memory storing computer-readable code that, when executed by the one or more processors, causes the system to perform the method according to any one of the preceding claims.

25. A computer program product, characterized in that, Includes computer-readable code, which, when executed by a computing device, causes the computing device to perform the method according to any one of claims 1 to 23.

Images