Generating 3D object models from 2D images

Abstract

This specification discloses a method and system for generating a three-dimensional model of a deformable object from a two-dimensional image. According to one aspect of this disclosure, a computer-implemented method for generating a three-dimensional model of a deformable object from a two-dimensional image is described. The method includes: receiving a two-dimensional image as input to an embedding neural network, wherein the two-dimensional image includes an image of an object; generating an embedding representation of the two-dimensional image using the embedding neural network; inputting the embedding representation into a learned decoder model; and generating parameters of a three-dimensional model of the object from the embedding representation using the learned decoder model.

Description

Technical Field

[0001] This specification discloses methods and systems for generating three-dimensional models of deformable objects from two-dimensional images. Background Technology

[0002] Monocular 3D reconstruction of objects can facilitate a wide range of applications in human-computer interaction, augmented reality, and virtual telepresence. In some applications, 3D reconstruction of hands can be particularly useful, such as for automatic sign language recognition / generation and augmented reality control / interaction.

[0003] The limitation of current existing methods is that they are not easily generalized to samples captured in non-laboratory environments (e.g., "in the wild"), and are often based on estimating the location of sparse keypoint sets (e.g., the location of hand joints and fingertips when generating a 3D model of a hand). Summary of the Invention

[0004] According to one aspect of this disclosure, a computer-implemented method for generating a three-dimensional model of a deformable object from a two-dimensional image is described. The method includes: receiving a two-dimensional image as input to an embedding neural network, wherein the two-dimensional image includes an image of an object; generating an embedding representation of the two-dimensional image using the embedding neural network; inputting the embedding representation into a learned decoder model; and generating parameters of a three-dimensional model of the object from the embedding representation using the learned decoder model.

[0005] According to another aspect of this disclosure, a computer-implemented method for training a learned decoder model to generate a 3D model of a deformable object from a 2D image is described. The method includes: receiving a 2D training image as input to an embedding neural network, wherein the 2D training image includes an object and is associated with predetermined parameters of a 3D model of the deformable object; generating an embedding representation of the 2D training image using the embedding neural network; inputting the embedding representation into a decoder model; generating parameters of a 3D model of the object from the embedding representation using the decoder model; comparing the generated parameters of the 3D model of the object with predetermined parameters of the 3D model of the object associated with the 2D training image using a loss function; and updating the parameters of the decoder model based on the comparison.

[0006] The parameters of the object's 3D model may include the positions of multiple nodes in a mesh, and the loss function includes a vertex term that compares the generated node positions with the corresponding predetermined positions of nodes associated with the 2D training image. The loss function may also include an edge term that compares the distances between the generated node positions with the distances between the corresponding predetermined positions of nodes associated with the 2D training image. The predetermined parameters of the object's 3D model associated with the 2D training image can be obtained by fitting the 3D object model to the 2D training image.

[0007] The methods above can each be combined with one or more of the features below, individually or in combination.

[0008] Embedded neural networks can include residual neural networks.

[0009] The 3D model can include a mesh model, and the parameters of the object's 3D model can include mesh parameters. The decoder model can include a decoder neural network.

[0010] The decoder model may include a decoder neural network comprising one or more two-dimensional convolutional layers, wherein the parameters of the three-dimensional model of the hand include the three-dimensional coordinates of the nodes of the mesh. The decoder neural network may include one or more lifting layers.

[0011] The initial layer of a decoder neural network can include fully connected layers and remodeling layers.

[0012] The decoder model may include linear layers, and the parameters of the three-dimensional model of the hand include the three-dimensional coordinates of the nodes of the mesh.

[0013] A deformable object can be a hand.

[0014] According to another aspect of this disclosure, a computer-implemented method is described for generating a training dataset for training a machine learning model to generate a 3D model of a deformable object from 2D images. The method includes: extracting a plurality of 2D images from one or more segments of video data, each 2D image comprising one or more deformable objects; for each of the plurality of 2D images: identifying a plurality of object keypoints in the 2D image using a keypoint recognition model; and determining parameters of a 3D parametric model of the object, the parameters corresponding to the deformable object in the 2D image, wherein determining the parameters includes fitting the 3D parametric model of the object to the 2D image using the identified object keypoints and corresponding object keypoints in the 3D parametric model; and storing the corresponding parameters of the 2D image and the 3D parametric model in computer memory.

[0015] One or more segments of video data may include one or more segments of video showing the use of sign language.

[0016] Fitting a 3D parametric model of an object to a 2D image can include applying an iterative optimization process to an objective function. This objective function includes a reprojection term that compares the positions of object keypoints identified in the 2D image with the corresponding positions of object keypoints in the 2D projection of the 3D parametric model. The reprojection term can include applying weights to a mask of the corresponding object keypoints in the 2D projection. The objective function can also include a length term that compares the distances between object keypoints identified in the 2D image with the distances between corresponding object keypoints in the 2D projection of the 3D parametric model. The objective function can also include a regularization term that penalizes deviations from a predefined average object pose.

[0017] The 3D parametric model of an object can be a mesh model parameterized by parameters including one or more shape parameters and / or one or more pose parameters. The parameters of the 3D parametric model of an object can also include one or more camera device parameters. Corresponding object keypoints in the 3D parametric model can be identified from the mesh model using a sparse regression model.

[0018] The deformable object can be a hand. One or more segments of video data can include multiple segments of a video of sign language usage, wherein the multiple segments of video include the use of sign language from different sign languages. The keypoint extraction model can be a two-dimensional pose estimation model configured to identify the joint positions and / or fingertip positions of the hand in a two-dimensional image.

[0019] According to another aspect of this disclosure, a computer-implemented method for training a machine learning model to generate a three-dimensional model of an object is described, the method comprising using a training dataset generated according to the method described above.

[0020] According to another aspect of this disclosure, a computer-implemented method for generating a three-dimensional model of an object is described, the method comprising using a machine learning model trained using any of the training methods disclosed herein.

[0021] According to another aspect of this disclosure, an apparatus including a processor and a memory is described, the memory including computer-readable instructions that, when executed by the processor, cause the apparatus to perform any or more of the methods disclosed herein.

[0022] According to another aspect of this disclosure, a computer program product comprising computer-readable code is described, which, when executed by a computer, causes the computer to perform any or more of the methods disclosed herein. Attached Figure Description

[0023] The embodiments will now be described by way of non-limiting example with reference to the accompanying drawings, in which:

[0024] Figure 1 An overview of an example method for generating a 3D model of an object from a 2D image is shown;

[0025] Figure 2 A flowchart illustrating an example method for generating a 3D model of an object from a 2D image is shown.

[0026] Figure 3 An example of the structure of a decoder neural network is shown;

[0027] Figure 4 An overview of an example method for training a model to generate a 3D model of an object from a 2D image is shown;

[0028] Figure 5 A flowchart is shown illustrating an example method for training a model to generate a 3D model of an object from a 2D image;

[0029] Figure 6 An overview of an example method for generating a training dataset for training a machine learning model to generate 3D models of objects from 2D images is shown.

[0030] Figure 7 A flowchart illustrating an example method for generating a training dataset for training a machine learning model to generate 3D models of objects from 2D images is shown; and

[0031] Figure 8 An example of a system / apparatus for performing the methods disclosed herein is shown. Detailed Implementation

[0032] The example implementation provides a system and method for reconstructing monocular 3D object models.

[0033] This paper introduces a method / system for generating 3D object models from 2D images, using a neural network architecture that includes an image encoder and a subsequent decoder model. The method / system can be trained based on a direct 3D object mesh reconstruction loss. A training dataset for use in training can be automatically generated from a video containing example images of an object of a given type (e.g., a hand). While the methods and systems used in this paper can be used to generate models for a wide range of object types, they are particularly well-suited for generating 3D models of hands in images.

[0034] Compared to current methods, the method described in this paper results in lower pose estimation errors for 3D object models generated from field images. The method also outperforms previous methods for object pose estimation, particularly hand pose estimation, without overfitting or adjusting for specific datasets. Therefore, the method described in this paper provides a robust and accurate means of determining 3D object models from 2D images, from which accurate estimates of object pose can be derived. Accurate object pose information, especially hand pose estimation, has numerous applications in human-computer interaction, such as automatic sign language recognition and augmented / virtual reality control.

[0035] Furthermore, the dataset generation method described in this paper can generate labeled datasets of object images from raw field hand images / videos without requiring manual labeling.

[0036] The following describes methods and systems for generating 3D object models from 2D images, primarily concerning the object of the hand; that is, generating 3D models of the hand from 2D images containing the hand. However, it will be appreciated that the methods can be alternatively or additionally applied to other objects. Examples include, but are not limited to: faces; human / animal torsos and / or parts thereof; vehicles, such as cars or airplanes; and / or robots and / or robot parts. The methods and systems can be used with objects that possess deformable models (i.e., deformable objects). A deformable object can be considered as an object / object category whose surface can be parameterized using a mesh with a fixed / predetermined topology.

[0037] As used herein, the term "hand" is preferably used to refer to a human hand. However, the term "hand" may alternatively or additionally refer to one or more of the following: animal claws / hands; mechanical hands or grippers; and / or appendages of fictional creatures.

[0038] Figure 1 An overview of an example method for generating a 3D model of an object from a 2D image is shown. In this example, the object is a hand, but could alternatively be another type of deformable object. Method 100 takes a 2D image 102 comprising one or more hands 104 as input. An encoder model 106, such as an encoder neural network, processes the 2D image 102 to generate an embedding 108, E of the image. The embedding 108 is input to a decoder model 110 (in this example, a decoder neural network), which processes the embedding 108 to generate a 3D model 112 of one or more hands 104 in the 2D image 102.

[0039] A two-dimensional image 102, I, includes a set of pixel values ​​corresponding to a two-dimensional array. For example, in a color image, Here, H is the height of the image in pixels, W is the height of the image in pixels, and the image has three color channels (e.g., RGB or CIELAB). In some implementations, the two-dimensional image can be black and white / grayscale.

[0040] The encoder neural network 106 is a model for generating embeddings (e.g., latent representations / vectors) of an input image. The encoder neural network 106 may be a neural network pre-trained for image embeddings. Such a neural network may include one or more convolutional layers. The neural network may include one or more skip connections. For example, the resulting embedding 108 may include a distribution over a set of image features. An example of such a neural network is a ResNet neural network, such as the ResNet-50 network. Other examples include, but are not limited to, HRNet (see “Deep High-Resolution Representation Learning for Human Pose Estimation”, K. Sun et al., CVPR 2019), MobileNetV2 (see “MobileNetV2: Inverted Residuals and Linear Bottlenecks”, M. Sandler et al., CVPR 2019), and EfficientNet (see “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks”, M. Tan et al., ICML 2019).

[0041] Decoder model 110 takes image embedding 108 as input and processes it to generate an output 3D model 112 of the hand 104 in the input image. In an implementation where decoder model 110 is a neural network, the image embedding is input into the input layer of the decoder neural network. The embedding 108 is then processed through a series of decoder neural network layers to generate the output 3D model 112 of the hand 104 in the input image.

[0042] Decoder model 110 is a learned model (i.e., a trained model) that has been trained on a set of training data to generate an output 3D model 112 of the hand 104 in the input image from the embedding of the input image. An example of training such a model is described below.

[0043] Decoder model 110 may include one or more spatial convolutional layers. The spatial convolutions of these layers can operate in pixel space. Decoder model 110 may include one or more fully connected layers. For example, the initial layer may be a fully connected layer. Alternatively or additionally, the decoder model may include one or more lifting layers for increasing the dimension of its input. In some implementations, the decoder model includes a single linear layer or consists of a single linear layer.

[0044] Examples of the structure of the decoder neural network / model are shown below. Figure 3 Further detailed description. Generally, a neural network comprises multiple layers of nodes, each node associated with one or more parameters. The parameters of each node in the neural network may include one or more weights and / or biases. A node takes one or more outputs from nodes in previous layers as input. The one or more outputs from nodes in previous layers are used by the node to generate activation values ​​using the parameters of the neural network and an activation function. The activation function may include the ReLU activation function, but other activation functions (e.g., sigmoid or tanh activation functions) may be used alternatively or additionally.

[0045] The output 3D model 112 includes a 3D representation of the hand 104 in the input image 102. Generally, any parameters defining the 3D model of the hand corresponding to the hand 104 in the input image 102 can be output by the decoder neural network 112. For example, the 3D model 112 can be a mesh model defined by a set of mesh parameters. Mesh parameters can include: vertex / node positions; edges defined between pairs of vertices / nodes in the mesh; and / or faces (i.e., a set of closed edges, such as triangular or quadrilateral faces). An example of such a mesh model for a hand is the MANO model, but other mesh models can be used alternatively. In some implementations, the output of the decoder neural network 512 may simply be the positions of mesh vertices / nodes in a predefined model, rather than the entire mesh itself.

[0046] Figure 2 A flowchart illustrating an example method for generating a 3D model of an object from a 2D image is shown. This method can be implemented on a computer. (See above regarding...) Figure 1 The object described could be a hand.

[0047] At operation 2.1, a two-dimensional image is received as input to the embedded neural network. The two-dimensional image includes one or more objects of a given type, such as one or more hands.

[0048] At operation 2.2, an embedded neural network is used to generate an embedded representation of the two-dimensional image. The embedded neural network may include a residual neural network, i.e., a neural network with one or more skip connections. The embedded neural network may include one or more convolutional layers, each applying one or more convolutional filters to the output and / or input image of the previous layer in the embedded neural network.

[0049] At operation 2.3, the embedded representation is input into the decoder model. The decoder model can be a decoder neural network. The initial layer of the decoder neural network (e.g., one of the input layers or a layer following the input layer) can include a fully connected layer. The fully connected layer can be an input layer of the decoder neural network, which takes the embedded representation as input and generates a feature map from it. The feature map can be defined in a pixel-like space (e.g., an array of one or more pixel values) or a grid space (e.g., values ​​associated with grid nodes). A reshaping layer can be added after the fully connected layer to reshape the feature map for use by subsequent layers.

[0050] At operation 2.4, the parameters of a 3D model of the hand are generated from the embedding of the 2D image using a decoder model. The 3D model of the hand can be a mesh model, where the parameters are mesh parameters. For example, mesh parameters may include: the positions of nodes in the mesh; edges connecting pairs of nodes in the mesh; and / or the mesh shape (e.g., a mesh triangle or a mesh quadrilateral).

[0051] In some implementations, the decoder model may include a single linear layer or consist of a single linear layer. A single linear layer may provide direct object regression for a 3D model. A linear layer may include a matrix with learned values ​​for its matrix elements, which takes as input a vector obtained by reshaping an image embedding from the output of an embedded neural network.

[0052] In some implementations, the decoder model may include a convolutional neural network comprising one or more convolutional layers. The convolutional neural network may also include one or more lifting layers configured to increase the dimension of the output of the convolutional layers.

[0053] Figure 3 An example of a decoder neural network structure is shown. The decoder neural network 300 is configured to generate a 3D mesh model of the hand from an embedding of a 2D image of the hand.

[0054] The decoder neural network may include one or more two-dimensional convolutional layers 302, i.e., each configured to apply one or more two-dimensional convolutional filters to, for example, an array of pixel values ​​and / or a feature map. In embodiments using such two-dimensional convolutional layers, the output of the decoder neural network may be a set of node / vertex coordinates of a grid.

[0055] The decoder neural network may include one or more boosting layers 304. Boosting layers 304 are configured to increase the dimension of their input by a predetermined factor. In the example shown, boosting layer 304 boosts its input by a factor of two, but other boosting factors may be used alternatively. Boosting layer 304 may be a deconvolutional layer.

[0056] The initial layer of the decoder neural network (e.g., one of the input layers or a layer following the input layer) may include a fully connected layer 306. The fully connected layer 306 can be an input layer of the decoder neural network, taking the embedded representation as input and generating a feature map from it. The feature map can be defined in a pixel-like space (e.g., an array of one or more pixel values) or a grid space (e.g., values ​​associated with grid nodes). In the example shown, the fully connected layer 306 outputs a (51×48) dimensional vector.

[0057] Reshaping layer 308 can reshape feature maps after fully connected layers for use by subsequent layers. Reshaping layer 308 can convert the vector output by fully connected layer 306 into a tensor. In the example shown, reshaping layer 308 converts the (51×48) dimensional vector output by fully connected layer 306 into a 51×48 dimensional matrix. Depending on the nature of the subsequent layers of neural network 300, other reshaping is also possible.

[0058] In some implementations, the output of the decoder neural network includes or consists of the three-dimensional coordinates of the nodes of the grid.

[0059] In some implementations, the decoder model can be a linear regression model. The problem can be reinterpreted as one of the direct regressions from image paths in a 2D image to grid vertex locations. In these implementations, an embedded neural network is used as described above, where a single layer operates on top of an embedded neural network block. The resulting tensor is reshaped into a D-dimensional vector and then multiplied by a matrix of dimension (N*3)×D to give the three-dimensional coordinates of N grid vertices, where N is the number of vertices in the grid. The values ​​of the matrix elements used in the matrix can, for example, use the following... Figure 5 and Figure 6 The method described is for learning. In some implementations, a (N*3) vector of learned parameters can also be added to the output of the matrix multiplication.

[0060] For example, the embedding network could be a ResNet-50 network. The output 5×5×512 tensor representing the embedding is reshaped into a D = 12800-dimensional vector. The reshaped vector is multiplied by a (722*3)×12800 matrix to provide the 3D coordinates of 722 grid vertices.

[0061] Implementations based on linear regression models are particularly well-suited for lightweight implementations on mobile devices. They do not use any specialized convolutions, such as geometric convolutions, nor do they use deconvolution-based filters. Instead, they rely on general convolutional network layers. The models can also be extended to be applied to the entire image in a fully convolutional manner, rather than processing individual patches.

[0062] Figure 4 An overview of a method 400 for training a model to generate a 3D model of object 412 from a 2D image 402 is shown. In the example shown, object 412 is a hand, but the model can alternatively be trained to generate 3D models of other object types.

[0063] The training images 402, including the object (in this case, a hand) 404, are selected from a training dataset containing images of hands. Each training image 402 in the training dataset is associated with a predetermined (i.e., known) 3D model 414 of the object 404 in the image. For example, the following can be used... Figure 6 and Figure 7 The described method generates the training dataset. Alternatively, the training dataset can be generated manually. Generally, any method used to generate a dataset of images of objects with corresponding 3D models can be used.

[0064] A predetermined 3D model 414 of the hand can be stored as a full mesh model of the hand associated with a corresponding 2D image. Alternatively, the predetermined 3D model 414 of the hand can be generated from the parameters of a stored parametric model associated with the corresponding 2D image (e.g., parameters of a MANO model).

[0065] The selected image 404 is input into the embedding neural network 406. The embedding neural network 406 generates an embedding 408 of the input image 402, that is, a latent representation of the image.

[0066] Embedsion 408 is input into decoder neural network 410, which generates a 3D model of the hand from the embedding. Decoder neural network 410 can be, for example, a convolutional neural network as described above. The 3D model 412 of the hand can be a mesh model (i.e., parameters representing the mesh of the hand, such as vertex / node positions, edges, and / or mesh polygons).

[0067] The generated 3D model 412 of the hand is compared with a predetermined 3D model 414 of the hand associated with the input image 402. This can be achieved using a loss function 416. To perform the comparison, the loss function compares the properties of the generated 3D model 412 of the hand with the properties of a predetermined 3D model 414 of the hand associated with the input image 402.

[0068] Loss function 416 may include vertex terms. The vertex term compares the positions of vertices / nodes in the generated object's 3D model 412 with the positions of vertices / nodes in a predetermined 3D model 414 of the object associated with the input image 402. The vertex term may depend on the difference between the positions of vertices / nodes in the generated object's 3D model 412 and the corresponding positions of vertices / nodes in the predetermined 3D model 414 of the object associated with the input image 402. An example of such a vertex loss function term is given by the following equation:

[0069]

[0070] in, These are the vertex positions of the ground-truth mesh (i.e., the positions of vertices / nodes in the predetermined 3D model 414 of the object associated with the input image 502). It refers to the vertex positions of the generated mesh (i.e., the positions of vertices / nodes in the 3D model 412 of the generated object, generated from the same input image 402), and λ. vertex This is a hyperparameter that controls the relative importance of a term in the loss function when the loss function includes multiple terms. Hyperparameter λ vertex It can be, for example, between 0.001 and 1, between 0.001 and 0.1, between 0.005 and 0.015, for example, λ. vertex = 0.001. In the example above, the L1 norm was used, but it will be recognized that other norms, such as the L2 norm, can be used alternatively.

[0071] Loss function 416 may alternatively or otherwise include edge terms. The edge term can alternatively be called the "edge length preservation" component. The edge term compares the lengths of the edges between nodes in the generated object's 3D model 412 with the lengths of the edges between nodes in the predetermined 3D model 414 of the object associated with the input image 502. The edge term may depend on the difference between the lengths of the edges in the generated object's 3D model 412 and the corresponding edge lengths in the predetermined 3D model 414 of the object associated with the input image 502. An example of such an edge loss function is given by the following equation:

[0072]

[0073] Where, ε mesh It is a set of mesh edges (e.g., defining the paired nodes (u,v) at the endpoints of the edges). and These are the nodes corresponding to the endpoints of the edge (u,v) in the basic truth grid 514, and and It is the node corresponding to the endpoint of the edge (u,v) in the generated mesh 412. λ edge This is a hyperparameter that controls the relative importance of a term in the loss function when the loss function includes multiple terms. Hyperparameter λ edge It can be, for example, between 0.001 and 1, between 0.001 and 0.1, between 0.005 and 0.015, for example, λ. edge = 0.001. In the example above, the L2 norm was used, but it will be recognized that other norms, such as the L1 norm, can be used alternatively.

[0074] As shown below, the loss function for the combination can be given by the sum of the vertex terms and the edge terms:

[0075]

[0076] In some implementations, λ edge =λ vertex The loss function may include additional terms such as pose estimation loss. However, it has been found that including pose estimation loss has a limited impact on training results, and therefore it can be omitted in some implementations.

[0077] The loss function 416 is used to determine the parameter updates 418 of the decoder model 410. For example, the weights and / or biases of the decoder neural network 418 are updated based on a comparison between the generated 3D model 412 of the object and the known 3D model 414 of the object. For example, an optimization process can be applied to the loss function 416 to determine the parameter updates 418. Examples of such optimization processes include, but are not limited to, gradient descent methods such as stochastic gradient descent, simulated annealing, momentum methods (e.g., Adam), etc.

[0078] In some implementations, the parameters of the encoder neural network 406 can also be updated based on the loss function 416 using any of the training methods described herein. Alternatively, the parameters of the encoder neural network 406 can be fixed during the training of the decoder model 412.

[0079] In some implementations, parameter updates 418 are determined once a batch of training images 402 has been processed. The parameter updates 418 can be determined using the average / sum of the loss function 416 over the batch of training images. This batch of training images comprises multiple distinct training images 402 from the training dataset. The batch size (i.e., the number of distinct training images in a batch) can be, for example, between 10 and 100 images, preferably between 25 and 50 images, such as 32 images.

[0080] Training can be iterated until a threshold condition is met. The threshold condition can be one or more of the following: the number of threshold iterations; the number of threshold training epochs (i.e., through forward and backward propagation of the training dataset); the threshold accuracy obtained on a test dataset comprising two-dimensional images of hands with known three-dimensional hand models; a loss function optimized to within the threshold; and / or some other convergence criterion or standard. When using the threshold number of epochs, the threshold number can be, for example, between 100 and 200 epochs, preferably between 125 and 175 epochs, such as 150 epochs.

[0081] Parameter updates can be determined based on the learning rate of the optimization process used (e.g., the "step size" in gradient descent). The learning rate can be constant throughout training. Alternatively, the learning rate can be varied throughout training, for example, by varying it from the initial learning rate by a predetermined factor during predefined training epochs. As an example, the initial learning rate using the Adam optimizer could be 10. -4 It decays by a factor of 0.1 at epochs 90 and 120. Alternatively, other learning rate decay routines can be used.

[0082] In some implementations, the input image and / or 3D model may undergo preprocessing and / or post-processing. For example, the input image 402 may be enhanced with random cropping and / or random transformations (e.g., translation, rotation, scaling, etc.). This can improve the performance of the trained model on real-world examples where the input image cannot be cropped based on ground truth annotations. Alternatively or additionally, the input image 502 may be normalized with the mean and standard deviation from a known image dataset such as ImageNet. The output grid 412 may be normalized based on statistics computed on a subset of the known grid 414.

[0083] Figure 5 A flowchart illustrates an example method for training a neural network to generate 3D models of objects of a given type from a 2D image. The method can be implemented on a computer. The object type could be a hand.

[0084] At operation 5.1, a two-dimensional training image is received as input to the embedded neural network. The two-dimensional training image comprises images of objects of a given type and is associated with a predetermined three-dimensional model corresponding to the object in the image. The predetermined three-dimensional object model can be derived by fitting the three-dimensional object model to the two-dimensional training image.

[0085] In operation 5.2, as mentioned above... Figure 1 and Figure 2 The described method uses an embedding neural network to generate embeddings for two-dimensional training images. The embedding neural network can be a residual neural network.

[0086] At operation 5.3, the embedded representation is input into the decoder model. The decoder model includes, for example, a parameterized model of a decoder neural network or a linearized regression model, which can be trained using machine learning techniques.

[0087] In operation 5.4, as mentioned above... Figures 1 to 3 As described, a 3D model of a hand in a 2D image is generated using a decoder model (e.g., parameters of the 3D model).

[0088] In operation 5.5, the generated 3D model is compared with the predetermined 3D model of the object (i.e., the parameters of the generated 3D model are compared with the parameters of the predetermined 3D model). The comparison is performed using a loss function (which may also be called an objective function).

[0089] The loss function may include a vertex term that compares the positions of the generated node / mesh vertices with the predetermined positions of the corresponding node / mesh vertices in a predetermined model associated with the 2D training image. The loss function may alternatively or additionally include an edge term that compares the distances between the positions of the generated node / mesh vertices with the distances between the predetermined positions of the corresponding node / mesh vertices in a predetermined 3D model associated with the 2D training image.

[0090] In operation 5.6, the parameters of the decoder model are updated based on the comparison. Optimization routines such as stochastic gradient descent can be used to update the parameters. Operations 5.1 through 5.5 can be iterated multiple times, each iteration using different training images from the training batch. The parameters of the decoder model are updated based on the expected value of the loss / objective function obtained on a batch of training images.

[0091] Operations 5.1 through 5.6 can be iterated until a threshold condition is met. The threshold condition can be the number of thresholds during training epochs. The threshold condition can include, for example, one or more convergence criteria defined on a set of validation images with a corresponding predetermined 3D model.

[0092] Figure 6 This document provides an overview of an example method for generating a training dataset for training a machine learning model to generate a 3D model of a hand from a 2D image. The method can be executed by a computer. (The text then repeats itself, seemingly unrelated to the previous sentence.) Figure 6 and Figure 7 The training set generated by the described method can be used as a reference for... Figure 4 and Figure 5 The training dataset / validation dataset in the described method.

[0093] Training models for 3D object reconstruction are often limited by the availability of labeled images that can be used in training. Labeled images are typically manually annotated and / or captured under laboratory conditions, and therefore may not capture the full range of object locations / poses, etc., that occur in the field.

[0094] Video data can provide multiple images of a specific type of object. For example, when the object of interest is a hand, a video used in sign language can provide two-dimensional images of hands in a wide range of realistic poses. Such videos can be obtained from publicly available video repositories such as internet video sites.

[0095] Automatically fitting 3D object models to 2D images extracted from such video data can produce training datasets that more accurately reflect the poses of objects in the field. Training 3D object generation models on such training data yields more accurate model-generating models compared to manually labeled images and / or images captured under laboratory conditions.

[0096] Method 600 includes: receiving video data 602, the video data comprising multiple frames; and extracting one or more two-dimensional images 604 comprising an object of a given type (e.g., in the example shown, the object type is a hand). The video data 602 may be taken from a publicly available video data source, such as a video hosting website. This helps ensure that the video data 602 represents a “real-world” example of the object type. In the implementation where the object type is a hand, the video data may include one or more segments of video data depicting the use of sign language.

[0097] Each extracted image is fed into a keypoint recognition model 606, which identifies multiple object keypoints 608 in the input image. The keypoint extraction model 606 can be, for example, a pre-trained neural network. The pre-trained neural network may include one or more convolutional layers. An example of such a model is OpenPose, but alternatively, other models trained to recognize keypoints of a given object type in an image can be used.

[0098] The location Y of the identified keypoint 608 is used to iteratively fit a 3D deformable model (i.e., parametric model) of the object to the image. For example, a deformable model of a hand (e.g., a MANO model) can be iteratively fitted to an image of the hand.

[0099] The deformable model predicts N mesh vertices on the surface of an object using a differentiable function M(β,θ), which maps the shape parameter β and pose parameter θ (parameter 612) to an instance of the model represented by an N-vertex mesh. In some implementations, the deformable model can also predict the positions of multiple keypoints K of the model. In an implementation where the object type is a hand, the keypoints may, for example, represent the joints of the hand. The pose parameter θ may be related to the angles of the joints in the hand. For example, in the MANO model, the pose parameter represents K = 16 joints of the hand, and

[0100] The deformable model can also take one or more camera device parameters as input. Camera device parameters may include, for example, scaling parameters and translation parameters. Global orientation can be included in the pose parameter θ. In such an example, the deformable model can be represented as:

[0101]

[0102] Wherein, φ is a set of tilted model parameters.

[0103] To prevent unrealistic object poses, such as impossible joint angles in the hand, pose parameters can be constrained to the convex hull of some pre-computed cluster centers. For example, when the pose parameters represent the angles of K joints of the hand, C Euler angle clusters P can be obtained through the k-mean of each joint i. 1 i ... P C i The pose variable of each joint can then be expressed as...

[0104]

[0105] This will affect the posture angle θ i The constraint is to take a reasonable value, while allowing for the unconstrained variable w. c Optimization can be performed. The constrained angles can be represented using a parameter matrix. This indicates that global orientation is allowed to be unrestricted.

[0106] A 3D model 610 can be initialized using an initial set of shape and pose parameters. The 3D model can be used to determine one or more model keypoints 614 (i.e., model keypoints) corresponding to keypoints 608 identified in the 2D image 604. Model keypoints 614 can include K keypoints predicted by the differentiable model M. Model keypoints can additionally or alternatively include one or more other keypoints F. In a hand model, the K keypoints can include joints (e.g., K = 16 in a MANO model), and the F other keypoints can include the five fingertips of the hand. A sparse regression model can be used. The hand pose is obtained from keypoint 714 of the N-vertex mesh regression model.

[0107]

[0108] The model is iteratively fitted to the identified keypoints 608 in the two-dimensional image 604 by comparing the model keypoints 614 with the identified image keypoints 608. The comparison can be performed using an objective function 614. Fitting can be accomplished by applying an iterative optimization process, such as gradient descent, to the objective function 616.

[0109] The objective function may include a 2D reprojection term E. 2D The 2D reprojection term compares the positions of model keypoints 714 in the 2D projection of the 3D model with the positions of the identified image keypoints 608. Norms such as L2 or L1 can be used as the objective function.

[0110] Mask Λ 2D Projections applied to 3D models can reduce the influence of some keypoints and enhance the influence of others. For example, in a hand model, a mask can reduce the influence of the metacarpophalangeal joints (MCPs, i.e., the base of each finger) and enhance the influence of the fingertips and wrist. MCP joint locations can be multiplied by a predetermined factor less than 1, such as between 0.2 and 0.9, or between 0.5 and 0.8, for example, 0.7. Fingertips can be multiplied by a predetermined factor greater than 1, such as between 1.1 and 2.5, or between 1.6 and 1.8, for example, 1.7. Wrists can be multiplied by a predetermined factor greater than 1, such as between 1.5 and 3.5, or between 2 and 3, for example, 2.5.

[0111] An example of a 2D reprojection term can be symbolically represented as:

[0112]

[0113] Where Y is the position of the identified image key point 608, ∏ K It is a projection of the inherent camera device in 2D and Λ 2D It's a mask.

[0114] The objective function may also include a feature length term E. f In implementations where the object type is a hand, this can be referred to as "bone loss." The feature length term ensures that the lengths of edges in the object skeleton tree ε are preserved. The feature length term compares the lengths of edges between keypoints in the 2D reprojection of the 3D model with the lengths of edges between identified keypoints 608 in the 2D image 604. Norms such as L2 or L1 can be used as the objective function.

[0115] An example of a feature length term can be symbolically represented as:

[0116]

[0117] Here, and are obtained at paired points (i,j) in the object skeleton tree ε. It is the position of feature i in the 2D projection of the 3D model.

[0118] In some implementations, the objective function may also include a regularization term E. reg The regularization term penalizes deviations from the average object pose and helps ensure realistic deformation. Assuming parameters θ and β are defined such that the average object pose corresponds to θ = 0 and β = 0, an example of the regularization term is given by the following equation:

[0119] E reg (β, θ) = λ θ ||θ|| 2 +λ β ||β|| 2

[0120] Where, λ β and λ θ It is the hyperparameter chosen for the experiment. In some implementations, λ θ It can be less than 1, for example, in the range of 0.01 and 0.5, such as between 0.05 and 0.2, for example, 0.1. In some implementations, λ β It can be greater than 100, for example, in the range of 100 and 10000, for example, between 500 and 1500, for example, 1000.

[0121] The objective function may include one or more of a reprojection term, a feature length term, and a regularization term. The goal of the optimization process may be to minimize the following objectives:

[0122]

[0123] For example, the Adam optimizer can be used to apply the optimization process to the objective function, and different learning rates (e.g., 10⁻⁶ for each) can be applied to the camera device, pose, and shape parameters. -2 10 -2 and 10 -5After a predetermined number of iterations of the optimization routine, the learning rate can decay by a predetermined factor. For example, the learning rate might decay by a factor of 0.95 every 500 iterations. During some iteration sequences, one or more parameters in the parameter set may be frozen. For example, in an implementation where the object is a hand, the first number of iterations might apply the optimization process to the camera device parameters and global orientation, where the joint set is reduced to the wrist and the MCP joint excluding the thumb. The first number of iterations could be between 500 and 3000 iterations, for example, between 1000 and 2000 iterations, such as 1500 iterations. Then, for the second number of iterations, the optimization process can be applied to all parameters. The second number of iterations could be between 500 and 4000 iterations, for example, between 2000 and 3000 iterations, such as 2500 iterations.

[0124] As mentioned above, data collection methods can be derived from publicly available video sources (e.g., YouTube). TM Collect object images. Methods could include, for example, iterating through a list of video links, downloading videos, extracting frames, running OpenPose, fitting a 3D model (e.g., a MANO of a hand) to each frame, and selecting a small subset of filtered samples. The depth of the projected mesh can be proportional to the ratio of the standard deviation of the X-coordinate of the projected mesh to its world position. Integrity checks can be performed by thresholding the total OpenPose confidence score, the confidence score per joint, and / or the mean squared error between the projected model (e.g., a MANO) joints and the OpenPose predictions normalized by distance from the camera device.

[0125] As an example, to create the training dataset, 102 videos are obtained, and up to 500 samples per video are randomly selected that meet a threshold condition. In implementations involving hands, most samples can cover sign language dialogues. These sign language dialogues can be performed by people from various countries and / or ethnicities. Some videos may include, for example, 100 people worldwide using common sign language phrases to webcams. Validation and test sets can cover, for example, seven videos that have no intersection with the training set. Test videos can be selected to be diverse and challenging, and can include, for example, dialogues captured outdoors, dance moves, and sign languages ​​from different regions (e.g., US, Australian, and Taiwanese sign languages). Alternatively, the method can be implemented on the COCO dataset, for example, where 7048 hand images are extracted for training. Combined training, validation, and test sets can be created by combining the extracted COCO images with images extracted from the videos.

[0126] Figure 7 A flowchart is shown as an example method for generating a training dataset for training machine learning models to generate 3D models of objects from 2D images.

[0127] At operation 7.1, multiple two-dimensional images are extracted from one or more segments of video data, each two-dimensional image comprising one or more objects of a given type, such as a hand. The video data can be downloaded from an external storage library, for example, a publicly available video streaming / sharing website. In an implementation where the object type is a hand, the video may include one or more video segments depicting the use of sign language.

[0128] At step 7.2, for a given image in the extracted image, a keypoint extraction model is used to identify multiple object keypoints in the 2D image. For example, the keypoint extraction model could be a 2D pose estimation model configured to identify the joint positions and / or fingertip positions of a hand in a 2D image. An example of such a model is OpenPose, but any keypoint extraction model capable of identifying object / hand keypoints can be used alternatively.

[0129] In step 7.3, the parameters of the 3D parametric object model corresponding to the 2D image are determined. Determining the parameters involves fitting the 3D parametric object model to the 2D image using the identified object keypoints and the corresponding object keypoints in the 3D parametric object model.

[0130] A 3D parametric hand model can be defined by a set of parameters, including a shape parameter β and a pose parameter θ. The parameters may also include the scaling and position parameters of the camera used to observe the mesh. The model uses these parameters to generate a 3D hand mesh via a differentiable function. An example of such a model is the MANO model, but other parametric hand models can be used alternatively.

[0131] Keypoints in a 3D hand mesh can be identified using a sparse regression model. Keypoints can be projected onto an image plane and compared, for example, with keypoints detected in a 2D image using a loss / objective function. The loss / objective function may include a reprojection term that compares the positions of the hand keypoints identified in the 2D image to the positions of their corresponding hand keypoints in the 2D projection of the 3D parametric model. A mask that applies weights to the hand keypoints in the corresponding 2D projection can be used in the reprojection term. The loss / objective function also includes a bone length term that compares the distances between hand keypoints identified in the 2D image to the distances between their corresponding hand keypoints in the 2D projection of the 3D parametric model. The loss / objective function further includes a regularization term that penalizes deviations from a predefined average hand pose.

[0132] Based on comparisons, optimization processes can be used, for example, to update the parameters of the 3D hand model. Examples of such optimization processes include Adam and / or other gradient descent methods.

[0133] In step 7.4, the two-dimensional image and the parameters of the corresponding three-dimensional parametric hand model are stored in computer memory. The two-dimensional image and the parameters of the corresponding three-dimensional parametric hand model can be stored as part of a training dataset, a validation dataset, or a test dataset.

[0134] Operations 7.2 through 7.4 can be repeated for each image in the extracted images (or a subset of all extracted images). A subset of the extracted images that meets a certain threshold condition can be used. For example, the threshold condition could be that the confidence score of an image containing an object of a given type (e.g., a hand) is higher than a threshold (e.g., 90%).

[0135] The training set generated in this way can be used in any training method described in this paper.

[0136] Figure 8 A schematic example of a system / apparatus for performing any of the methods described herein is shown. The system / apparatus shown is an example of a computing device. Those skilled in the art will recognize that other types of computing devices / systems can be alternatively used to implement the methods described herein, such as distributed computing systems.

[0137] Device (or system) 800 includes one or more processors 802. One or more processors control the operation of other components of system / device 800. One or more processors 802 may include, for example, general-purpose processors. One or more processors 802 may be single-core or multi-core devices. One or more processors 802 may include a central processing unit (CPU) or a graphics processing unit (GPU). Alternatively, one or more processors 802 may include dedicated processing hardware, such as a RISC processor or programmable hardware with embedded firmware. Multiple processors may be included.

[0138] The system / device includes working or volatile memory 804. One or more processors can access the volatile memory 804 to process data and can control the storage of data in the memory. The volatile memory 804 may include any type of RAM such as static RAM (SRAM), dynamic RAM (DRAM), or it may include flash memory such as an SD card.

[0139] The system / device includes non-volatile memory 806. The non-volatile memory 806 stores a set of operation instructions 808 for controlling the operation of processor 802 in the form of computer-readable instructions. The non-volatile memory 806 can be any type of memory, such as read-only memory (ROM), flash memory, or magnetically driven memory.

[0140] One or more processors 802 are configured to execute operation instructions 808 to cause the system / device to perform any of the methods described herein. Operation instructions 808 may include code relating to hardware components of the system / device 800 (i.e., drivers) and code relating to the basic operation of the system / device 800. Generally, one or more processors 602 use volatile memory 804 to execute one or more instructions of operation instructions 808 that are permanently or semi-permanently stored in non-volatile memory 806 to temporarily store data generated during the execution of said operation instructions 808.

[0141] The methods described herein can be implemented as digital electronic circuit systems, integrated circuit systems, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These may include computer program products (e.g., software stored on, for example, a disk, optical disk, memory, or programmable logic device), which include computer-readable instructions, which are configured by a computer, for example, regarding... Figure 8 When the computer is executed, it causes the computer to perform one or more of the methods described herein.

[0142] Any system feature described herein can also be provided as a methodological feature, and vice versa. As used herein, device-plus-functional features can be alternatively expressed according to their corresponding structures. In particular, methodological aspects can be applied to system aspects, and vice versa.

[0143] Furthermore, any, some, and / or all features of one aspect may be applied in any suitable combination to any, some, and / or all features of any other aspect. It should also be recognized that specific combinations of the various features described and defined in any aspect of the invention may be independently implemented and / or provided and / or used.

[0144] Although several embodiments have been shown and described, those skilled in the art will recognize that changes may be made to these embodiments without departing from the principles of this disclosure, the scope of which is defined in the claims.

Claims

1. A computer-implemented method for generating a three-dimensional model of a deformable object from a two-dimensional image, the method comprising: receiving the two-dimensional image as input to an embedding neural network, wherein the two-dimensional image comprises an image of an object; generating an embedding representation of the two-dimensional image using the embedding neural network; inputting the embedding representation into a learned decoder model, wherein the decoder model is a linear regression model; and generating parameters of a three-dimensional model of the object from the embedding representation using the learned decoder model as a linear regression model.

2. The method of claim 1, wherein, the embedding neural network comprises a residual neural network.

3. The method of claim 1, wherein, the three-dimensional model comprises a mesh model and the parameters of the three-dimensional model of the object comprise mesh parameters.

4. The method of claim 1, wherein, the deformable object is a hand.

5. The method of claim 4, wherein, the decoder model comprises a linear layer and wherein the parameters of the three-dimensional model of the hand comprise three-dimensional coordinates of nodes of a mesh.

6. A computer-implemented method for training a learned decoder model to generate a three-dimensional model of a deformable object from a two-dimensional image, the method comprising: receiving a two-dimensional training image as input to an embedding neural network, wherein the two-dimensional training image comprises an object and is associated with predetermined parameters of a three-dimensional model of the deformable object; generating an embedding representation of the two-dimensional training image using the embedding neural network; inputting the embedding representation into a decoder model, wherein the decoder model is a linear regression model; generating parameters of a three-dimensional model of the object from the embedding representation using the decoder model as a linear regression model, comparing the generated parameters of the three-dimensional model of the object to the predetermined parameters of the three-dimensional model of the object associated with the two-dimensional training image using a loss function; updating parameters of the decoder model in dependence on the comparison.

7. The method of claim 6, wherein, the parameters of the three-dimensional model of the object comprise positions of a plurality of nodes in a mesh and wherein the loss function comprises a vertex term that compares the generated positions of nodes to predetermined positions of corresponding nodes associated with the two-dimensional training image.

8. The method of claim 7, wherein, the loss function comprises an edge term that compares distances between the generated positions of nodes to distances between the predetermined positions of corresponding nodes associated with the two-dimensional training image.

9. The method of any one of claims 6-8, wherein, the predetermined parameters of the three-dimensional model of the object associated with the two-dimensional training image are derived by fitting a three-dimensional object model to the two-dimensional training image.

10. The method of any one of claims 6-8, wherein, the embedding neural network comprises a residual neural network.

11. The method of any one of claims 6 to 8, wherein, the three-dimensional model comprises a mesh model and the parameters of the three-dimensional model of the object comprise mesh parameters.

12. The method of any one of claims 6 to 8, wherein, the deformable object is a hand.

13. The method of claim 12, wherein, the decoder model comprises a linear layer and wherein the parameters of the three-dimensional model of the hand comprise three-dimensional coordinates of nodes of a mesh.

14. The method of claim 6, further comprising: extracting a plurality of two-dimensional images from one or more segments of video data, the two-dimensional images each comprising one or more deformable objects; for each two-dimensional image of the plurality of two-dimensional images: identifying a plurality of object keypoints in the two-dimensional image using a keypoint identification model; and ​ determining parameters of a three-dimensional parametric model of an object, the parameters corresponding to the deformable object in the two-dimensional image, wherein determining the parameters comprises fitting the three-dimensional parametric model of the object to the two-dimensional image using the identified object keypoints and corresponding object keypoints in the three-dimensional parametric model of the object; and storing the two-dimensional image and the corresponding parameters of the three-dimensional parametric model in a computer memory as a training data set.

15. The method of claim 14, wherein, The one or more segments of video data comprise one or more segments of video of the use of sign language.

16. The method of claim 14 or 15, wherein, Fitting the three-dimensional parametric model of the object to the two-dimensional image comprises applying an iterative optimization process to an objective function, wherein the objective function comprises a re-projection term that compares the positions of the identified object keypoints in the two-dimensional image to the positions of corresponding object keypoints in a two-dimensional projection of the three-dimensional parametric model.

17. The method of claim 16, wherein, The re-projection term comprises a mask that applies a weight to the corresponding object keypoints in the two-dimensional projection.

18. The method of claim 16, wherein, The objective function further comprises a length term that compares distances between the identified object keypoints in the two-dimensional image to distances between corresponding object keypoints in the two-dimensional projection of the three-dimensional parametric model.

19. The method of claim 16, wherein, The objective function further comprises a regularization term that penalizes deviations from a pre-defined average object pose.

20. The method of claim 14 or 15, wherein, The three-dimensional parametric model of the object is a mesh model parameterized by parameters comprising one or more shape parameters and / or one or more pose parameters.

21. The method of claim 20, wherein, The parameters of the three-dimensional parametric model of the object further comprise one or more camera parameters.

22. The method of claim 20, wherein, The corresponding object keypoints in the three-dimensional parametric model are identified from the mesh model using a sparse regression model.

23. The method of claim 14 or 15, wherein, The deformable object is a hand.

24. The method of claim 23, wherein, The one or more segments of video data comprise a plurality of segments of video of the use of sign language, wherein the plurality of segments of video comprise the use of sign language from different sign languages.

25. The method of claim 23, wherein, The keypoint identification model is a two-dimensional pose estimation model configured to identify joint positions and / or fingertip positions of a hand in the two-dimensional image.

26. The method of claim 14 or 15, wherein, The two-dimensional training images and the corresponding predetermined parameters of the three-dimensional model of the deformable object associated with the two-dimensional training images are retrieved from the training data set.

27. A computer-implemented method of training a machine learning model to generate a three-dimensional model of an object, the method comprising using a training data set generated using the method of any of claims 14 to 26.

28. A computer-implemented method of generating a three-dimensional model of an object, the method comprising using a machine learning model trained using the method of claim 27.

29. An apparatus comprising a processor and a memory, the memory comprising computer-readable instructions which, when executed by the processor, cause the apparatus to perform the method of any of claims 1 to 28.

30. A computer program product comprising computer readable code which, when executed by a computer, causes the computer to carry out the method of any one of claims 1 to 28.

Images