3D Object Reconstruction

Abstract

The present disclosure relates to reconstructing a three-dimensional model of an object from a two-dimensional image. According to a first aspect, the present specification describes a computer-implemented method for creating a three-dimensional reconstruction from a two-dimensional image, the method comprising: receiving a two-dimensional image; identifying an object in the image to be reconstructed and identifying the type of the object; spatially anchoring a predetermined set of object landmarks within the image; extracting a two-dimensional image representation from each object landmark; estimating a corresponding three-dimensional representation for the corresponding two-dimensional image representation; and combining the corresponding three-dimensional representations to obtain a fused three-dimensional representation of the object.

Description

Technical Field

[0001] The present disclosure relates to reconstructing a three-dimensional model of an object from a two-dimensional image. Background Art

[0002] 3D reconstruction from a single image is a fundamentally ill-posed problem, yet humans routinely perform this operation when viewing pictures. Prior information about geometry can exploit multiple cues, such as object silhouettes, surface-to-image correspondences, or occlusions. Nevertheless, perhaps the greatest contribution to monocular 3D reconstruction comes from semantics: constrained variability in known object categories can easily resolve ambiguities in 3D reconstruction, for example if the object's shape necessarily lies in a low-dimensional space. Summary of the invention

[0003] According to a first aspect, the present specification describes a computer-implemented method for creating a three-dimensional reconstruction from a two-dimensional image, the method comprising: receiving a two-dimensional image; identifying an object in the image to be reconstructed and identifying the type of the object; spatially anchoring a set of predetermined object landmarks within the image; extracting a two-dimensional image representation from each object landmark; estimating a corresponding three-dimensional representation for the corresponding two-dimensional image representation; and combining the corresponding three-dimensional representations to obtain a fused three-dimensional representation of the object.

[0004] The steps of spatially anchoring, extracting, estimating, and combining may be performed in an encoder neural network. The encoder neural network may be one of a plurality of encoder networks, the method further comprising selecting the encoder neural network based on the identified object type.

[0005] The two-dimensional image may be a single image.Receiving may further comprise receiving a plurality of images, wherein the steps of spatially anchoring, extracting, estimating and combining are completed for each received image, for example in real time.

[0006] The combination may further include: decoupling respective estimated three-dimensional representations of kinematically independent object landmarks; and wherein each respective estimated three-dimensional representation includes a corresponding set of possible orientation angles for the respective object landmark, the method further including applying a weighting to each of the set of possible orientation angles based on a kinematic association of the respective object landmark with the respective possible orientation and landmark visibility in the received image. Prior to weighting, the method may further include identifying, within a set of predetermined object landmarks, object landmarks that are not visible in the two-dimensional image.

[0007] The method may further include applying the fused three-dimensional representation of the object to a part-based three-dimensional shape representation model. The part-based three-dimensional shape representation model may be a kinematic tree.

[0008] According to another aspect, the present specification describes a computing unit for creating a three-dimensional reconstruction from a single two-dimensional image, the computing unit comprising: a memory; and at least one processor, wherein the at least one processor is configured to execute any of the methods described herein for creating a three-dimensional reconstruction from a two-dimensional image.

[0009] According to another aspect, the present specification describes a computer-readable medium storing a set of instructions executable by at least one processor of a computing unit for creating a three-dimensional reconstruction from a two-dimensional image to cause the computing unit to perform any of the methods described herein for creating a three-dimensional reconstruction from a two-dimensional image.

[0010] According to another aspect, the present specification describes a computer program product comprising instructions which, when executed by a computer, cause the computer to perform any of the methods described herein for creating a three-dimensional reconstruction from a two-dimensional image.

[0011] According to another aspect, the present specification describes a method for updating a fused three-dimensional representation of an object included in a two-dimensional image, the method comprising: projecting the fused three-dimensional representation onto a two-dimensional image plane to obtain a projected representation; comparing corresponding positions of the projected representation with objects in the two-dimensional image; determining an error value based on the comparison; and adjusting parameters of the fused three-dimensional representation based on the error value, wherein the comparison, measurement, and adjustment are iterated until the measured error value is lower than a predetermined threshold or exceeds a threshold number of iterations.

[0012] The comparing may further include comparing corresponding positions of object landmarks and surface-based points of the projected representation and the two-dimensional image. The method may further include obtaining a Cartesian sampling or mapping of a known surface model based on the detected object type of the object in the two-dimensional image.

[0013] According to another aspect, the present specification describes a computing unit for creating a three-dimensional reconstruction from a single two-dimensional image, the computing unit comprising: a memory; and at least one processor, wherein the at least one processor is configured to perform any of the methods described herein for updating a fused three-dimensional representation of an object.

[0014] According to another aspect, the present specification describes a computer-readable medium storing a set of instructions executable by at least one processor of a computing unit for creating a three-dimensional reconstruction from a two-dimensional image to cause the computing unit to perform any of the methods described herein for updating a fused three-dimensional representation of an object.

[0015] According to another aspect, the present specification describes a computer program product comprising instructions that, when executed by a computer, cause the computer to perform any of the methods for updating a fused three-dimensional representation of an object described herein.

[0016] According to another aspect, the present specification describes a system for providing a three-dimensional representation of an object identified in a two-dimensional image, the system comprising: a first computing unit configured to perform any of the methods described herein for creating a three-dimensional reconstruction from the two-dimensional image; and a second computing unit configured to perform any of the methods described herein for updating a fused three-dimensional representation of the object.

[0017] In the following, examples and embodiments will be described with reference to 3D human reconstruction. However, it will be appreciated that the methods disclosed herein may be applied to the reconstruction of other 3D objects, such as animals. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Embodiments will now be described by way of non-limiting examples with reference to the accompanying drawings, in which:

[0019] Figure 1 A schematic overview showing a method for part-based three-dimensional reconstruction from a two-dimensional image;

[0020] Figure 2 A flow chart illustrating an example method for creating a three-dimensional reconstruction from a two-dimensional image;

[0021] Figure 3 An example showing the Euler angles cluster centers used before Mixture-of-experts rotation;

[0022] Figure 4 A schematic overview showing a method for refining a three-dimensional representation of an object;

[0023] Figure 5 A flow chart illustrating an example method for updating a fused three-dimensional representation of an object;

[0024] Figure 6a Table showing DensePose and 3D keypoint errors achieved using various refinement techniques;

[0025] Figure 6b Examples showing qualitative results of the methods disclosed herein; and

[0026] Figure 7 An example of a computing unit is shown. DETAILED DESCRIPTION

[0027] Part-based models are used in monocular 3D human reconstruction methods. Because they are part-based, the method can operate in natural scenes, gracefully handling severe occlusions and large pose changes. A multi-task network including 2D, 3D and Dense Pose estimation is trained to drive the 3D reconstruction task through a synergistic approach that combines bottom-up cues with top-down model fitting. In addition, an iterative refinement method that aligns model-based 3D reconstructions with their image-based estimates can be used, which is attributed to bottom-up cues, thereby achieving both model-based global consistency and high spatial accuracy. The method allows both accurate joint and 3D surface estimation for a variety of data sets. Part-based estimation reduces the 3D joint estimation error of the best published method from 56.8 to 50.56, while refinement further reduces it to 46.52.

[0028] Deformable models have been previously used for monocular 3D face reconstruction. To extend it to the more complex articulated structure of the human body, monocular human reconstruction has previously combined part-based representations, sampling-based reasoning, spatiotemporal reasoning, and bottom-up / top-down. In the context of deep learning, monocular 3D reconstruction has witnessed both generic class and human-specific restorations. Most of these works rely on an effective parameterization of the human body in terms of a parametric deformable model of the human body (also referred to herein as a “parameterized model”). Examples of such models include, but are not limited to, skinned linear models, such as the Skinned Multi-Person Linear (SMPL) model (e.g., "SMPL: A Skinned Multi-Person Linear Model", M. Loper et al., ACM SIGGRAPH Asia, 2015), linear hybrid skinning models (e.g., "Joint-dependent Local Deformations for Hand Animation and Object Grasping" N. Magnenat-Thalmann et al., Proceedings on Graphics Interface'88, Canadian Information Processing Society), and / or the SCAPE model ("SCAPE: Shape Completion and Animation of People", D. Anguelov et al., ACM SIGGRAPH 2005). Exploiting the fact that parametric deformable models of human models provide a low-dimensional differentiable representation of the human body, these works train the system to regress model parameters by minimizing the reprojection error between parameterized deformable models based on 3D key points and 2D joint annotations of the human body, human segmentation masks, and 3D volume projections or even refinement to the level of body parts.

[0029] At the same time, the accuracy of 3D human joint estimation has been significantly improved by moving from the classic "structure-from-motion" approach to a 3DCNN-based architecture that directly localizes 3D joints in the volumetric output space through a hybrid of classification and regression.

[0030] Recent work on DensePose estimation (e.g., "DensePose: Dense human pose estimation in the wild" RAGuler et al., The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018) also shows that dense correspondences between RGB images and parametric models can be estimated by training a general bottom-up detection system to associate image pixels with surface-level UV coordinates. However, while DensePose establishes a direct link between the image and the surface, it does not discover the underlying 3D geometry, but rather gives strong hints about it.

[0031] Connecting these individual lines of research into a synergistic architecture that combines the capabilities of different methods can improve the performance of monocular 3D reconstruction. Model-based monocular 3D reconstruction methods estimate model parameters via a single linear layer applied on top of CNN features extracted within the object bounding box. Such a holistic system can be challenged by feature changes due to bounding box misalignment, such as occlusion, rotation, or global translation. For example, by the definition of the convolution operation, translation leads to a rearrangement of feature sizes.

[0032] Figure 1 A schematic overview of a method 100 for part-based three-dimensional reconstruction 102 from a two-dimensional image 104 is shown. A keypoint detector indicates 2D keypoint (e.g., joint) locations, thereby driving feature pooling. These in turn are used to determine 3D model parameters (e.g., 3D SMPL joint angles) of a parametric model. The method can be implemented on one or more computing devices. The method follows a part-based modeling paradigm, extracting localized features around keypoints (human joints in the example shown). The resulting model can be referred to as a fused three-dimensional representation.

[0033] The method uses a rigorous prior model of the object's shape (such as a deformable parametric model for human body reconstruction) in combination with multiple pose estimation methods to derive an accurate monocular 3D reconstruction. In embodiments related to 3D human reconstruction, the articulated nature of the human body is taken into account, which can significantly improve performance over the overall baseline. A refinement process can also be used that allows the shape prediction results of the single-shot system to be adjusted to satisfy the geometric constraints imposed by the complementary fully convolutional network / neural network branch.

[0034] The 3D reconstruction method relies on a prior model of the target shape. The prior model can be a parametric model of the object being reconstructed. The type of prior model used may depend on the object being reconstructed. For example, in an embodiment where the shape is a human body, the human body can be parameterized using a parametric deformable model. In the following, the method will be described with reference to a parametric deformable model, but it will be understood that other parametric shape models may be used instead. In some embodiments, SMPL is used. SMPL uses two parameter vectors to capture pose and shape: θ includes 23 3D rotation matrices corresponding to each joint in the kinematic tree of the human body pose, and β captures the shape variability across the object in terms of a 10-dimensional shape vector. SMPL uses these parameters to obtain a triangulated mesh of the human body through linear skinning and blended shapes (as differential functions of θ, β). In some embodiments, as described below with respect to Figure 3 As described, a parametric deformable model can be modified to allow it to be used more efficiently by a convolutional neural network.

[0035] A two-dimensional image 104 is received and input into a neural network 106. The two-dimensional image I includes a set of pixel values ​​corresponding to a two-dimensional array. For example, in a color image, Where H is the image height in pixels, W is the image height in pixels, and the image has three color channels (eg, RGB or CIELAB). In some embodiments, the two-dimensional image may be black and white / grayscale.

[0036] The neural network may include a backbone network 206a, a keypoint localization head 106b, and a feature extraction head 106c. The neural network 106 may include one or more convolutional and / or deconvolutional layers / networks.

[0037] The neural network includes multiple node layers, each node is associated with one or more parameters. The parameters of each node of the neural network may include one or more weights and / or biases. The node takes one or more outputs of the node in the previous layer as input, or as input data in the initial layer. Each node uses one or more outputs of the node in the previous layer to generate an activation value using an activation function and the parameters of the neural network. One or more layers of the neural network may be a convolutional layer.

[0038] The backbone network 106a provides the initial shared layers of the neural network 106. In the example shown, the backbone network is a ResNet-50 network, but it will be appreciated that other neural networks may alternatively be used as the backbone network. The output of the backbone neural network 206a is used as input to each of the following neural network heads 106b, 106c.

[0039] The neural network outputs the keypoint locations in the image 108 from the keypoint localization branch 106b of the neural network. The keypoints correspond to a predetermined set of object landmarks within the image. Determining the locations of these keypoints can effectively spatially anchor the object landmarks within the image.

[0040] The feature extraction branch 106c of the neural network is used to extract features 110. The location of the extracted features varies with the keypoint location. Therefore, the features are invariant to global transformations and can focus on fine-grained local patterns that can better reveal the underlying 3D geometry.

[0041] The extracted features 110 are pooled to corresponding keypoints (such as visible joint locations in the image) using bilinear interpolation. A part-based regressor is used that pools convolutional features around the 2D keypoint locations estimated by the keypoint localization head 106b. This allows for the extraction of refined local features that are largely invariant to joints while tracking the presence / absence of parts. In particular, a part-based attention mechanism is used that limits the influence of any part to the parameters of the parametric model associated with it. This can limit the impact that a missing part may have on the parameters of the global parametric model and make the model robust to occlusion.

[0042] Each feature extracted around a 2D keypoint / joint can deliver a separate parametric deformable model parameter estimate. But intuitively, a 2D keypoint / joint should have a larger influence on the parameters of the parametric deformable model to which it is more related - for example, the left wrist joint should affect the left arm parameters, rather than the parameters of dynamically independent parts (such as the right arm, head or feet). Furthermore, the fact that some keypoints / joints may be missing in an image means that we cannot simply concatenate the features in a larger feature vector, but need to use a form that can handle possible missing parts well.

[0043] A linear layer 112 applied to the pooled extracted features is used to estimate the parameters 114 of the parametric deformable model. The linear layer may, for example, output weights ω i k,j 116, for determining parameters 114 of the parametric deformable model.

[0044] In order to determine the parameter deformable model angle θi , using information from the neighborhood N(i) of angle i. In some embodiments, angle θ i This can be determined using the following methods:

[0045]

[0046] We can use an arg-soft-max operation on the angle cluster and fuse the information from multiple 2D joints: i k,j Indicates the assignment of 2D joint j to the i-th model parameter θ i The score of cluster k. Clusters can be defined as described below with respect to the “Expert Mixture Prior”. Based on kinematic tree correlations, a neighborhood of i is constructed by checking which model parameters directly affect the 2D joints of the human. By utilizing the maximum value of the 2D joint detection module, the joints can be found in the image. If the maximum value is below a threshold (for example, 0:4 in some embodiments), it is considered that no joints are observed in the image. In this case, each summation term corresponding to the missing joint is not included in the summation of the numerator and denominator, so that the above equation can still provide an angular distribution. If all elements of N(i) are absent, θ is replaced by i Set to rest pose.

[0047] Figure 2 A flow chart is shown of an exemplary method 200 for creating a three-dimensional reconstruction from a two-dimensional image. The method may be performed by one or more computing devices.

[0048] At operation 2.1, a two-dimensional image is received. As described above, a 2D image may include a set of pixel values ​​corresponding to a two-dimensional array. In some embodiments, a single image is received. Alternatively, multiple images may be received. If multiple images (e.g., of a moving person) are taken, a three-dimensional representation may account for these changes.

[0049] At operation 2.2, an object to be reconstructed is identified in the image. An object type of the identified object is also identified.

[0050] At operation 2.3, a predetermined set of object landmarks within the image are spatially anchored. Spatially anchoring may refer to locking to a portion of the image that is of interest for computation. An object landmark may refer to a portion of an object to be reconstructed that is identified as an area of ​​interest for computation. For example, in the case where the object is a human, the object landmark may be a joint of the human. In some embodiments, a training set or training database that may be annotated by a human operator may be employed to perform the pre-determination of the object landmarks.

[0051] At operation 2.4, a two-dimensional image is extracted from each object landmark.

[0052] At operation 2.5, a corresponding three-dimensional representation is estimated for the corresponding two-dimensional image representation. Each corresponding three-dimensional representation may include a corresponding set of possible orientation angles of the corresponding object landmark, and the method may further include: applying a weighting to each orientation angle in the set of possible orientation angles based on a kinematic association of the corresponding object landmark with the corresponding possible orientation and a visibility of the landmark in the received image.

[0053] Possible orientation angles may refer to all possible positions that a landmark may have. Each landmark may initially include a default set of possible orientations. Given an estimated three-dimensional representation of a given landmark, some embodiments focus on determining more likely orientations within the default set. For example, where the object is a human and the object landmark is a shoulder, the orientation angle may represent different rotational positions that a shoulder-torso-upper arm joint (e.g., a body part associated with or affected by the shoulder) may take. In this example, a dynamic association may refer to a more likely orientation angle given another estimated object landmark. Visibility refers to whether a predetermined object landmark is visible in a received image. If an object landmark is not visible in the received image, the orientation angle associated with the invisible landmark will not be affected by the angle.

[0054] Prior to weighting, the method may further include identifying object landmarks within a predetermined set of object landmarks that are not visible in the two-dimensional image. Object landmarks that are expected to be present due to the identified object type may not be present in the image. For example, if the object type is a human, parts of the human body may not be visible due to posture or other items that obstruct the view. This embodiment discusses identifying such invisible landmarks so that they can be given a lower weight because they will not be useful in the three-dimensional reconstruction.

[0055] At operation 2.6, the corresponding three-dimensional representations are combined to generate a fused three-dimensional representation of the object.

[0056] Combining the corresponding three-dimensional representations may further include decoupling the corresponding estimated 3D representations of kinematically independent object landmarks. The term kinematically independent refers to how the motion of different object landmarks relates to the assumed three-dimensional representation. For example, if the object is a human: an object landmark of an ankle will be kinematically independent from the angle of the torso-leg joint, because the motion of the leg does not affect the motion of the ankle. In contrast, if the object landmark is an elbow, it will not be kinematically independent from the arm-torso motion, because the motion of the arm will likely affect the motion or position of the elbow.

[0057] The method may further comprise applying the fused three-dimensional representation of the object to a part-based three-dimensional shape representation model. An example of such a model is a kinematic tree.

[0058] In embodiments where multiple images are received, operations 2.3 to 2.6 may be performed for each of the multiple images. This may be performed in real time as each image is received.

[0059] Operations 2.3 to 2.6 may be performed by a neural network such as an encoder neural network. The neural network may have multiple heads, each head outputting data related to a different aspect of the method.

[0060] In some embodiments, multiple neural networks are available, and the neural network used is selected based on the type of object identified. According to some example embodiments, the neural network may be specific to the object type. For example, a neural network may be dedicated to a three-dimensional representation of a human being, while a different neural network may be dedicated to a three-dimensional representation of an animal. It should be understood that a set of predetermined object landmarks may also be specific to the type of object identified. It should be understood that the neural network and object landmarks may be specific to, for example, humans, animals, etc.

[0061] In some embodiments, the parametric model can be modified so that it interfaces more efficiently with the convolutional neural network architecture. Examples of such modifications include "Cartesian surface parameterization" and "expert hybrid rotation priors". Such modifications can be combined with the parametric model together or independently. A unified Cartesian mapping of the UV space within each part can be used, effectively re-parameterizing the characteristic shapes and blend shapes and basis of the parametric model. This can more easily implement mesh-level operations, resulting in a simple and fast GPU-based implementation of model refinement.

[0062] Cartesian surface parameterization reparameterizes the body surface using a local Cartesian coordinate system. Even though the body surface is a continuous structure, it can be discretized using a triangular mesh. This means that associating a pair of continuous UV coordinates with a mesh attribute (such as a 3D position) first requires identifying the facet containing the UV coordinates, looking up the vertex values ​​supporting the facet, and then interpolating these values ​​using the barycentric coordinates of the point. This can be inefficient, especially if different memory locations need to be accessed for different vertices.

[0063] It is advantageous to reparameterize the body surface in a local Cartesian coordinate system. This allows the above process to be replaced by bilinear interpolation and use of space transformer layers to efficiently handle large numbers of points. To perform this reparameterization, "multidimensional scaling" is first performed to flatten parts of the parametric model surface into two dimensions, and then these parts are uniformly sampled on a grid.

[0064] Figure 3An example of Euler angle cluster centers 300 used in an expert hybrid rotation prior is shown. The range of possible joint angle values ​​302a-d for joints 304a-d is limited by human body mechanics. This can be exploited to improve the accuracy of our angular joint estimates. By explicitly forcing the predictions of joint angles 302a-d to lie on the manifold of plausible shapes for the joints 304a-d, a simple yet rigorous prior can be constructed.

[0065] Unlike early work that aimed to analytically model joint angles, recent work uses classification as a proxy for rotation estimation (see, for example, S. Tulsiani et al., "Viewpoints and keypoints", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1510-1519, 2015): instead of predicting Euler angles, rotation estimation is converted into a classification problem where the categories correspond to disjoint angular partitions. This approach is consistent with the empirical observation that CNNs can improve their regression accuracy by exploiting classification within regression.

[0066] Based on this direction, a "mixture of experts" angle regression layer can be built using data collected, for example, by recording the joint angles of humans when they stretch. They are expected to cover the space of possible joint angles very well. For each joint, the joint rotation is represented as Euler angles θ, and K rotation clusters θ1...θ are calculated via K-means. K These clusters provide a set of representative angle values. This allows the method / system to predict any rotation value within the convex hull of these clusters by using a softmax weighted combination of the clusters. In particular, the Euler rotation θ of the i-th parameter model joint i It can be calculated as:

[0067]

[0068] where ω k is the real-valued input to this layer (e.g., a set of weights output by the linear layer 112). Figure 3 As shown in , this forms a reasonable approximation to the underlying angle distribution while completely avoiding the need for adversarial training since, by design, the estimated angles will come from this prior distribution. The output space of the joint regressor is restricted to the convex hull at the center for achievable joint rotations.

[0069] Figure 4 A schematic overview of a method 400 for refining a three-dimensional representation of an object 402 is shown.

[0070] An additional set of data, such as DensePose and 3D joint estimates, can be used to improve the accuracy of 3D reconstruction. This can be achieved both by introducing an additional loss that can regularize the training in a standard multi-task learning setting and by providing bottom-up CNN-based cues to drive top-down model-based 3D reconstruction. To this end, instead of relying on a CNN-based system to deliver accurate results in a single shot at test time, predictions from the CNN are used as initialization for an iterative fitting process that uses the pose estimation results delivered by the 2D decoder to drive model parameter updates. This can lead to effective 3D reconstruction. To this end, the parametric model parameters are updated to align the parametric model-based and CNN-based pose estimates as captured by a combination of the DensePose-based loss and the distance between the parametric model-based and CNN-based estimates of the 3D joints. This allows real-time updates to the parametric model parameters to better match the CNN-based localization results.

[0071] The above about Figure 1-2 The described method provides a "bottom-up" estimate of the 3D surface of the human body in a single shot (i.e., via a forward pass through the network). In the same feed-forward manner, 2D keypoints, 3D joint locations, and / or Dense-Pose

[13] estimates can be obtained via a fully convolutional network. Figure 1-2 During the described approach, 2D keypoints have been obtained as part of part-based pooling, but the combined processing of all tasks can improve the ultimate goal of accurate 3D surface reconstruction.

[0072] Above about Figure 1-3 The described method performs 3D reconstruction. However, supervision is generally lacking for this task. In contrast, large-scale datasets exist for 2D pose estimation. These datasets can be exploited by training the network in a multi-task setting such that it not only provides the parameters of the parametric deformable model, but also DensePose, 2D and 3D joint predictions. Due to the multi-task processing, this can improve the parameter estimation of the parametric deformable model and can also provide a general pose estimation model. In addition to this improvement, external pose datasets can be further exploited in a way that exploits the geometric properties of the task.

[0073] Having a 3D surface estimate allows predicting 3D joint locations, their 2D projections, and dense surface-to-image correspondences in a differentiable way. Therefore, external pose information can be used to construct a loss function that indicates the quality of our surface estimate in terms of geometric distances. This is achieved in two ways: First, multiple pose estimation cues are used for network supervision during training. Second, a refinement process is used that enforces the model-based 3D geometry to be consistent with the neural network's predictions through an iterative scheme. In both cases, the geometric nature of the problem is exploited. A loss that penalizes the deviation between the 3D model-based predictions and the pose information provided by the complementary cues is used.

[0074] Method 400 is similar to the above Figure 1-3 The method described above is performed in a similar manner. A two-dimensional image 404 is received and input into a neural network 406. The neural network 406 processes the input image 404 using a part-based 3D shape reconstruction 410 (e.g., as described above) to generate an initial 3D model 408 (i.e., a fused three-dimensional model / 3D surface estimate) of an object (e.g., a human body) in the image. The neural network further outputs 2D key point locations (e.g., 2D joints), 3D key point locations (e.g., 3D joints), and DensePose estimates. Each of these can be output from a different head 406b-e of the neural network 406. The neural network 406 may include one or more convolutional and / or deconvolutional layers / networks. The initial layers of the neural network 406 may be shared and form a backbone network 406a. In the example shown, the backbone network 406a is a ResNet-50 network, but other networks may be used instead.

[0075] The initial 3D model 408 may be refined using a 3D shape refinement process 412 . The 3D shape refinement process 412 takes as input the 3D model of the corresponding 2D image, 2D keypoint locations, 3D keypoint locations, and DensePose estimates, and uses them to refine the 3D model to generate a refined 3D model of the object 402 .

[0076] The 3D shape refinement process 412 may include projecting the 3D model 408 onto a 2D image plane to obtain a projected representation. The locations in the projected representation are compared to objects in the input image 404 to determine an error value (e.g., using a loss function). Parameters of the 3D model 408 may then be adjusted based on the error value to determine an updated 3D model. The process may be iterated until a threshold condition is met.

[0077] The loss function used to determine the error may include one or more geometric losses, such as a 2D keypoint loss (L Kps-2D ), 3D key point loss (L Kps-3D) and / or DensePose loss (L DP ). These loss functions can also be used when training the neural network 406.

[0078] 2D keypoint loss (L Kps-2D ) compares the position of the 2D keypoint in the projected representation with the position of the corresponding keypoint in the input image 204 that has been determined by the 2D keypoint head 406d of the neural network 406. A geometric distance may be used as the comparison.

[0079] 3D keypoint loss (L Kps-3D ) compares the position of the 3D keypoint in the 3D model 408 with the position of the corresponding 3D keypoint in the input image 204 that has been determined by the 3D keypoint head 406e of the neural network 406. A geometric distance may be used as the comparison.

[0080] DensePose loss (L DP ) compares the DensePose position of the 3D model 408 with the DensePose estimate that has been determined by the DensePose head 406c of the neural network 406.

[0081] DensePose associates image positions x = (x1, x2) with intrinsic surface coordinates u = (u1, u2). Given a set of parameters of a 3D parametric deformable model (PDM), φ (e.g., SMPL parameters, φ = (θ, β)), each u vector can be associated with a 3D position X(φ), which can in turn be projected to a 2D position estimate, DensePose loss penalty The geometric distance between and x = (x1, x2) requires (φ) to produce a shape that is correctly projected in 2D. In summary, DensePose is associated with the following process and loss:

[0082]

[0083]

[0084] in is a model-based estimate of what x should be, and i covers image positions associated with surface coordinates.

[0085] The use of these losses can exploit the complementary strengths of the outputs of the heads 406b-e of the neural network 406. On the one hand, the model-based estimates are inherently dense, 3D, compact, and necessarily correspond to plausible human poses - which is generally not the case for the neural network 406 estimates, as its feed-forward architecture makes it difficult to impose lateral constraints between parts. On the other hand, the neural network 406 based estimates examine and exhaustively evaluate each image patch, allowing us to precisely localize human structures in the image. In contrast, the model-based estimates can be severely off due to, for example, some miscalculated angles at the beginning of the kinematic tree.

[0086] The use of these losses reconciles these complementary factors by forcing the model-based estimate of the 3D structure to be consistent with the DensePose / 2D / 3D keypoint predictions based on the neural network 406. To this end, the neural network-based predictions are treated as the initialization of an iterative fitting scheme driven by the sum of the geometric losses.

[0087] The Cartesian parameterization outlined above can also be used to quickly evaluate the loss using, for example, a Space Transformer network, and backpropagate through the loss. This can provide a GPU-based implementation of 3D model fitting to 2D images.

[0088] Figure 5 A flowchart showing an example method 500 for updating a fused three-dimensional representation of an object. The method may correspond to Figure 4 3D shape refinement process 412 .

[0089] At operation 5.1, the fused three-dimensional representation is projected onto a two-dimensional image plane to generate a projected representation.

[0090] At operation 5.2, the corresponding position of the projected representation is compared with the corresponding object in the corresponding two-dimensional image.

[0091] The comparison may include comparing the projection representation with corresponding positions of object landmarks and surface-based points of the single two-dimensional image. According to some example embodiments, both the object landmarks of the projection representation and the surface-based points may be used in the comparison. This may provide an accurate measurement of errors due to misalignment.

[0092] The method may further comprise obtaining a Cartesian sampling or mapping of a known surface model based on the object type of the object detected in the single two-dimensional image. This need not necessarily be performed at each iteration. The obtaining step may be performed so as to provide an optimization method based on initial learning.

[0093] At operation 5.3, an error value is determined based on the comparison. The error may also be referred to as a refinement loss. The refinement loss may include many components. For example, the loss function may include one or more of the following: 2D keypoint loss (LKps-2D ); 3D key point loss (L Kps-3D ); and / or DensePose loss (L DP ).

[0094] At operation 5.4, a threshold condition is checked. The threshold condition may be an error value falling below a predetermined threshold and / or when a threshold number of iterations is exceeded.

[0095] If the threshold condition is met, the method ends and the fused 3D representation is output.

[0096] If the threshold condition is not met, the method continues.

[0097] At operation 5.5, parameters of the fused three-dimensional representation are adjusted based on the error value. The adjustment may be performed using a learning-based optimization method.

[0098] It will be appreciated that the evaluation of the threshold condition may alternatively occur after the parameters have been updated in operation 5.5.

[0099] Figure 6a and Figure 6b An example showing the results of the refinement method. Figure 6a Table 600 shows the DensePose and 3D keypoint errors achieved using various refinement techniques. The effect of different refinement loss choices on the performance of two tasks (dense correspondence estimation and 3D keypoint localization in natural scenes) is shown. The second column 602 shows the base performance. The base performance refers to the output of the neural network without refinement, such as the above Figure 1-3 The third column 604 shows the loss of 2D keypoints (L Kps-2D ) is used for the refinement method. The fourth column 606 shows the error when the 3D keypoint loss (L Kps-3D ) is used for the refinement method. The fifth column 608 shows the error when the DensePose loss (L DP ) is used for the refinement method. The sixth column 610 shows the error when the 2D keypoint loss and the DensePose loss (L Kps-2D +L DP ) are both used for the refinement method. The seventh column 612 shows the error when the 3D keypoint loss and the DensePose loss (L Kps-3D +L DP ) are both used to account for errors when refining the method.

[0100] Figure 6b Examples 614 of qualitative results of the methods disclosed herein are shown. The first column 616 shows examples of input images. The second column 618 shows examples of 3D models resulting from using HMR. The third column 620 shows examples of 3D models generated by Figure 1-3The fourth column 622 shows an example of a 3D model obtained by the method disclosed in Figure 4-5 The fifth column 624 and the sixth column 626 show examples of 3D models obtained by applying the refinement.

[0101] In some embodiments, the neural network used in the above method includes a ResNet-50 network pre-trained on ImageNet as the backbone of the system. Each dense prediction task can use a deconvolution head, following the architecture choice of "Simplebaselines for human pose estimation and tracking" (B. Xiao et al., arXiv:1804.06208, 2018), in which three 4x4 deconvolution layers are applied with batch norm and ReLU. These are followed by linear layers to obtain outputs of the desired dimensions. Each deconvolution layer has a stride of 2, assuming a 256x256 image as input, producing an output resolution of 64x64.

[0102] In some embodiments, the neural network described herein can be trained in two stages. In the first stage, the 3D key points and 2D key points are trained. A combination of the MPII and H36m datasets can be used to train the 3D key points and 2D key points. An example of a method that can be used to train the 3D key points and 2D key points can be found in "Integral human pose regression" (X. Sun et al., arXiv:1711.08229, 2017).

[0103] In the second training phase, the DensePose head and the part-based 3D reconstruction head can be attached to the neural network. The entire system can then be trained end-to-end. For example, the part-based 3D reconstruction head can be trained using 2D and 3D keypoints and the dense correspondences present in all the above datasets.

[0104] In some embodiments, a neural mesh synthesis model can be used; the use of a distributed representation can more easily accommodate multimodal distributions, covering male, female, and child surfaces that are currently handled by separate shape models. In addition, more accurate geometric modeling can be used, for example, by incorporating perspective projection, surface normals, and contour information. Depth data, multiple views, and / or temporal information can also be used to help disambiguate 3D reconstruction errors.

[0105] Figure 7An example of a computing unit / device is shown. According to an exemplary embodiment, the apparatus 700 includes a processing device 702 and a memory 704. Computer readable code 706 may be stored on the memory 704 and, when executed by the processing device 702, may cause the apparatus 700 to perform the methods described herein.

[0106] The processing device 702 may have any suitable composition and may include one or more processors of any suitable type or combination of suitable types. In fact, the term "processing device" should be understood to cover computers with different architectures (such as single / multi-processor architectures and sequencer / parallel architectures). For example, the processing device may be a programmable processor that interprets computer program instructions and processes data. The processing device may include multiple programmable processors. Alternatively, the processing device may be programmable hardware, for example, with embedded firmware. The processing device may alternatively or additionally include a graphics processing unit (GPU), or one or more dedicated circuits, such as a field programmable gate array FPGA, an application specific integrated circuit (ASIC), a signal processing device, etc. In some cases, the processing device may be referred to as a computing device or a processing unit.

[0107] The processing device 702 is coupled to the memory 704 and is operable to write data to the memory 704 or read data from the memory 704. The memory 704 may include a single storage unit or multiple storage units on which computer-readable instructions (or codes) are stored. For example, the memory may include both volatile memory and non-volatile memory. In this example, the computer-readable instructions / program code may be stored in the non-volatile memory and may be executed by the processing device using the volatile memory for temporary storage of data or data and instructions. Examples of volatile memory include RAM, DRAM, SDRAM, etc. Examples of non-volatile memory include ROM, PROM, EEPROM, flash memory, optical memory, magnetic memory, etc.

[0108] The various example embodiments described herein are described in the general context of method steps or processes, which can be implemented in one aspect by a computer program product embodied in a computer-readable medium, which includes computer executable instructions such as program code executed by a computer in a networked environment. Computer-readable media may include removable and non-removable storage devices, including but not limited to read-only memory (ROM), random access memory (RAM), compact disk (CD), digital versatile disk (DVD), etc. In general, program modules may include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Computer executable instructions, associated data structures, and program modules represent examples of program codes for performing the steps of the methods disclosed herein. A specific sequence of such executable instructions or associated data structures represents an example of corresponding actions for implementing the functions described in such steps or processes.

[0109] In the foregoing description, embodiments have been described with reference to many specific details that may vary with implementation. Certain changes and modifications may be made to the described embodiments. Other embodiments will be apparent to those skilled in the art in view of the description and practice of the invention disclosed herein. It is intended that the description and embodiments are considered to be exemplary only, with the true scope and spirit of the invention being indicated by the appended claims. It is also intended that the order of steps shown in the accompanying drawings is for illustrative purposes only and is not intended to be limited to any particular order of steps. Thus, it will be appreciated by those skilled in the art that these steps may be performed in different orders while implementing the same method.

[0110] In the drawings and the specification, exemplary embodiments have been disclosed. However, many variations and modifications may be made to these embodiments. Therefore, although specific terms are employed, they are used only in a general and descriptive sense, and not for the purpose of limitation, and the scope of the embodiments is limited by the exemplary embodiments set forth herein.

Claims

1. A computer-implemented method for creating a three-dimensional reconstruction from a two-dimensional image, the method comprising: receiving a two-dimensional image; identifying an object in the image to be reconstructed and identifying the type of the object; spatially anchoring a predetermined set of object landmarks within the image; extracting a two-dimensional image representation from each object landmark in the predetermined set of object landmarks to generate a plurality of two-dimensional images each corresponding to a different object landmark; identifying one or more missing object landmarks that are missing in the identified object; applying a function to the two-dimensional image representation to determine an orientation angle for each of a plurality of joints of the object, the function comprising a sum of a numerator and a denominator, the one or more missing object landmarks not being included in the sum of the numerator and the denominator; estimating a respective three-dimensional representation of each of the plurality of two-dimensional images, the estimating comprising limiting an effect of the one or more missing object landmarks to generate a plurality of separate three-dimensional representations of each of different object landmarks corresponding to each respective one of the plurality of two-dimensional images; as well as The corresponding three-dimensional representations are combined to obtain a fused three-dimensional representation of the object, the corresponding three-dimensional representation comprising the multiple individual three-dimensional representations of each of the different object landmarks.

2. The method according to claim 1, wherein: The steps of spatially anchoring, extracting, estimating, and combining are performed in the encoder neural network.

3. The method according to claim 2, wherein: The encoder neural network is one of a plurality of encoder networks, and the method further comprises selecting the encoder neural network based on the identified object type.

4. The method according to any one of claims 1 to 3, wherein: The receiving further comprises receiving a plurality of images, wherein the steps of spatially anchoring, extracting, estimating and combining are performed in real time for each received image.

5. The method according to any one of claims 1 to 3, wherein: The combination further comprises: decoupling corresponding estimated three-dimensional representations of kinematically independent object landmarks; and wherein each respective estimated three-dimensional representation comprises a corresponding set of possible orientation angles of the respective object landmark, the method further comprising applying a weighting to each of the set of possible orientation angles based on a kinematic association of the respective object landmark with the respective possible orientation and a visibility of the landmark in the received image.

6. The method according to claim 5, wherein: Prior to weighting, the method further comprises identifying object landmarks within a predetermined set of object landmarks that are not visible in the two-dimensional image.

7. The method according to any one of claims 1 to 3, wherein: The method further includes applying the fused three-dimensional representation of the object to a part-based three-dimensional shape representation model.

8. The method according to claim 7, wherein: The part-based three-dimensional shape representation model is a kinematic tree.

9. The method according to claim 1, further comprising: calculating a maximum value of the output of the two-dimensional joint detection module to identify the set of object landmarks; comparing the maximum value with a threshold value; as well as In response to determining that the maximum value does not exceed the threshold, it is determined that an object landmark in the set of object landmarks corresponds to the one or more object landmarks that are missing from the identified object.

10. The method according to claim 1, wherein: Each of the plurality of two-dimensional images corresponds to a different joint of the plurality of joints on the human body.

11. The method according to claim 1, further comprising: The one or more object landmarks that are determined to be missing from the identified object are not visible in the two-dimensional image.

12. The method according to claim 11, further comprising: The identification identifies a plurality of object landmarks that are expected to be present based on the type of the identified object.

13. The method of claim 1, further comprising weighting the one or more object landmarks that are missing from the identified object less heavily in generating the fused three-dimensional representation of the object.

14. The method according to claim 1, further comprising: Determine the type of object that corresponds to the animal; as well as A neural network specific to the animal is selected to perform estimation of a corresponding three-dimensional representation of the corresponding two-dimensional image representation.

15. A computing unit for creating a three-dimensional reconstruction from a single two-dimensional image, the computing unit comprising: Memory; as well as At least one processor, wherein the at least one processor is configured to perform the method of any one of claims 1 to 14.

16. A computer readable medium storing a set of instructions executable by at least one processor of a computing unit for creating a three-dimensional reconstruction from a single two-dimensional image, so as to cause the computing unit to perform the method according to any one of claims 1 to 14.

17. A computer program product comprising instructions, which, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 14.

Images