Reconstructing a three-dimensional scene from two-dimensional images

Abstract

This specification relates to using neural networks to reconstruct three-dimensional (3D) scenes from two-dimensional (2D) images. According to a first aspect of the present specification, a method for creating a three-dimensional reconstruction of a scene having multiple objects from a single two-dimensional image is described, the method comprising: receiving a single two-dimensional image; identifying all objects in the image to be reconstructed and identifying the type of the objects; estimating a three-dimensional representation of each identified object; estimating a three-dimensional plane that physically supports all three-dimensional objects; and positioning all three-dimensional objects in space relative to the support plane.

Description

TECHNICAL FIELD

[0001] This specification relates to using neural networks to reconstruct three-dimensional (3D) scenes from two-dimensional (2D) images. BACKGROUND

[0002] Apart from the comparatively easy face problem, there is little research for real-time integration of 3D human bodies (or other objects) into augmented reality applications. Bodies are generally more difficult than faces because of their joint deformations, multiple and self-occlusions, complex interactions with other objects and people, and greater appearance variability due to clothing.

[0003] Pose estimation algorithms usually aim at localizing certain sparse points in the image, such as skeletal joints and facial landmarks, or more recently dense surface coordinates. Despite their increasing accuracy, these representations still do not satisfy downstream applications, such as augmented reality, motion capture, gaming, or graphics. This requires three-dimensional access to the underlying human surface and currently resorts to multi-camera setups or depth sensors.

[0004] Research on morphable models showed that by using a low-dimensional parameterization of face surfaces and appearances and turning the reconstruction task into an optimization problem, accurate monocular surface reconstruction can be performed. Extending this to the more complex human articulated structure, monocular human reconstruction has been extensively studied over the past decade, combining part-based representations, sampling-based inference, spatio-temporal inference, and bottom-up / top-down approaches. Monocular 3D reconstruction witnessed a renaissance in the context of deep learning, both for generic categories and specifically for humans. Previous research relied on effective human parameterizations in terms of skinned linear models, in particular the Skinned Multi-Person Linear (SMPL) model. Leveraging the fact that the SMPL model provides a low-dimensional, differentiable representation of the human body, these studies trained systems to regress the model parameters by minimizing reprojection errors between SMPL-based 3D keypoints and 2D joint annotations, human segmentation masks, and 3D volume projections, or even refining to body part level.

[0005] At the same time, 3D human pose estimation also saw great improvements in accuracy by evolving from classical structure from motion approaches to 3D convolutional neural network (CNN) based architectures that directly localize 3D joints in the volumetric output space through a mix of classification and regression.

[0006] Finally, recent research on dense pose estimation shows that dense correspondences between RGB images and SMPL models can be estimated by training a general-purpose, bottom-up detection system to associate image pixels with surface-level UV coordinates. It should be noted that while DensePose establishes a direct link between images and surfaces, it does not reveal the underlying 3D geometry, but gives strong hints about it. Other recent research relies on parametric human shape models, such as SMPL, which allow 3D human surfaces to be described in terms of a low-dimensional parameter vector. Most of these researches train CNNs to regress the deformable model parameters and update them in an iterative and time-consuming optimization process. SUMMARY

[0007] According to a first aspect of the present specification, a method for creating a three-dimensional reconstruction of a scene with multiple objects from a single two-dimensional image is described, the method comprising: receiving a single two-dimensional image; identifying all objects in the image to be reconstructed and identifying the type of the objects; estimating a three-dimensional representation of each of the identified objects; estimating a three-dimensional plane that physically supports all three-dimensional objects; and positioning all three-dimensional objects in space relative to the supporting plane.

[0008] The step of estimating a three-dimensional representation can be performed in a deep machine learning model comprising an output layer and one or more hidden layers each applying a non-linear transformation to received input to generate an output. The deep machine learning model predicts three-dimensional landmark positions of multiple objects by concatenating feature data from one or more intermediate layers of the neural network and estimates the predicted three-dimensional positions simultaneously for the predicted type of object described in each region.

[0009] The step of estimating a plane that supports multiple objects can be performed for a single frame by using the estimated three-dimensional positions of all visible objects to reconstruct a two-dimensional plane that passes through them. The step of estimating a plane that supports multiple objects can be performed for a series of frames using relative camera pose estimation and plane positioning using correspondences between points at consecutive frames.

[0010] The receiving can also comprise receiving multiple images, wherein the steps of estimating three-dimensional representations of multiple objects and positioning them on a plane are performed for each received image, e.g. in real-time. The processing can be performed over multiple consecutive frames by combining the hidden layer responses at consecutive frames, e.g. by averaging them.

[0011] Digital graphical objects can be synthetically added to the three-dimensional scene reconstruction in a given relationship to the estimated three-dimensional object positions and then projected back to the two-dimensional image.

[0012] According to another aspect of the present specification, a computing unit for creating a three-dimensional reconstruction of a scene with multiple objects from a single two-dimensional image is described, the computing unit comprising: a memory; and at least one processor, wherein the at least one processor is configured to perform the method according to the first aspect.

[0013] According to another aspect of the present specification, a computer-readable medium storing a set of instructions is described, the set of instructions being executable by at least one processor of a computing unit for creating a three-dimensional reconstruction of a scene with multiple objects from a single two-dimensional image, thereby causing the computing unit to perform the method according to the first aspect.

[0014] According to another aspect of the present specification, a computer program product comprising instructions which, when executed by a computer, cause the computer to perform the method according to the first aspect is described.

[0015] According to an additional aspect of the present specification, a method for training a deep machine learning model for creating a three-dimensional reconstruction of a scene with multiple objects from a single two-dimensional image is described, the method comprising: receiving a single two-dimensional image; obtaining a training signal for a three-dimensional reconstruction by fitting a three-dimensional model of an object to the two-dimensional image; using the resulting three-dimensional model fit as a supervisory signal for training the deep machine learning model.

[0016] The step of fitting a three-dimensional model can be performed by: projecting a three-dimensional representation onto a two-dimensional image plane, thereby resulting in a projected representation; comparing respective positions of the projected representation to the object in the single two-dimensional image; measuring an error value based on the comparison; and adjusting parameters of the fused three-dimensional representation based on the error value, wherein the comparison, measuring and adjusting are iteratively repeated until a threshold condition is met. The threshold condition can be that the measured error value is below a predetermined threshold or that a threshold number of iterations is exceeded. If multiple views are available, the step of projecting can be performed by taking into account the effect of perspective projection and by utilizing multiple views of the same object.

[0017] According to another aspect of the present specification, a computing unit for creating a three-dimensional reconstruction of a scene with multiple objects from a single two-dimensional image is described, the computing unit comprising: a memory; and at least one processor, wherein the at least one processor is configured to perform the method according to the additional aspect.

[0018] According to another aspect of the present specification, a computer-readable medium storing a set of instructions is described, the set of instructions being executable by at least one processor of a computing unit for creating a three-dimensional reconstruction of a scene with multiple objects from a single two-dimensional image, thereby causing the computing unit to perform the method according to the additional aspect.

[0019] According to another aspect of the present specification, a computer program product comprising instructions which, when executed by a computer, cause the computer to carry out the method according to the additional aspect is described.

[0020] According to another aspect of the present specification, a system for providing a three-dimensional reconstruction of a scene having multiple objects from a single two-dimensional image is described, the system comprising: a first computing unit for carrying out any method according to the first aspect; and a second computing unit configured to carry out a method according to the additional aspect, wherein the second unit is configured to train a model with the results of the first computing unit. BRIEF DESCRIPTION OF DRAWINGS

[0021] Embodiments and examples will be described with reference to the accompanying drawings, in which:

[0022] Figure 1 A schematic overview of an exemplary method for creating a three-dimensional reconstruction of a scene having multiple objects from a single two-dimensional image is shown;

[0023] Figure 2 A flowchart of a method for creating a three-dimensional reconstruction of a scene having multiple objects from a single two-dimensional image is shown;

[0024] Figure 3 A schematic overview of an exemplary method for training a neural network for reconstructing a three-dimensional scene having multiple objects from a single two-dimensional image is shown;

[0025] Figure 4 A flowchart of a method for training a neural network for reconstructing a three-dimensional scene having multiple objects from a single two-dimensional image is shown;

[0026] Figure 5 Another example of a method for training a neural network for reconstructing a three-dimensional scene and object re-identification is shown; and

[0027] Figure 6 A schematic example of a system / apparatus for carrying out any of the methods described herein is shown. DETAILED DESCRIPTION

[0028] This specification describes methods and systems for reconstructing 3D scenes containing multiple objects (e.g., humans) from a single 2D image (e.g., an RGB image). The example methods and systems described herein can recover accurate 3D reconstructions of multiple objects at more than 30 frames per second on a mobile device (e.g., a mobile phone), while also recovering information about the 3D camera position and world coordinates. The methods and systems described herein can be applied to real-time augmented reality applications involving, for example, an entire human body, allowing a user to control objects positioned on it, such as graphical assets attached to their hands, while also allowing the objects to interact with the human, such as a ball that bounces back from the human body once it comes into contact.

[0029] Furthermore, this specification describes CNN design choices that allow the 3D shape of multiple (potentially hundreds) of objects (e.g., humans) to be estimated at more than 30 frames per second on a mobile device. Using the methods described herein, results can be obtained on a Snapdragon 855 Neural Processing Unit (NPU) at a constant time of about 30 milliseconds per frame, regardless of the number of objects (e.g., humans) in the scene.

[0030] Several aspects described herein can contribute to these effects, alone or in combination.

[0031] In some embodiments, network purification is used to construct a supervisory signal for monocular image reconstruction. A detailed time-consuming model fitting process is performed offline to recover 3D interpretations for all images in a training set. The fitting results are then used to train one or more neural networks (e.g., convolutional neural networks) to efficiently process input test images with a single feed-forward pass through the network. This allows complex constraints imposed by complementary ground truth signals (e.g., region alignment, sparse and dense re-projection errors based on keypoints and dense poses, respectively, etc.) to be incorporated during model fitting without impacting speed at test time.

[0032] In some embodiments, an efficient encoder-only neural network architecture is used for monocular 3D human pose estimation. Early work relied on parameterizations of the human surface, manipulated models, and decoder-based networks for exhaustive, accurate localization of human body joints in 2D. In contrast, a repurposed (standard) deep neural network for image classification can be used, configured so that the last layer outputs 3D coordinates for each vertex of an object (e.g., human) mesh. This can significantly speed up test-time inference, and also make it simple to deploy the network on a mobile device.

[0033] In some embodiments, the neural network can have a single-stage, fully convolutional architecture that densely processes the input 2D images and emits 3D pose estimates for a large number of image locations in a single pass through the (standard) convolutional network. This can result in an inference time that is independent of the number of objects (e.g., people) in the scene, while also greatly simplifying the smoothing of the 3D reconstruction over time, as the network layers can be averaged over time, rather than smoothed a posteriori.

[0034] In some embodiments, a human-based self-calibration method is used that estimates the floor position in a three-dimensional scene by fitting a plane to the estimated 3D positions of objects / sub-objects (e.g., human feet) in 2D images. This allows recovering the world geometry in a way that eliminates the scaling effects of the perspective projection, while also recovering the camera position relative to the plane. In turn, recovering the world geometry allows augmenting the recovered scene by inserting objects that, for example, can fall on the floor and bounce back, while respecting the laws of physics. The method can operate both when the camera is stationary (where a Simultaneous Localization and Mapping (SLAM) method fails), and also in the case of a moving camera, for example, by aligning the estimated floor position with a plane recovered by SLAM.

[0035] In some embodiments, a distributed, part-based variant of the method collects information about the mesh parts from multiple image locations, as indicated by the estimated object part positions. The body mesh is obtained by combining the part-level meshes output by the neural network together, thereby allowing better handling of occlusions, large joint connections, while maintaining exactly the same memory and computational cost (except for the memory lookup operations).

[0036] In some embodiments, the neural network can additionally adapt to the task of object / person re-identification (REID). A teacher-student network distillation method can be used to train the network to perform such a task. REID embeddings are extracted from object (e.g., human) segments in 2D images using a pre-trained REID network, and these embeddings are used as a supervisory signal for training the REID branch of the neural network. This branch provides REID embeddings that mimic those of the teaching network, but can be fully convolutional, meaning that its running speed is independent of the number of objects in the image.

[0037] Figure 1 An illustrative overview of an exemplary method 100 for creating a three-dimensional reconstruction of a scene having multiple objects from a single two-dimensional image is shown. The method can be performed by one or more computing devices operating at one or more locations. For example, the method can be performed by a mobile computing device, such as a mobile phone.

[0038] One or more 2D images 102 each comprising a plurality of objects 104 of a given type (e.g., people) are input into a neural network 106. In some embodiments, only a single input image 102 is used. The neural network 106 processes the input data to identify the objects 104 in the image and generate output data 108 comprising an estimated 3D representation 110 of each (potentially) identified object 104 in the input 2D image 102. In some embodiments, the output data 108 can also include estimated coordinates 112 of a bounding box of a potential object 104 within the input 2D image 102 and / or a probability that an object 104 is present at a location within the input 2D image 102. The output data 108 is further processed (i.e., post-processed) to generate a three-dimensional reconstruction 116 of the scene in the input 2D image 102. In some embodiments, the output data 108 can also include an embedding vector (not shown) associated with each object 104 in the image. The embedding vector provides a REID embedding of the object such that a given object 104 can be identified and / or tracked across multiple input images 102.

[0039] The 2D input image 102i comprises a set of pixel values corresponding to a two- dimensional array. For example, in a color image, where H is the height of the image in pixels, W is the width of the image in pixels, and the image has three color channels (e.g., RGB or CIELAB). In some embodiments, the 2D image can be black and white / gray scale.

[0040] In some embodiments, the method 100 can use multiple (e.g., a series) of input 2D images 102 (e.g., images of multiple moving people). The 3D representation 110 can account for these changes. The multiple input 2D images 102 can be received, for example, from a camera on a mobile device, substantially in real-time (e.g., at 30 fps). The neural network 106 can process each of these input 2D images 102 individually, with the hidden layer responses for each input 2D image 102 being combined during post-processing when generating the 3D scene.

[0041] The neural network 106 takes as input the 2D images 102 and processes them through a plurality of neural network layers to generate the output data 108. The neural network 106 is a deep machine learning model that includes an output layer and one or more hidden layers, each of which applies a non-linear transformation to received input to generate an output. The neural network can predict three-dimensional landmark positions for multiple objects by concatenating feature data from one or more intermediate layers of the neural network. The predicted three-dimensional positions can be estimated simultaneously for a predicted type of object depicted in each region.

[0042] Each layer of the neural network 106 includes a plurality of nodes (also referred to herein as “neurons”), each associated with one or more neural network parameters (e.g., weights and / or biases). Each node accepts as input the output from one or more nodes in a previous layer (or input for the first layer of the neural network) and applies a transformation to its input based on the parameters associated with that node. The transformation can be a non-linear transformation. Some nodes can instead apply a linear transformation.

[0043] The neural network 106 can include one or more convolutional layers, each configured to apply one or more convolutional filters to the output of a previous layer in the network. In some embodiments, the neural network 106 is fully convolutional. The neural network can include one or more fully connected layers, where each node in a fully connected layer receives input from every node in a previous layer. The neural network 106 can include one or more skip connections. The neural network 106 can include a residual neural network, such as ResNet-50. The residual network can be used as a backbone network.

[0044] The neural network 106 can be a single-stage system that jointly detects objects (e.g., humans) and estimates their 3D shape by performing a single forward pass through the neural network 106 (which can be a fully convolutional neural network). Instead of cropping image patches around the detected regions of objects and then processing them again, the task of extracting region-specific features is delegated to the neurons of successive layers of the neural network 106 with increasingly larger receptive fields. A ResNet-50 backbone can be used, which provides an excellent trade-off between speed and accuracy. A dilated convolution (see, e.g., L. Chen et al., “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” PAMI, March 2017, the contents of which are incorporated by reference herein) can be used, allowing us to increase the spatial density of the hypotheses of estimated objects (e.g., humans) and can reduce the number of missed detections.

[0045] The task of the last layer of the neural network 106 (e.g., a fully convolutional neural network) is to predict the output data 108 (e.g., a plurality of outputs) at each of its neurons, corresponding to the attributes of a person hypothesized to be at the respective location.

[0046] In some embodiments, one or more of the layers of the neural network 106 can be 1 x 1 convolutional layers. For example, the last layer of the neural network can include one or more 1 x 1 convolutions. Examples of these layers are described in "Fully Convolutional Networks for Semantic Segmentation" (E. Shelhamer et al., IEEE Transactions on Pattern Analysis and Machine Intelligence 39(4): 640-651 (2017)) and "Overfeat: Integrated recognition, localization and detection using convolutional networks" (P. Sermanet et al., International Conference on Learning Representations, Second International Conference on Learning Representations, 2014), the contents of which are hereby incorporated by reference.

[0047] The output data 108 includes a mesh representation for each detected object 110. This can be in the form of a K = N x 3 dimensional vector that captures the shape of the object, where N is the number of nodes in the mesh. N can be 563, but other numbers of mesh nodes are possible. The output data 108 can also include a probability of the presence of an object (e.g., a person) 114. The output data 108 can also include the corners of a bounding box 112 of the object in the input image 102.

[0048] In some embodiments, the predictions of the neural network 106 can be specific to an object type. For example, the neural network 106 can predict the 3D locations of the facial parts, fingers, and limbs of a human, while a different neural network 106 can be specialized to predict the locations of the wheels, windows, doors, and lights of a car. It will be appreciated that object landmarks expected to be present due to the identified object type can not be present in the image. For example, if the object type is a person, parts of the body can not be visible due to obstructed views from poses or other items.

[0049] Using a neural network 106 as described above can result in several advantages. First, the inference is efficient, avoiding the need for a deconvolution-based decoder for high-accuracy pose estimation. Instead, this architecture is "encoder-only," reducing the time needed to process M x M (e.g., 233 x 233) patches to a few milliseconds. Second, the resulting network is easy to port to mobile devices, as they rely entirely on general convolutional network layers, rather than rigged body models used in earlier research. Third, it is simple to extend the resulting model, making it possible to process entire images in a fully convolutional manner, rather than processing individual patches.

[0050] In addition to simplicity, the resulting architecture also makes it simple to perform temporal smoothing on the reconstructed 3D shape. Instead of the complex, parameter tracking based approach used previously, a running average of the penultimate layer activations of the network 106 can be obtained. This essentially stabilizes the 3D shape recovered by the network, while incorporating it as a processing step in a mobile device is virtually effortless.

[0051] In some embodiments, the neural network 106 can output a complete object mesh V for an object (e.g., a person) based on high-dimensional feature vectors F computed at locations i aligned with predetermined points in the object (e.g., a person’s sternum). The feature vectors can be output by a layer in the neural network, e.g., corresponding to the activations of a hidden layer in the network. The mapping can be represented as:

[0052] V[i] = M(F[i]) (1)

[0053] where M indicates a mapping of the feature vectors F to the output mesh V. In some embodiments, M can be implemented as a linear layer in the neural network.

[0054] The simplicity of this “centralized” (i.e., aligned with a single predetermined point in the object) approach is offset by the challenge of counting for all the details and variations of the object (e.g., a human) mesh by F[i], which can become a bottleneck for computation. In this approach, the position of a given node in the output mesh 110 can be based on the activations of neurons in the neural network associated with locations far away from the node of the mesh. This can introduce inaccuracies to the resulting output mesh 110.

[0055] In some embodiments, a part-based approach can be used alternatively to reconstruct a 3D representation 110 of an object. The part-based approach uses a distributed approach for mesh regression, which can be significantly more accurate than the centralized approach. The part-based approach is useful in cases where an object 104 can be divided into parts. For example, a human body can be naturally divided into several parts, such as the head, torso, arms, hands, etc.

[0056] Taking the human body as an example, neurons associated with locations closer to the object parts (e.g., hands, feet, etc.) that are further away from a predetermined point (e.g., the sternum) can provide more reliable estimates for the corresponding mesh parts than neurons associated with locations further away from the predetermined point. These closer neurons act like “distributed part experts” that provide reliable information about the body parts in their vicinity. Information from these distributed part experts can be combined using a coordinator node located at the predetermined point (e.g., the sternum) to determine the output mesh 110. Information from neurons associated with locations further away from a given part (i.e., locations not associated with the part) about a particular mesh node associated with the given part can be compressed / dropped during the determination of the output mesh 110. This can be achieved by using a part-level attention matrix A.

[0057] The part locations providing information to the coordinator can be determined by a human body skeleton computation stage, associating the sternum location with corresponding part locations. The part locations can be based on, for example, the joint locations of the human body.

[0058] In the part-based approach, the neural network 106 outputs a separate part mesh V[p] for each part p of the object 104 in a predetermined list of object parts. Each part mesh V[p] is set in its own coordinate system, whose origin is offset with respect to a reference point (also referred to herein as a “coordinator node”) c (e.g., the human sternum). In other words, {p1,... pM} represents the location of each of the M parts associated with the coordinator (e.g., sternum) location c. The part locations can be based on key points of the object, such as joint locations in the human body. M} represents the location of each of the M parts associated with the coordinator (e.g., sternum) location c. The part locations can be based on key points of the object, such as joint locations in the human body.

[0059] To reconstruct the full mesh V from the part meshes V[p], the coordinator uses a part-level attention matrix A to select from V[p] the nodes relevant to part p. In some embodiments, A is a binary matrix indicating which part should be used for each vertex in the final mesh V. For such an attention matrix, the following can be used to reconstruct the final mesh V for each mesh vertex v in the final mesh:

[0060]

[0061] where V[c, v] represents the location of mesh node v with respect to the predetermined point c, p v indicates the location of the part that is the “active expert” for node v (i.e., its A[p, v] = 1 in the binary example), and V[p, v] represents the location of mesh node v in the mesh associated with part p. (c - p v ) is an offset vector that takes into account the relative location between the part and the reference location c (e.g., the sternum).

[0062] In some embodiments, the partial-level attention matrix A can be a learned matrix determined during training. In such examples, the position of a given mesh node v can be based on information from one or more of the partial levels. Thus, using Equation (2) can yield a weighted sum of more than one partial level, rather than Equation (3).

[0063] In some embodiments, the output data 108 further includes estimated coordinates 112 of a bounding box of the potential object 104 within the input 2D image 102 and / or a probability of the presence of the object 104 at the location within the input 2D image 102.

[0064] In some embodiments, the estimated 3D representation 110 of each (potential) identified object 104 in the input 2D image 102 is in the form of an N-vertex mesh V. For example, the estimated 3D representation 110 of each identified object 104 can be in the form of an (Nx3) -dimensional vector giving the 3D coordinates of the N vertices in the mesh. In some embodiments, N = 536, but other numbers of mesh nodes are possible.

[0065] For each potential object, the coordinates of the bounding box can include the x and y positions of the bounding box vertices in the input 2D image 102, e.g., {(xi, yi), (x2, y2), (x3, y3), (x4, y4)} for each potential object. Each bounding box can be associated with a corresponding probability that the bounding box contains an object of a given type (e.g., a person).

[0066] The 3D reconstruction 116 of the scene in the input 2D image 102 is generated by taking into account the perspective projection when estimating the world- within coordinates and removing its effects. The 3D reconstruction 116 is generated by estimating a 3D plane 118 (also referred to herein as a “support plane”) on which the identified objects 104 in the image 102 are physically supported (e.g., a floor, ground) and positioning all 3D objects 110 in space relative to the support plane.

[0067] Estimating a 3D plane 118 on which the identified objects 104 in the image 102 are physically supported can be based on the assumption that objects (e.g., humans) are supported by a single physical plane. In some embodiments, this assumption is relaxed by using a mixture model and using expectation maximization to assign different people to different planes.

[0068] Estimating a 3D plane 118 can also be based on the assumption that the height of objects (e.g., humans) is approximately equal. This latter assumption can be relaxed if a series of input images 102 are available in which individual objects can be tracked over time and the effects of perspective projection on each object over time are monitored.

[0069] To position the 3D objects 110 in space relative to the support plane, the scaling of each mesh is estimated in order to bring it to world coordinates. The scaling is inversely proportional to the distance of the object (e.g. person) from the camera that captured the scene in the input image 102. Thus, the scaling can be used to position the mesh in world coordinates along the line connecting the mesh vertex to the center of the camera.

[0070] As an example, the 3D objects 110 can be provided by the neural network 106 in the form of meshes estimated under orthographic projection. Given the i-th mesh in the scene, its vertices are m i = {v i，1 …, v i，K}, where v i，k = (x i，k , y i，k , z i，k ) e R 3 are the mesh coordinates estimated in scaled orthographic, and the scaling is s i By removing the effect of the perspective projection, the mesh vertices are positioned in 3D world coordinates. For example, the world coordinates V i，k = (X i，k , Y, i，k , Z, i，k ) of the k-th point in the i-th mesh can be estimated from the corresponding mesh coordinates estimated in scaled orthographic by “back- projecting” the mesh depth in world coordinates with the inverse of the scale factor, and setting the X and Y world coordinates so that they project back correctly to the corresponding pixel coordinate values x and y. Symbolically, this can be represented as:

[0071]

[0072]

[0073]

[0074] where the camera calibration matrix has center c x = W / 2 and c y = H / 2 (where W and H are the image dimensions, i.e. image width and height), and focal length f. These can be set manually or through camera calibration.

[0075] For each mesh corresponding to an identified object, the lowest (i.e. lowest Y value) is determined and used to estimate the contact point between the object and the support plane (e.g. floor). Once at least four such points (collectively referred to as M) have been determined, the determined points can be used to estimate the support plane in world coordinates. In some embodiments, a least squares method can be used to estimate the plane in world coordinates, e.g.:

[0076]

[0077] where V1= (a, b, c) is a vector normal to the support plane. In some embodiments, this vector can be normalized. The world coordinate axes R = [v1 T , v2 T , v3 T ] T are defined by finding two complementary directions V2and V3that are both orthogonal to V1and to each other. In some embodiments, V2is chosen in the Z direction, and V3= V1x V2. In some embodiments, the vectors V2and V3may be normalized, i.e. the set {V1, V2, V3} forms an orthonormal basis.

[0078] The world coordinate center T can also be specified as a 3D point lying on the plane. For example, it can be set to a point three meters away from the camera and projected to y = H / 2.

[0079] In some embodiments, the above can be used to define a transformation between the world coordinate system and the pixel positions in the 2D image 102. The world-to-camera transformation and the camera calibration matrix are in a single 3x4 perspective projection matrix P. For example, it can be given by:

[0080]

[0081] Here, the direction V vectors are shown as rows instead of columns because an inverse rotation matrix is used, and R -1 = R T .

[0082] Using homogeneous coordinates, the world coordinates C can be converted to pixel coordinates c using:

[0083] c = PC, where and

[0084] In some embodiments, the coordinate transformation can be used to introduce objects into a scene / image that follow the laws of physics and / or interact with other objects in a meaningful / realistic way. This can lead to interactive applications that require real-world coordinates, for example augmented reality games such as a game where players try to hit each other with swords or laser beams. In some embodiments, the 3D reconstructed mesh can be projected back into the input image 102, showing the accuracy of the 3D human pose estimation.

[0085] In embodiments using multiple input 2D images 102, estimating the plane supporting multiple can be performed for a series of frames using relative camera pose estimation and plane localization (using correspondences between points of consecutive frames). Processing (e.g. post-processing) can be done over multiple consecutive frames by combining (e.g. by averaging) the hidden layer responses at consecutive frames.

[0086] According to some example embodiments, the synthetic objects can be controlled by or interact with the objects (e.g., people) in the input images. For example, the added synthetic objects can be a sword or a laser beam that a person controls with their arm in a computer game - or objects that move towards the person and bounce off when they contact the estimated three-dimensional person location.

[0087] The method can be integrated into a graphics engine, such as Unity. A user can interact with their own image / video in real-time and / or see a mesh overlaid on themselves.

[0088] Combining multiple measurements over time obtained from successive input images 102 (e.g., in a video) gives a further improved estimate, while combining the method with camera tracking, as allowed in simultaneous localization and mapping (SLAM). For the combination of deformable objects (like humans) with rigid 3D scenes, both are reconstructed in metric coordinates.

[0089] Figure 2 A flowchart showing an example method for creating a three-dimensional reconstruction of a scene with multiple objects from a single two-dimensional image is shown. The method can be performed by one or more computers operating in one or more locations. The method can correspond to the method described with respect to Figure 1 the method described.

[0090] At operation 2.1, a 2D image is received. The 2D image includes an array of pixel values, such as an RGB image. A single 2D input image can be received. In some embodiments, a series of single images can be received.

[0091] At operation 2.2, objects to be reconstructed in the image are identified. Object types can also be identified. Operation 2.2 can be performed by one or more layers of a neural network, such as the neural network described above with respect to Figure 1 the method described.

[0092] At operation 2.3, a 3D representation of each identified object is estimated. The 3D representation can take the form of a mesh, for example, K = Nx3 dimensional vectors of 3D N-vertex positions. Operation 7.3 can be performed by one or more layers of a neural network, such as the neural network described above with respect to Figure 1 the method described.

[0093] The operation of estimating a 3D representation can be performed in a deep machine learning model (e.g., a neural network) that includes an output layer and one or more hidden layers that each apply a non-linear transformation to received inputs to generate an output. The deep machine learning model can predict 3D landmark positions for multiple objects by concatenating feature data from one or more intermediate layers of the neural network. For the predicted type of object depicted in each region, the predicted 3D positions are estimated simultaneously.

[0094] In the case of receiving a series of input images, the following is performed for each received image: estimate the three-dimensional representation of multiple objects and position them on a plane. If the images are received substantially in real-time (e.g., 30 fps), these operations can be performed substantially in real-time.

[0095] At operation 2.4, a 3D plane that physically supports all three-dimensional objects is estimated. Operation 2.4 can be performed in a post-processing step, i.e., after the neural network processes the input image. Reference is made above to Figure 1 The method of estimating the support frame is described in further detail.

[0096] Estimating the plane that supports multiple objects can be performed for a single frame by using the estimated three-dimensional positions of all visible objects to reconstruct a two-dimensional plane that passes through them. For example, the support plane can be estimated based on the estimated positions of contact points between the objects and the plane (e.g., the positions of human feet identified in the input image). Estimating the plane that supports multiple objects can be performed for a series of frames using relative camera pose estimation and plane localization (using correspondences between points in consecutive frames).

[0097] At operation 2.5, the 3D objects are positioned in space relative to the support plane. Operation 2.5 can be performed in a post-processing step, i.e., after the neural network processes the input image. Reference is made above to Figure 1 The method of positioning the 3D objects relative to the support frame is described in further detail.

[0098] Figure 3 An illustrative overview of an exemplary method 300 for training a neural network to reconstruct a three-dimensional scene with multiple objects from a single two-dimensional image is shown. The method can be performed by one or more computers operating in one or more locations.

[0099] A 2D training image 302 is obtained from a training dataset that includes multiple 2D images. The 2D image includes one or more objects 304 of a given type (e.g., one or more humans). A 3D morphable model is fitted (e.g., iteratively fitted) to the objects in the input image 302 to extract one or more supervisory signals 306 for monocular 3D reconstruction. The training image 302 is input into a neural network 308, and the training image 302 is processed through a series of neural network layers using current parameters of the neural network 308 to generate output data 310 that includes a candidate 3D representation of each (expected) object in the input training image 302. The output data is compared to the supervisory signals 306 of the input image 302 to determine parameter updates 312 for the neural network 308.

[0100] The method can be performed iteratively until a threshold condition is met.

[0101] The input images 302 and the neural network 308 can take the same form as described with respect to Figure 1 the same form as described with respect to

[0102] During the morphable model fitting / optimization phase, all available 2D and 3D ground truth data is used to provide the energy terms (i.e., the supervisory signal 306) that are minimized to obtain the estimated 3D shape. For example, the method described in “Holopose: Holistic 3d human reconstruction in-the-wild” (R. Alp Guler and I. Kokkinos, In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019. 2, 3, the contents of which are incorporated by reference herein) can be used, which uses a CNN-driven morphable model for iterative fitting of 2D images.

[0103] The step of fitting the three-dimensional model can be performed by projecting the three-dimensional representation onto a two-dimensional image plane, thereby producing a projected representation. If multiple views are available, the projection step can be performed by taking into account the effects of perspective projection and by utilizing multiple views of the same object. The individual positions of the projected representation can then be compared to the object in the single two-dimensional image, and based on the comparison, an error value is determined, for example by determining the difference in object landmark positions in the training image 302 and the re-projected image. The parameters of the fused three-dimensional representation are then adjusted based on the error value. The projection, comparison, measurement, and adjustment can be iteratively repeated until the measured error value is below a predetermined threshold or a threshold number of iterations is exceeded.

[0104] Once the morphable model fitting converges, the estimated 3D surface is used as the target, or supervisory signal 306 for network 308 training. For example, a “low polygon” representation of the object 304 (e.g., a human body) to cover N (e.g., N = 536) vertex representation meshes of salient objects (e.g., human landmarks (e.g., facial parts, limb connections, etc.).

[0105] From the morphable model fitting to the image, the estimated 3D positions of those vertices around each object present in the image are measured. The neural network 308 is then trained to regress this K = Nx3 dimensional vector when presented with the object 304 in its receptive field. In the case of multiple objects, different responses are expected in different image locations. The position of the bounding box of the object in the input image 302 and / or the presence / absence of the object at the location in the input image can additionally be used as the supervisory signal 306.

[0106] The task of extracting region-specific features for neurons in successive layers with increasingly large receptive fields. A ResNet-50 backbone can be used, which has an excellent trade-off between speed and accuracy. A-trous convolutions can be used, allowing to increase the spatial density with which the hypotheses employed to evaluate are assessed and to reduce the number of missed detections. The task of the last layer of the resulting fully convolutional layer is to predict multiple outputs at each of its neurons, corresponding to attributes of the person at the respective location where the hypothesis is at. In some embodiments, the output data 310 also includes a probability of the presence of the object (e.g., person). The corners of the object bounding box can also be regressed and form part of the output data 310.

[0107] The comparison of the output data to the supervision signal 306 of the input image 302 can be performed using a loss / objective function L. The loss function includes a term that penalizes the difference between the candidate 3D representation and the 3D representation in the supervision signal 306. Many examples of such loss functions are known in the art, such as LI or L2 loss. The loss function can also include a term that penalizes the difference between the position of the candidate bounding box output by the neural network 308 and the ground truth bounding box position in the input training image 302. Many examples of such loss functions are known in the art, such as LI or L2 loss. The loss function can also include a term that penalizes the output probability of the neural network 308 based on the actual presence / absence of an object at that location. Many examples of such loss functions are known in the art, such as a classification loss function.

[0108] In some embodiments, the loss can only penalize the 3D representation and the bounding box position in the case of the presence of an object (e.g., person) - it should be understood that in the case of the absence of an object, the box and the 3D shape can take arbitrary values, but the object detector will discard the resulting hypothesis.

[0109] The parameter updates 312 can be determined by applying an optimization procedure to the loss / objective function, such as stochastic gradient descent. The loss function can be averaged over a batch of training images before determining each set of parameter updates. The procedure can be iterated over multiple batches in the training dataset.

[0110] In embodiments where mesh estimation is performed using a part-based approach, the 3D representations in the supervision signal 306 can each be divided into parts. Each supervision signal 306 can be divided into parts based on the keypoints of the object 304. For example, the supervision signal 306 can be divided into body parts (e.g., hands, feet, etc.) based on the joint positions. When determining the parameter updates 312, the part meshes V[p] output by the neural network 308 are compared to the corresponding part meshes in the supervision signal 306.

[0111] Figure 4A flowchart showing a method for training a neural network for reconstructing a three-dimensional scene with multiple objects from a single two-dimensional image is shown. The method can be performed by one or more computers operating at one or more locations. The method can correspond to the method shown and described above with respect to Figure 3 the training method described.

[0112] At operation 4.1, one or more two-dimensional images are received. The 2D images can take the form of images described above with respect to Figure 1 Each 2D image includes one or more objects of a given type (e.g., humans).

[0113] At operation 4.2, training signals for three-dimensional reconstruction are obtained by fitting a three-dimensional model of the objects to each two-dimensional image. This effectively generates a labeled training dataset comprising multiple 2D images, each 2D image being associated with one or more 3D representations of the objects (e.g., humans) within the image.

[0114] Fitting the three-dimensional model can be performed by projecting the three-dimensional representation onto the two-dimensional image plane, thereby producing a projected representation; comparing respective positions of the projected representation to the objects in the single two-dimensional image; measuring an error value based on the comparison; and adjusting parameters of the fused three-dimensional representation based on the error value, wherein the comparison, measurement, and adjustment are iteratively repeated until a threshold condition is met. The threshold condition can be that the measured error value is below a predetermined threshold and / or a threshold number of iterations is exceeded. If multiple views are available, the projection step can be performed by taking into account the effects of perspective projection and by utilizing multiple views of the same object.

[0115] At operation 4.3, the resulting three-dimensional model fitting results are used as supervisory signals for training a deep machine learning model. The deep machine learning model can be trained as described above with respect to Figure 3 the training method described.

[0116] Figure 5 Another example of a method 500 for training a neural network for reconstructing a three-dimensional scene and object re-identification is shown. The method can be performed by one or more computers operating at one or more locations.

[0117] The method 500 is an extension of the method shown and described above with respect to Figure 3 the training method described, to include training of a branch of the neural network 508 to output one or more re-identification (REID) embedding vectors 516a, 516b as an additional portion of the output data 510. Thus, any features described with respect to Figure 3 may additionally be combined with features described with respect to Figure 5 the training method described. In accordance with the training method described with respect to Figure 5The described method trains a neural network 508 that is endowed with the ability to maintain object / person identity across a series of images. This is crucial for object / person specific parameter smoothing and accumulation of information over time - as well as for video game experiences, where each user is consistently associated with a unique character over time.

[0118] In the literature, person / object re-identification is typically solved by a two-stage architecture, where persons / objects are first detected by a person / object detection system, and for each detected person / object, the image is cropped and sent as input to a separate person recognition network. The latter provides a high-dimensional embedding that acts like a discriminative person / object signature, trained to be invariant to nuisance parameters (e.g. camera position, person / object pose, lighting, etc.). This strategy has a complexity that scales linearly in the number of persons, and is also difficult to implement on mobile devices, being two-stage.

[0119] Alternatively, with respect to Figures 1 to 4 The described neural network can be extended to adapt to the person / object re-identification (REID) task using a teacher-student network distillation approach. REID embeddings are extracted from the person / object patches using a pre-trained REID network. These REID embeddings are used as a supervisory signal to train a REID branch of the neural network. This branch provides REID embeddings that mimic those of the teaching network, but can be, for example, fully convolutional, meaning that its running speed is independent of the number of persons.

[0120] Combining a network trained in this way with a re-identification and tracking algorithm would allow to maintain person / object identity in scenes with high person overlap, interaction, and occlusions, even in cases where a user temporarily disappears from the scene. An example of such a re-identification and tracking algorithm can be found in “Memory based online learning of deep representations from video streams” (F. Pernici et al., CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 2324-2334), the content of which is incorporated herein by reference. Maintaining person / object recognition across multiple images in a scene can allow different virtual persons / skins to be persistently associated with each person / object in the images over time.

[0121] 2D training images 502 are obtained from a training dataset comprising a plurality of 2D images. The 2D images include one or more objects 504a, 504b of a given type (e.g., humans) (in the illustrated example, two objects 504a, 504b, but there can be any number of objects in general). A 3D morphable model is fitted (e.g., iteratively fitted) to the objects in the input images 502 to extract one or more supervisory signals 506 for monocular 3D reconstruction. In addition, each object 504a, 504b is cropped from the image (e.g., using a ground truth bounding box for the object) and input separately into a pre-trained REID network 518. The pre-trained REID network 518 processes the patch of each input object 504a, 504b to generate a“teacher” re-identification embedding 520a, 520b (denoted as e T i ), which encodes the identity of the input object 504a, 504b. The teacher re-identification embedding e T i , 520a, 520b is used as a supervisory signal for training a re-identification branch of the neural network 508.

[0122] The training images 502 are input into the neural network 508 and processed through a series of neural network layers using the current parameters of the neural network 508 to generate output data 510, which includes a candidate 3D representation 514a, 514b for each (intended) object in the input training images 502 and a“student” re-identification embedding 516a, 516b for each (intended) object in the input training images 502. The output data 510 is compared to the supervisory signals 506 and teacher embeddings 520a, 520b of the input images 502 to determine parameter updates 512 for the neural network 508.

[0123] The teacher REID embeddings 520a, 520b and student REID embeddings 516a, 516b are each high-dimensional vectors representing individual object identities. For example, each REID embedding can be an N-dimensional vector. For example, N can be 256, 512, or 1024.

[0124] In the case where a loss / objective function is used to determine the parameter updates 512, the loss / objective function can include a re-identification loss that compares each teacher REID embedding 520a, 520b to its corresponding student REID embedding 516a, 516b. The re-embedding loss can be an LI or L2 loss between each teacher REID embedding 520a, 520b and its corresponding student REID embedding 516a, 516b.

[0125] Figure 6Illustrated are schematic examples of systems / apparatuses for performing any of the methods described herein. The illustrated systems / apparatuses are examples of computing devices. Those skilled in the art will appreciate that other types of computing devices / systems can alternatively be used to implement the methods described herein, such as distributed computing systems. One or more of these systems / apparatuses can be used to perform the methods described herein. For example, a first computing device (e.g., a mobile computing device) can be used to perform the methods described above with respect to Figure 1 and Figure 2 a first computing device (e.g., a mobile computing device) can be used to perform the methods described above with respect to Figures 3 to 5 a first computing device (e.g., a mobile computing device) can be used to perform the methods described above with respect to

[0126] The apparatus (or system) 600 includes one or more processors 602. The one or more processors control the operation of the other components of the system / apparatus 600. The one or more processors 602 may, for example, include a general purpose processor. The one or more processors 602 can be a single core device or a multiple core device. The one or more processors 602 can include a central processing unit (CPU) or a graphics processing unit (GPU). Alternatively, the one or more processors 602 can include special purpose processing hardware, such as a RISC processor with embedded firmware or programmable hardware. Multiple processors can be included.

[0127] The system / apparatus includes a research or volatile memory 604. The one or more processors can access the volatile memory 604 in order to process data and can control the storage of data in the memory. The volatile memory 604 can include any type of RAM, such as static RAM (SRAM), dynamic RAM (DRAM), or it can include flash memory such as an SD card.

[0128] The system / apparatus includes a non-volatile memory 606. The non-volatile memory 606 stores a set of operating instructions 608 in the form of computer readable instructions for controlling the operation of the processor 602. The non-volatile memory 606 can be any type of memory, such as read only memory (ROM), flash memory, or magnetic drive memory.

[0129] The one or more processors 602 are configured to execute the operating instructions 608 to cause the system / apparatus to perform any of the methods described herein. The operating instructions 608 can include code related to the hardware components of the system / apparatus 600 (i.e., drivers) as well as code related to the basic operation of the system / apparatus 600. Generally speaking, the one or more processors 602 execute one or more instructions of the operating instructions 608, which are permanently or semi-permanently stored in the non-volatile memory 606, using the volatile memory 604 to temporarily store data generated during the execution of the operating instructions 608.

[0130] Implementation of the methods described herein can be implemented in a digital electronic circuit, integrated circuit, specially designed ASIC (application specific integrated circuit), computer hardware, firmware, software, and / or combinations thereof. These can include a computer program product such as software stored on a non-transitory computer readable medium (e.g., magnetic disk, optical disk, memory, programmable logic devices, and the like) which can be executed by a computer (e.g., about Figure 6 the computer described herein) to cause the computer to perform one or more of the methods described herein.

[0131] Any system features described herein can also be provided as method features, and vice versa. As used herein, apparatus plus functional features can alternatively be expressed as their corresponding structural application. In particular, method aspects can apply to system aspects, and vice versa.

[0132] Furthermore, any, some and / or all features in one aspect can be applied to any, some and / or all features in any of the other aspects, in any appropriate combinations. It is also to be understood that specific combinations of the various features described and defined in any aspects of the application can be independently achieved and / or provided and / or used.

[0133] While several embodiments have been shown and described, one skilled in the art will understand, from the teachings and disclosures provided herein, that changes might be made in the embodiments without departing from the principles of the disclosure, the scope of which is defined in the claims.

[0134] The various exemplary embodiments described herein are described in the general context of method steps or processes, which can be implemented in one aspect by a computer program product, embodied in a computer-readable medium, including computer-executable instructions, such as program code, executed by computers in networked environments. A computer-readable medium can include removable and non-removable storage devices including, but not limited to, Read Only Memory (ROM), Random Access Memory (RAM), compact discs (CDs), digital versatile discs (DVDs), etc. Generally, program modules can include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of program code for executing steps of the methods disclosed herein. The particular sequence of such executable instructions or associated data structures represents examples of corresponding acts for implementing the functions described in such steps or processes.

[0135] In the foregoing specification, embodiments have been described with reference to numerous specific details that can vary with implementation. Certain adaptations and modifications of the described embodiments can occur to those skilled in the art. Other embodiments can be apparent to those of ordinary skill in the art having the benefit of this disclosure. The specification and examples are to be considered exemplary only, with the true scope and spirit of the application indicated by the following claims. The sequence of steps shown in the figures is for illustrative purposes only and is not intended to be limited to any particular sequence of steps. Thus, those skilled in the art will appreciate that the steps can be performed in different orders while still implementing the same method.

[0136] In the drawings and specification, there have been disclosed exemplary embodiments. However, many variations and modifications can be made to these embodiments. Consequently, it is intended that the application be limited only by the scope of the appended claims, and not by the specific examples provided herein.

Claims

1. A method for creating a three-dimensional reconstruction of a scene with multiple objects from a single two-dimensional image, the method comprising: receiving a single two-dimensional image; identifying all objects of a given type in the image that are to be reconstructed; estimating a three-dimensional representation of each identified object using a neural network, wherein the neural network is trained using as one or more supervisory signals for a training process results of fitting two-dimensional projections of a three-dimensional deformable model to objects of the given type in a set of two-dimensional training images; estimating a three-dimensional plane that physically supports all identified objects in the image; and positioning the three-dimensional representation of each identified object in space relative to the support plane. The step of estimating a three-dimensional representation is performed using a deep neural network comprising an output layer and one or more hidden layers that each apply a non-linear transformation to received input to generate an output.

2. The method of claim 1, wherein, The deep neural network predicts three-dimensional landmark positions for multiple identified objects by concatenating feature data from one or more intermediate layers of the deep neural network, and estimates the predicted three-dimensional positions simultaneously for identified objects described in each region.

3. The method of claim 2, wherein, The step of estimating the support plane is performed for a single frame by using the estimated three-dimensional positions of all visible identified objects to reconstruct a two-dimensional plane that passes through them.

4. The method of claim 1, wherein, The step of estimating the support plane is performed for a series of frames using relative camera pose estimation and plane localization that uses correspondences between points in consecutive frames.

5. The method of claim 1, wherein, The receiving further comprises receiving multiple images, wherein the steps of estimating a three-dimensional representation of identified objects and positioning them on a plane are performed for each received image, e.g., in real time.

6. The method of any preceding claim, wherein, Consecutive frames are processed by averaging their hidden layer responses.

7. The method of claim 6, wherein, Digital graphic objects are synthetically added to the three-dimensional scene reconstruction in a given relationship to the estimated three-dimensional object positions, and then projected back to the two-dimensional image.

8. The method of any one of claims 4, 5, and 7, wherein, Digital graphic objects are synthetically added to the three-dimensional scene reconstruction in a given relationship to the estimated three-dimensional object positions, and then projected back to the two-dimensional image.

9. The method of claim 6, wherein, 10. A first computing unit for creating a three-dimensional reconstruction of a scene with multiple objects from a single two-dimensional image, the first computing unit comprising a memory; and at least one processor, wherein the at least one processor is configured to perform the method according to any one of claims 1 to 9.

11. A computer readable medium storing a set of instructions executable by at least one processor of a first computing unit for creating a three-dimensional reconstruction of a scene with multiple objects from a single two-dimensional image, thereby causing the first computing unit to perform the method according to any one of claims 1 to 9.

12. A computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method according to any one of claims 1 to 9. ​ 13. A method for training a deep neural network to create a three-dimensional reconstruction of a scene with multiple objects from a single two-dimensional image, the method comprising: receiving a single two-dimensional image; obtaining training signals for a three-dimensional reconstruction by fitting a two- dimensional projection of a three-dimensional deformable model of an object to objects of a given type in a set of two-dimensional training images; and using the resulting three-dimensional model fitting results as a supervisory signal for training the deep neural network.

14. The method of example claim 13, wherein, The step of fitting the three-dimensional model is performed by projecting a three- dimensional representation onto a two-dimensional image plane resulting in a projected representation; comparing respective positions of the projected representation to the objects in the single two-dimensional image; measuring an error value based on the comparison; and adjusting parameters of the fused three-dimensional representation based on the error value, wherein the comparing, measuring and adjusting are iteratively repeated until the measured error value is below a predetermined threshold or a threshold number of iterations is exceeded.

15. The method of claim 14, wherein, The step of projecting is performed by taking into account the effects of perspective projection and by utilizing multiple views of the same object if multiple views are available.

16. A second computing unit for creating a three-dimensional reconstruction from a single two-dimensional image for estimating a three-dimensional representation of an object included in a single two-dimensional image, the second computing unit comprising a memory; and at least one processor, wherein the at least one processor is configured to perform the method according to any one of claims 13 to 15.

17. A computer readable medium storing a set of instructions executable by at least one processor of a second computing unit for updating a fused three-dimensional representation of an object included in a single two-dimensional image, thereby causing the second computing unit to perform the method according to any one of claims 13 to 15.

18. A computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method according to any one of claims 13 to 15.

19. A system for providing a three-dimensional reconstruction of a scene with multiple objects from a single two-dimensional image, the system comprising: a first computing unit according to claim 10; and a second computing unit according to claim 16, wherein the first computing unit is trained with the results of the second computing unit.

Images