Scene reconstruction in three-dimensions from two-dimensional images

Abstract

The present specification relates to reconstructing three-dimensional (3D) scenes from two-dimensional (2D) images using a neural network. According to a first aspect of the present specification, a method for generating a three-dimensional reconstruction of a scene having a plurality of objects from a single two-dimensional image is described, the method comprising: receiving a single two-dimensional image; identifying all objects in the image to be reconstructed and identifying the types of said objects; estimating a three-dimensional representation of each identified object; estimating a three-dimensional plane that physically supports all three-dimensional objects; and positioning all three-dimensional objects in space with respect to the supporting plane.

Description

Technology Field

[0001] This specification relates to reconstructing three-dimensional (3D) scenes from two-dimensional (2D) images using a neural network. Background Technology

[0002] Aside from the easier issues with faces, there is almost no research on integrating 3D human bodies (or other objects) into augmented reality applications in real time. Bodies are generally more difficult than faces due to their articulation deformities, numerous occlusions and magnetic occlusions, complex interactions with other objects and people, and greater appearance variability caused by clothing.

[0003] Human pose estimation algorithms typically aim to locate specific sparse points within images, such as skeletal joints and facial landmarks, or more recently, dense surface-level coordinates. Despite their increasing accuracy, such representations are insufficient for serving downstream applications such as augmented reality, motion capture, gaming, or graphics. These require access to a three-dimensional underlying human body surface and currently rely on multiple camera setups or depth sensors.

[0004] Research on morphable models has demonstrated that accurate monocular surface reconstruction can be performed by utilizing low-dimensional parametric representations of facial surfaces and appearances and casting the reconstruction task as an optimization problem. Extending this to the more complex joint structures of the human body, monocular human body reconstruction has been extensively studied over the past decade, along with part-based representations, sampling-based inference, spatiotemporal inference, and bottom-up / bottom-up methods. Monocular 3D reconstruction has witnessed a resurgence in the context of deep learning for general categories, particularly humans. Previous studies rely on the efficient parameterization of the human body in terms of skinned linear models, and specifically skinned multilinear (SMPL) models. Utilizing the fact that SMPL models provide low-dimensional differentiable representations of the human body, these studies trained systems to regress model parameters by minimizing reprojection errors between SMPL-based 3D key points, 2D joint annotations, human segmentation masks, and 3D volume projections, or even refining them to the level of body parts.

[0005] In parallel with this, 3D human joint estimation showed a dramatic increase in accuracy by transitioning from classical motion-to-structure approaches to 3D convolutional neural network (CNN)-based architectures that directly localize 3D joints in volume output space through hybrids of classification and regression.

[0006] Finally, recent research on dense pose estimation has demonstrated that dense correspondences between RGB images and SMPL models can be estimated by training a general bottom-up detection system to associate image pixels with surface-level UV coordinates. It should be noted that while dense poses establish direct links between images and surfaces, they do not cover the underlying 3D geometry but rather provide strong hints regarding it. Other recent studies rely on parametric human models of geometry, such as SMPL, which allow for describing 3D human surfaces in terms of low-dimensional parameter vectors. Most of these studies train CNNs to regress deformable model parameters and update them in an iterative, time-demanding optimization process.

[0007] According to a first aspect of the present specification, a method for generating a three-dimensional reconstruction of a scene having a plurality of objects from a single two-dimensional image is described, and the method comprises: receiving a single two-dimensional image; identifying all objects in the image to be reconstructed and identifying the types of said objects; estimating a three-dimensional representation of each identified object; estimating a three-dimensional plane that physically supports all three-dimensional objects; and positioning all three-dimensional objects in space with respect to the support plane.

[0008] The step of estimating a 3D representation can be performed in a deep machine learning model, and the deep machine learning model includes an output layer and one or more hidden layers that apply a non-linear transformation to the received input to generate the output. The deep machine learning model predicts the 3D landmark locations of multiple objects by connecting feature data from one or more intermediate layers of a neural network, and the predicted 3D locations are simultaneously estimated for the predicted type of object described in each region.

[0009] The step of estimating a plane supporting multiple objects can be performed for a single frame using the estimated 3D positions of all visible objects to reconstruct a 2D plane passing through all visible objects. The step of estimating a plane supporting multiple objects can be performed for a series of frames using relative camera pose estimation and plane localization using correspondences between points of consecutive frames.

[0010] The receiving step may further include the step of receiving a plurality of images, wherein the steps of estimating three-dimensional representations of a plurality of objects and positioning them at a predetermined location are performed, for example, in real time for each received image. Processing may occur over a plurality of consecutive frames by combining hidden layer responses in consecutive frames, for example, by averaging them.

[0011] Digital graphic objects can be synthetically added to a 3D scene reconstruction with a given relationship to the estimated 3D object positions and then projected back onto a 2D image.

[0012] According to a further aspect of the present specification, a computational unit for generating a three-dimensional reconstruction of a scene having a plurality of objects from a single two-dimensional image is described, wherein the computational unit comprises a memory; and at least one processor, and the at least one processor is configured to perform a method according to a first aspect.

[0013] According to a further aspect of the present specification, a computer-readable medium is described that stores a set of instructions executable by at least one processor of a computing unit to enable a computing unit to perform a method according to a first aspect for generating a three-dimensional reconstruction of a scene having a plurality of objects from a single two-dimensional image.

[0014] According to a further aspect of the present specification, a computer program product is described that includes instructions that, when executed by a computer, cause the computer to perform a method according to a first aspect.

[0015] According to a further aspect of the present specification, a method for training a deep machine learning model to generate a three-dimensional reconstruction of a scene having a plurality of objects from a single two-dimensional image is described, the method comprises: receiving a single two-dimensional image; obtaining a training signal for the three-dimensional reconstruction by adapting a three-dimensional model of an object to the two-dimensional image; and using the resulting three-dimensional model fitting results as a supervisory signal for training the deep machine learning model.

[0016] The step of fitting a 3D model may be performed by: a step of generating a projected representation by projecting a 3D representation onto a 2D image plane; a step of comparing each location of the projected representation with an object of a single 2D image; a step of measuring an error value based on the comparison; and a step of adjusting the parameters of the fused 3D representation based on the error value—the comparison, measurement, and adjustment steps are repeated iteratively until a threshold condition is satisfied. The threshold condition may be that the measured error value falls below a predetermined threshold value or that a threshold number of iterations is exceeded. The projection step may be performed by taking into account the effects of perspective projection, and, if multiple views are available, by using multiple views of the same object.

[0017] According to a further aspect of the present specification, a computational unit for generating a three-dimensional reconstruction of a scene having a plurality of objects from a single two-dimensional image is described, wherein the computational unit comprises a memory; and at least one processor, and the at least one processor is configured to perform according to a further aspect.

[0018] According to a further aspect of the present specification, a computer-readable medium is described that stores a set of instructions executable by at least one processor of a computing unit to enable a computing unit to perform a method according to a further aspect to generate a three-dimensional reconstruction of a scene having a plurality of objects from a single two-dimensional image.

[0019] According to a further aspect of the present specification, a computer program product is described that includes instructions that, when executed by a computer, cause the computer to perform a method according to a further aspect.

[0020] According to a further aspect of the present specification, a system for providing a three-dimensional reconstruction of a scene having a plurality of objects from a single two-dimensional image is described, and the system comprises: a first computing unit for performing any of the methods of the first aspect; and a second computing unit configured to perform a method according to an additional aspect, wherein the second unit is configured to train a model with the results of the first computing unit. Brief explanation of the drawing

[0021] Examples and embodiments will be described with reference to the accompanying drawings, in which: FIG. 1 illustrates a schematic overview of an exemplary method for generating a three-dimensional reconstruction of a scene having multiple objects from a single two-dimensional image; FIG. 2 illustrates a flowchart of a method for generating a three-dimensional reconstruction of a scene having multiple objects from a single two-dimensional image; FIG. 3 illustrates a schematic overview of an exemplary method for training a neural network to reconstruct a three-dimensional scene having multiple objects from a single two-dimensional image; FIG. 4 illustrates a flowchart of a method for training a neural network to be used to reconstruct a three-dimensional scene having multiple objects from a single two-dimensional image; FIG. 5 illustrates an additional example of a method for training a neural network for use in 3D scene reconstruction and object re-identification; FIG. 6 illustrates a schematic example of a system / device for performing any of the methods described herein. Specific details for implementing the invention

[0022] This specification describes methods and systems for reconstructing a 3D scene containing multiple objects (e.g., humans) from a single 2D image (e.g., an RGB image). The exemplary methods and systems described herein can recover accurate 3D reconstructions of multiple objects at more than 30 frames per second on a mobile device (e.g., a mobile phone), and also recover information regarding 3D camera positions and world coordinates. The methods and systems described herein can be applied, for example, to real-time augmented reality applications involving the whole human body, allowing users to control objects placed on them, for example, graphic assets attached to their hands, and also allowing objects to interact with humans, for example, balls to bounce off humans when they come into contact with the human body.

[0023] Additionally, this specification describes CNN design choices that allow for the estimation of 3D shapes of a number (potentially hundreds) of objects (e.g., humans) at more than 30 frames per second on a mobile device. Using the methods described herein, results can be obtained within a constant time of about 30 milliseconds per frame on a SnapDragon 855 neural processing unit (NPU), regardless of the number of objects (e.g., people) in a scene.

[0024] The various aspects described herein may contribute to these effects, either individually or in combination.

[0025] In some embodiments, network distillation is used to construct a supervised signal for monocular image reconstruction. A detailed time-demand model fitting procedure is performed offline to recover the 3D interpretation of all images in the training set. The fitting results are then used to train one or more neural networks (e.g., convolutional neural networks) to efficiently process incoming test images in a single feedforward pass through the network. This allows for the integration of complex constraints imposed by complementary real-world information signals during model fitting (e.g., region alignment, sparse and dense reprojection errors based on key points and dense poses, respectively) without compromising speed during test time.

[0026] In some embodiments, an efficient encoder-only neural network architecture is used for monocular 3D human pose estimation. Early studies relied on parametric modeling of human surfaces and decoder-based networks for thorough and accurate localization of human joints in 2D. Instead, a repurposed (standard) deep neural network used for image classification is utilized, and the final layer can be configured to output 3D coordinates for each vertex of an object (e.g., human) mesh. This can substantially accelerate test-time inference and also simplify the deployment of networks on mobile devices.

[0027] In some embodiments, the neural network may have a single-stage full convolutional architecture that densely processes input 2D images and emits 3D pose estimates for multiple image locations in a single pass through a (standard) convolutional network. This results in time-inferential inference independent of the number of objects (e.g., people) in the scene, and can also dramatically simplify the smoothing of 3D reconstructions over time, because the network layers can be averaged over time instead of performing post-processing smoothing.

[0028] In some embodiments, a human-based self-correction method is used to estimate floor positions in a 3D scene by fitting planes to the estimated 3D positions of objects / sub-objects (e.g., human feet) in a 2D image. This allows for the recovery of world geometry in a way that nullifies the scaling effects of perspective projection, while also recovering the camera position relative to the planes. In turn, recovering world geometry allows for the augmentation of the recovered scene, for example, by inserting objects that can fall and bounce back down while adhering to the laws of physics. The method can operate, for example, by aligning the estimated floor positions with planes recovered by SLAM, both when the camera is static—in which case simultaneous localization and mapping (SLAM) methods fail—and also when the camera is moving.

[0029] In some embodiments, a distributed part-based variant of this approach collects information about mesh parts from multiple image locations, as indicated by estimated object part locations. The body mesh is obtained by synthesizing the part-level meshes output by the neural network, thereby allowing for better processing of occlusions and large joints while maintaining exactly the same memory and computational costs, aside from memory retrieval operations.

[0030] In some embodiments, the neural network may additionally accommodate the task of object / person re-identification (REID). A teacher-student network distillation approach may be used to train the network to perform such a task. REID embeddings are extracted from crops of objects (e.g., humans) from 2D images using a pre-trained REID network, and these are used as supervised signals to train the REID branch of the neural network. This branch carries REID embeddings that mimic those of the training network, but may be full convolutions, which means that its execution speed is independent of the number of objects in the image.

[0031] FIG. 1 illustrates a schematic overview of an exemplary method (100) for generating a three-dimensional reconstruction of a scene having multiple objects from a single two-dimensional image. The method may be performed by one or more computing devices operating at one or more locations. For example, the method may be performed by a mobile computing device, such as a mobile phone.

[0032] One or more 2D images (102), each containing multiple objects (104) of a given type (e.g., people), are input to a neural network (106). In some embodiments, only a single input image (102) is used. The neural network (106) processes the input data to identify objects (104) within the image and to generate output data (108) containing an estimated 3D representation (110) of each (potential) identified object (104) within the input 2D image (102). In some embodiments, the output data (108) may further include, estimated coordinates (112) of bounding boxes of potential objects (104) within the input 2D image (102) and / or probabilities of the presence of objects (104) at locations within the input 2D image (102). The output data (108) is further processed (i.e., post-processed) to generate a three-dimensional reconstruction (116) of the scene within the input 2D image (102). In some embodiments, the output data (108) may further include an embedding vector (not shown) associated with each object (104) within the image. The embedding vector provides the REID embedding of the object, and thus, a given object (104) can be identified and / or tracked across multiple input images (102).

[0033] The 2D input image (102, I) contains a set of pixel values ​​corresponding to a 2D array. For example, in a color image, I ∈ R HxWx3 And, where H is the height of the image in pixels and W is the width of the image in pixels, and the image has three color channels (e.g., RGB or CIELAB). In some embodiments, the 2D image may be black and white / grayscale.

[0034] In some embodiments, the method (100) may use a number (e.g., a series) of input 2D images (102) (e.g., a number of moving people). 3D representations (110) may take these changes into account. The number of input 2D images (102) may be received, for example, from a camera on a mobile device in substantially real-time (e.g., at 30 fps). The neural network (106) may process each of these input 2D images (102) individually, and the hidden layer responses for each input 2D image (102) are combined during post-processing when generating a 3D scene.

[0035] A neural network (106) takes a 2D image (102) as input and processes it through multiple neural network layers to generate output data (108). The neural network (106) is a deep machine learning model comprising an output layer and one or more hidden layers that each apply a non-linear transformation to the received input to generate the output. The neural network can predict the 3D landmark locations of multiple objects by connecting feature data from one or more intermediate layers of the neural network. The predicted 3D locations can be simultaneously estimated for the predicted type of object depicted in each area.

[0036] Each layer of the neural network (106) comprises a plurality of nodes (also referred to herein as “neurons”), and each of the nodes is associated with one or more neural network parameters (e.g., weights and / or biases). Each node takes as input an output from one or more nodes in a previous layer (or an input to the first layer of the neural network) and applies a transformation to the input based on the parameters associated with that node. The transformation may be a non-linear transformation. Alternatively, some of the nodes may apply a linear transformation.

[0037] The neural network (106) may include one or more convolutional layers, each configured to apply one or more convolutional filters to the output of the previous layer within the network. In some embodiments, the neural network (106) is a full convolution. The neural network may include one or more fully connected layers, wherein each node in the fully connected layer receives input from all nodes in the previous layer. The neural network (106) may include one or more skip connections. The neural network (106) may include a residual neural network, e.g., ResNet-50. The residual network may be used as a backbone network.

[0038] The neural network (106) may be a single-stage system that detects objects (e.g., humans) and estimates the 3D shape of the objects by performing a single forward pass through the neural network (106), which may be a fully convolutional neural network. Instead of cropping image patches around object detection regions and then reprocessing them, the task of extracting region-specific features is delegated to neurons in successive layers of the neural network (106) having increasingly larger receptive fields. A ResNet-50 backbone may be used, which provides an excellent trade-off between speed and accuracy. a-trous convolutions can be used (e.g., see L. Chen et al., “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs”, PAMI, 2017.3, the contents of which are incorporated herein by reference), allowing for an increase in the spatial density at which object (e.g., person) hypotheses are evaluated and reducing the number of missed detections.

[0039] The final layer of the neural network (106) (e.g., a full convolutional neural network) is responsible for predicting output data (108) (e.g., multiple outputs) at each of the neurons corresponding to attributes of a person assumed to be at each location.

[0040] In some embodiments, one or more of the neural network (106) layers may be 1x1 convolution layers. For example, the final layer of the neural network may include one or more 1x1 convolutions. Examples of such layers are described in “Fully Convolutional Networks for Semantic Segmentation” (E. Shelhamer et al., IEEE Trans. Pattern Anal. Mach. Intell. 39(4): 640-651(2017)) and “Overfeat: Integrated recognition, localization and detection using convolutional networks” (P. Sermanet et al., 2nd International Conference on Learning Representations, 2014), the contents of which are incorporated by reference.

[0041] The output data (108) includes a mesh representation for each detected object (110). This may be in the form of a K = N x 3D vector capturing the shape of the object, where N is the number of nodes in the mesh. N may be 563, but other numbers of mesh nodes are possible. The output data (108) may further include the probability of the existence of an object (e.g., a person) (114). The output data (108) may further include the corners of the bounding boxes (112) of the objects in the input image (102).

[0042] In some embodiments, the predictions of the neural network (106) may be specific to the object type. For example, the neural network (106) may predict the 3D positions of facial parts, fingers, and limbs for humans, whereas a different neural network (106) may be dedicated to the positions of wheels, windows, doors, and lights for cars. It should be understood that object landmarks expected to be present due to the identified object type may not be present in the image. For example, if the object type is a human, parts of the human body may not be visible due to postures or other items that obstruct the view.

[0043] The use of a neural network (106) as described above can yield several advantages. First, inference is efficient and bypasses the need for deconvolution-based decoders for high-accuracy pose estimation. Instead, this architecture is "encoder-only" and reduces the time required to process M x M (e.g., 233 x 233) patches to a few milliseconds. Second, the resulting networks are easily portable on mobile devices because they rely exclusively on general convolution network layers, in contrast to the chair body models used in previous studies. Third, it is simple to extend the resulting model so that the entire image can be processed in a fully convolutional manner instead of processing individual patches.

[0044] Beyond simplicity, the resulting architecture also simplifies performing temporal smoothing of the reconstructed 3D shapes. Instead of the complex parameter tracking-based methods previously used, the moving average of the second layer activations at the end of the network (106) can be taken. This substantially stabilizes the 3D shapes recovered by the network without substantially requiring effort to integrate them as a processing step into a mobile device.

[0045] In some embodiments, the neural network (106) may output a whole object mesh (V) of an object (e.g., a person) based on a high-dimensional feature vector (F) calculated at a position (i) aligned with a predetermined point within the object (e.g., a sternum of a person). The feature vector may be output by a layer within the neural network and may correspond, for example, to activations of a hidden layer within the network. The mapping may be expressed as follows:

[0046]

[0047] Here, M represents mapping a feature vector (F) to an output mesh (V). In some embodiments, M can be implemented as a linear layer in a neural network.

[0048] The simplicity of this 'centralized' (i.e., aligned with a predetermined single point within the object) method is offset by the difficulty of coefficients for both details and variations of the object (e.g., human) mesh through F[i], which can act as a computational bottleneck. In such a method, the location of a given node within the output mesh (110) may be based on the activations of neurons within the neural network associated with locations far from the mesh node. This may introduce inaccuracies into the resulting output mesh (110).

[0049] In some embodiments, the part-based method may alternatively be used to reconstruct a 3D representation (110) of an object. The part-based approach utilizes a distributed approach to mesh regression, which can be substantially more accurate than a centralized method. The part-based approach is useful in situations where objects (104) can be divided into parts. For example, the human body can naturally be divided into parts, such as a head, torso, arms, hands, etc.

[0050] Taking the human body as an example, neurons associated with locations near object parts (e.g., hands, feet, etc.) that are far from a predetermined point (e.g., sternum) can provide more reliable estimates for each mesh part than neurons associated with more distant locations. These closer neurons act as "distributed part experts" that provide reliable information about body parts in their vicinity. Information from these distributed part experts can be combined using a coordinator node located at a predetermined point (e.g., sternum) to determine the output mesh (110). Information regarding a specific mesh node associated with a given part, coming from neurons associated with locations farther from the part (i.e., not associated with the part), can be suppressed / discarded during the determination of the output mesh (110). This can be achieved using a part-level attention matrix (A).

[0051] The partial locations that provide information to the coordinator are determined by the human skeletal calculation stage, and the sternum position can be associated with corresponding partial locations. The partial locations can be based, for example, on the joint positions of the human body.

[0052] In a part-based approach, the neural network (106) outputs individual part meshes (V[p]) for each part (p) of an object (104) within a predetermined list of object parts. Each part mesh (V[p]) is provided in its own coordinate system, with its origin offset from a reference point (also referred to herein as a "coordinator node") (c) (e.g., a human sternum). That is, {p1, ... p M} represents the locations of each of the M parts associated with the coordinator (e.g., sternum) location (c). The part locations may be based on key points of the objects, e.g., joint locations within the human body.

[0053] To reconstruct the entire mesh (V) from partial meshes (V[p]), the coordinator selects nodes related to the partial (p) from V[p] using a partial-level attention matrix (A). In some embodiments, A is a binary matrix indicating which part should be used for each vertex within the final mesh (V). For such an attention matrix, the final mesh (V) can be reconstructed for each mesh vertex (v) within the final mesh using the following:

[0054]

[0055]

[0056] Here, V[c,v] represents the position of the mesh node (v) for a predetermined point (c), and p v represents the location of the "active expert" part for node (v) (i.e., the node where A[p,v]=1 in the binary example), and V[p,v] represents the location of the mesh node (v) within the mesh associated with the part (p). (cp v ) is an offset vector describing the relative position between the part and the reference position (c) (e.g., the sternum).

[0057] In some embodiments, the partial-level attention matrix (A) may be a learned matrix determined during training. In such examples, the position of a given mesh node (v) may be based on information from one or more parts. Thus, using Equation (2) rather than Equation (3) may result in a weighted sum regarding more than one part.

[0058] In some embodiments, the output data (108) further includes, in some embodiments, estimated coordinates (112) of bounding boxes of potential objects (104) in the input 2D image (102) and / or probabilities of the existence of objects (104) at locations in the input 2D image (102).

[0059] The estimated 3D representation (110) of each (potential) identified object (104) within the input 2D image (102) is, in some embodiments, in the form of an N-vertex mesh (V). For example, the estimated 3D representation (110) of each identified object (104) may be in the form of an (Nx3)-dimensional vector providing 3D coordinates of N vertices within the mesh. In some embodiments, N=536, but other numbers of mesh nodes are possible.

[0060] The coordinates of the bounding boxes may include the x and y positions of the vertices of the bounding boxes within the input 2D image (102) for each potential object, for example, {(x1, y1), (x2, y2), (x3, y3), (x4, y4)} for each potential object. Each bounding box may be associated with a corresponding probability that the bounding box contains an object of a given type (e.g., a person).

[0061] A 3D reconstruction (116) of a scene within an input 2D image (102) is generated by considering perspective projection and nullifying its effects when estimating coordinates within the world. The 3D reconstruction (116) is generated by estimating a 3D plane (118) (also referred to herein as a “support plane”) that physically supports the objects (104) (e.g., floor, ground) identified in the image (102) and by placing all 3D objects (110) in space relative to the support plane.

[0062] Estimating the 3D plane (118) that physically supports the objects (104) identified in the image (102) may be based on the assumption that the objects (e.g., humans) are supported by a single physical plane. In some embodiments, this assumption is relaxed by using mixed models and using expectation maximization to assign different humans to different planes.

[0063] Estimating the 3D plane (118) may also be based on the assumption that the heights of the objects (e.g., humans) are approximately the same. This latter assumption may be mitigated if a series of input images (102) in which individual objects can be tracked over time is available and the effects of perspective projection over time for each object are monitored.

[0064] The scaling of each mesh is estimated to place 3D objects (110) in space relative to a support plane and to bring them into world coordinates. The scale is inversely proportional to the distance of the object (e.g., a person) from the camera capturing the scene in the input image (102). Thus, the scaling can be used to position the mesh in world coordinates along a line connecting the mesh vertices to the center of the camera.

[0065] For example, 3D objects (110) may be provided by the neural network (106) in the form of meshes estimated under orthogonal projection. Vertices (m i ={v i,1 ,..., v i,K Assuming the i-th mesh in a scene having}), here v i,k = (x i,k , y i,k , z i,k ) ∈ R 3 is the scaled orthogonal projection and scale(s i These are mesh coordinates estimated under ), and the mesh vertices are located in 3D world coordinates by nullifying the effects of perspective projection. For example, the world coordinates of the k-th point in the i-th mesh (V i,k =(X i,k , Y ,i,k , Z ,i,k)) can be estimated from the corresponding mesh coordinates estimated under scaled orthogonality by "pushing back" the mesh depth in world coordinates by the inverse of the scale factor and setting the X and Y world coordinates so that they are correctly re-projected to their corresponding pixel coordinate values ​​in x and y. Symbolically, this can be expressed as follows:

[0066]

[0067] Here, the camera correction matrix is ​​the center (c x = W / 2 and c y = H / 2, where W and H are image dimensions, i.e., image width and height) and focal length (f). These can be set manually or by camera calibration.

[0068] For each mesh corresponding to an identified object, the lowest point (i.e., the lowest Y value) is determined and used to estimate the contact points between the object and a support plane (e.g., the floor). Once at least four of such points are determined (collectively denoted as M), the determined points can be used to estimate the support plane in world coordinates. In some embodiments, the least squares method can be used to estimate the plane in world coordinates, for example:

[0069]

[0070] Here, V1=(a, b, c) is a vector perpendicular to the support plane. In some embodiments, this vector may be normalized. World coordinate axes (R=[v1 T , v2 T , v3 T ] T) is defined by finding two complementary directions (V2 and V3)—both of which are orthogonal to V1 and to each other. In some embodiments, V2 is chosen as the direction of Z and V3 = V1 x V2. In some embodiments, the vectors (V2 and V3) can be normalized, that is, the set {V1, V2, V3} forms an orthonormal basis.

[0071] The world coordinate center (T) can also be assigned as a 3D point placed on a plane. For example, it can be set to a point 3 meters from the camera and projected at y=H / 2.

[0072] In some embodiments, the above may be used to define a transformation between the world coordinate system and pixel locations within a 2D image (102). For example, in a single 3x4 perspective projection matrix (P), the world-to-camera transformation and the transformation and camera correction matrix may be given as follows:

[0073]

[0074] Here, the direction (V) vectors appear as rows rather than columns because the inverse rotation matrix is ​​being used, and R -1 = R T am.

[0075] Using homogeneous coordinates, world coordinates (C) can be converted into pixel coordinates (c) using the following:

[0076]

[0077] In some embodiments, coordinate transformation can be used to introduce objects into a scene / image that comply with the laws of physics and / or interact with other objects in a meaningful / realistic manner. This can lead to interaction applications requiring real-world coordinates, e.g., augmented reality games, such as games where a player attempts to hit another player with a sword or laser beam. In some embodiments, 3D reconstructed meshes can be projected back onto the input image (102) to demonstrate the accuracy of 3D human pose estimations.

[0078] In embodiments where a plurality of input 2D images (102) are used, the step of estimating a plane supporting the plurality can be performed for a series of frames using relative camera pose estimation and plane positioning using correspondence between points of consecutive frames.

[0079] Processing (e.g., post-processing) can occur over a number of consecutive frames by combining hidden layer responses in consecutive frames, for example, by averaging them.

[0080] According to some of the exemplary embodiments, composite objects may be controlled by or interact with objects (e.g., people) within the input image. For example, the added composite object may be a sword or laser beam that a person is controlling with their arm in a computer game, or an object that moves toward a person and bounces back when it comes into contact with an estimated 3D position of the person.

[0081] This method can be integrated into graphics engines, such as Unity. Users can interact with their own images / videos in real time and / or view meshes superimposed on themselves.

[0082] For example, in video, combining multiple time-lapse measurements taken from consecutive input images (102) provides improved estimates by combining methods with camera tracking, as simultaneous localization and mapping (SLAM) allows.

[0083] In the case of a combination of deformable objects such as humans and robust 3D scenes, both are reconstructed into metric coordinates.

[0084] FIG. 2 illustrates a flowchart of an exemplary method for generating a three-dimensional reconstruction of a scene having multiple objects from a single two-dimensional image. The method may be performed by one or more computers operating at one or more locations. The method may correspond to the method described in relation to FIG. 1.

[0085] In operation (2.1), a 2D image is received. The 2D image includes an array of pixel values, for example, an RGB image. A single 2D input image may be received. In some embodiments, a series of single images may be received.

[0086] In operation (2.2), objects within the image to be reconstructed are identified. Object types may also be identified. Operation (2.2) may be performed by a neural network, for example, one or more layers of the neural network described above in relation to FIG. 1.

[0087] In operation (2.3), a 3D representation of each identified object is estimated. The 3D representation may be in the form of a mesh, for example, a K=Nx3D vector of N vertex positions in 3D. Operation (7.3) may be performed by a neural network, for example, one or more layers of the neural network described above in relation to FIG. 1.

[0088] The operation of estimating 3D representations can be performed in a deep machine learning model, for example, a neural network, which includes an output layer and one or more hidden layers that apply a non-linear transformation to the received input to generate the output. The deep machine learning model can predict the 3D landmark locations of multiple objects by concatenating feature data from one or more intermediate layers of the neural network. The predicted 3D locations are simultaneously estimated for the predicted types of objects depicted in each region.

[0089] When a series of input images are received, the step of estimating three-dimensional representations of multiple objects and placing them on a plane is performed for each received image. When images are received in substantially real-time (e.g., 30 fps), these operations can be performed in substantially real-time.

[0090] In operation (2.4), a 3D plane that physically supports all three-dimensional objects is estimated. Operation (2.4) can be performed in a post-processing step, that is, after processing the input image by a neural network. Methods for estimating the support frame are described in more detail above with reference to FIG. 1.

[0091] Estimating a plane supporting multiple objects can be performed for a single frame using the estimated 3D positions of all visible objects to reconstruct a 2D plane passing through all visible objects. For example, the support plane can be estimated based on the estimated positions of contact points between the objects and the plane, e.g., the positions of people's feet identified in the input image. Estimating a plane supporting multiple objects can be performed for a series of frames using relative camera pose estimation and plane localization using correspondences between points in consecutive frames.

[0092] In operation (2.5), 3D objects are positioned in space with respect to a support plane. Operation (2.5) can be performed in a post-processing step, that is, after processing of the input image by a neural network. Methods for positioning 3D objects with respect to a support frame are described in more detail above with reference to FIG. 1.

[0093] FIG. 3 illustrates a schematic overview of an exemplary method (300) for training a neural network to reconstruct a three-dimensional scene having multiple objects from a single two-dimensional image. The method may be performed by one or more computers operating at one or more locations.

[0094] A 2D training image (302) is obtained from a training dataset containing multiple 2D images. The 2D images contain one or more objects (304) of a given type (e.g., one or more humans). A 3D deformable model is fitted (e.g., iteratively fitted) to the objects in the input image (302) to extract one or more supervised signals (306) for monocular 3D reconstruction. To generate output data (310) containing candidate 3D representations of each (expected) object in the input training image (302), the training image (302) is input into a neural network (308) and processed through a series of neural network layers using the current parameters of the neural network (308). The output data is compared with the supervised signals (306) of the input image (302) to determine parameter updates (312) for the neural network (308).

[0095] The method can be performed repeatedly until a critical condition is satisfied.

[0096] The input images (302) and the neural network (308) may be of the same form as described in relation to FIG. 1.

[0097] During the deformable model fitting / optimization stage, all available 2D and 3D real-world data are used to provide energy terms that are minimized to obtain the estimated 3D shape (i.e., supervised signal (306)). As an example, the method described in “Holopose: Holistic 3d human reconstruction in-the-wild” (R. Alp Guler and I. Kokkinos, The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019, 2, 3, the contents of which are incorporated herein by reference) which uses CNN-style iterative fitting of deformable models into 2D images may be used.

[0098] The step of fitting the 3D model can be performed by projecting the 3D representation onto a 2D image plane to generate a projected representation. The projection step can be performed by considering the effects of perspective projection and, if multiple views are available, by using multiple views of the same object. Then, each location of the projected representation can be compared with an object within a single 2D image, and based on the comparison, an error value can be determined by determining, for example, the difference between the object landmark locations in the training image (302) and the reprojected image. Then, the parameters of the fused 3D representation are adjusted based on the error value. Projection, comparison, measurement, and adjustment can be repeated repeatedly until the measured error value is below a predetermined threshold or the threshold number of iterations is exceeded.

[0099] Once the deformable model fitting converges, the estimated 3D surface is used as a supervised signal (306) or target for training the network (308). For example, a “low-poly” representation of objects (304) (e.g., human bodies) represents a mesh in terms of N (e.g., N=536) vertices covering important objects (e.g., human landmarks (e.g., facial parts, limb joints, etc.)).

[0100] From fittings of a deformable model to an image, the estimated 3D locations of the vertices around each object present in the image are measured. Then, a neural network (308) is trained to regress this K=Nx3D vector when an object (304) is present in its receptive field. When multiple objects are present, different responses are expected at different image locations. The locations of the bounding boxes of objects within the input image (302) and / or the presence / absence of objects at locations within the input image may additionally be used as supervision signals (306).

[0101] The task of extracting region-specific features is delegated to neurons in successive layers having increasingly larger receptive fields. A ResNet-50 backbone, which offers an excellent trade-off between speed and accuracy, may be used. Atros convolutions may be used, allowing for an increase in the spatial density at which human hypotheses are evaluated and a reduction in the number of missed detections. The final layer of the resulting full convolution layer is tasked with predicting multiple outputs at each of those neurons corresponding to the attributes of a human assumed to be at each location. In some embodiments, the output data (310) further includes the probability of the presence of an object (e.g., a person). The corners of the object bounding box may also be regressed and form part of the output data (310).

[0102] Comparison of output data with respect to supervised signals (306) of an input image (302) can be performed using a loss / objective function (L). The loss function includes terms that penalize differences between candidate 3D representations and 3D representations in the supervised signals (306). Many examples of such loss functions, such as L1 or L2 losses, are known in the relevant art. The loss function may also include terms that penalize differences between the locations of candidate bounding boxes output by the neural network (308) and the locations of actual information bounding boxes within the input training image (302). Many examples of such loss functions, such as L1 or L2 losses, are known in the relevant art. The loss function may also include terms that penalize the output probabilities of the neural network (308) based on the actual presence or absence of objects at those locations. Many examples of such loss functions, such as classification loss functions, are known in the relevant art.

[0103] In some embodiments, the loss may penalize only the 3D representation and bounding box location in the presence of objects (e.g., people) - in the absence of objects, the box and 3D shape may take any values, but it is understood that the object detector will filter out the resulting hypotheses.

[0104] Parameter updates (312) can be determined by applying an optimization procedure, such as stochastic gradient descent, to the loss / objective function. The loss function can be averaged over batches of training images before determining each set of parameter updates. The process can be repeated over multiple batches within the training dataset.

[0105] In embodiments where a part-based approach to mesh estimation is used, each 3D representation within the supervised signal (306) can be divided into parts. The division into parts of each supervised signal (306) can be based on key points of the object (304). For example, the supervised signal (306) can be divided into body parts (e.g., hands, feet, etc.) based on joint positions. When determining parameter updates (312), the part mesh (V[p]) output by the neural network (308) is compared with the corresponding part meshes within the supervised signal (306).

[0106] FIG. 4 illustrates a flowchart of a method for training a neural network to be used to reconstruct a three-dimensional scene having multiple objects from a single two-dimensional image. The method may be performed by one or more computers operating at one or more locations. The method may correspond to the training method described with reference to FIG. 3.

[0107] In operation (4.1), one or more two-dimensional images are received. The 2D images may be in the form of the images described with reference to FIG. 1. Each 2D image contains one or more objects of a given type (e.g., human).

[0108] In operation (4.2), training signals for three-dimensional reconstruction are obtained through adaptation of a three-dimensional model of an object to each two-dimensional image. This effectively generates a labeled training dataset containing multiple 2D images, each associated with one or more 3D representations of objects (e.g., humans) within the images.

[0109] Fitting a 3D model may be performed by the steps of: generating a projected representation by projecting a 3D representation onto a 2D image plane; comparing each location of the projected representation with an object of a single 2D image; measuring an error value based on the comparison; and adjusting the parameters of the fused 3D representation based on the error value—the comparison, measurement, and adjustment steps are repeated iteratively until a threshold condition is satisfied. The threshold condition may be that the measured error value falls below a predetermined threshold value and / or exceeds a threshold number of iterations. The projection step may be performed by taking into account the effects of perspective projection and, if multiple views are available, by using multiple views of the same object.

[0110] In operation (4.3), the resulting three-dimensional model fitting results are used as supervisory signals for training a deep machine learning model. The deep machine learning model can be trained as described above in relation to FIG. 3.

[0111] FIG. 5 illustrates an additional example of a method (500) for training a neural network for use in three-dimensional scene reconstruction and object re-identification. The method may be performed by one or more computers operating at one or more locations.

[0112] Method (500) is an extension of the method described and illustrated above in relation to FIG. 3, comprising training a branch of the neural network (508) to output one or more re-identification (REID) embedding vectors (516a, 516b) as an additional part of the output data (510). Thus, any of the features described in relation to FIG. 3 may be additionally combined with the features described in relation to FIG. 5. The neural network (508) trained according to the methods described in relation to FIG. 5 is endowed with the ability to maintain object / person identity across a series of images. This may be important for video game experiences as well as for the accumulation of information over time and the smoothing of object / person specific parameters, where every user is consistently associated with a unique character over time.

[0113] Person / object re-identification is generally addressed in the literature through two-stage architectures, where people / objects are first detected by a person / object detection system, and for all detected people / objects, images are cropped and transmitted as input to a separate person identification network. The latter provides high-dimensional embeddings that act as unique person / object signatures trained to be invariant to disturbance parameters, such as camera position, person / object pose, and lighting. Such strategies have complexity that scales linearly with the number of people and are also difficult to implement as two-stage on mobile devices.

[0114] Instead, the neural networks described in relation to Figures 1-4 can be extended to accommodate the task of object / person re-identification (REID) using a teacher-student network distillation approach. REID embeddings are extracted from object / person crops using a pre-trained REID network. These REID embeddings are used as supervised signals to train REID branches for the neural network. These branches deliver REID embeddings that mimic those of the training network, but may be, for example, full convolutions, which means that their execution speed is independent of the number of objects.

[0115] Combining a network trained in such a manner with re-identification and tracking algorithms allows object / person identities to be maintained in scenes with high person overlap, interactions, and occlusions, even when the user briefly disappears from the scene. Examples of such re-identification and tracking algorithms can be found in "Memory based online learning of deep representations from video streams" (F. Pernici et al., CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 2324-2334), the content of which is incorporated by reference. Maintaining object / person identification across multiple images within a scene can allow different virtual objects / skins to be continuously associated with each object / person within the images over time.

[0116] A 2D training image (502) is obtained from a training dataset containing multiple 2D images. The 2D images contain one or more objects (504a, 504b) of a given type (e.g., humans) (generally, any number of objects may exist, but in the illustrated example, two objects (504a, 504b)). A 3D deformable model is fitted (e.g., iteratively fitted) to the objects in the input image (502) to extract one or more supervised signals (506) for monocular 3D reconstruction. Additionally, each object (504a, 504b) is cropped from the image (e.g., using a real information bounding box for the object) and individually input into a pre-trained REID network (518). A pre-trained REID network (518) processes a crop of each input object (504a, 504b) to encode the identity of the input objects (504a, 504b) into "teacher" re-identification embeddings (520a, 520b) (for object (i) in the input image (502) e T i Generates (indicated by ). Teacher re-identification embeddings (e T i , 520a, 520b) are used as supervisory signals to train the re-identification branch of the neural network (508).

[0117] To generate output data (510) including candidate 3D representations (514a, 514b) of each (expected) object in the input training image (502) and "student" re-identification embeddings (516a, 516b) of each (expected) object in the input training image (502), the training image (502) is input into a neural network (508) and processed through a series of neural network layers using the current parameters of the neural network (508). The output data (510) is compared with the teacher embeddings (520a, 520b) and the supervised signals (506) of the input image (502) to determine parameter updates (512) for the neural network (508).

[0118] The teacher REID embeddings (520a, 520b) and student REID embeddings (516a, 516b) are each high-dimensional vectors representing the identity of individual objects. For example, each REID embedding may be an N-dimensional vector. N may be, for example, 256, 512, or 1024.

[0119] When a loss / objective function is used to determine parameter updates (512), the loss / objective function may include a re-identification loss that compares each teacher REID embedding (520a, 520b) with its corresponding student REID embedding (516a, 516b). The re-embedding loss may be an L1 or L2 loss between each teacher REID embedding (520a, 520b) and its corresponding student REID embedding (516a, 516b).

[0120] FIG. 6 illustrates a schematic example of a system / device for carrying out any of the methods described herein. The illustrated system / device is an example of a computing device. It will be understood by those skilled in the art that other types of computing devices / systems, e.g., distributed computing systems, may alternatively be used to implement the methods described herein. One or more of these systems / devices may be used to carry out the methods described herein. For example, a first computing device (e.g., a mobile computing device) may be used to carry out the method described above in relation to FIGS. 1 and 2, and / or a second computing device may be used to carry out the method described above in relation to FIGS. 3 through 5. A model used in the first computing unit may have been trained using the second computing unit.

[0121] The device (or system) (600) includes one or more processors (602). One or more processors control the operation of other components of the system / device (600). One or more processors (602) may include, for example, a general-purpose processor. One or more processors (602) may be a single-core device or a multi-core device. One or more processors (602) may include a central processing unit (CPU) or a graphics processing unit (GPU). Alternatively, one or more processors (602) may include specialized processing hardware, for example, a RISC processor or programmable hardware with embedded firmware. Multiple processors may be included.

[0122] The system / device includes a working or volatile memory (604). One or more processors may access the volatile memory (604) to process data and control the storage of data within the memory. The volatile memory (604) may include any type of RAM, for example, static RAM (SRAM), dynamic RAM (DRAM), or flash memory such as an SD card.

[0123] The system / device includes non-volatile memory (606). The non-volatile memory (606) stores a set of operation instructions (608) for controlling the operation of the processors (602) in the form of computer-readable instructions. The non-volatile memory (606) may be any type of memory, such as read-only memory (ROM), flash memory, or magnetic drive memory.

[0124] One or more processors (602) are configured to execute operation instructions (608) that cause a system / device to perform any of the methods described herein. The operation instructions (608) may include code (i.e., drivers) for hardware components of the system / device (600), as well as code for basic operation of the system / device (600). Generally speaking, one or more processors (602) execute one or more instructions of the operation instructions (608) that are permanently or semi-permanently stored in non-volatile memory (606), and use volatile memory (604) to temporarily store data generated during the execution of said operation instructions (608).

[0125] Implementations of the methods described herein may be realized as digital electronic circuits, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These may include computer program products (e.g., software stored on magnetic disks, optical disks, memory, programmable logic devices) comprising computer-readable instructions that, when executed by a computer as described in connection with FIG. 6, cause the computer to perform one or more of the methods described herein.

[0126] Any system features as described herein may also be provided as method features, and vice versa. As used herein, functional description features may alternatively be expressed in terms of their corresponding structures. In particular, method aspects may be applied to system aspects, and vice versa.

[0127] Additionally, any, some, and / or all features in one aspect may be applied to any, some, and / or all features in any other aspect in any suitable combination. It should also be understood that specific combinations of the various features described and defined in any aspect of the invention may be independently implemented, supplied, and / or utilized.

[0128] Although some embodiments have been illustrated and described, those skilled in the art will understand that modifications to these embodiments may be made without departing from the principles of the present disclosure, the scope of which is defined in the claims.

[0129] The various exemplary embodiments described herein are described in the general context of method steps or processes that may be implemented by a computer program product embodied on a computer-readable medium, which includes computer-executable instructions, such as program code, executed by computers in networked environments in one aspect. The computer-readable medium may include removable and non-removable storage devices, including but not limited to read-only memory (ROM), random access memory (RAM), compact discs (CDs), digital multifunction discs (DVDs), etc. Generally, program modules may include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of program code for executing the steps of the methods disclosed herein. A particular set of such executable instructions or associated data structures represents examples of corresponding operations for implementing the functions described in such steps or processes.

[0130] In the foregoing specification, embodiments have been described with reference to a number of specific details that may vary by implementation. Specific adaptations and modifications to the described embodiments may be made. Other embodiments may be apparent to a person skilled in the art by considering the practice of the invention disclosed herein and the specification. The specification and examples are intended to be regarded only as illustrative examples having the true scope and spirit of the invention as set forth in the following claims. Furthermore, the series of steps illustrated in the drawings are intended only for illustrative purposes and are not intended to be limited to any particular order of steps.

[0131] Thus, a person skilled in the art can understand that these steps can be performed in different orders while implementing the same method.

[0132] In the drawings and in this specification, exemplary embodiments have been disclosed. However, many variations and modifications may be made to these embodiments. Accordingly, specific terms are employed, but are used only in a general and descriptive sense and not for limiting purposes, and the scope of the embodiments is defined by the exemplary embodiments presented herein.

Claims

Claim 1 A method comprising: extracting object / person re-identification (REID) embeddings from object / person crops by a teacher-student network; training a REID branch for a neural network using the REID embeddings as a supervised signal; processing the REID embeddings generated by the teacher-student network to track object / person identities across multiple images in a scene having object overlaps and occlusions; generating a 3D reconstruction of the scene having multiple objects from a single 2D image; estimating a 3D representation of each identified object using a deep machine learning model; estimating a 3D plane that physically supports the multiple objects based on the 3D positions of the 3D representations; measuring an error value based on comparing the 3D plane with a 2D plane depicted in the single 2D image; and adjusting the 3D plane based on the error value. Claim 2 delete Claim 3 The method of claim 1, wherein the deep machine learning model comprises an output layer and one or more hidden layers that each apply a non-linear transformation to a received input to generate an output. Claim 4 A method according to paragraph 3, wherein the deep machine learning model predicts three-dimensional landmark locations of a plurality of objects by connecting feature data from one or more intermediate layers of a neural network, and the predicted three-dimensional landmark locations are simultaneously estimated for the predicted type of object described in each area. Claim 5 A method according to claim 1, wherein the step of estimating the three-dimensional plane supporting the plurality of objects is performed on a sequence of frames using plane positioning and relative camera pose estimation using correspondence relationships between points of consecutive frames. Claim 6 A method according to claim 1, further comprising the step of receiving a plurality of images, wherein the step of estimating the three-dimensional representations of the plurality of objects and the step of displaying the three-dimensional representations on the three-dimensional plane are performed in real time for each received image. Claim 7 In claim 6, the deep machine learning model comprises one or more hidden layers, and the method further comprises the step of combining hidden layer responses in consecutive frames by averaging the hidden layer responses. Claim 8 A method according to claim 1, wherein digital graphic objects are synthetically added to the three-dimensional reconstruction of the scene in relation to the three-dimensional positions and then projected back onto the single two-dimensional image. Claim 9 A method according to claim 1, wherein the teacher-student network is trained using a pre-trained REID network to generate high-dimensional embeddings. Claim 10 A method according to claim 1, further comprising the step of using the REID embeddings to maintain object / person identity across a sequence of video frames. Claim 11 A method according to claim 1, wherein the REID branch of the neural network is a complete convolution, allowing real-time processing independent of the number of objects in the scene. Claim 12 A method according to claim 1, further comprising the step of integrating the REID embeddings with a tracking algorithm to process the occlusion and reappearance of objects / people in video frames. Claim 13 A method according to claim 1, wherein the REID embeddings are used to associate virtual objects or skins across video frames with tracked objects / people. Claim 14 A method according to claim 1, further comprising the step of improving the accuracy of object / person-specific parameter smoothing over time using the REID embeddings. Claim 15 A method according to claim 1, wherein the REID embeddings are used to generate a unique signature for each object / person that is invariant to changes in camera position and lighting conditions. Claim 16 A method according to claim 1, further comprising the step of employing a memory-based online learning approach to update the REID embeddings as new video frames are processed. Claim 17 A system comprising: memory; and at least one processor, wherein the at least one processor is configured to perform operations, the operations comprising: extracting object / person re-identification (REID) embeddings from object / person crops by a teacher-student network; training a REID branch for a neural network using the REID embeddings as a supervised signal; processing the REID embeddings generated by the teacher-student network to track object / person identities across multiple images in a scene having object overlaps and occlusions; generating a 3D reconstruction of the scene having multiple objects from a single 2D image; estimating a 3D representation of each identified object using a deep machine learning model; estimating a 3D plane that physically supports the multiple objects based on the 3D positions of the 3D representation; measuring an error value based on comparing the 3D plane with a 2D plane depicted in the single 2D image; and adjusting the 3D plane based on the error value. Claim 18 A system according to claim 17, wherein the operations include employing a memory-based online learning approach to update the REID embeddings as new video frames are processed. Claim 19 In claim 17, the system is used to generate a unique signature for each object / person that is invariant to changes in camera position and lighting conditions. Claim 20 A computer-readable medium storing a set of instructions executable by at least one processor to enable at least one processor to perform operations, wherein the operations include: extracting Object / Person Re-identification (REID) embeddings from object / person crops by a teacher-student network; training a REID branch for a neural network using the REID embeddings as a supervised signal; processing the REID embeddings generated by the teacher-student network to track object / person identities across multiple images in a scene having object overlaps and occlusions; generating a 3D reconstruction of the scene having multiple objects from a single 2D image; estimating a 3D representation of each identified object using a deep machine learning model; estimating a 3D plane physically supporting the multiple objects based on the 3D positions of the 3D representation; and measuring an error value based on comparing the 3D plane with a 2D plane depicted in the single 2D image. A computer-readable medium comprising an operation to adjust the three-dimensional plane based on the error value.

Images