System and method for end-to-end scene reconstruction from multi-view images - Patents.com

Abstract

A system and method for generating a three-dimensional (3D) reconstruction of a scene or environment surrounding a user of a spatial computing system, such as a virtual reality, augmented reality, or mixed reality system, using only multi-view images comprising RGB images and without requiring a depth sensor or depth data from sensors. Features are extracted from a sequence of frames of the RGB images and back-projected into a 3D voxel volume using known camera intrinsic and extrinsic properties, with each pixel of the voxel volume being mapped to a ray within the voxel volume. The back-projected features are fused into the 3D voxel volume. The 3D voxel volume is passed through a 3D convolutional neural network to refine the features and regress a truncated signed distance function value at each voxel of the 3D voxel volume.

Description

[Technical field]

[0001] FIELD OF THEINVENTION The present invention relates to computing, learning network configurations, and connected mobile computing systems, methods, and configurations, and more particularly to systems and methods for generating three-dimensional reconstructions of a scene from multi-view images that may be utilized in mobile computing systems, methods, and configurations having at least one wearable component configured for virtual and / or augmented reality operation. [Background technology]

[0002] (background) Modern computing and display technologies are facilitating the development of systems for so-called virtual reality ("VR"), augmented reality ("AR"), and / or mixed reality ("MR") experiences, collectively referred to as "cross-reality" environments or experiences. This can be done by presenting computer-generated images to the user through a head-mounted display. The images create a sensory experience that immerses the user in the simulated environment. The data can, for example, describe virtual objects that can be rendered such that the user can sense or perceive and interact with the virtual objects as part of the physical world. The user can experience these virtual objects as a result of data being rendered and presented through a user interface device, such as a head-mounted display device. The data can be displayed to the user so that it can be seen, or played audibly to the user, can control audio, or can control a tactile (or haptic) interface, allowing the user to experience touch sensations that the user senses or perceives as they feel the virtual objects.

[0003] XR systems can be useful for many applications, ranging from the areas of scientific visualization, medical training, engineering design and prototyping, remote operation and telepresence, and personal entertainment. VR systems typically involve the presentation of digital or virtual image information without transparency to actual real-world visual input.

[0004] AR systems generally complement the real-world environment with simulated elements. For example, AR systems may provide a user with a view of the surrounding real-world environment via a head-mounted display. Computer-generated images may also be presented on the head-mounted display to enhance the surrounding real-world environment. The computer-generated images may include elements that are contextually related to the surrounding real-world environment. Such elements may include simulated text, images, objects, and the like. MR systems also introduce simulated objects into the real-world environment, but these objects are typically characterized by a degree of interaction that exceeds that of AR systems.

[0005] AR / MR scenarios often involve the presentation of virtual image elements in relation to real-world objects. For example, an AR / MR scene is depicted such that a user of the AR / MR technology sees a real-world scene featuring the environment surrounding the user, including structures, objects, etc. In addition to these features, the user of the AR / MR technology perceives that they "see" computer-generated features (i.e., virtual objects), even if such features do not exist in the real-world environment. Thus, in contrast to VR, AR and MR involve one or more virtual objects in relation to real objects in the physical world. The virtual objects also interact with real-world objects, and thus AR / MR systems may also be referred to as "spatial computing" systems in relation to the system's interaction with the 3D world surrounding the user. The experience of virtual objects interacting with real objects greatly enhances the user's enjoyment when using an AR / MR system, and also opens up possibilities for a variety of applications that present realistic and easily understandable information about how the physical world can be altered.

[0006] The visual centers of the brain derive valuable perceptual information from the movement of the eyes and their components relative to one another. Vergence movement of the two eyes relative to one another (i.e., the rotational movement of the pupils toward or away from one another to converge and fixate the gaze of the eyes on an object) is closely linked to the accommodation (or focusing) of the eye's lenses. Under normal conditions, accommodating the eyes, or changing the focus of the eye's lenses to focus on an object at a different distance, will automatically produce a matching change in vergence to the same distance, a relationship known as the "accommodation-vergence reflex." Similarly, a change in vergence will induce a matching change in accommodation under normal conditions. Working against this reflex, as in most conventional stereoscopic VR / AR / MR configurations, is known to cause eye strain, headaches, or other forms of discomfort to the user.

[0007] Stereoscopic wearable glasses generally feature two displays (one for the left eye, one for the right eye) configured to display images with slightly different element presentations such that a three-dimensional perspective is perceived by the human visual system. Such configurations have been found to be uncomfortable for many users due to a mismatch between vergence and accommodation ("vergence-accommodation conflict"), which must be overcome to perceive images in three dimensions. In fact, some users are unable to tolerate stereoscopic configurations. These limitations apply to VR, AR, and MR systems. Thus, most conventional VR / AR / MR systems are not optimally suited to present rich binocular three-dimensional experiences in a manner that would be comfortable and maximally useful to users, in part because conventional systems are unable to address some of the fundamental aspects of the human perceptual system, including vergence-accommodation conflict.

[0008] Various systems and methods have been disclosed to address vergence-accommodation conflicts. For example, U.S. Utility Patent Application No. 14 / 555,585 discloses a VR / AR / MR system and method that addresses vergence-accommodation conflicts by using one or more light-guiding optical elements to project light to the user's eyes such that the light and the images rendered by the light appear to originate from multiple depth planes. The light-guiding optical elements are designed to in-couple virtual light corresponding to a digital or virtual object, propagate it by total internal reflection ("TIR"), and then out-couple the virtual light and display the virtual object to the user's eyes. In an AR / MR system, the light-guiding optical elements are also designed to be transparent to light from (e.g., reflected from) actual real-world objects. Thus, a portion of the light-guiding optical elements is designed to be transparent to real-world light from real-world objects in the AR / MR system while reflecting virtual light for propagation via TIR.

[0009] AR / MR scenarios often involve interactions between virtual objects and the real-world physical environment. Similarly, some VR scenarios involve interactions between fully virtual objects and other virtual objects. Delineating objects in the physical environment facilitates interaction with virtual objects by defining the boundaries and borders of those interactions (e.g., by defining the extent of a particular structure or object in the physical environment). For example, if an AR / MR scenario involves a virtual object (e.g., a tentacle or fist) extending from a particular object in the physical environment, defining the extent of the object in three dimensions allows the AR / MR system to present a more realistic AR / MR scenario. Conversely, if the extent of the object is not defined or is defined inaccurately, artifacts or errors will occur in the displayed image. For example, a virtual object may appear to extend partially or completely from the air adjacent to the object, instead of from the surface of the object. In another example, if an AR / MR scenario involves a virtual character walking on a particular horizontal surface in a physical environment, inaccurately defining the extent of the surface may result in the virtual character not falling but instead appearing to be floating in the air and walking away from the surface.

[0010] Therefore, to provide an accurate and effective three-dimensional (3D) XR experience, an XR system needs to be capable of generating an accurate, yet computationally efficient (e.g., near real-time) 3D reconstruction of the surrounding environment. Indeed, reconstructing the world around us has been a long-standing goal of computer vision.

[0011] Reconstructing a 3D model of a scene usually involves obtaining the depth of features over a sequence of images and fusing the depth maps using a 3D data structure. The most common 3D structure for depth accumulation is the voxel volume used by TSDF fusion. However, surfels (oriented point clouds) are starting to gain popularity. These methods are usually used in conjunction with physical depth sensors, but can also be applied to depth maps predicted from monocular or stereo images.

[0012] With the rise of deep learning, monocular depth estimation has improved significantly; however, its accuracy is still far below state-of-the-art stereo methods. A common classical approach to stereo is to use mutual information and semi-global matching to compute the difference between two images. More recently, several end-to-end plane sweeping algorithms have been proposed, such as DeepMVS, which uses a patch matching network, and MVDepthNet, which builds a cost volume from raw pixel measurements, performs 2D convolutions, and treats the plane as a feature channel. GPMVS builds on this and aggregates information over a long sequence into a cost volume using Gaussian processes. MVSNet and DPSNet use 2D CNNs to build a cost volume from features extracted from the image. They then use 3D convolutions on a 4D tensor to filter the cost volume. All of these methods require picking a target image, predicting the depth, and then finding a suitable nearby reference image.

[0013] Recent binocular stereo methods use a similarity-cost volume approach, but avoid frame selection by using a fixed baseline stereo pair. Depth maps across sequences are computed independently (or in some cases, weakly coupled). In contrast to these approaches, our method directly builds a single coherent 3D model from a sequence of input images.

[0014] TSDF fusion is simple and effective, but it cannot reconstruct partially occluded geometry and requires averaging many measurements to reduce noise. Therefore, learned methods are proposed to improve fusion. OctNetFusion uses a 3D encoder-decoder to aggregate multiple depth maps into a TSDF and shows results on single objects and parts of scans. ScanComplete builds on this and shows results for the entire room. SG-NN improves on ScanComplete by increasing the resolution using sparse convolutions and training using a novel self-supervised training scheme. 3D-SIC focuses on 3D instance segmentation using region proposals and adds a head per instance completion. Routed fusion uses 2D filtering and 3D convolutions within the view frustum to improve the aggregation of depth maps.

[0015] Several networks have been disclosed that take one or more images and directly predict a 3D representation. For example, 3D-R2N2 encodes the image into a latent space and then decodes the voxel occupancy volume. Octtree-Gen increases the resolution by using an octree data structure to improve the efficiency of 3D voxel volumes. Deep SDF chooses to learn a generative model that can output SDF values ​​for any input location, instead of discretizing the volume. A point set generation network has been disclosed that learns to generate a point cloud with a fixed number of points. Pixel2Mesh++ uses a graph convolution network to directly predict a triangulated mesh. These methods encode the input into a small latent code and mainly report results for single objects from Shapenet. Therefore, it is not obvious how to extend them to work on full scene reconstructions. Mesh-RCNN builds on 2D object detection by adding an additional head, which predicts a voxel occupancy grid for each instance and then refines them using a graph convolutional network on the mesh.

[0016] Backprojecting image features into voxel volumes and then refining them using 3D CNNs has also been used for human pose estimation. These work on regressive 3D heatmaps that are used to localize joint locations.

[0017] Subsequent work on deep voxel and scene representation networks accumulates features into 3D volumes, forming an unsupervised representation of the world, which can then be used to render novel views without the need to form an explicit geometric intermediate representation.

[0018] Other approaches to 3D reconstruction use monocular, binocular, or multi-view stereo methods, which take red-green-blue (RGB) images (one, two, or multiple, respectively) from an image sensor and predict a depth map for the images. For example, using monocular stereo methods to generate 3D reconstructions using RGB images, methods are described in the following publications: Fu, H., Gong, M., Wang, C. Batmanhelich, K., Tao, D.: Deep ordinal regression network for monocular depth estimation; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2002-2011 (2018); Lasinger, K., Ranftl, R., Schindler, K., Koltun, V.: Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer; arXiv preprint arXiv: 1907.01341 (2019); and Lee, JH, Hang, MK, Ko, DW, Suh, IH: From big to small: Multi-scale local planar guidance for monocular depth estimation; arXiv preprint arXiv: Disclosed in 1907.10326 (2019).One method of using binocular stereo methods to generate a 3D reconstruction using RGB images is disclosed in the following publications: Chabra, R., Straub, J., Sweeney, C., Newcombe, R., Fuchs, H.: Stereodrnet: Dilated residual stereonet; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; pp. 11786-11795 (2019); and Chang, AX, Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q. Li, Z., Savarese, S., Savva, M., Song, S., Su, H., et al.; Shapenet: An information-rich 3d model repository; arXiv preprint arXiv: 1512.00312 (2015).Also, various methods using multi-view stereo methods to generate 3D reconstructions using RGB images are described in the following publications: Hirschmuller, H.: Stereo processing by semiglobal matching and mutual information; IEEE Transaction on pattern analysis and machine intelligence 30(2), 328-341 (2007); Huang, PH, Matzen, K., Kopf, J., Ahuja, N., Huang, JB; Deepmvs: Learning multi-view stereopsis; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; pp. 2821-2830 (2018); I, S., Jeon, HG, Lin, S., Kweon, IS; DPSnet: End-to-end deep plane sweep stereo; 7th International Conference on Learning Representations, ICLR 2019; International Conference on Learning Representations, ICLR (2019) and Want, K., Shen, S.; Mvdepthnet: real-time multiview depth estimation neural network; 2018 International Conference on Computer Vision; pp. 2088-2096 (2017). However, despite a lot of recent work, these methods that generate 3D reconstructions using only RGB images are still much less accurate than depth sensors and do not produce satisfactory results when fused into a 3D model.

[0019] In addition to reconstructing geometry, many XR applications require 3D semantic labeling (i.e., 3D semantic segmentation) of the 3D reconstruction to provide a richer representation. In general, there are two approaches to solving this problem: 1) using a 2D segmentation network to predict semantics on 2D input images and backprojecting the labels to 3D; and 2) directly predicting semantic labels in 3D space. All these methods assume that depth is provided by a depth sensor. A notable exception is Kimera, which uses multi-view stereo to predict depth; however, results have only been shown on synthetic data and ground truth 2D segmentation.

[0020] SGPN formulates instance segmentation as a 3D point cloud clustering problem: predict a similarity matrix, cluster the 3D point cloud, and derive semantics and instance labels. 3D-SIS improves on these approaches by fusing 2D features within the 3D representation. RGB images are encoded and backprojected onto the 3D geometry reconstructed from the depth map using a 2D CNN. A 3D CNN is then used to predict 3D object bounding boxes and semantic labels. SSCN predicts semantics on high-resolution voxel volumes enabled by sparse convolutions.

[0021] Thus, there is a need for more efficient and accurate methods and systems for generating 3D reconstructions of a scene in an end-to-end manner using RGB images, without the need for a depth sensor. Furthermore, there is a need for such 3D reconstructions to be usable in presenting virtual image information on an XR display in multiple focal planes (e.g., two or more) to be practical for various use cases without exceeding acceptable tolerances for vergence-accommodation mismatch. In addition, there is a need for an XR system that implements a display that is lightweight, low-cost, has a small form factor, has a wide virtual image field of view, and is as transparent as possible. Furthermore, there is a need for 3D semantic segmentation of the 3D reconstruction, without the use of a depth sensor. Summary of the Invention [Means for solving the problem]

[0022] (summary) The embodiments disclosed herein are directed to systems and methods that generate a three-dimensional (3D) reconstruction of a scene or environment surrounding a user of a spatial computing system, such as an XR system, using only multi-view images, comprising RGB images, without the need for a depth sensor or depth data from a sensor. The 3D reconstruction can be utilized by the spatial computing system to provide, for example, an accurate and effective 3D XR experience. The resulting 3D XR experience may be displayed in a rich, binocular, three-dimensional experience that is comfortable and maximally useful to the user, in part because it may present images in a way that addresses some of the fundamental aspects of the human perceptual system, such as vergence-accommodation misalignment. For example, the 3D reconstruction with accurate depth data enables 3D images to be displayed in multiple focal planes. The 3D reconstruction also enables accurate management of interactions between virtual objects, other virtual objects, and / or real-world objects.

[0023] Thus, one embodiment of the present invention is directed to a method of generating a three-dimensional (3D) reconstruction of a scene using multi-view images. First, a sequence of frames of red-green-blue (RGB) images is acquired. This can be accomplished by one or more suitable cameras. Features are then extracted from the sequence of frames of RGB images using a two-dimensional convolutional neural network (2D CNN). The extracted features from each frame are then back-projected into a 3D voxel volume using known camera intrinsic and extrinsic properties, and each pixel of the voxel volume is mapped to a ray within the voxel volume. In contrast to previous methods of generating 3D reconstructions using monocular, binocular, and multi-view images, which require selecting a target image, predicting depth, and then finding a suitable nearby reference image, the present process avoids the need to select a target image and allows for the fusion of the entire sequence of frames into a single volume.

[0024] The backprojected features from each frame are accumulated (i.e., fused) into a 3D voxel volume. The 3D voxel volume is passed through a 3D Convolutional Neural Network (3D CNN) with an encoder-decoder to refine the features in the 3D voxel volume. The 3D CNN also regresses the output Truncated Signed Distance Function (TSDF) value at each voxel of the 3D voxel volume.

[0025] In another aspect, the frames may be fused into a single 3D feature volume using a moving average, such as a simple moving average or a weighted moving average.

[0026] In another aspect, the method may further include passing the refined features in the 3D voxel volume and the TSDF values ​​at each voxel of the 3D voxel volume through a batch normalization (batch norm) function and a rectified linear unit (reLU) function after passing the 3D voxel volume through all layers of the 3D CNN.

[0027] In yet another aspect of the method, the 3D CNN may include an additional skip connection from the encoder to the decoder of the 3D CNN. The method then uses the additional skip connection to skip one or more features in the 3D voxel volume from the encoder to the decoder of the 3D CNN.

[0028] In yet another aspect, the method may be configured to handle null voxels in the 3D voxel volume that have not had features backprojected into them. For example, the null voxels may correspond to voxels that were not observed during a sequence of frames of an RGB image. In such a case, the method further includes not using additional skip connections from the encoder for the null voxels and passing the null voxels through a batch norm function and a relu function to match the magnitude of the voxels that receive the skip connections.

[0029] In another aspect of the method, the 3D CNN may have multiple layers, each with a set of 3x3x3 residual blocks, and the 3D CNN may implement downsampling using a 3x3x3 stride-2 convolution and upsampling using a 1x1x1 convolution after trilinear interpolation.

[0030] In another aspect of the method, the 3D CNN further comprises an additional head for predicting a semantic segmentation. The method further includes the 3D CNN predicting a semantic segmentation of features within the 3D voxel volume.

[0031] In another aspect, the method further includes training the 2D CNN using a short frame sequence that covers a portion of the scene. The short frame sequence may include 10 or fewer frame sequences. In yet another aspect, the training of the 2D CNN may be fine-tuned using a larger frame sequence having more frame sequences than the short frame sequence. The larger frame sequence may include, for example, 100 or more frame sequences.

[0032] Another disclosed embodiment is directed to a cross reality system configured to generate a 3D reconstruction of a scene or environment surrounding a user of the cross reality system using only multi-view images comprising RGB images, without the need for depth data from a depth sensor or sensors. The cross reality system comprises a head mounted display device having a display system. For example, the head mounted display may have a pair of eyepiece displays in a glasses-like configuration. A computing system is in operative communication with the head mounted display. A plurality of camera sensors are in operative communication with the computing system. For example, the camera sensors may be mounted on the head mounted display or on any other suitable structure. The computing system is configured to generate a three-dimensional (3D) reconstruction of the scene from a sequence of frames of the RGB images by a process including any configuration of the method described above. In additional aspects of the cross reality system, the process may include any one or more of the additional aspects of the method described above. For example, the process may include obtaining from the camera sensor a sequence of frames of red-green-blue (RGB) images of a scene within a field of view of the camera sensor. Features from a sequence of frames of RGB images are extracted using a two-dimensional convolutional neural network (2D CNN). Features from each frame are backprojected into a 3D voxel volume using known camera intrinsic and extrinsic properties, and each pixel in the voxel volume is mapped to a ray within the voxel volume. Features from each frame are fused into the 3D voxel volume. The 3D voxel volume is passed through a 3D convolutional neural network (3D CNN) with an encoder-decoder to refine the features in the 3D voxel volume and regress the output truncated signed distance function (TSDF) value at each voxel of the 3D voxel volume.

[0033] The cross-reality system may then utilize the 3D reconstruction to generate a 3D XR experience in which 3D images are displayed within a rich, binocular, three-dimensional experience, such as displaying 3D images in multiple focal planes on a head-mounted display. The cross-reality system may also utilize the 3D reconstruction to manage interactions between virtual objects, other virtual objects, and / or real-world objects. The present invention provides, for example, the following: (Item 1) 1. A method for generating a three-dimensional (3D) reconstruction of a scene from multi-view images, the method comprising: obtaining a sequence of red-green-blue (RGB) image frames; extracting features from the sequence of frames of the RGB image using a two-dimensional convolutional neural network (2D CNN); back-projecting the features from each frame into a 3D voxel volume using known camera intrinsic and extrinsic properties, where each pixel of the voxel volume is mapped to a ray within the voxel volume; Fusing / accumulating features from each frame into the 3D voxel volume; passing the 3D voxel volume through a 3D convolutional neural network (3D CNN) having an encoder-decoder to refine the features in the 3D voxel volume and regressing output truncated signed distance function (TSDF) values ​​at each voxel of the 3D voxel volume; A method comprising: (Item 2) 2. The method of claim 1, wherein the frames are fused into a single 3D feature volume using a moving average. (Item 3) 3. The method of claim 2, wherein the moving average is a simple moving average. (Item 4) 4. The method of claim 3, wherein the moving average is a weighted moving average. (Item 5) Item 1, the method of item 1 further comprising passing the refined features in the 3D voxel volume and the TSDF values ​​at each voxel of the 3D voxel volume through a batch normalization (batch norm) function and a rectified linear unit (reLU) function after passing the 3D voxel volume through all layers of the 3D CNN. (Item 6) An additional skip connection is included from the encoder to the decoder of the 3D CNN, 2. The method of claim 1, further comprising: skipping one or more features in the 3D voxel volume from the encoder to the decoder of the 3D CNN using the additional skip connections. (Item 7) One or more null voxels of the 3D voxel volume do not have features backprojected therein, corresponding to voxels that were not observed during a sequence of frames of the RGB images, and the method further comprises: not using the additional skip connection from the encoder for the null voxels, the additional skip connection being for the null voxels; passing the null voxels through the batch norm function and a relu function to match the magnitude of the voxels receiving the skip connections; The method according to item 6, comprising: (Item 8) Item 1, wherein the 3D CNN has multiple layers, each with a set of 3x3x3 residual blocks, and the 3D CNN implements downsampling using a 3x3x3 stride 2 convolution and upsampling using a 1x1x1 convolution after trilinear interpolation. (Item 9) The 3D CNN further comprises an additional head for predicting semantic segmentation; 2. The method of claim 1, further comprising the 3D CNN predicting a semantic segmentation of the features within the 3D voxel volume. (Item 10) Item 1. The method of item 1, further comprising training the 2D CNN using a short-frame sequence that covers a portion of a scene. (Item 11) Item 11. The method of item 10, wherein the short frame sequence comprises a 10 or fewer frame sequence. (Item 12) Item 12. The method of item 11, further comprising fine-tuning the training of the 2D CNN using a larger frame sequence having more frames than the short frame sequence. (Item 13) Item 13. The method of item 12, wherein the larger frame sequence comprises a sequence of 100 or more frames. (Item 14) A cross reality system, comprising: a head mounted display device having a display system; a computing system in operative communication with the head mounted display; a plurality of camera sensors in operative communication with the computing system; Equipped with the computing system is configured to generate, by a process, a three-dimensional (3D) reconstruction of the scene from a sequence of frames of RGB images captured by the camera sensor; The process comprises: acquiring from the camera sensor a sequence of red-green-blue (RGB) image frames of a scene within a field of view of the camera sensor; extracting features from the sequence of frames of the RGB image using a two-dimensional convolutional neural network (2D CNN); back-projecting the features from each frame into a 3D voxel volume using known camera intrinsic and extrinsic properties, where each pixel of the voxel volume is mapped to a ray within the voxel volume; fusing the features from each frame into the 3D voxel volume; passing the 3D voxel volume through a 3D convolutional neural network (3D CNN) having an encoder-decoder to refine the features in the 3D voxel volume and regressing output truncated signed distance function (TSDF) values ​​at each voxel of the 3D voxel volume; Including, the system. (Item 15) Item 15. The system of item 14, wherein the frames are fused into a single 3D feature volume using a moving average. (Item 16) Item 16. The system of item 15, wherein the moving average is a simple moving average. (Item 17) Item 17. The system of item 16, wherein the moving average is a weighted moving average. (Item 18) Item 15. The system of item 14, wherein the process for generating a three-dimensional (3D) reconstruction of the scene from the sequence of frames of the RGB images further includes, after passing the 3D voxel volume through all layers of the 3D convolutional encoder-decoder, passing the refined features in the 3D voxel volume and the TSDF values ​​at each voxel of the 3D voxel volume through a batch normalization (batch norm) function and a rectified linear unit (reLU) function. (Item 19) An additional skip connection is included from the encoder to the decoder of the 3D CNN, Item 15. The system of item 14, wherein the process for generating a three-dimensional (3D) reconstruction of the scene from the sequence of frames of the RGB images further includes skipping one or more features in the 3D voxel volume from the encoder to the decoder of the 3D CNN using the additive skip connections. (Item 20) The process for generating a three-dimensional (3D) reconstruction of the scene from the sequence of frames of RGB images further comprises: not backprojecting into one or more null voxel features of the 3D voxel volume that correspond to voxels that were not observed during the sequence of frames of the RGB images; not using the additional skip connection from the encoder for the null voxels, the additional skip connection being for the null voxels; passing the null voxels through the batch norm function and a relu function to match the magnitude of the voxels receiving the skip connections; 20. The system according to item 19, comprising: (Item 21) The 3D CNN further comprises an additional head for predicting semantic segmentation; 15. The system of claim 14, wherein the method further comprises the 3D CNN predicting a semantic segmentation of the feature within the 3D voxel volume. [Brief description of the drawings]

[0034] The drawings illustrate the design and utility of preferred embodiments of the present disclosure, in which like elements are referred to by common reference numerals. In order to better understand how the above and other advantages and objects of the present disclosure are obtained, a more particular description of the present disclosure, briefly described above, will be given by reference to specific embodiments thereof, which are illustrated in the accompanying drawings. With the understanding that these drawings depict only exemplary embodiments of the present disclosure and therefore are not to be considered as limiting its scope, the present disclosure will be described and explained with additional specificity and detail through the use of the accompanying drawings.

[0035] [Figure 1] FIG. 1 is a schematic diagram of an exemplary cross-reality system for providing a cross-reality experience, according to one embodiment.

[0036] [Diagram 2] FIG. 2 shows a comparison of evaluations performed on Scannet of 3D reconstructions performed according to an embodiment of an embodiment of the method disclosed herein compared to a baseline method that utilizes traditional TSDF fusion after deep multi-view stereo.

[0037] [Diagram 3] FIG. 3 is a flowchart illustrating a method for generating a 3D reconstruction of a scene using RGB images, according to one embodiment.

[0038] [Figure 4] 4A-4C illustrate a comparison of a schematic of backprojection of features into a 3D voxel volume (FIG. 4A), a schematic of features using simple skip connections in a 3D CNN (FIG. 4B), and a schematic of features using masked skip connections to reduce artifacts and better complete the geometry of unobserved regions (FIG. 4C), according to one embodiment.

[0039] [Diagram 5] FIG. 5 is a schematic diagram of a 3D encoder-decoder architecture according to one embodiment.

[0040] [Figure 6] 6A-6E are a series of images illustrating the preparation of ground truth for training a CNN in an embodiment described herein.

[0041] [Figure 7] FIG. 7 is a series of images illustrating a comparison of 3D reconstructions generated in an embodiment described according to the methods disclosed herein, 3D reconstructions generated using DPSNet, and ground truth.

[0042] [Figure 8] FIG. 8 is a series of images comparing qualitative 3D semantically segmented landmarks generated in an embodiment described according to the methods disclosed herein against landmarks transferred to ground truth and ground truth landmarks.

[0043] [Figure 9] FIG. 9 is a table comparing 3D reconstructions generated in an embodiment described according to the methods disclosed herein against 3D reconstructions according to DPSNet and GPMVS using standard 2D depth and 3D metrics.

[0044] [Figure 10] FIG. 10 describes a process for passing a sequence of images through a 2D CNN backbone, extracting features, and backprojecting the features into a 3D voxel volume, according to one embodiment.

[0045] [Figure 11]FIG. 11 describes a process for accumulating feature volumes over a sequence of images using a weighted moving average, according to one embodiment.

[0046] [Figure 12] FIG. 12 sets forth equations that represent an exemplary skip connection, according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0047] (Detailed Description) The following describes various embodiments of systems and methods for generating a three-dimensional (3D) reconstruction of a scene or environment surrounding a user of a spatial computing system, such as an XR system, in an end-to-end reconstruction utilizing multi-view RGB images and without using depth or distance sensors. Various embodiments will now be described in detail with reference to the drawings, which are provided as illustrative examples of the present disclosure to enable those skilled in the art to practice the present disclosure. It should be noted that the following figures and examples are not meant to limit the scope of the present disclosure. Where certain elements of the present disclosure can be implemented partially or completely using known components (or methods or processes), only a portion of such known components (or methods or processes) necessary for understanding the present disclosure will be described, and detailed descriptions of other portions of such known components (or methods or processes) will be omitted so as not to obscure the present disclosure. Furthermore, various embodiments encompass current and future known equivalents of components referenced herein as examples.

[0048] Furthermore, the systems and methods for generating a three-dimensional (3D) reconstruction of a scene or environment surrounding a user of a spatial computing system may also be implemented independent of an XR system, and the embodiments depicted herein are described in conjunction with an AR / MR system for illustrative purposes only.

[0049] Referring to Figure 1, an exemplary XR system 100 is illustrated, according to one embodiment. The XR system 100 includes a head-mounted display device 2 (also referred to as a head-mounted viewing component 2), a handheld controller 4 (also referred to as a handheld controller component 4), and an interconnected ancillary computing system or controller 6 (also referred to as an interconnected ancillary computing system or controller component 6), which may be configured to be worn on a user as a beltpack or equivalent. Each of these components operatively communicate (i.e., are operatively coupled to) each other and other connected resources 8 (such as cloud computing or cloud storage resources) via wired or wireless communication connections 10, 12, 14, 16, 17, 18, such as those defined by IEEE 802.11, Bluetooth (RTM), and other connectivity standards and configurations. The head mounted display device includes two rendered optical elements 20 through which the user may view the world around them along with video images and visual components produced by associated system components, a pair of image sources (e.g., microdisplay panels) and viewing optics for displaying computer-generated images on the optical elements 20 for an augmented reality experience. As illustrated in FIG. 1, the XR system 100 also includes various sensors configured to provide information about the user's environmental surroundings, including, but not limited to, various camera type sensors 22, 24, 26 (such as monochrome, color / RGB, and / or thermal), a depth camera sensor 28, and / or a sound sensor 30 (such as a microphone).

[0050] In various embodiments, a user wears an augmented reality system, such as the XR system 100 depicted in FIG. 1, which may also be referred to as a "spatial computing" system in reference to the interaction of such system with the three-dimensional world around the user when operated. The cameras 22, 24, 26 are configured to map the user's surroundings and / or create a "mesh" of such environment, comprising various points representing the geometry of various objects within the user's surroundings, such as walls, floors, chairs, and the like. The spatial computing system may be configured to map or mesh the environment around the user and run or operate software, such as that available from Magic Leap, Inc. (Plantation, Florida), which may be configured to utilize the room map or mesh to assist the user in placing, manipulating, visualizing, creating, and modifying various objects and elements within the three-dimensional space around the user. As shown in FIG. 1, the XR system 100 may also be operatively coupled to additional connected resources 8, such as other computing systems, by cloud or other connectivity configurations.

[0051] It should be understood that the methods, systems, and configurations described herein are broadly applicable to a variety of scenarios outside the realm of wearable spatial computing, such as the XR system 100, depending on the appropriate sensors and associated data available.

[0052] One of the challenges in spatial computing relates to utilizing data captured by various operatively coupled sensors of the XR system 100 (such as elements 22, 24, 26, 28 of the system 100 of FIG. 1) in making decisions that are useful and / or important to a user, such as in computer vision and / or object recognition tasks that may relate to the three-dimensional world around the user. Disclosed herein are methods and systems for generating a 3D reconstruction of a scene, such as a 3D environment surrounding a user of the XR system 100, using only RGB images, such as RGB images from cameras 22, 24, and 26, without using depth data from the depth sensor 28.

[0053] In contrast to previous methods of generating 3D reconstructions using only RGB images described above, which produce relatively inaccurate depth and relatively unsatisfactory 3D image models, the methods and systems disclosed herein produce accurate full 3D models and also support efficient computation of other reconstruction data, including semantic segmentation.

[0054] In general, an approach is disclosed that directly regresses a truncated distance function ("TSDF") for a set of pose-attached RGB images. A two-dimensional (2D) CNN (convolutional neural network) is configured to independently extract features from each image. These features are backprojected and accumulated into a voxel volume using camera intrinsic and extrinsic properties (features for each pixel are placed along the entire ray). After accumulation, the voxel volume is passed through a three-dimensional (3D) CNN configured to refine the features and predict a TSDF value. Additional heads may be added to predict color, semantics, and instance labels with minimal excess computational resources. As described in more detail herein, the method is evaluated on Scannet, and it was determined that such method significantly outperforms state-of-the-art baselines (deep multi-view stereo followed by traditional TSDF fusion) both quantitatively and qualitatively, as shown in FIG. 2. The resulting 3D semantic segmentation was compared to previous methods using depth sensors, as previous work did not attempt to solve the problem using only RGB input. The methods and configurations of the present disclosure are broadly applicable to a variety of scenarios outside the realm of wearable spatial computing, depending on the appropriate sensors and associated data available.

[0055] It is observed that depth maps are typically merely intermediate representations that are then fused with other depth maps into a full 3D model. In contrast, the disclosed method takes a sequence of RGB images and directly predicts a full 3D model in an end-to-end trainable manner. This allows the network to fuse more information, learn better geometric initializations for the world, and produce much better reconstructions. Furthermore, it reduces the complexity of the system by eliminating steps such as frame selection, and reduces the required computations by amortizing the cost over the entire sequence.

[0056] The method of the present disclosure begins by acquiring a sequence of frames of RGB images, such as images acquired by cameras 22, 24, and 26, or other suitable cameras. Features from each of the frames are then extracted using a 2D CNN. These features are then backprojected into a 3D volume using known camera intrinsic and extrinsic properties. However, unlike previous cost volume approaches that use image warping to backproject features into a target view frustum, the present method backprojects features from each frame into a reference 3D voxel volume, where each pixel is mapped to a ray within the volume. This process avoids the need to select a target image and allows for the fusion of an entire sequence of frames into a single volume. The features in all of the frames are then each fused into a 3D voxel volume using a simple moving average. The 3D voxel volume is then passed through a 3D convolutional encoder-decoder to refine the features. Finally, the resulting 3D voxel feature volume is used to regress a TSDF value at each voxel.

[0057] The network is trained and evaluated on real scans of indoor rooms from the Scannet and RIO datasets. As shown herein, the disclosed method outperforms state-of-the-art multi-view stereo baselines by producing accurate and complete meshes. Furthermore, because the disclosed method requires launching the large 3D CNN only once at the end of the sequence, the total amount of computation required to generate a mesh for the entire scene is much lower than previous multi-view stereo methods.

[0058] As an added benefit, an additional head is added to the 3D CNN to also predict the semantic segmentation, for minimal redundant computation. The problem of 3D semantic and instance segmentation has received a lot of attention recently, but all previous methods assume that depth is obtained using a depth sensor. Although the 3D segmentation disclosed herein does not compete with the top performers on the Scannet benchmark scoreboard, the 3D segmentation establishes a strong baseline for the new task of 3D semantic segmentation from 3D reconstructions from multi-view RGB images.

[0059] With reference to FIG. 3, an outline of one embodiment of the method 110 is depicted. In step 112, the method 110 takes as input an arbitrarily long sequence of RGB images 114, each with known camera intrinsic properties and pose. In step 116, these images 114a, 114b, 114c are passed through a 2D CNN 118 backbone to extract features 120. This step 116 is depicted in equation (1) of FIG. 10. As illustrated in FIG. 10, the 2D features are then backprojected into a 3D voxel volume using known camera intrinsic and extrinsic properties, assuming a pinhole camera model. The process of FIG. 10 results in all voxels along the camera ray being filled with the same feature corresponding to that pixel (see FIGS. 4A-4C). In step 122, the features 120a, 120b, 120c are then backprojected into individual 3D voxel volumes 124. This step 122 is depicted in equation (2) of Figure 10. Figure 10 is referenced below.

number

[0060] Still referring to Figure 3, in step 125, features 120a, 120b, and 120c are accumulated using a moving average into a 3D voxel volume 126. Step 124 is depicted in equations (3) and (4) of Figure 11, which are quoted below.

number

[0061] Once the features 124 are accumulated in the 3D voxel volume 126, in step 128, the 3D voxel volume is passed through a 3D convolutional encoder-decoder network 130 to refine the features and regress the output TSDF. Each layer of the encoder and decoder uses a set of 3x3x3 residual blocks. Downsampling may be implemented using 3x3x3 stride 2 convolutions, while upsampling may utilize 1x1x1 convolutions followed by trilinear interpolation to vary the feature dimensions. The feature dimensions are doubled with each downsampling and bisected with each upsampling. All convolutional layers are then followed by a batch norm (batch normalization) function and a ReLU (rectified linear unit) function. Figure 5 illustrates an example.

[0062] Still referring to FIG. 3, once the image features 120 are fused into the 3D voxel volume 126, in step 128a, the 3D voxel volume 126 is passed through a 3D CNN 130 to refine the features and directly regress the TSDF 132 using the 3D CNN. This step 128 is depicted in FIG. 5, which shows diagrammatically the encoder-decoder architecture. The 3D CNN predicts the TSDF in a coarse-to-fine fashion, with previous layers being used to sparsify the next resolution. In step 128b, the 3D CNN may also include an additional head, which predicts a semantic segmentation of the features within the 3D voxel volume.

[0063] Additional skip connections from the encoder to the decoder may also be included to complete the geometry in unobserved regions. The encoder features are passed through a 1x1x1 convolution followed by a batch norm function and a Relu function. However, there may be voxels that were never observed during the sequence and therefore do not have any features backprojected into them. The large receptive fields of the coarser resolution layers in the network are able to smooth and fill across these areas, but adding zero values ​​from the early layers of the decoder would undo this and bring back zeros. This significantly reduces the ability of the 3D CNN to complete the geometry in unobserved regions. Therefore, for these voxels, we do not use skips from the encoder. Instead, the decoder features are passed through the same batch norm function and a Relu function to match the magnitude of the standard skip connections and add them. An exemplary skip connection is shown in equation (5) of FIG. 12, which is quoted below.

number

[0064] After the encoder-decoder, a tanh activation followed by a 1x1x1 convolution of the 3D CNN is used to regress the final TSDF value 132. In addition, an intermediate output head may be included in the 3D CNN for each resolution prior to upsampling. This is used as an intermediate teacher to help make the network training faster and to guide later resolutions to focus on refining predictions near the surface and ignore the open sky regions where the coarser resolutions are already confident. For the semantic segmentation model, an additional 1x1x1 convolution may be included to predict the segmentation logits (only at the final resolution).

[0065] Because features are backprojected along the entire ray, the voxel volume is densely packed, and therefore the method cannot take advantage of sparse convolutions in the encoder. However, by applying a hard threshold to the intermediate output TSDF, the decoder can be sparsified to allow the use of sparse convolutions similar to previous methods. In practice, we have shown that models do not require sparse convolutions and can achieve resolution of 4 cm. 3 It has been found that the feature volume can be trained at voxel resolution. The feature volume is not sparsified, but the multi-resolution output is used to sparsify the final predicted TSDF. Any voxels predicted to be beyond a fixed distance threshold are truncated at the next resolution.

[0066] Working Example:

[0067] The following describes an example use case of the method for generating a 3D reconstruction of a scene from a sequence of RGB images. Resnet50-FPN is used, followed by the merging of the 32 output feature channels as our 2D backbone with the method used in Kirilov, A., Girshick, R., He, K., Dollar, P.: Panoptic feature pyramid networks; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; pp. 6399-6408 (2019). Features are 4cm 3 The 3D CNN consists of a 4-scale resolution pyramid, where we double the number of channels each time we halve the resolution. The encoder consists of a (1,2,3,4) residual block at each scale, and the decoder consists of a (3,2,1) residual block.

[0068] We first train the network end-to-end using short sequences that cover part of the room, since all frames need to be kept in memory for backpropagation. We train with 10 frame sequences, an initial learning rate of 1e-3, and a 96x96x56 voxel grid. After 35k iterations, we freeze the 2D network and fine-tune the 3D network. This removes the need to keep all activations from the 2D CNN in memory, allows in-place accumulation of feature volumes, and makes memory independent of the number of frames. We fine-tune the network with 100 frame sequences, at a learning rate of 4e-4.

[0069] During testing, as during fine-tuning, we accumulate feature volumes in-place, allow ourselves to operate on sequences of arbitrary length (often thousands of frames, for Scannet), and use a voxel grid of size 400x400x104.

[0070] Training the network to completion takes approximately 36 hours on eight Titan RTX GPUs using a batch size of 16 and synchronized batch norm.

[0071] Ground truth preparation and loss:

[0072] With reference to Figures 6A-6E, we use the l1 losses on the ground truth TSDF values ​​to supervise the multi-scale TSDF reconstruction. Following the use of the 3D encoder-decoder in Dai, A., Qui, CR, Niebner, M.: Shape completion using 3d-encoder predictor cnns and shape synthesis (2016), we log-transform the prediction and target values ​​before applying the l1 losses, and back-propagate the losses only for voxels that were observed in the ground truth (i.e., have TSDF values ​​strictly less than 1). However, to prevent the network from hallucinating artifacts behind walls and outside the room, we also mark all voxels whose entire vertical column is equal to 1, and penalize in these areas as well. The intuition for this is that if an entire vertical column was not observed, it was probably not present in the room.

[0073] Furthermore, to learn more details in the finer resolution layers, we only calculate loss for voxels that did not exceed a certain fraction (0.97) of the cutoff distance in the previous resolution. Without this, the later layer loss would be dominated by a large number of voxels that are far from the surface and easily classified as empty, preventing it from learning effectively.

[0074] To construct the ground truth TSDF, we run TSDF fusion on the full sequence at each resolution prior to training. This results in less noise and a more complete ground truth than simply fusing short training batch sequences on the fly. However, this adds a complication where we now need to find the appropriate region of the TSDF for the training batch. We solve this in a two-step process.

[0075] During training, we use the camera frustum to crop the relevant parts of this TSDF.

[0076] We first backproject all depth points from a batch of frames to crop the relevant parts. The centroids of these points are used for the centers of the points in the reconstruction volume. We also apply a random rotation around the vertical axis for data augmentation. If we always center the visible geometry in our volume during training, the network has no chance to learn not to hallucinate geometry that is far beyond the walls (it takes advantage of the fact that the volume boundary fits the visible area). This does not let the network know how to cope when the volume is much larger during testing. Therefore, after centering, we apply a random shift along the camera's viewing direction (thus forcing the network to learn not to hallucinate geometry behind the visible geometry).

[0077] Since even perfect ground truth reconstructions are imperfect, we adopt a similar loss scheme to that disclosed in Dai, A., Diller, C., Niebner, M.; SG-nn Sparse generative neural networks for self-supervised scene completion of rgb-d scans, arXiv preprint arXiv:1912.00036 (2019) and only apply losses where the ground truth TSDF is strictly less than 1 (i.e., known empty voxels (T=-1) and near-surface (|T|<1)). However, we also mark voxels with T=1 that are outside the scene and penalize on them as well, which helps with the hallucination problem mentioned above.

[0078] We want the network to learn to complete the geometry, but asking it to perfectly hallucinate geometry that is completely outside the view is very difficult. Therefore, we further reduce to the portion of the TSDF that we penalize by clipping the visibility frustum. We build a mask from the voxels that are visible (T<1) in the batch reconstructions, and then let it extend a few voxels (allowing the network to complete the geometry slightly beyond the visibility frustum). Furthermore, any instances that are partially visible are completely contained within the mask. This mask is applied to the full TSDF used for training.

[0079] result:

[0080] Datasets and metrics:

[0081] We evaluate the example on ScanNet, which consists of 2.5M images across 707 distinct spaces. We use a standard train / validation / test split.

[0082] We evaluate our 3D reconstructions using both standard 2D depth metrics (see Fig. 9) and 3D metrics qualitatively (see Fig. 7). Fig. 9 shows that the 3D reconstruction according to the embodiment is more accurate in all respects than the 3D reconstructions generated using DPSNet and GPMVS.

[0083] Since previous work has not attempted to reconstruct a full 3D scene from multi-view images, we compare it to state-of-the-art multi-view stereo algorithms. To evaluate these in 3D, we take their outputs and fuse them into a TSDF using standard TSDF fusion.

[0084] We evaluate our semantic segmentation by transferring the predicted landmarks on our mesh onto the ground truth mesh using nearest neighbor lookup on the vertices, and then report the standard IOU benchmark defined in Dai, A., Chang, AX, Savva, M., Halber M., Funkhouser, T., Niebner, M.; Scannet: Richly-annotated 3d reconstructions of indoor scenes; Proc. Computer Vision and Pattern Recognition (CVPR), IEEE (2017), as shown in Figure 8. Figure 8 shows that our semantic segmentation according to the method disclosed herein accurately segments the 3D scene without using a depth sensor. Previous work has not attempted to perform 3D semantic segmentation from RGB images alone.

[0085] Conclusion:

[0086] In this work, we present a novel approach to 3D scene reconstruction. Notably, our approach does not require depth input, is not time-constrained, allows for the synthesis of long frame sequences, predictively completes meshes, and supports efficient computation of other quantities such as semantics. We hope that this work will open new avenues for solving 3D scene reconstruction.

[0087] Various exemplary embodiments of the present invention are described herein. These examples are referred to in a non-limiting sense. They are provided to illustrate the more broadly applicable aspects of the present invention. Various modifications may be made to the described invention, and equivalents may be substituted without departing from the true spirit and scope of the present invention. In addition, many modifications may be made to adapt a particular situation, material, composition, process, process action, or step to the objective, spirit, or scope of the present invention. Furthermore, as will be understood by those skilled in the art, each of the individual variations described and illustrated herein has discrete components and features that can be easily separated from or combined with the features of any of the other several embodiments without departing from the scope or spirit of the present invention. All such modifications are intended to be within the scope of the claims associated with the present invention.

[0088] The present invention includes methods that may be implemented using the subject devices. The method may comprise the act of providing such a suitable device. Such provision may be performed by an end user. In other words, the act of "providing" merely requires the end user to obtain, access, approach, locate, configure, activate, power on, or otherwise act to provide the device required in the subject method. The methods recited herein may occur in any order of the recited events that is logically possible, as well as the recited order of events.

[0089] Exemplary aspects of the invention have been described above, together with details regarding material selection and manufacture. As for other details of the invention, these may be understood in conjunction with the aforementioned reference patents and publications, and are generally known or may be understood by those skilled in the art. The same may be true with respect to the method-based aspects of the invention in terms of additional acts as generally or theoretically adopted.

[0090] In addition, the present invention has been described with reference to several embodiments, optionally incorporating various features, but the present invention is not limited to those described or illustrated as each variation of the invention is considered. Various modifications may be made to the invention described, and equivalents (whether recited herein or not included for purposes of some brevity) may be substituted without departing from the spirit and scope of the invention. In addition, when a range of values ​​is provided, it is to be understood that all intervening values ​​between the upper and lower limits of that range, and any other stated value or intervening values ​​within the stated range, are encompassed within the invention.

[0091] It is also contemplated that any optional features of the described inventive variations may be described and claimed independently or in combination with any one or more of the features described herein. Reference to a singular item includes the possibility that there are multiple identical items. More specifically, as used in this specification and the claims associated herewith, the singular forms "a," "an," "said," and "the" include plural references unless specifically stated otherwise. In other words, the use of articles allows for "at least one" of the items of the subject matter in the above description as well as in the claims associated herewith. Furthermore, it is noted that such claims may be drafted to exclude any optional element. Thus, the language is intended to serve as a predicate for the use of exclusive terminology such as "solely," "only," and the like in connection with the recitation of claim elements, or the use of a "negative" limitation.

[0092] Without the use of such exclusive terminology, the term "comprising" in a claim associated with the present invention shall be deemed to permit the inclusion of any additional elements, regardless of whether a given number of elements are recited in the claim, or the addition of features may be deemed to change the nature of the elements recited in such claim. Except as specifically defined herein, all technical and scientific terms used herein should be given the broadest possible commonly understood meaning while maintaining the legitimacy of the claims.

[0093] The scope of the present invention should not be limited to the examples provided and / or this specification, but rather should be limited only by the scope of the claim terms associated with this disclosure.

Claims

1. 1. A method for generating a three-dimensional (3D) reconstruction of a scene from multi-view images, the method comprising: Obtaining a sequence of frames of red-green-blue (RGB) images captured by a camera; extracting features from the sequence of frames of the RGB image using a two-dimensional convolutional neural network (2D CNN); backprojecting the features from each frame into a 3D voxel volume using known intrinsic and extrinsic camera parameters, where each pixel in the 3D voxel volume is mapped to a ray within the 3D voxel volume; Fusing features from each frame into the 3D voxel volume; passing the 3D voxel volume through a 3D convolutional neural network (3D CNN) with an encoder-decoder to refine the features within the 3D voxel volume and regressing output truncated signed distance function (TSDF) values ​​at each voxel of the 3D voxel volume; A method comprising:

2. The method of claim 1 , in which the frames are fused into a single 3D feature volume using a moving average.

3. The method of claim 2 , wherein the moving average is a simple moving average.

4. The method of claim 3 , wherein the moving average is a weighted moving average.

5. 2. The method of claim 1, further comprising: after passing the 3D voxel volume through all layers of the 3D CNN, passing the refined features in the 3D voxel volume and the TSDF values ​​at each voxel of the 3D voxel volume through a batch normalization (batch norm) function and a regularized linear unit (reLU) function.

6. An additional skip connection is included from the encoder to the decoder of the 3D CNN; 6. The method of claim 5, further comprising: skipping one or more features in the 3D voxel volume from the encoder to the decoder of the 3D CNN using the additional skip connections.

7. one or more null voxels of the 3D voxel volume do not have features backprojected into them, corresponding to voxels that were not observed during a sequence of frames of the RGB image, the method comprising: not using the additional skip connections from the encoder for the null voxels; and passing the null voxels through the batch norm function and the reLU function to match the magnitude of the voxels that receive the additional skip connections; The method of claim 6 further comprising:

8. 2. The method of claim 1, wherein the 3D CNN has multiple layers, each with a set of 3x3x3 residual blocks, and the 3D CNN implements downsampling with a 3x3x3 stride-2 convolution and upsampling using a 1x1x1 convolution after trilinear interpolation.

9. The 3D CNN further comprises an additional head for predicting semantic segmentation; The method of claim 1 , further comprising the 3D CNN predicting a semantic segmentation of the features within the 3D voxel volume.

10. The method of claim 1 , further comprising training the 2D CNN using short-frame sequences that cover a portion of a scene.

11. The method of claim 10 , wherein the short frame sequence comprises a sequence of 10 or fewer frames.

12. 12. The method of claim 11, further comprising fine-tuning the training of the 2D CNN using a larger frame sequence having more frame sequences than the short frame sequence.

13. The method of claim 12 , wherein the larger frame sequence comprises a sequence of 100 or more frames.

14. A cross reality system, comprising: a head mounted display device having a display system; a computing system in operative communication with the head mounted display device; a plurality of camera sensors in operative communication with the computing system; Equipped with the computing system is configured to generate, by a process, a three-dimensional (3D) reconstruction of a scene from a sequence of frames of RGB images captured by the camera sensor; The process comprises: acquiring from the camera sensor a sequence of frames of red-green-blue (RGB) images of a scene within a field of view of the camera sensor; extracting features from the sequence of frames of the RGB image using a two-dimensional convolutional neural network (2D CNN); backprojecting the features from each frame into a 3D voxel volume using known intrinsic and extrinsic parameters of the camera sensor, where each pixel of the 3D voxel volume is mapped to a ray within the 3D voxel volume; fusing the features from each frame into the 3D voxel volume; passing the 3D voxel volume through a 3D convolutional neural network (3D CNN) with an encoder-decoder to refine the features within the 3D voxel volume and regressing output truncated signed distance function (TSDF) values ​​at each voxel of the 3D voxel volume; Including, the system.

15. The system of claim 14 , wherein the frames are fused into a single 3D feature volume using a moving average.

16. The system of claim 15 , wherein the moving average is a simple moving average.

17. The system of claim 16 , wherein the moving average is a weighted moving average.

18. 15. The system of claim 14, wherein the process for generating a three-dimensional (3D) reconstruction of the scene from the sequence of frames of the RGB images further comprises, after passing the 3D voxel volume through all layers of the 3D convolutional encoder-decoder, passing the refined features in the 3D voxel volume and TSDF values ​​at each voxel of the 3D voxel volume through a batch normalization (batch norm) function and a regularized linear unit (reLU) function.

19. An additional skip connection is included from the encoder to the decoder of the 3D CNN, 20. The system of claim 18, wherein the process for generating a three-dimensional (3D) reconstruction of the scene from the sequence of frames of the RGB images further comprises skipping one or more features in the 3D voxel volume from the encoder to the decoder of the 3D CNN using the additive skip connections.

20. One or more null voxels of the 3D voxel volume correspond to voxels that were not observed during the sequence of frames of the RGB images, and do not have features backprojected into them, and the process for generating a three-dimensional (3D) reconstruction of the scene from the sequence of frames of the RGB images comprises: not using the additional skip connections from the encoder for the null voxels; and passing the null voxels through the batch norm function and the reLU function to match the magnitude of the voxels that receive the additional skip connections; 20. The system of claim 19, further comprising:

21. The 3D CNN further comprises an additional head for predicting semantic segmentation; 15. The system of claim 14, wherein the method further comprises the 3D CNN predicting a semantic segmentation of the features within the 3D voxel volume.

Images