System and method for end-to-end scene reconstruction from multi-view images

Abstract

A system and method for generating a three-dimensional (3D) reconstruction of a scene or environment around a user of a spatial computing system, such as a virtual reality, augmented reality, or mixed reality system, using only multi-view images including RGB images without the need for depth sensors or depth data from sensors. Features are extracted from a sequence of frames of RGB images and back projected into a 3D voxel volume using known camera intrinsic and extrinsic functions, where each pixel of the voxel volume is mapped to light in the voxel volume. The back projection features are fused into the 3D voxel volume. The 3D voxel volume is passed through a 3D convolutional neural network to refine and regress truncated symbol distance function values at each voxel of the 3D voxel volume.

Description

[0001] This application is a divisional application of the Chinese patent application with application number 202180017749.5. The application date of the original application is March 5, 2021, the priority date is March 5, 2020, and the date of entering the Chinese national phase is August 30, 2022. The name of the invention is "System and method for end-to-end scene reconstruction from multi-view images". Technical Field

[0002] The present invention relates to mobile computing systems, methods and configurations for computing, learning network configurations and connectivity, and more particularly to systems and methods for generating three-dimensional reconstructions of scenes from multi-view images that can be used in mobile computing systems, methods and configurations having at least one wearable component configured for virtual and / or augmented reality operations. Background Art

[0003] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality" ("VR"), "augmented reality" ("AR"), and / or "mixed reality" ("MR") environments or experiences (collectively, "cross-reality" environments or experiences). This can be accomplished by presenting computer-generated images to a user via a head-mounted display. The images create a sensory experience that immerses the user in a simulated environment. For example, the data may describe virtual objects that may be rendered in a manner that the user feels or perceives as part of the physical world, and the virtual objects may be interacted with. The user may experience these virtual objects as a result of the data being rendered and presented via a user interface device (such as, for example, a head-mounted display device). The data may be displayed to the user for viewing, or may control audio that is played to the user for hearing, or may control a tactile (or haptic) interface that enables the user to experience the sensation of touch that the user feels or perceives as feeling a virtual object.

[0004] XR systems can be used for many applications, spanning the fields of scientific visualization, medical training, engineering design and prototyping, remote operation and telepresence, and personal entertainment. VR systems typically involve the presentation of digital or virtual image information, without transparency to actual real-world visual input.

[0005] AR systems typically use simulated elements to supplement the real-world environment. For example, an AR system can provide a user with a view of the surrounding real-world environment via a head-mounted display. Computer-generated images can also be presented on the head-mounted display to enhance the surrounding real-world environment. The computer-generated images may include elements that are contextually relevant to the surrounding real-world environment. Such elements may include simulated text, images, objects, etc. MR systems also introduce simulated objects into the real-world environment, but these objects typically have a greater degree of interactivity than AR systems.

[0006] AR / MR scenes typically include the presentation of virtual image elements associated with real-world objects. For example, an AR / MR scene is described in which a user of AR / MR technology sees a real-world scene featuring the environment around the user, including structures, objects, etc. In addition to these features, users of AR / MR technology perceive that they "see" computer-generated features (i.e., virtual objects) even if such features do not exist in the real-world environment. Therefore, compared to VR, AR and MR include one or more virtual objects associated with real objects in the physical world. Virtual objects also interact with real-world objects, so that AR / MR systems can also be referred to as "spatial computing" systems related to the interaction of the system with the 3D world around the user. The experience of virtual objects interacting with real objects significantly increases the user's enjoyment of using the AR / MR system, and also opens the door to a variety of applications that provide real and easy-to-understand information about how to change the physical world.

[0007] The brain's visualization center obtains valuable perceptual information from the movement of the two eyes relative to each other and their components. The convergence movement of the two eyes relative to each other (i.e., the rolling movement of the pupils toward or away from each other to converge the eyes' line of sight to look at an object) is closely associated with the accommodation (or focusing) of the eye lens. Under normal circumstances, adapting the eyes, or changing the focus of the eye lens to focus on objects at different distances, automatically results in a matching change in vergence to the same distance in a relationship known as the "accommodation-convergence reflex." Similarly, under normal circumstances, a change in vergence will trigger a matching change in accommodation. As with most traditional stereoscopic VR / AR / MR configurations, activities that counter this reflex are known to produce eyestrain, headaches, or other forms of discomfort in the user.

[0008] Stereoscopic wearable glasses typically have two displays (one for the left eye and one for the right eye) that are configured to display images with slightly different element renderings so that the human visual system perceives a three-dimensional perspective. This configuration has been found to be uncomfortable for many users due to the mismatch between convergence and accommodation (the "vergence-accommodation conflict"), which must be overcome to perceive a three-dimensional image. In fact, some users cannot tolerate a stereoscopic configuration. These limitations apply to VR, AR, and MR systems. As a result, most conventional VR / AR / MR systems are not best suited to present a rich binocular three-dimensional experience in a way that is comfortable and maximally useful to the user, in part because previous systems have failed to address some fundamental aspects of the human perceptual system, including the convergence-accommodation conflict.

[0009] Various systems and methods for resolving the vergence-accommodation conflict have been disclosed. For example, U.S. Utility Patent Application Serial No. 14 / 555,585 discloses VR / AR / MR systems and methods that resolve the vergence-accommodation conflict by projecting light onto a user's eyes using one or more light-guiding optical elements so that the light and images rendered by the light appear to come from multiple depth planes. The light-guiding optical elements are designed to couple virtual light corresponding to a digital or virtual object, propagate it via total internal reflection ("TIR"), and then outcouple the virtual light to display the virtual object to the user's eyes. In an AR / MR system, the light-guiding optical elements are also designed to be transparent to light from (e.g., reflect) actual real-world objects. Thus, portions of the light-guiding optical elements are designed to reflect virtual light for propagation via TIR while being transparent to real-world light from real-world objects in the AR / MR system.

[0010] AR / MR scenes typically include interactions between virtual objects and real-world physical environments. Similarly, some VR scenes include interactions between completely virtual objects and other virtual objects. Depicting objects in a physical environment facilitates interaction with virtual objects by defining the limits and boundaries of these interactions (e.g., by defining the scope of a specific structure or object in the physical environment). For example, if an AR / MR scene includes a virtual object (e.g., a tentacle or fist) extending from a specific object in the physical environment, defining the scope of the object in three dimensions allows the AR / MR system to present a more realistic AR / MR scene. In contrast, if the scope of the object is undefined or inaccurately defined, artifacts and errors will appear in the displayed image. For example, a virtual object may appear to extend partially or completely from mid-air adjacent to the object, rather than from the surface of the object. As another example, if an AR / MR scene includes a virtual character walking on a specific horizontal surface in a physical environment, inaccurately defining the scope of the surface may cause the virtual character to appear to be walking on the surface without falling, but floating in mid-air.

[0011] Therefore, in order to provide an accurate and effective three-dimensional (3D) XR experience, the XR system needs to be able to accurately and computationally efficiently (e.g., in near real-time) generate a 3D reconstruction of the surrounding environment. In fact, reconstructing the world around us has been a long-standing goal of computer vision.

[0012] Reconstructing a 3D model of a scene usually involves taking the characteristic depth of an image sequence and fusing the depth map using a 3D data structure. The most common 3D structure for depth accumulation is the voxel volume used by TSDF fusion. However, surfels (oriented point clouds) are starting to gain popularity. These methods are usually used with physical depth sensors, but can also be applied to depth maps predicted from monocular or stereo images.

[0013] With the rise of deep learning, monocular depth estimation has improved dramatically, but its accuracy is still far below that of state-of-the-art stereo methods. A popular classical stereo method uses mutual information and semi-global matching to compute the disparity between two images. Recently, several end-to-end plane scanning algorithms have been proposed, such as DeepMVS, which uses a block matching network; and MVDepthNet, which constructs a cost volume from raw pixel measurements and performs 2D convolutions, treating planes as feature channels. GPMVS builds on this and uses Gaussian processes to aggregate information into long sequences of cost volumes. MVSNet and DPSNet use 2D CNNs to construct cost volumes from features extracted from images. They then filter the cost volume using 3D convolutions on 4D tensors. All of these methods require selecting a target image to predict depth and then finding suitable adjacent reference images.

[0014] Recent binocular stereo methods use a similar cost volume approach, but avoid frame selection by using a fixed reference stereo pair. The depth maps over the sequence are computed independently (or in some cases weakly coupled). In contrast to these methods, our method builds a single coherent 3D model directly from the input image sequence.

[0015] While TSDF fusion is simple and effective, it cannot reconstruct partially occluded geometry and requires averaging many measurements to reduce noise. Therefore, learning methods have been proposed to improve fusion. OctNetFusion aggregates multiple depth maps into a TSDF using a 3D encoder-decoder and shows results for single objects and partial scans. ScanComplete builds on this and shows results for entire rooms. SG-NN improves on ScanComplete by increasing the resolution using sparse convolutions and training with a novel self-supervised training scheme. 3D-SIC focuses on 3D instance segmentation using region proposals and adds a per-instance completion head. Route Fusion uses 2D filtering and 3D convolutions in the view cone to improve the aggregation of depth maps.

[0016] Some networks have been published that take one or more images and directly predict a 3D representation. For example, 3D-R2N2 encodes an image into a latent space and then decodes the voxel occupancy volume. Octtree-Gen improves the efficiency of 3D voxel volumes by using an octree data structure to increase resolution. Deep SDF chooses to learn a generative model that can output SDF values ​​for any input location instead of discretizing the volume. Point set generation networks have been published, which learn to generate point clouds with a fixed number of points. Pixel2Mesh++ uses a graph convolutional network to directly predict a triangular mesh. These methods encode the input into a small latent code and report results for a single object, mainly from Shapenet. Therefore, it is unclear how to extend them to full scene reconstruction. Mesh-RCNN builds on 2D object detection and adds an additional head to predict a voxel occupancy grid for each instance, and then refines it using a graph convolutional network on the grid.

[0017] Back-projecting image features into voxel volumes and then refining them using 3DCNNs has also been used for human pose estimation. These works regress a 3D heatmap for localizing joint locations.

[0018] Subsequent work on deep voxel and scene representation networks accumulates features into 3D volumes, forming an unsupervised representation of the world that can then be used to render new views without forming an explicit geometric intermediate representation.

[0019] Other methods for 3D reconstruction use monocular, binocular or multi-view stereo methods that obtain red, green and blue (RGB) images (one, two or more respectively) from an image sensor and predict a depth map of the image. For example, monocular stereo methods that generate 3D reconstructions using RGB images are disclosed in the following publications: Fu, H., Gong, M., Wang, C. Batmanhelich, K., Tao, D.: Deep ordinal regression network for monocular depth estimation; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2002-2011 (2018); Lasinger, K., Ranftl, R., Schindler, K., Koltun, V.: Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer; arXiv preprint arXiv:1907.01341 (2019); and Lee, JH, Hang, MK, Ko, DW, Suh, IH: From big to small: Multi-scale local planarguidance for monocular depth estimation; arXiv preprint arXiv:1907.10326(2019).Certain methods for generating 3D reconstructions using RGB images using binocular stereo methods are disclosed in the following publications: Chabra, R., Straub, J., Sweeney, C., Newcombe, R., Fuchs, H.: Stereodrnet: Dilatedresidual stereonet; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; pp. 11786-11795 (2019); and Chang, A. X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q. Li, Z., Savarese, S., Savva, M., Song, S., Su, H., et al.; Shapenet: An information-rich 3d model repository; arXiv preprint arXiv:1512.00312 (2015).In addition, various methods of multi-view stereo methods for generating 3D reconstruction using RGB images are disclosed in the following publications: Hirschmuller, H.: Stereo processing by semiglobal matching and mutual information; IEEE Transaction on pattern analysis and machine intelligence 30(2), 328-341(2007); Huang, PH, Matzen, K., Kopf, J., Ahuja, N., Huang, JB; Deepmvs: Learning multi-view stereopsis; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; pp.2821-2830(2018); I, S., Jeon, HG, Lin, S., Kweon, IS; DPSnet: End-to-end deep plane sweep stereo; 7th International Conference on Learning Representations, ICLR 2019; International Conference on Learning Representations, ICLR (2019); and Want, K., Shen, S.; Mvdepthnet: real-time multiview depth estimation neural network; 2018 International Conference on Computer Vision; pp. 2088-2096 (2017). However, despite a plethora of recent research, these methods that generate 3D reconstructions using only RGB images are still far less accurate than depth sensors and do not produce satisfactory results when fused into 3D models.

[0020] In addition to reconstructing geometry, many XR applications require 3D semantic labeling of 3D reconstructions (i.e., 3D semantic segmentation) to provide a richer representation. In general, there are two approaches to this problem: 1) predicting the semantics of 2D input images using a 2D segmentation network and back-projecting the labels to 3D; and 2) directly predicting semantic labels in 3D space. All of these methods assume that depth is provided by a depth sensor. A notable exception is Kimera, which uses multi-view stereo to predict depth, however, the results are only shown on synthetic data and real data 2D segmentation.

[0021] SGPN formulates instance segmentation as a 3D point cloud clustering problem. A similarity matrix is ​​predicted and the 3D point cloud is clustered to derive semantic and instance labels. 3D-SIS improves on these methods by fusing 2D features in a 3D representation. RGB images are encoded using a 2D CNN and back-projected onto 3D geometry reconstructed from a depth map. The 3D CNN is then used to predict 3D object bounding boxes and semantic labels. SSCN predicts semantics on high-resolution voxel volumes enabled by sparse convolutions.

[0022] Therefore, there is a need for more efficient and accurate methods and systems for generating 3D reconstructions of scenes in an end-to-end manner using RGB images and without the need for a depth sensor. In addition, it is necessary that such 3D reconstructions can be used to present virtual image information on an XR display in multiple focal planes (e.g., two or more) so as to be applicable to a variety of use cases without exceeding an acceptable range of convergence-accommodation mismatch. In addition, the XR system also needs to implement a display that is lightweight, low cost, small in size, has a wide virtual image field of view, and is as transparent as possible. In addition, there is a need for 3D semantic segmentation of the 3D reconstruction without the use of a depth sensor. Summary of the invention

[0023] Embodiments disclosed herein are directed to systems and methods for generating a three-dimensional (3D) reconstruction of a scene or environment surrounding a user of a spatial computing system (such as an XR system) using only multi-view images including RGB images, and without requiring a depth sensor or depth data from a sensor. For example, a spatial computing system can utilize 3D reconstruction to provide an accurate and efficient 3D XR experience. The resulting 3D XR experience can be displayed in a rich binocular 3D experience that is comfortable and maximally useful to the user, in part because it can present images in a way that addresses some fundamental aspects of the human perceptual system, such as convergence-accommodation mismatch. For example, 3D reconstruction with accurate depth data enables 3D images to be displayed in multiple focal planes. 3D reconstruction also enables accurate management of interactions between virtual objects, other virtual objects, and / or real-world objects.

[0024] Therefore, one embodiment of the present invention relates to a method for generating a three-dimensional (3D) reconstruction of a scene using multi-view images. First, a frame sequence of red, green and blue (RGB) images is obtained. This can be done by one or more suitable cameras. Features are then extracted from the frame sequence of RGB images using a two-dimensional convolutional neural network (2D CNN). The features extracted from each frame are then back-projected into a 3D voxel volume using known camera intrinsic and extrinsic functions, where each pixel of the voxel volume is mapped to a ray in the voxel volume. Compared to previous methods of generating 3D reconstructions using monocular, binocular and multi-view images that require selecting a target image to predict depth and then finding suitable adjacent reference images, this process avoids the need to select a target image and allows the entire frame sequence to be fused into a single volume.

[0025] The back-projected features from each frame are accumulated (i.e., fused) into a 3D voxel volume. The 3D voxel volume is passed through a 3D convolutional neural network (3D CNN) with an encoder-decoder to refine the features in the 3D voxel volume. The 3D CNN also regresses the value of the output truncated signed distance function (TSDF) at each voxel of the 3D voxel volume.

[0026] In another aspect, the frames may be fused into a single 3D feature volume using a running average, such as a simple running average or a weighted running average.

[0027] On the other hand, the method may further include, after passing the 3D voxel volume through all layers of the 3D CNN, passing the refined features of the 3D voxel volume and the TSDF value at each voxel of the 3D voxel volume through a batch normalization (batchnorm) function and a rectified linear unit (reLU) function.

[0028] In another aspect of the method, the 3D CNN may include an additive skip connection from an encoder to a decoder of the 3D CNN. The method then uses the additive skip connection to jump one or more features of the 3D voxel volume from the encoder to the decoder of the 3D CNN.

[0029] In another aspect, the method can be configured to process empty voxels in the 3D voxel volume that do not have features back-projected therein. For example, an empty voxel may correspond to a voxel that is not observed during a frame sequence of an RGB image. In this case, the method further includes not using an additive skip connection from the encoder for the empty voxel, and passing the empty voxel through a batchnorm function and a reLU function to match the amplitude of the voxel undergoing the skip connection.

[0030] In another aspect of the method, the 3D CNN can have multiple layers, each layer has a set of 3×3×3 residual blocks, the 3D CNN can use 3×3×3 stride 2 convolutions to achieve downsampling, and upsampling can use trilinear interpolation followed by 1×1×1 convolutions.

[0031] In another aspect of the method, the 3D CNN further includes an additional head for predicting semantic segmentation. The method further includes the 3D CNN predicting semantic segmentation of features in the 3D voxel volume.

[0032] In another aspect, the method further includes training the 2D CNN using a short frame sequence covering a portion of the scene. The short frame sequence may include ten or fewer frame sequences. In another aspect, the training of the 2D CNN may be fine-tuned using a larger frame sequence having more frame sequences than the short frame sequence. For example, the larger frame sequence may include 100 or more frame sequences.

[0033] Another disclosed embodiment relates to a cross-reality system that is configured to generate a 3D reconstruction of a scene or environment around a user of the cross-reality system using only multi-view images including RGB images without a depth sensor or depth data from the sensor. The cross-reality system includes a head-mounted display device having a display system. For example, the head-mounted display may have a near-eye display with a pair of glasses-like structures. The computing system is in operative communication with the head-mounted display. Multiple camera sensors are in operative communication with the computing system. For example, the camera sensor may be mounted on the head-mounted display, or on any other suitable structure. The computing system is configured to generate a three-dimensional (3D) reconstruction of the scene from a frame sequence of RGB images through a process including any configuration of the above method. In additional aspects of the cross-reality system, the process may include any one or more of the additional aspects of the above method. For example, the process may include obtaining a frame sequence of red, green, and blue (RGB) images of a scene within the field of view of the camera sensor from a camera sensor. Features are extracted from the frame sequence of RGB images using a two-dimensional convolutional neural network (2D CNN). The features from each frame are back-projected into a 3D voxel volume using known camera intrinsics and extrinsics, where each pixel of the voxel volume is mapped to a ray in the voxel volume. The features from each frame are fused into the 3D voxel volume. The 3D voxel volume is passed through a 3D convolutional neural network (3DCNN) with an encoder-decoder to refine the features of the 3D voxel volume and regress the value of the output truncated signed distance function (TSDF) at each voxel of the 3D voxel volume.

[0034] The cross-reality system can then utilize the 3D reconstruction to generate a 3D XR experience displayed in a rich binocular three-dimensional experience, such as displaying 3D images in multiple focal planes on a head-mounted display. The cross-reality system can also utilize the 3D reconstruction to manage interactions between virtual objects, other virtual objects, and / or real-world objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings illustrate the design and practicality of preferred embodiments of the present disclosure, wherein similar elements are represented by common reference numerals. In order to better understand how to obtain the above and other advantages and purposes of the present disclosure, the present disclosure briefly described above will be described in more detail with reference to the specific embodiments shown in the accompanying drawings. Understanding that these drawings depict only typical embodiments of the present disclosure and therefore should not be considered as limiting the scope thereof, the present disclosure will be described and explained with additional specificity and detail through the use of the accompanying drawings.

[0036] Figure 1 is a schematic diagram of an exemplary cross-reality system for providing a cross-reality experience, according to one embodiment.

[0037] Figure 2 Shown is a comparison of an evaluation of Scannet performance of 3D reconstruction performed according to an embodiment of the method disclosed herein compared to a baseline method utilizing deep multi-view stereo followed by traditional TSDF fusion.

[0038] Figure 3 is a flow chart illustrating a method of generating a 3D reconstruction of a scene using an RGB image according to one embodiment.

[0039] Figures 4A-4C A diagram showing back-projection of features into a 3D voxel volume according to one embodiment ( Figure 4A ), Figure 2. Features of using naive skip connections in 3DCNN ( Figure 4B ) and features using masked skip connections to reduce artifacts and better complete the geometry of unobserved regions ( Figure 4C ) comparison.

[0040] Figure 5 is a schematic diagram of a 3D encoder-decoder architecture according to one embodiment.

[0041] Figures 6A-6E is an image sequence showing the preparation of real data for training the CNN in the examples described here.

[0042] Figure 7 is a sequence of images showing a comparison of a 3D reconstruction generated in the example according to the method disclosed herein with a 3D reconstruction generated using DPSNet and real data.

[0043] Figure 8 is an image sequence that compares the qualitative 3D semantic segmentation labels generated in the example according to the method disclosed herein with the labels transferred to the real data and the real data labels.

[0044] Fig. 9 is a table comparing the 3D reconstruction generated in the described example according to the methods disclosed herein with the 3D reconstructions according to DPSNet and GPMVS using standard 2D depth metrics and 3D metrics.

[0045] Fig.10 A process for passing an image sequence through a 2D CNN backbone to extract features and back-project the features into a 3D voxel volume is described according to one embodiment.

[0046] Fig.11 A process for accumulating feature volumes over an entire sequence of images using a weighted running average is described according to one embodiment.

[0047] Fig.12 An equation representing an exemplary skip connection according to one embodiment is set forth. DETAILED DESCRIPTION

[0048] Various embodiments of systems and methods for generating a three-dimensional (3D) reconstruction of a scene or environment around a user of a spatial computing system (such as an XR system) with end-to-end reconstruction are described below, using multi-view RGB images without using depth or distance sensors. Various embodiments are described in detail with reference to the accompanying drawings, which are provided as illustrative examples of the present disclosure to enable those skilled in the art to practice the present disclosure. It is worth noting that the following figures and examples are not meant to limit the scope of the present disclosure. In the case where certain elements of the present disclosure can be partially or fully implemented using known components (or methods or processes), only those parts of such known components (or methods or processes) necessary for understanding the present disclosure will be described, and detailed descriptions of other parts of such known components (or methods or processes) will be omitted so as not to confuse the present disclosure. In addition, various embodiments cover current and future known equivalents of the components mentioned herein by way of illustration.

[0049] In addition, systems and methods for generating three-dimensional (3D) reconstructions of a scene or environment surrounding a user of a spatial computing system may also be implemented independently of an XR system, and the embodiments described herein are described with respect to AR / MR systems for illustrative purposes only.

[0050] refer to Figure 1, shows an exemplary XR system 100 according to one embodiment. The XR system 100 includes a head-mounted display device 2 (also referred to as a head-mounted viewing component 2), a handheld controller 4 (also referred to as a handheld controller component 4), and an interconnected auxiliary computing system or controller 6 (also referred to as an interconnected auxiliary computing system or controller component 6), which can be configured to be worn on a user as a belt pack or the like. Each of these components is in operative communication (i.e., operatively coupled) with each other and with other connected resources 8 (such as cloud computing or cloud storage resources) via wired or wireless communication connections 10, 12, 14, 16, 17, 18 (such as those specified by IEEE 802.11, Bluetooth (RTM), and other connection standards and configurations). The head-mounted display device includes two depicted optical elements 20 through which a user can see the world around them and video images; and visual components produced by associated system components, including a pair of image sources (e.g., microdisplay panels) and viewing optics for displaying computer-generated images on the optical elements 20 for an augmented reality experience. As Figure 1 As shown in , the XR system 100 also includes various sensors that are configured to provide information about the user's surroundings, including but not limited to various camera type sensors 22, 24, 26 (such as monochrome, color / RGB, and / or thermal sensors), a depth camera sensor 28 and / or a sound sensor 30 (such as a microphone).

[0051] In various embodiments, the user wears a Figure 1 The augmented reality system of the XR system 100 depicted in FIG. 1 , which may also be referred to as a “spatial computing” system, is associated with such a system interacting with the three-dimensional world around a user when in operation. The cameras 22, 24, 26 are configured to map the environment around the user, and / or create a “mesh” of such an environment that includes various points representing the geometry of various objects within the environment around the user (such as walls, floors, chairs, etc.). The spatial computing system can be configured to map or mesh the environment around the user and run or operate software, such as software available from Magic Leap, Inc. of Plantation, Florida, which can be configured to utilize a map or mesh of a room to assist the user in placing, manipulating, visualizing, creating, and modifying various objects and elements in the three-dimensional space around the user. As Figure 1 As shown in , the XR system 100 can also be operably coupled to additional connected resources 8, such as other computing systems, through a cloud or other connection configuration.

[0052] It should be understood that the methods, systems, and configurations described herein may be broadly applicable to a variety of scenarios outside the field of wearable spatial computing (such as XR system 100), provided that appropriate sensors and associated data are available.

[0053] One of the challenges in spatial computing involves utilizing the various operatively coupled sensors (such as Figure 1 The invention relates to a method and system for generating a 3D reconstruction of a scene, such as a 3D environment around a user of XR system 100, using only RGB images, such as those from cameras 22, 24, and 26, without using depth data from depth sensor 28.

[0054] Compared to previous methods of generating 3D reconstructions using only the above-mentioned RGB images, which produced relatively inaccurate depth and relatively unsatisfactory 3D image models, the methods and systems disclosed herein produce accurate full 3D models and also support efficient computation of other reconstruction data (including semantic segmentation).

[0055] In general, a method is disclosed for directly regressing a truncated distance function ("TSDF") for a set of posed RGB images. A two-dimensional (2D) CNN (convolutional neural network) is configured to extract features from each image independently. These features are back-projected and accumulated into a voxel volume using camera intrinsic and extrinsic functions (features for each pixel are placed along the entire ray). After accumulation, the voxel volume is passed through a three-dimensional (3D) CNN, which is configured to refine the features and predict TSDF values. Additional heads can be added to predict color, semantic, and instance labels with minimal additional computational resources. As explained in more detail herein, the method was evaluated on Scannet, and it was determined that this method significantly outperformed state-of-the-art baselines (deep multi-view stereo followed by traditional TSDF fusion) both quantitatively and qualitatively, such as Figure 2 The resulting 3D semantic segmentation is compared to previous methods using depth sensors, as previous work has not attempted to solve the problem using only RGB input. The currently disclosed methods and configurations can be widely applied to various scenarios outside the field of wearable spatial computing, provided that appropriate sensors and associated data are available.

[0056] It has been observed that depth maps are often only intermediate representations, which are then fused with other depth maps into a full 3D model. In contrast, currently published methods take a sequence of RGB images and directly predict a full 3D model in an end-to-end trainable manner. This allows the network to fuse more information and learn better geometric priors about the world, resulting in better reconstructions. Furthermore, it reduces the complexity of the system by eliminating steps like frame selection and reduces the required computation by amortizing the cost over the entire sequence.

[0057] The currently disclosed method begins by obtaining a sequence of frames of RGB images (such as images obtained by cameras 22, 24 and 26 or other suitable cameras). Then, features of each frame in the frame are extracted using a 2D CNN. These features are then back-projected into a 3D volume using known camera intrinsic and extrinsic functions. However, unlike previous cost volume methods that use image warping to back-project features into a target frustum, the present method back-projects features from each frame into a canonical 3D voxel volume, where each pixel is mapped to a ray in the volume. This process avoids the need to select a target image and allows the entire sequence of frames to be fused into a single volume. Then, each of the features in all frames is fused into the 3D voxel volume using a simple running average. The 3D voxel volume is then passed through a 3D convolutional encoder-decoder to refine the features. Finally, the resulting 3D voxel feature volume is used to regress the TSDF value at each voxel.

[0058] The network is trained and evaluated on real scans of indoor rooms from the Scannet and RIO datasets. As shown here, the current disclosed method significantly outperforms the state-of-the-art multi-view stereo benchmark by producing accurate and complete meshes. Furthermore, since the current disclosed method only requires running a large 3D CNN at the end of the sequence, the total computation required to generate a mesh for the entire scene is much lower than previous multi-view stereo methods.

[0059] As an added bonus, for minimal additional computation, an additional head is added to the 3D CNN to also predict semantic segmentation. While the problem of 3D semantic and instance segmentation has received a lot of attention recently, all previous approaches assume that depth is obtained using a depth sensor. While the 3D segmentation presented here is not competitive with the top performers on the Scannet benchmark leaderboard, the 3D segmentation establishes a strong benchmark for the new task of 3D semantic segmentation from 3D reconstruction of multi-view RGB images.

[0060] refer to Figure 3, depicts a schematic diagram of one embodiment of the present method 110. At step 112, the method 110 takes as input a sequence 114 of RGB images of arbitrary length, each image having known camera intrinsics and pose. At step 116, these images 114a, 114b, 114c are passed through a 2D CNN 118 backbone to extract features 120. This step 116 is Fig.10 The equation (1) is as follows. Fig.10 As shown in , a pinhole camera model is assumed and the 2D features are then back-projected into a 3D voxel volume using known camera intrinsics and extrinsics. Fig.10 The process results in all voxels along the camera ray being filled with the same feature corresponding to that pixel (see Figures 4A-4C ). At step 122 , the features 120 a , 120 b , 120 c are then back-projected into corresponding 3D voxel volumes 124 . Fig.10 This step 122 is depicted in equation (2) of Fig.10 .

[0061] make is a sequence of T RGB images. We use standard 2D CNN to extract features where c is the feature dimension. Assuming a pinhole camera model, these 2D features are then back-projected into a 3D voxel volume using known camera intrinsics and extrinsics. Considering the voxel volume

[0062]

[0063] Where P t and K t are the extrinsic and intrinsic matrices of image t, Π is the perspective map, and : is the slice operator. Here (i, j, k) are the voxel coordinates in world space, and are pixel coordinates in image space. Note that this means filling all voxels along the camera ray.

[0064] Still reference Figure 3 At step 125, features 120a, 120b, and 120c are accumulated into a 3D voxel volume 126 using a running average. Fig.11 is depicted in equations (3) and (4), cited below.

[0065] These feature volumes are accumulated over the entire sequence using a weighted running average similar to TSDF fusion.

[0066]

[0067] For the weights, we use a binary mask Wt (i,j,k)∈{0,1}, which stores whether the voxel (i,j,k) is inside or outside the camera’s frustum.

[0068] Feature volumes are accumulated over the entire sequence using a weighted running average similar to TSDF fusion. For the weights, a binary mask is used to store whether the voxel is inside or outside the camera's frustum.

[0069] Once the features 124 are accumulated into the 3D voxel volume 126, at step 128, the 3D voxel volume passes through a 3D convolutional encoder-decoder network 130 to refine the features and regress the output TSDF. Each layer of the encoder and decoder uses a set of 3×3×3 residual blocks. Downsampling can be achieved using 3×3×3 convolutions with a stride of 2, while upsampling can utilize trilinear interpolation followed by 1×1×1 convolutions to change the feature dimension. The feature dimension doubles with each downsampling, and the feature dimension is halved with each upsampling. All convolutional layers are followed by a Batchnorm (batch normalization) function and a ReLU (rectified linear unit) function. Figure 5 An example is shown.

[0070] Still reference Figure 3 Once the image features 120 have been fused into the 3D voxel volume 126 , at step 128 a , the 3D voxel volume 126 is passed through a 3D CNN 130 to refine the features and directly regress the TSDF 132 using the 3D CNN. Figure 5 This step 128 is depicted in FIG, which schematically illustrates an encoder-decoder architecture. The 3D CNN predicts the TSDF in a coarse-to-fine manner, with the previous layer being used to sparse the next resolution. At step 128b, the 3D CNN may also include an additional head that predicts the semantic segmentation of features in the 3D voxel volume.

[0071] Additive skip connections from the encoder to the decoder may also be included in order to complete geometry in unobserved regions. The encoder features pass through a 1×1×1 convolution followed by a batchnorm function and a relu function. However, there may be voxels that have never been observed during the sequence and therefore do not have any features backprojected into them. The large receptive fields of the coarser resolution layers in the network are able to smooth and fill in these areas, but adding zero values ​​from the early layers of the decoder undoes this effect and brings the zeros back. This significantly reduces the ability of the 3D CNN to complete geometry in unobserved regions. Therefore, for these voxels, no skips from the encoder are used. Instead, the decoder features pass through the same batchnorm function and relu function to match the magnitude of the standard skip connection and add them. Fig.12 An exemplary skip connection is shown in equation (5) of , cited below.

[0072]

[0073] Where: x is the feature from the decoder

[0074] y is the feature jumped from the encoder

[0075] f is the convolution

[0076] g is batchnorm and relu function

[0077] Figures 4A-4C The use of these masked skip connections is shown to complete the geometry of the unobserved regions. Figure 4A The back projection of the feature into a 3D voxel volume is shown, and the unobserved region within the coverage rectangle is shown. Figure 4B Showing how naive skip connections in 3D CNNs can lead to significant artifacts. Figure 4C Showing how masked skip connections reduce artifacts and allow 3D CNNs to better complete the geometry of unobserved regions.

[0078] After the encoder decoder, a 1×1×1 convolution of the 3D CNN followed by a tanh activation is used to regress the final TSDF value 132. In addition, before upsampling, for each resolution, an intermediate output head can be included in the 3D CNN. This is used as intermediate supervision to help the network train faster, as well as to guide later resolutions to focus on refining predictions near the surface and ignore large empty regions that are already credible at coarser resolutions. For semantic segmentation models, additional 1×1×1 convolutions can be included to predict the segmentation logit (only at the final resolution).

[0079] Since the features are back-projected along the entire ray, the voxel volume is densely packed, and therefore this method cannot take advantage of sparse convolutions in the encoder. However, by applying a hard threshold to the intermediate output TSDF, the decoder can be sparsified, allowing the use of sparse convolutions similar to previous methods. In practice, it was found that the model can be trained at 4cm3 voxel resolution without the need for sparse convolutions. Although the feature volume is not sparsified, the multi-resolution output is used to sparsify the final predicted TSDF. Any voxel whose prediction is beyond a fixed distance threshold is truncated at the following resolution.

[0080] Example:

[0081] An example use case of our method for generating 3D reconstructions of a scene from a sequence of RGB images is described below. A Resnet50-FPN was used, after incorporating the method used in Kirilov, A., Girshick, R., He, K., Dollar, P.: Panoptic feature pyramid networks; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; pp. 6399-6408 (2019), with 32 output feature channels as our 2D backbone. The features are back-projected into a 4 cm3 voxel grid. Our 3D CNN consists of a four-scale resolution pyramid, with the number of channels doubling each time the resolution is halved. The encoder consists of (1, 2, 3, 4) residual blocks at each scale, and the decoder consists of (3, 2, 1) residual blocks.

[0082] Initially, we train the network end-to-end using short sequences covering parts of the room, since all frames need to be kept in memory for backpropagation. We train with sequences of 10 frames, an initial learning rate of 1e-3, and a 96×96×56 voxel grid. After 35k iterations, we freeze the 2D network and fine-tune the 3D network. This eliminates the need to keep all activations of the 2D CNN in memory and allows accumulating feature volumes in place, breaking the memory dependency on the number of frames. We fine-tune the network with sequences of 100 frames at a learning rate of 4e-4.

[0083] At test time, similar to during fine-tuning, we accumulate feature volumes in place, allowing us to operate on sequences of arbitrary length (typically thousands of frames for Scannet), and we use a voxel grid of size 400×400×104.

[0084] With a batch size of 16 and synchronized batchnorm, training the network takes about 36 hours to complete on 8 Titan RTX GPUs.

[0085] Real data preparation and loss:

[0086] refer to Figures 6A-6E, we supervise the multi-scale TSDF reconstruction using an l1 loss to the true data TSDF values. Using the 3D encoder-decoder from Dai, A., Qui, CR, Niebner, M. Shape completion using 3d-encoder predictor cnns and shape synthesis (2016), we log-transform the predicted and target values ​​before applying the l1 loss, and backpropagate the loss only on voxels that were observed in the true data (i.e., TSDF values ​​strictly less than 1). However, to prevent the network from hallucinating artifacts behind walls and outside the room, we also mark all voxels where the entire vertical column is equal to 1 and penalize these areas as well. The intuition for this is that if the entire vertical column is not observed, it is probably not in the room.

[0087] Additionally, to force finer resolution layers to learn more details, we only compute the loss for voxels that are no closer than a small fraction (0.97) of the cutoff distance in the previous resolution. Without this setting, the later layer loss would be dominated by a large number of voxels that are far away from the surface and easily classified as empty, preventing it from learning effectively.

[0088] To construct the TSDF for real data, we run TSDF fusion on the full sequence at each resolution before training. This produces less noise and more complete ground truth than simply fusing short training batches of sequences on the fly. However, this adds complexity, as we now have to find the right TSDF region for the training batch. We solve this problem in a two-step process.

[0089] During training, we crop the relevant parts of this TSDF using the camera frustum.

[0090] To crop the relevant parts, we first back-project all the depth points from this batch of frames. The centroid of these points is used to center the points in the reconstructed volume. We also apply random rotations around the vertical axis for data augmentation. If we always center the visible geometry in our volume at training time, the network has no chance to learn to not hallucinate geometry far away from the walls (the network exploits the fact that the boundary of the volume fits into the visible area). This causes the network to not know what to do when the volume is much larger at test time. Therefore, after centering, we apply random shifts along the viewing direction of the camera (so the network is forced to learn to not hallucinate geometry behind the visible geometry).

[0091] Because even the complete ground-truth reconstruction is incomplete, we adopt a similar loss scheme to that published in Dai, A., Diller, C., Niebner, M., SG-nn Sparse generative neural networks for self-supervised scene completion of RGB-D scans, arXiv preprint arXiv:1912.00036 (2019), and only apply the loss when the ground-truth TSDF is strictly less than 1 (i.e., known empty voxels (T=-1) and voxels near the surface (|T|<1). However, we also mark and penalize voxels outside the scene with T=1 to help address the hallucination problem mentioned above.

[0092] We want the network to learn to complete geometry, but asking it to completely hallucinate geometry that is completely out of view is too difficult. Therefore, we further reduce the part of the TSDF that we penalize by cropping the visible frustum. We construct a mask from the voxels that are visible in the batch reconstruction (T<1), and then dilate it by a few voxels (forcing the network to complete geometry that is slightly beyond the visible frustum). Furthermore, any partially visible instances are fully included in the mask. This mask is applied to the full TSDF for training.

[0093] result:

[0094] Datasets and metrics:

[0095] We evaluate examples on ScanNet, which consists of 2,500,000 images spanning 707 different spaces. We use the standard train / validation / test split.

[0096] We use both the standard 2D depth metric and the 3D metric (see Fig. 9 ) and qualitatively (see Figure 7 ) to evaluate our 3D reconstruction. Fig. 9 It is shown that the 3D reconstruction according to the example is more accurate in various aspects than the 3D reconstructions generated using DPSNet and GPMVS.

[0097] Since there is no prior work that attempts to reconstruct a complete 3D scene from multi-view images, we compare to state-of-the-art multi-view stereo algorithms. To evaluate these in 3D, we obtain their outputs and fuse them into a TSDF using standard TSDF fusion.

[0098] We evaluate our semantic segmentation by transferring the labels predicted on our grid to the real data grid using nearest neighbor lookup on vertices, and then report the standard IOU benchmark defined in Dai, A., Chang, AX, Savva, M., Halber M., Funkhouser, T., Niebner, M.; Scannet: Richly-annotated 3dreconstructions of indoor scenes; Proc. Computer Vision and Pattern Recognition (CVPR), IEEE (2017), as Figure 8 as shown in . Figure 8 We show that our semantic segmentation according to the method disclosed herein accurately segments 3D scenes without using a depth sensor. No prior work has attempted 3D semantic segmentation from RGB images alone.

[0099] in conclusion:

[0100] In this work, we propose a new approach to 3D scene reconstruction. Notably, our approach does not require depth input; is unbounded in time, allowing integration of long sequences of frames; completes meshes predictively; and supports efficient computation of other quantities such as semantics. We hope that this work opens up another avenue for solving 3D scene reconstruction.

[0101] Various exemplary embodiments of the present invention are described herein. Reference is made to these examples in a non-restrictive manner. They are provided to illustrate aspects to which the present invention is more widely applicable. Various changes may be made to the described invention and equivalents may be substituted without departing from the true spirit and scope of the present invention. In addition, many modifications may be made to adapt specific circumstances, materials, compositions of matter, processes, process actions or steps to the goals, spirit or scope of the present invention. In addition, as will be appreciated by those skilled in the art, each individual variant described and illustrated herein has discrete components and features, which may be easily separated from or combined with the features of several other embodiments without departing from the scope or spirit of the present invention. All of these modifications are intended to fall within the scope of the claims associated with the present disclosure.

[0102] The present invention includes methods that can be performed using subject equipment. The method can include the action of providing such suitable equipment. This provision can be performed by an end user. In other words, the "providing" action only requires the end user to obtain, access, approach, locate, set, activate, power on or otherwise act to provide the necessary equipment in the subject method. The methods listed here can be performed in any order of the listed events that is logically possible and in the listed order of events.

[0103] The exemplary aspects of the present invention and the details about material selection and manufacturing have been described above. As for other details of the present invention, these can be understood in conjunction with the patents and publications cited above and the contents generally known or understood by those skilled in the art. For the method-based aspects of the present invention, the same is true in terms of additional actions that are usually or logically adopted.

[0104] In addition, although the present invention has been described with reference to several examples, which optionally incorporate various features, the present invention is not limited to what is described or indicated with respect to the consideration of each variant of the present invention. Various changes may be made to the described invention and equivalents (whether listed herein or not included for the sake of brevity) may be substituted without departing from the true spirit and scope of the present invention. In addition, where a numerical range is provided, it should be understood that each intermediate value between the upper and lower limits of the range and any other specified or intermediate values ​​within the specified range is included in the present invention.

[0105] In addition, it is contemplated that any optional features of the described variants of the invention may be independently set forth and claimed, or combined with any one or more features described herein. References to singular items include the possibility of multiple identical items. More specifically, as used herein and in the claims associated therewith, the singular forms "one", "an", "said", and "the" include plural indicators unless otherwise expressly stated. In other words, the use of the article allows for "at least one" subject item in the above description and claims associated with the present disclosure. It should also be noted that such claims may be drafted to exclude any optional elements. Therefore, this statement is intended to serve as a prior basis for the use of exclusive terms such as "only", "only", etc. when combining reference claim elements or using "negative" limitations.

[0106] In the absence of such exclusive terms, the term "comprising" in the claims associated with the present disclosure shall allow for the inclusion of any additional elements - regardless of whether a given number of elements are recited in such claim or the addition of a feature may be considered to change the nature of the elements recited in such claim. Unless otherwise specifically defined herein, all technical and scientific terms used herein shall be given as broad a meaning as commonly understood while maintaining claim validity.

[0107] The breadth of the present invention is not limited to the examples provided and / or subject specification, but is limited only by the scope of the claim language associated with the present disclosure.

Claims

1. A method for generating a three-dimensional (3D) reconstruction of a scene from multi-view images, the method include: Obtain a frame sequence of a red, green, and blue (RGB) image; Extracting features from the frame sequence of RGB images using a two-dimensional convolutional neural network (2D CNN); back-projecting the features from each frame into a 3D voxel volume using known camera intrinsics and extrinsics, wherein each pixel of the voxel volume is mapped to a ray in the voxel volume; and The 3D reconstruction is generated without using depth data from a depth sensor.

2. The method according to claim 1, further comprising: include: Features from each frame are fused / accumulated into the 3D voxel volume.

3. The method according to claim 2, in, The frames are fused into a single 3D feature volume using a running average.

4. The method according to claim 3, in, The running average is a simple running average.

5. The method according to claim 4, in, The running average is a weighted running average.

6. A cross-reality system, include: A head mounted display device having a display system; a computing system in operable communication with the head mounted display; a plurality of camera sensors in operable communication with the computing system; wherein the computing system is configured to generate a three-dimensional (3D) reconstruction of the scene from a sequence of frames of RGB images captured by the camera sensor by a process comprising: obtaining from the camera sensor a sequence of frames of red, green, and blue (RGB) images of a scene within a field of view of the camera sensor; Extracting features from the frame sequence of RGB images using a two-dimensional convolutional neural network (2D CNN); back-projecting the features from each frame into a 3D voxel volume using known camera intrinsics and extrinsics, wherein each pixel of the voxel volume is mapped to a ray in the voxel volume; and The 3D reconstruction is generated without using depth data from a depth sensor.

7. A method for generating a three-dimensional (3D) reconstruction of a scene from multi-view images, the method include: Obtaining a frame sequence of an image; extracting features from the sequence of frames of the image using a two-dimensional convolutional neural network (2D CNN); back-projecting the features from each frame into a 3D voxel volume, wherein each pixel of the 3D voxel volume is mapped to a ray in the 3D voxel volume; Passing the 3D voxel volume through a 3D convolutional neural network (3DCNN) with an encoder-decoder to refine the features in the 3D voxel volume and regressing a truncated signed distance function (TSDF) value at each voxel of the 3D voxel volume; and After the 3D voxel volume passes through all layers of the 3D CNN, the refined features in the 3D voxel volume and the TSDF value at each voxel in the 3D voxel volume pass through a batch normalization (batchnorm) function and a rectified linear unit (reLU) function, The 3D reconstruction is generated without using depth data from a depth sensor.

8. A cross-reality system, include: A head mounted display device having a display system; a computing system in operable communication with the head mounted display device; a plurality of camera sensors in operable communication with the computing system; wherein the computing system is configured to generate a three-dimensional (3D) reconstruction of a scene from a sequence of frames of images captured by the camera sensor by a process comprising: obtaining a sequence of frames of images of the scene from the camera sensor, wherein the sequence of frames of images are within a field of view of the camera sensor; extracting features from the sequence of frames of the image using a two-dimensional convolutional neural network (2D CNN); back-projecting the features from each frame into a 3D voxel volume, wherein each pixel of the 3D voxel volume is mapped to a ray in the 3D voxel volume; Passing the 3D voxel volume through a 3D convolutional neural network (3D CNN) with an encoder-decoder to refine the features in the 3D voxel volume and regressing a truncated signed distance function (TSDF) value at each voxel of the 3D voxel volume; and After passing the 3D voxel volume through all layers of the 3D convolutional encoder-decoder, the refined features in the 3D voxel volume and the TSDF value at each voxel in the 3D voxel volume are passed through a batch normalization (batchnorm) function and a rectified linear unit (reLU) function, The 3D reconstruction is generated without using depth data from a depth sensor.

9. A method for generating a three-dimensional (3D) reconstruction of a scene from multi-view images, the method include: back-projecting features from each frame of the sequence of frames into a 3D voxel volume, wherein each pixel of the 3D voxel volume is mapped to a ray in the 3D voxel volume; Passing the 3D voxel volume through a 3D convolutional neural network (3DCNN) with an encoder-decoder to refine the features in the 3D voxel volume and regressing a truncated signed distance function (TSDF) value at each voxel of the 3D voxel volume; and After the 3D voxel volume passes through all layers of the 3D CNN, the refined features in the 3D voxel volume and the TSDF value at each voxel in the 3D voxel volume pass through a batch normalization (batchnorm) function and a rectified linear unit (reLU) function, The 3D reconstruction is generated without using depth data from a depth sensor.

10. A cross-reality system, include: A head mounted display device having a display system; a computing system in operable communication with the head mounted display device; a plurality of camera sensors in operable communication with the computing system; wherein the computing system is configured to generate a three-dimensional (3D) reconstruction of a scene from a sequence of frames of images captured by the camera sensor by a process comprising: back-projecting features from each frame of the sequence of frames into a 3D voxel volume, wherein each pixel of the 3D voxel volume is mapped to a ray in the 3D voxel volume; Passing the 3D voxel volume through a 3D convolutional neural network (3D CNN) with an encoder-decoder to refine the features in the 3D voxel volume and regressing a truncated signed distance function (TSDF) value at each voxel of the 3D voxel volume; and After passing the 3D voxel volume through all layers of the 3D convolutional encoder-decoder, the refined features in the 3D voxel volume and the TSDF value at each voxel in the 3D voxel volume are passed through a batch normalization (batchnorm) function and a rectified linear unit (reLU) function, The 3D reconstruction is generated without using depth data from a depth sensor.

Images