Multi-resolution voxel meshing

Abstract

The present disclosure relates to multi-resolution voxel gridding. Various specific implementations disclosed herein generate a grid that represents a surface in a physical environment. The grid is generated using multi-resolution voxels based on detected depth information (e.g., from a depth camera). These techniques may use multiple hash tables to store multi-resolution voxel data. For example, a hash table may store the 3D position of each voxel and a truncated signed distance field (TSDF) value corresponding to the distance of each voxel from the closest surface. Each of the multiple hash tables may include data corresponding to different resolution levels, and those resolutions may depend on distance / noise or other factors. For example, voxels close to a depth camera may have a finer resolution and a smaller size than voxels farther from the depth camera. The techniques disclosed herein may involve the use of a gridding algorithm that combines multi-resolution voxel information stored in multiple hash tables to generate a single grid.

Description

Technical Field

[0001] The present disclosure relates generally to generating three-dimensional geometric representations of physical environments, and in particular to systems, methods, and devices for generating geometric representations based on depth information detected in the physical environment. Background Art

[0002] The physical environment has been modeled (e.g., reconstructed) by generating a three-dimensional (3D) grid. These grids represent 3D surface points and other surface features of the floor, walls, and other objects of the physical environment. Such reconstructions can be generated based on images and depth measurements of the physical environment (e.g., using an RGB camera and a depth sensor). The reconstruction technique can use voxels to generate a grid to provide reconstruction. As used herein, a voxel refers to a volume pixel. Existing reconstruction techniques use voxels of fixed size that are spaced in a regularly spaced grid in 3D space without gaps between voxels. For example, such reconstruction techniques can use a truncated signed distance function (TSDF) to accumulate information volumetrically, which provides a signed distance value for a voxel within a threshold distance of a surface in the physical environment, wherein these values ​​represent the distance of such a voxel to the closest corresponding surface in the physical environment. When such techniques use relatively large voxels, these techniques may not be able to fully represent the detailed features of the physical environment. In contrast, when such techniques use relatively small voxels, these techniques may produce inaccurate reconstructions due to inadequate noise filtering and may require undesirable resource usage, e.g., requiring processing that cannot be performed in real time or undesirable or unavailable levels of memory storage. As a result, existing reconstruction techniques may not provide sufficiently accurate and efficient reconstructions of the physical environment. Summary of the invention

[0003] Various embodiments disclosed herein include devices, systems, and methods for generating a mesh representing a surface in a physical environment (e.g., vertices forming connected triangles) using multi-resolution voxels generated based on detected depth information. In some embodiments, the reconstruction technique uses multiple hash tables to store multi-resolution voxel data. For example, the hash table may store the 3D position of each voxel and a truncated signed distance field (TSDF) value corresponding to the distance of each voxel from the closest surface. Each of the multiple hash tables may include data corresponding to different resolution levels. Those resolutions can be selected based on the distance of the voxel from the sensor, the noise in the depth data associated with different voxels, or other factors. For example, voxels close to a depth camera may have a finer resolution and a smaller size than voxels farther from the depth camera.

[0004] The techniques disclosed herein can achieve various advantages by using multi-resolution voxels. Using multi-resolution voxels provides smaller voxels for certain parts of the reconstruction to provide finer resolution and thus potentially higher accuracy and fidelity, and provides larger voxels for other parts of the reconstruction to provide coarser resolution and thus lower accuracy and fidelity. If small voxels have been used for all parts of the reconstruction, the total number of voxels of all sizes may be less than the total number of voxels that would otherwise exist, thereby reducing the resources required to determine and process voxel information. In addition, voxel size (e.g., which voxels are small and which voxels are large) can be determined using criteria that provide for areas where the use of smaller voxels is unlikely to result in higher accuracy (e.g., areas where there is more noise in the data) rather than areas where the use of smaller voxels is unlikely to result in higher accuracy (e.g., areas where there is less noise in the data). Therefore, using voxels of appropriately selected and varying sizes can facilitate fast and efficient reconstruction, for example, providing reconstruction with sufficient accuracy and speed for real-time applications.

[0005] The technology disclosed herein can use multi-resolution voxel data to generate a mesh that reconstructs the geometry of a physical environment. This can involve using a meshing algorithm that combines multi-resolution voxel information stored in multiple hash tables to generate a single mesh. Such algorithms are configured to take into account the different resolutions of the voxels represented in the multiple hash tables.

[0006] Some specific implementations of the present disclosure relate to an exemplary method for generating a mesh (e.g., vertices forming connected triangles) representing a surface in a 3D environment based on depth sensor measurements. The method involves obtaining depth data of a physical environment using a sensor. For example, the depth data may include pixel depth values ​​from a viewpoint and sensor position and orientation data.

[0007] The exemplary method also involves generating a first hash table based on the depth data, the first hash table storing 3D positions of a first set of voxels (e.g., large voxels) having a first resolution and signed distance values ​​representing distances to surfaces of the physical environment (e.g., to the closest surface). The distance field can be defined as a scalar field whose value at any given point is equal to the distance from the point to the closest surface. For example, in some specific implementations, the signed distance value includes a TSDF value that can be used to represent the voxel distance of each voxel to the closest surface of the surface of the physical environment. The TSDF value can save storage space by including only values ​​within a truncated band in the representation (e.g., only storing voxel data within a threshold distance of the surface).

[0008] The example method also involves generating a second hash table based on the depth data, the second hash table storing 3D positions of a second set of voxels (e.g., small voxels) having a second resolution and a signed distance value representing a distance to a surface of the physical environment, wherein the second resolution is different from the first resolution. For example, the resolution level for each voxel can be determined based on distance from the sensor, noise, semantics, etc. For example, voxels close to the depth camera can have a finer resolution and a smaller size than voxels farther from the depth camera.

[0009] The exemplary method also involves generating a mesh representing the surface based on the first hash table and the second hash table. The mesh can be generated by locating vertices along a line connecting a first voxel of the first group of voxels (e.g., a position at the center of the first voxel) and a second voxel of the second group of voxels (e.g., a position at the center of the second voxel). Additionally or alternatively, vertices are generated between voxels within the same resolution (e.g., whenever there is a zero crossing, the TSDF value changes from negative to positive at the zero crossing). For example, a walking cube meshing algorithm technique can be used to generate a mesh that identifies lines connecting points associated with voxels in each hash table and interpolates to identify vertices along those lines corresponding to the surface. Executing the walking cube meshing algorithm may involve parsing one or more hash tables and generating mesh triangles using vertices representing surfaces in the volume represented by the hash tables. For each hash entry, the algorithm can construct a vertex list of adjacent voxels of the cube to which it belongs and identify triangular surfaces that pass through the cube. In some implementations, generating the mesh includes generating lines connecting points associated with voxels in each of a plurality of hash tables (e.g., between a first hash table and a second hash table, between a second hash table and a third hash table, etc.), and interpolating along the lines to identify vertices of the mesh corresponding to the surfaces. In some implementations, the meshing algorithm extracts a duplicate culling mesh and a manifold mesh.

[0010] In some implementations, the exemplary method further involves generating a third hash table based on the depth data, the third hash table storing 3D positions of a third set of voxels having a third resolution different from the first resolution and the second resolution and a signed distance value representing a distance to a surface of the physical environment, wherein a mesh is generated based on the third hash table. Additional hash tables may be generated for each resolution level required. In an exemplary implementation, at least four hash tables are used for four different resolutions.

[0011] In some implementations, the exemplary method further involves determining whether to represent the 3D position as voxels having a first resolution or voxels having a second resolution. In some implementations, the exemplary method further involves determining whether to represent the 3D position as voxels having a first resolution or voxels having a second resolution based on determining noise in the depth data. In some implementations, the exemplary method further involves determining whether to represent the 3D position as voxels having a first resolution or voxels having a second resolution based on a distance of a surface closest to the voxel from a source of the depth data (e.g., a depth camera position). In some implementations, the voxels of the first group of voxels have a first size and the voxels of the second group of voxels have a second size, wherein the first size is greater than the second size.

[0012] In some implementations, the first hash table and the second hash table use 3D positions as keys to generate memory addresses storing voxel information. In some implementations, the first hash table and the second hash table have memory addresses storing signed distance values.

[0013] In some implementations, depth data is obtained using one or more depth cameras. For example, one or more depth cameras can acquire depth based on structured light (SL), passive stereo (PS), active stereo (AS), time of flight (ToF), etc. The depth camera can provide per-pixel depth information aligned with image pixels from a standard camera. In some implementations, the depth data includes pixel depth values ​​from a viewpoint and a sensor position.

[0014] According to some specific implementations, a device includes one or more processors, non-volatile memory, and one or more programs; the one or more programs are stored in the non-volatile memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing or causing the execution of any of the methods described herein. According to some specific implementations, a non-volatile computer-readable storage medium stores instructions that, when executed by one or more processors of the device, cause the device to perform or cause the execution of any of the methods described herein. According to some specific implementations, a device includes: one or more processors, non-volatile memory, and means for performing or causing the execution of any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] So that the present disclosure may be understood by those of ordinary skill in the art, a more detailed description may be obtained with reference to aspects of some exemplary implementations, some of which are illustrated in the accompanying drawings.

[0016] Figure 1 is a block diagram of an exemplary operating environment according to some implementations.

[0017] Figure 2 is a block diagram of an example controller according to some implementations.

[0018] Figure 3 is a block diagram of an exemplary device according to some implementations.

[0019] Figure 4 is a flowchart representation of an exemplary method for generating a mesh representing a surface in a 3D environment using a multi-resolution hashing data structure based on depth information, according to some implementations.

[0020] Figure 5 is a block diagram illustrating exemplary truncated signed distance field (TSDF) values ​​for a region of depth data according to some implementations.

[0021] Figure 6 is a block diagram illustrating a hash data structure according to some specific implementations.

[0022] Figure 7 is a block diagram illustrating a multi-resolution hash data structure representation and a grid representation according to some specific implementations.

[0023] FIG. 8A to FIG. 8C is a block diagram illustrating an exemplary image during 3D reconstruction using dense TSDF mapping according to certain implementations.

[0024] 9A to 9C is a block diagram illustrating an exemplary image during 3D reconstruction using a hash table of a sparse TSDF map and a single-resolution grid, according to certain implementations.

[0025] FIG. 10A to FIG. 10C is a block diagram illustrating an exemplary image during 3D reconstruction using multiple hash tables of a sparse TSDF map and a multi-resolution grid, according to certain implementations.

[0026] Fig.11 is a system flow diagram for exemplary generation of a mesh representing a surface in a 3D environment using a multi-resolution hashing data structure based on depth information, according to certain implementations.

[0027] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, the sizes of the various features may be arbitrarily expanded or reduced for clarity. In addition, some of the drawings may not depict all of the components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. DETAILED DESCRIPTION

[0028] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail in order to avoid obscuring more relevant aspects of the exemplary implementations described herein.

[0029] Figure 1 1 is a block diagram of an exemplary operating environment 100 according to some implementations. In this example, the exemplary operating environment 100 shows an exemplary physical environment 105 including an object 125 and a table 135. Although relevant features are shown, one of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the exemplary implementations disclosed herein. To this end, as a non-limiting example, the operating environment 100 includes a controller 110 and a device 120.

[0030] In some implementations, the controller 110 is configured to manage and coordinate the user experience. In some implementations, the controller 110 includes a suitable combination of software, firmware, and / or hardware. Figure 2 Controller 110 is described in more detail. In some implementations, controller 110 is a computing device that is located locally or remotely relative to physical environment 105. In one example, controller 110 is a local server located within physical environment 105. In another example, controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside of physical environment 105. In some implementations, controller 110 is communicatively coupled to device 120 via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.).

[0031] In some implementations, the device 120 is configured to present an environment to a user. In some implementations, the device 120 includes a suitable combination of software, firmware, and / or hardware. Figure 3 Device 120 is described in further detail. In some implementations, the functionality of controller 110 is provided by and / or combined with device 120.

[0032] In some implementations, the user wears the device 120 on his / her head. As such, the device 120 may include one or more displays configured to display content. For example, the device 120 may encompass the user's field of view. In some implementations, the device 120 is a handheld electronic device (e.g., a smartphone or tablet) configured to present content to the user. In some implementations, the device 120 is replaced with a chamber, housing, or chamber configured to present content, wherein the user does not wear or hold the device 120.

[0033] Figure 2 is a block diagram of an example of a controller 110 according to some implementations. While some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some specific implementations, the controller 110 includes one or more processing units 202 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., a universal serial bus (USB), FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), infrared (IR), Bluetooth, ZIGBEE and / or similar type interfaces), one or more programming (e.g., I / O) interfaces 210, a memory 220, and one or more communication buses 204 for interconnecting these components and various other components.

[0034] In some implementations, the one or more communication buses 204 include circuits for interconnecting system components and controlling communications between system components. In some implementations, the one or more I / O devices 206 include at least one of a keyboard, a mouse, a touch pad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, and the like.

[0035] The memory 220 includes a high-speed random access memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some specific implementations, the memory 220 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 220 optionally includes one or more storage devices located away from the one or more processing units 202. The memory 220 includes a non-transitory computer-readable storage medium. In some specific implementations, the memory 220 or the non-transitory computer-readable storage medium of the memory 220 stores the following programs, modules, and data structures, or a subset thereof, including an optional operating system 230 and one or more application programs 240.

[0036] The operating system 230 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, the operating system 230 includes built-in CGR functionality, for example, including a CGR environment application or a viewer configured to be called from one or more applications 240 to display a CGR environment within a user interface. In some implementations, the application 240 is configured to manage and coordinate one or more experiences of one or more users (e.g., a single experience of one or more users, or multiple experiences of corresponding groups of one or more users).

[0037] Application 240 includes image integration unit 242, hash algorithm unit 244, grid algorithm unit 246, and CGR unit 248. Image integration unit 242, hash algorithm unit 244, grid algorithm unit 246, and CGR unit 248 may be combined into a single application or unit, or separated into one or more additional applications or units.

[0038] The image integration unit 242 is configured with instructions that can be executed by the processor to obtain image data (e.g., light intensity data, depth data, etc.) and integrate (e.g., fuse) the image data using one or more techniques disclosed herein. For example, the image integration unit 242 fuses the RGB image from the light intensity camera with the sparse depth map from the depth camera (e.g., time-of-flight sensor) and other sources of physical environment information to output a dense depth point cloud of information.

[0039] The hash algorithm unit 244 is configured with instructions that can be executed by the processor to generate a hash table based on the depth data using one or more techniques disclosed herein, the hash table storing the 3D position of the voxel and a signed distance value representing the distance to the surface of the physical environment (e.g., to the closest surface). For example, the hash algorithm unit 244 can determine the level of resolution from the image data and generate a hash table for each level of the determined resolution.

[0040] The mesh algorithm unit 246 is configured with instructions that can be executed by the processor to perform a meshing algorithm (e.g., a double-marching cube meshing algorithm, a Poisson meshing algorithm, a tetrahedron meshing algorithm, etc.) using one or more techniques disclosed herein and generate a mesh representing a room (e.g., the physical environment 105) and / or objects within the room (e.g., the object 125, the table 135, etc.). The computer generated reality (CGR) unit 248 is configured with instructions that can be executed by the processor to provide a CGR environment including a description of a physical environment containing real objects or virtual objects. Virtual objects can be positioned based on stored 3D models of real objects and virtual objects, such as using one or more techniques disclosed herein, based on detection, tracking, and representation of objects relative to each other in 3D space.

[0041] also, Figure 2 It is more of a functional description of various features present in a particular implementation, as opposed to a schematic diagram of the implementations described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 2 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a particular embodiment.

[0042] Figure 3is a block diagram of an example of a device 120 according to some implementations. While some specific features are shown, those skilled in the art will appreciate from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some embodiments, the device 120 includes one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, SPI, I2C, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more AR / VR displays 312, one or more internal-facing and / or external-facing image sensors 314, a memory 320, and one or more communication buses 304 for interconnecting these and various other components.

[0043] In some implementations, the one or more communication buses 304 include circuits for interconnecting and controlling communications between system components. In some implementations, the one or more I / O devices and sensors 306 include at least one of the following: an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, and / or one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0044] In some implementations, one or more displays 312 are configured to present an experience to a user. In some implementations, one or more displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emitter display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), microelectromechanical system (MEMS), and / or similar display types. In some implementations, one or more displays 312 correspond to diffractive, reflective, polarized, holographic, etc. waveguide displays. For example, device 120 includes a single display. As another example, device 120 includes a display for each eye of the user.

[0045] In some implementations, the one or more image sensor systems 314 are configured to obtain image data corresponding to at least a portion of the physical environment 105. For example, the one or more image sensor systems 314 include one or more RGB cameras (e.g., with complementary metal oxide semiconductor (CMOS) image sensors or charge coupled device (CCD) image sensors), monochrome cameras, IR cameras, event-based cameras, etc. In various implementations, the one or more image sensor systems 314 also include an illumination source that emits light, such as a flash. In various implementations, the one or more image sensor systems 314 also include an on-camera image signal processor (ISP) that is configured to perform a plurality of processing operations on the image data, including at least a portion of the processes and techniques described herein.

[0046] The memory 320 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices. In some specific implementations, the memory 320 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 320 optionally includes one or more storage devices located away from the one or more processing units 302. The memory 320 includes a non-transitory computer-readable storage medium. In some specific implementations, the memory 320 or the non-transitory computer-readable storage medium of the memory 320 stores the following programs, modules and data structures, or a subset thereof, including an optional operating system 330 and one or more application programs 340.

[0047] The operating system 330 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, the operating system 330 includes built-in CGR functionality, for example, including a CGR environment application or a viewer configured to be called from one or more applications 340 to display a CGR environment within a user interface. In some implementations, the application 340 is configured to manage and coordinate one or more experiences of one or more users (e.g., a single experience of one or more users, or multiple experiences of corresponding groups of one or more users).

[0048] Application 340 includes image integration unit 342, hash algorithm unit 344, grid algorithm unit 346, and CGR unit 348. Image integration unit 342, hash algorithm unit 344, grid algorithm unit 346, and CGR unit 348 may be combined into a single application or unit, or separated into one or more additional applications or units.

[0049] The image integration unit 342 is configured with instructions that can be executed by the processor to obtain image data (e.g., light intensity data, depth data, etc.) and integrate (e.g., fuse) the image data using one or more techniques disclosed herein. For example, the image integration unit 342 fuses the RGB image from the light intensity camera with the sparse depth map from the depth camera (e.g., time-of-flight sensor) and other sources of physical environment information to output a dense depth cloud.

[0050] The hash algorithm unit 344 is configured with instructions that can be executed by the processor to generate a hash table based on the depth data using one or more techniques disclosed herein, the hash table storing the 3D position of the voxel and a signed distance value representing the distance to the surface of the physical environment (e.g., to the closest surface). For example, the hash algorithm unit 344 can determine the level of resolution from the image data and generate a hash table for each level of the determined resolution.

[0051] The mesh algorithm unit 346 is configured with instructions that can be executed by the processor to perform a meshing algorithm (e.g., a double-marching cube meshing algorithm, a Poisson meshing algorithm, a tetrahedron meshing algorithm, etc.) using one or more techniques disclosed herein and generate a mesh representation of a room (e.g., the physical environment 105) and / or objects in the room (e.g., the object 125, the table 135, etc.). The CGR unit 348 is configured with instructions that can be executed by the processor to provide a CGR environment including a description of the physical environment containing real objects or virtual objects. Virtual objects can be positioned based on stored 3D models of real objects and virtual objects, for example, using one or more techniques disclosed herein, based on detection, tracking, and representation of objects relative to each other in 3D space.

[0052] Although these elements are shown as residing on a single device (e.g., device 120), it should be understood that in other implementations, any combination of the elements may be located in separate computing devices. Figure 3 It is more of a functional description of various features present in a particular implementation, as opposed to a schematic diagram of the implementations described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 3 Some functional modules (e.g., application 340) shown separately in the figure can be implemented in a single module, and the various functions of a single functional block can be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a particular embodiment.

[0053] Figure 4 4 is a flowchart representation of an exemplary method 400 for generating a grid using multiple hash tables representing voxels at multiple resolutions according to some implementations. In some implementations, the method 400 is performed by a device (e.g., Figure 1 and Figure 2 The method 400 may be performed on a device 120 having a screen for displaying 2D images and / or a screen for viewing stereoscopic images (e.g., Figure 1 and Figure 3 The method 400 is performed on a device 120 such as a head mounted display (HMD). In some implementations, the method 400 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 400 is performed by a processor executing code stored in a non-transitory computer readable medium (e.g., a memory).

[0054] Method 400 generates a mesh representing a surface in a 3D environment (e.g., vertices forming connected triangles) using a multi-resolution hash data structure based on depth information detected in a physical environment. Method 400 can provide a realistic and physically meaningful reconstruction of a physical environment. When an image capture device (e.g., a mobile phone) scans a physical environment (e.g., a room or hallway of a building), method 400 can efficiently use memory and quickly update the reconstruction. Method 400 can generate a mesh that can be used in a variety of applications. For example, method 400 can provide a mesh that facilitates realistic placement and movement of virtual objects that interact with physical objects in a CGR environment and segmentation of a physical environment in 3D to enable object replacement.

[0055] At block 402, method 400 obtains depth data of a physical environment using a sensor, the physical environment having a surface. For example, the depth data may include pixel depth values ​​from a viewpoint and sensor position and orientation data. In some implementations, depth data is obtained using one or more depth cameras. For example, one or more depth cameras may obtain depth based on structured light (SL), passive stereo (PS), active stereo (AS), time of flight (ToF), etc. Various techniques may be applied to obtain depth image data to allocate each portion of the image (e.g., at a pixel level or voxel level). Such techniques may involve obtaining and analyzing depth values, depth images, and depth histograms, and may result in the creation of one or more masks that identify portions of the image associated with different focal depth planes. For example, voxel data (e.g., a raster graphic on a 3D grid, with values ​​of length, width, and depth) may also include multiple scalar values, such as opacity, color, and density. In some implementations, depth data is obtained from a sensor or a 3D model of the content of the image. Part or all of the content of the image may be based on a real environment, such as a physical environment 105 around a depicting device 120. The image sensor can capture images of the physical environment 105 to be included in the image and depth information about the physical environment 105. In some implementations, a depth sensor on the device 120 determines a depth value for a voxel determined based on the image captured by the image sensor on the device 120. The physical environment 105 around the user can be 3D modeled based on one or more values, and subsequent depths of objects depicted in subsequent images of the physical environment can be determined based on the model and the camera position information. In some implementations, a virtual object can be presented in the CGR environment along with the image data, and the virtual object can have a pre-specified depth value or coordinates from which such a depth value can be determined.

[0056] At block 404, method 400 generates a first hash table based on the depth data that stores 3D positions of a first set of voxels having a first resolution and signed distance values ​​representing distances to surfaces of the physical environment. For example, the first resolution may include larger voxels at a particular distance from the surface of the physical environment (e.g., object 125 on table 135). In some implementations, multiple hash tables are used, and each hash table includes a memory address that stores signed distance values. As described herein with reference to Figure 5 As further discussed, in some implementations, truncated signed distance field (TSDF) values ​​may be used to represent the voxel distance of each voxel to the closest surface in the surface of the physical environment corresponding to the depth data. The TSDF values ​​may be used to save significant storage space by including only values ​​within a truncated band (e.g., within a threshold distance of a surface) in the representation.

[0057] In addition, for example, the 3D volume data may include distributed voxel addresses, and the stored 3D positions may be used as keys for hash table entries to provide (x, y, z) coordinates and generate memory addresses for storing voxel information. For example, in the 3D volume data, each bit may be unique, and the (x, y, z) coordinates of each voxel may be unique. In one exemplary implementation, an algorithm implemented in the system may utilize the unique voxel locations in the exemplary 3D volume data to provide an addressing scheme that minimizes out-of-order or redundant hash table entries. As described herein with reference to Figure 6 As further described, out-of-order or redundant hash table entries may be a result of a hash function employed in a hashing scheme that generates the same index for more than one key in the hash table.

[0058] At box 406, method 400 generates a second hash table based on the depth data, the second hash table storing the 3D positions of the second set of voxels with a second resolution and a signed distance value representing the distance to the surface of the physical environment. The generation of the second hash table in box 406 may be similar to the generation of the first hash table in box 404. However, at box 406, a second resolution different from the resolution of the first hash table is used. For example, the second resolution may be a finer resolution and include smaller voxels, and the first hash table may include a coarser resolution and include larger voxels. Different hash tables may include data corresponding to different resolution levels, and those resolutions may depend on distance, noise, or other factors. Distance can be used as an approximation of noise based on correlation (e.g., quadratic noise relative to distance, so that greater distance means greater noise).

[0059] According to some specific implementations, method 400 also includes determining whether to represent the 3D position as voxels with a first resolution or voxels with a second resolution. For example, the system may determine the number of resolution levels to be analyzed and represented by the hash table. According to some specific implementations, method 400 also includes determining whether to represent the 3D position as voxels with a first resolution or as voxels with a second resolution based on determining estimated noise in the depth data, as described herein. According to some specific implementations, method 400 also includes determining whether to represent the 3D position as a voxel with a first resolution or a voxel with a second resolution based on the distance of the surface closest to the voxel from the source of the depth data. For example, the system may determine the number of resolution levels to be analyzed based on the corresponding depth data of the depth camera position and the distance to the surface of an object in the physical environment (such as the distance to the surface of object 125).

[0060] Additionally or alternatively, the resolution level may be determined based on semantic tags that identify object types (e.g., table, teapot, chair, vase, etc.). In some implementations, semantic tagging uses a machine learning model, where a semantic segmentation model may be configured to identify semantic tags for pixels or voxels of image data. For example, if a voxel is labeled "wall", the system may select a larger / sparser resolution based on the assumption or description that a wall-type object will have consistent or insignificant texture that does not need to be represented using a fine resolution, regardless of how far the object (wall) is. If the voxel is labeled "cup", the system may select a smaller / finer resolution based on the assumption or description that a teacup-type object will have fine details that are worth trying to represent regardless of potential noise. In some implementations, the machine learning model is a neural network (e.g., an artificial neural network), a decision tree, a support vector machine, a Bayesian network, etc.

[0061] According to some specific implementations, a third hash table may be generated based on the depth data to store the 3D positions of a third set of voxels having a third resolution and a signed distance value representing the distance to the surface of the physical environment. The third resolution is different from the first resolution and the second resolution. A grid may be generated based on the third hash table. Depending on the specific implementation or situation, additional hash tables may be similarly generated for each further level of resolution.

[0062] At block 408, the method 400 generates a mesh representing the surface based on the first hash table and the second hash table, wherein the mesh is generated by positioning vertices of the mesh along lines connecting a first voxel of the first group of voxels with a second voxel of the second group of voxels. For example, the mesh may be generated by positioning vertices of the mesh along lines connecting a first voxel of the first group of voxels (e.g., a position located at the center of the first voxel) with a second voxel of the second group of voxels (e.g., a position located at the center of the second voxel). In some implementations, the mesh may be generated using a dual-stepping cube meshing algorithm technique that identifies lines connecting points associated with voxels in each hash table and interpolates to identify vertices along those lines corresponding to the surface. Additionally or alternatively, vertices are generated between voxels within the same resolution (e.g., whenever there is a zero crossing, the TSDF value changes from negative to positive at the zero crossing). Using a stepping cube meshing algorithm may involve parsing one or more hash tables and generating mesh triangles using vertices representing a surface in the volume. For each hash entry in the hash table, the algorithm can construct a list of vertices of neighboring voxels of the cube to which it belongs, and identify triangular faces that pass through the cube. In some implementations, generating a mesh includes generating lines connecting points associated with voxels in both the first hash table and the second hash table, and interpolating along these lines to identify vertices of the mesh corresponding to these faces. In some implementations, the meshing algorithm extracts a duplicate elimination mesh and a manifold mesh. Alternatively, other suitable meshing algorithms (e.g., Poisson meshing algorithm, tetrahedral meshing algorithm, etc.) can be used.

[0063] Figure 5 1 is a block diagram of an exemplary operating environment 500 illustrating exemplary truncated signed distance field (TSDF) values ​​for a region 520 of depth data according to some implementations. In this example, the exemplary operating environment 500 illustrates an environment including a device 510, an object 125, and a table 135. Although relevant features are illustrated, one of ordinary skill in the art will recognize from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure more relevant aspects of the exemplary implementations disclosed herein. To this end, as a non-limiting example, the operating environment 500 includes a device 510.

[0064] In some implementations, device 510 may include one or more depth sensors (e.g., structured light, time of flight, etc.). Figure 5As shown, the device 510 is tilted at an angle toward the object 125. The object 125 includes an edge 525 closest to the device 510. For schematic illustration, a region 520 of the object 520 is expanded at the voxel level in a two-dimensional (2D) representation in a grid format to show a truncated signed distance function (TSDF) value. If the voxel is within the truncation threshold, the signed distance value is stored. As shown, row 1 and row 1' are at "++" and "--" distances, respectively, and row 2 and row 2' are at "+" and "-" distances, respectively, each of which is within a positive truncation threshold 532 or a negative truncation threshold 534. However, row 3 and row 3' are outside the corresponding truncation thresholds, so the signed distance function is not stored. Or, in other words, for those voxels outside the corresponding truncation thresholds, the signed distance is truncated or ignored. In an exemplary implementation, the hashing and gridding system utilizes more than one resolution level. Thus, voxels 542 and 544 are shown as examples where voxels may be adjusted and divided into smaller voxels for finer resolution. Additionally, each of the four voxels within each of voxels 542 and 544 may be further divided depending on the resolution level desired. Each resolution level determined by the system to be utilized is then stored in a separate hash data structure, as described below with reference to Figure 6 Further described.

[0065] Figure 66 is a block diagram illustrating an exemplary environment 600 for a multi-resolution hash data structure according to some specific implementations. The exemplary environment 600 includes an 8×8×8 single-resolution orthogonal (uniform) voxel grid 602. The voxel grid 602 represents all information in the volume by pre-allocating a fixed 3D voxel grid in memory. Although the voxel grid 602 provides constant access time for storing and retrieving the TSDF value for each voxel, the fact that the memory must be pre-allocated makes it impractical to store large capacities, which can reach tens of gigabytes even for a moderately sized room. Each voxel (e.g., voxel 610) can include global (x, y, z) coordinates and TSDF values. A more compact representation can be made using a hash function H(x, y, z) that maps world coordinates to voxels. Therefore, the information in the voxels 610 in the exemplary environment 600 is shown as being stored in a hash table 620. The hash table 620 stores hash entries, each of which contains a pointer to an allocated voxel block (e.g., voxel 610). For each voxel, this information is stored in a memory region based on different parameters (e.g., the stored information may include world coordinates (x, y, z), TSDF values, and offset values ​​for redundant entries in the memory region). For example, the hash table 620 may be decomposed into a set of memory regions where each time slot is either unallocated or contains one entry (grey shading). Each memory region is then stored as a voxel block array 630 for each ordered voxel entry, removing all redundant entries.

[0066] Hash table 620 represents the volume as a hash map, where a hash function is used to access voxels, such as voxel 610. Hash table 620 allows dynamic allocation and management of voxels in space. In addition, in addition to TSDF information, hash table 620 also provides the ability to associate and retrieve metadata for each voxel in real time. For example, compared to an octree that may require a large amount of searching to find similar capabilities, the material or object classification attributes of each voxel can be quickly retrieved. Hash table 620 provides the ability to divide large volumes into smaller 3D grids and manage multiple local and global hash tables under different hierarchical structures (e.g., based on different resolution levels) and efficient merging and updating between them based on platform computing availability and application requirements. In other words, the advantage of using a hash table is that it does not impose restrictions on the volume to be reconstructed a priori, and the mapping is enlarged as needed by design.

[0067] Figure 7Included are block diagrams showing exemplary multi-resolution hash data structure representations (e.g., exemplary voxel representations 700A to 700D) and 2D grid representations (e.g., grid representation 700E) according to some specific implementations. Exemplary voxel representations 700A to 700D are 2D voxel representations. For example, exemplary voxel representation 700A is a representation showing voxels (e.g., voxels 711, 712, 713) at a first resolution for a first hash table, and exemplary voxel representation 700B is a representation showing voxels (e.g., voxels 721, 722, 723) at a second resolution different from the first resolution for a second hash table. The world coordinates for voxel representation 700A and the voxel size and shape of the voxels in the first hash table are used to determine that voxel 711 is adjacent to voxel 712, but not adjacent to voxel 713. The world coordinates for voxel representation 700B and the voxel size and shape of the voxels in the second hash table are used to determine that voxel 721 is adjacent to voxel 722, but not adjacent to voxel 723. In addition, the world coordinates and the voxel size and shape of the voxels in both the first hash table and the second hash table are used to determine that voxel 721 is adjacent to voxel 711, but not adjacent to voxel 712.

[0068] The "voxel-free regions" in the voxel representations 700A, 700B are used for illustrative purposes to illustrate that a hash table representation of a particular resolution only creates a hash table for that particular resolution. For example, the voxel representation 700A represents a finer resolution and is therefore displayed with smaller voxels (e.g., voxels 711, 712, 713), and the voxel representation 700B represents a sparser or coarser resolution and is therefore displayed with larger voxels (e.g., voxels 711, 712, 713) compared to the voxel size of the voxel representation 700A.

[0069] Although not shown, another voxel representation may be shown with smaller voxels to obtain a third finer resolution. For example, voxel 711 may be divided into four smaller voxels for separate hash tables representing the third resolution.

[0070] The exemplary voxel representation 700C is a 2D voxel representation composed of multiple hash tables (e.g., a first hash table for voxel representation 700A and a second hash table for voxel representation 700B) and lines connecting adjacent voxels and labels (e.g., + or -) showing the voxel distance signs. For example, the exemplary voxel representation 700C shows the topology of a voxel grid for two resolution levels, where each vertex of the voxel grid is located at the center of each voxel. The voxel distance signs (e.g., "+" and "-") are used to show which voxels are positive "+" or above the outer edge of the detected object surface and which voxels are negative "-" or below the outer edge of the detected object surface, as shown in the exemplary voxel representation 700D.

[0071] As shown in the exemplary voxel representation 700D, consider the lines between voxels with different sign values, and the edge 730 includes mesh vertices A, B, C, D selected along these lines based on interpolation and depth data. For example, the edge 730 in the exemplary voxel representation 700D may represent Figure 5 Edge 525 of object 125 in FIG.

[0072] 8 to 10 illustrate exemplary 3D mesh reconstructions using a dense TSDF map, a sparse TSDF map and a hash table for a single resolution, and a sparse TSDF map and multiple hash tables for a multi-resolution mesh, respectively. FIGS. 8 to 10 illustrate a comparison between meshes generated using a multi-resolution hash data structure (e.g., FIG. 10 ) compared to two other techniques (e.g., FIGS. 8 and 9 ) at three different data acquisition time intervals (e.g., 0 seconds, 6 seconds, and 12 seconds).

[0073] In one specific implementation, FIG. 8A to FIG. 8C is a block diagram illustrating exemplary images during 3D reconstruction of an object (e.g., a teapot) using dense TSDF mapping according to certain specific implementations. Specifically, during the 3D reconstruction meshing process, exemplary environments 800a–800c represent three different data acquisition time intervals, exemplary environment 800a at an initial time (e.g., t at 0 seconds), exemplary environment 800b at a first midpoint time (e.g., t at 6 seconds), and exemplary environment 800c at a subsequent midpoint time or an end time indicating the end of data acquisition (e.g., t at 12 seconds). Alternatively, more or less times may be used to acquire and generate the mesh. Additionally, the exemplary time intervals of 0 seconds, 6 seconds, and 12 seconds are used for exemplary purposes only to illustrate the process of generating the 3D reconstruction mesh. For example, screenshots 802a-802c represent dense point cloud data that has been integrated from acquired image data including depth data (e.g., RGB-D data). For example, from Figure 5 804a-804c represent dense truncated signed distance field data. The dense data is used for high resolution reconstruction. Screenshots 806a-806c represent 3D reconstructed meshes generated based on the received dense truncated signed distance field data.

[0074] In one specific implementation, 9A to 9Cis a block diagram illustrating exemplary images during 3D reconstruction of an object (e.g., a teapot) using a hash table of sparse TSDF mapping and a single-resolution mesh according to certain specific implementations. Specifically, during 3D reconstruction meshing, exemplary environments 900a–900c represent three different data acquisition time intervals, exemplary environment 900a at an initial time (e.g., t at 0 seconds), exemplary environment 900b at a first midpoint time (e.g., t at 6 seconds), and exemplary environment 900c at a subsequent midpoint time or an end time indicating the end of data acquisition (e.g., t at 12 seconds). Alternatively, more or less time may be used to acquire and generate the mesh. Additionally, the exemplary time intervals of 0 seconds, 6 seconds, and 12 seconds are used for exemplary purposes only to illustrate the process of generating a 3D reconstruction mesh. For example, screenshots 902a-902c represent dense point cloud data that has been integrated from acquired image data including depth data (e.g., RGB-D data). For example, from Figure 5 904a-904c represent sparse truncated signed distance field data using a single hash table algorithm for primary resolution. Screenshots 906a-906c represent 3D reconstructed meshes generated based on the received sparse truncated signed distance field data and the single hash table data.

[0075] In one exemplary implementation, FIG. 10A to FIG. 10C is a block diagram illustrating exemplary images during 3D reconstruction of an object (e.g., a teapot) using a sparse TSDF map and a hash table of multi-resolution meshes according to certain specific implementations. Specifically, during the 3D reconstruction meshing process, exemplary environments 1000a–1000c represent three different data acquisition time intervals, exemplary environment 1000a at an initial time (e.g., t at 0 seconds), exemplary environment 1000b at a first midpoint time (e.g., t at 6 seconds), and exemplary environment 1000c at a subsequent midpoint time or an end time indicating the end of data acquisition (e.g., t at 12 seconds). Alternatively, more or fewer time intervals may be used to acquire and generate the mesh 3D reconstruction. Additionally, the exemplary time intervals of 0 seconds, 6 seconds, and 12 seconds are used for exemplary purposes only to illustrate the process of generating the 3D reconstruction mesh. For example, screenshots 1002a-1002c represent dense point cloud data that has been integrated from acquired image data including depth data (e.g., RGB-D data). For example, from Figure 5The image data obtained by the device 510 in FIG. 1004a - 1004c show sparse truncated signed distance field data for multi-resolution mesh 3D reconstruction using multiple hash tables. The screenshots 1006a - 1006c show 3D reconstruction meshes generated based on the received data using multiple hash tables for multiple resolutions.

[0076] Fig.11 1 is a system flow diagram of an exemplary environment 1100 in which the system can generate a mesh representing a surface in a 3D environment (e.g., vertices forming connected triangles) using a multi-resolution hash data structure based on depth information detected in a physical environment. In some implementations, the system flow of the exemplary environment 1100 is performed on a device (e.g., Figure 1 and Figure 2 The system flow of the exemplary environment 1100 may be executed on a device 120 having a screen for displaying 2D images and / or a screen for viewing stereoscopic images (e.g., Figure 1 and Figure 3 The system flow of the exemplary environment 1100 is executed on a device 120 such as a head mounted display (HMD). In some implementations, the system flow of the exemplary environment 1100 is executed on a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the system flow of the exemplary environment 1100 is executed on a processor executing code stored in a non-transitory computer readable medium (e.g., a memory).

[0077] The system flow of the exemplary environment 1100 acquires image data of an object (e.g., a teapot) and generates a mesh 1144 representing a surface in a 3D environment using a multi-resolution hash data structure 1134. The mesh 1144 is based on depth information detected in the physical environment that is integrated (e.g., fused) to recreate the physical environment.

[0078] In one exemplary implementation, environment 1100 includes an image synthesis pipeline that acquires or obtains objects of a physical environment (e.g., Figure 1 and Figure 5125 of the physical environment 105 (e.g., image data from an image source 1110). Exemplary environment 1100 is an example of acquiring image data (e.g., light intensity data and depth data) for a plurality of image frames. Image source 1110 may include a depth camera 1114 that acquires depth data 1115 of the physical environment, and a light intensity camera 1112 (e.g., an RGB camera) that acquires intensity data 1113. Additionally or alternatively, in some implementations, the image synthesis pipeline includes virtual content (e.g., a virtual box on table 135) generated for a CGR environment, as described herein. For example, the image synthesis pipeline may include a CGR unit (e.g., Figure 2 CGR Unit 248 and Figure 3 The CGR unit 348 is configured with instructions executable by the processor to provide a CGR environment including a description of a physical environment including real physical objects and virtual content. The CGR unit can generate virtual depth data (e.g., a depth image of the virtual content) and virtual intensity data (e.g., a light intensity image (e.g., RGB) of the virtual content.

[0079] A CGR environment refers to a fully or partially simulated environment that people sense and / or interact with via an electronic system. In CGR, a subset of a person's physical movements, or representations thereof, are tracked, and in response, one or more features of one or more virtual objects simulated in the CGR environment are adjusted in a manner that complies with at least one physical law. For example, a CGR system may detect a person's head rotation, and in response, adjust the graphical content and sound field presented to the person in a manner similar to the way such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), adjustments to features of virtual objects in a CGR environment may be made in response to representations of physical movement (e.g., voice commands).

[0080] A person can sense and / or interact with a CGR object using any of their senses, including vision, hearing, touch, taste, and smell. For example, a person can sense and / or interact with an audio object that creates a 3D or spatial audio environment that provides the perception of a point audio source in a 3D space. As another example, an audio object can enable audio transparency that selectively introduces ambient sounds from a physical environment with or without computer-generated audio. In some CGR environments, a person can sense and / or interact only with an audio object. In some implementations, image data is aligned with image pixels of a physical environment 105 (e.g., RGB, depth, etc.) used with imaging processing techniques within the CGR environment described herein.

[0081] Examples of CGR include virtual reality and mixed reality. A virtual reality (VR) environment refers to a simulated environment designed to be based entirely on computer-generated sensory input for one or more senses. A VR environment includes virtual objects that a person can sense and / or interact with. For example, trees, buildings, and computer-generated images representing avatars of people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence within the computer-generated environment, and / or through a simulation of a subset of the person's physical movement within the computer-generated environment.

[0082] In contrast to VR environments, which are designed to be based entirely on computer-generated sensory input, a mixed reality (MR) environment refers to a simulated environment designed to include sensory input from the physical environment, or representations thereof, in addition to computer-generated sensory input (e.g., virtual objects). On the virtuality continuum, a mixed reality environment is anything between a fully physical environment at one end and a virtual reality environment at the other end, but not including both ends.

[0083] In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. In addition, some electronic systems used to render the MR environment can track position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical items from the physical environment or representations thereof). For example, the system can cause movement so that virtual trees appear stationary relative to the physical ground.

[0084] Examples of mixed reality include augmented reality and augmented virtuality. An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on a transparent or translucent display so that a person uses the system to perceive virtual objects superimposed on a physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of a physical environment that are representations of the physical environment. The system combines the image or video with the virtual object and presents the composition on an opaque display. People use the system to indirectly view the physical environment via an image or video of the physical environment and perceive virtual objects superimposed on the physical environment. As used herein, a video of a physical environment displayed on an opaque display is referred to as a "transparent video," meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting an AR environment on an opaque display. Further alternatively, the system may have a projection system that projects virtual objects into a physical environment, for example as a hologram or on a physical surface, so that a person using the system perceives the virtual objects superimposed on the physical environment.

[0085] An augmented reality environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing a pass-through video, the system may transform one or more sensor images to apply a selected perspective (e.g., a viewpoint) that is different from the perspective captured by the imaging sensor. For another example, a representation of a physical environment may be transformed by graphically modifying (e.g., enlarging) portions thereof so that the modified portions may be representative but not authentic versions of the original captured images. For another example, a representation of a physical environment may be transformed by graphically eliminating or blurring portions thereof.

[0086] An augmented virtual (AV) environment is a simulated environment in which a virtual or computer-generated environment combines one or more sensory inputs from a physical environment. The sensory input may be a representation of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, but the faces of people are realistically reproduced from images taken of physical people. For another example, a virtual object may take the shape or color of a physical object imaged by one or more imaging sensors. For another example, a virtual object may take a shadow that conforms to the positioning of the sun in the physical environment.

[0087] There are many different types of electronic systems that enable people to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on people's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smart phones, tablets, and desktop / laptop computers. A head-mounted system can have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system can be configured to accept an external opaque display (e.g., a smart phone). A head-mounted system can incorporate one or more imaging sensors for capturing images or videos of a physical environment, and / or one or more microphones for capturing audio of a physical environment. Instead of an opaque display, a head-mounted system can have a transparent or translucent display. A transparent or translucent display can have a medium through which light representing an image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In one specific implementation, a transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphic image onto a person's retina. The projection system can also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.

[0088] The exemplary environment 1100 also includes an image integration unit 1120 (e.g., Figure 2 The image integration unit 242 and / or Figure 3 An image integration unit 1120 is configured with instructions that can be executed by a processor to obtain image data (e.g., light intensity data, depth data, etc.) and integrate (e.g., fuse) the image data using one or more techniques disclosed herein. For example, the image integration unit 1120 receives depth image data 1115 and intensity image data 1113 from an image source 1110 (e.g., a light intensity camera 1112 and a depth camera 1114), and integrates the data and generates image integration data 1122. The image integration data 1122 may include dense point cloud data 1124 (e.g., imperfect depth maps and camera poses for multiple image frames around an object) that is sent to a hashing and gridding algorithm. For example, the image integration unit 1122 fuses an RGB image from a light intensity camera with a sparse depth map from a depth camera (e.g., a time-of-flight sensor) and other sources of physical environment information to output a dense depth point cloud of information.

[0089] The exemplary environment 1100 also includes a hash algorithm unit 1130 (e.g., Figure 2 The hash algorithm unit 244, and / or Figure 3 A hash algorithm unit 344 of the present invention is configured with instructions that can be executed by the processor to generate a hash table based on the depth data using one or more techniques disclosed herein, the hash table storing the 3D position of the voxel and a signed distance value representing the distance to the surface of the physical environment (e.g., to the closest surface). For example, the hash algorithm unit 1130 can determine the level of resolution from the image data and generate a hash table for each level of the determined resolution. The hash algorithm unit 1130 obtains the image integration data 1122 to use the referenced method of the present invention. Figure 7 and one or more hashing techniques disclosed in FIG. 8 generate hash data 1132, for Fig.11 For the exemplary teapot in FIG. 1 , the hash data is a multi-resolution hash data structure 1134 .

[0090] The exemplary environment 1100 also includes a grid algorithm unit 1140 (e.g., Figure 2 The grid algorithm unit 246, and / or Figure 3 A mesh algorithm unit 1140 is provided that is configured with instructions executable by a processor to perform a meshing algorithm (e.g., a dual marching cube meshing algorithm, a Poisson meshing algorithm, a tetrahedron meshing algorithm, etc.) and generate a mesh representation of a room (e.g., physical environment 105) and / or objects within the room (e.g., object 125, table 135, etc.) using one or more techniques disclosed herein. Mesh algorithm unit 1140 obtains hash data 1132 from hash algorithm unit 1130 and performs a meshing algorithm to generate mesh data 1142. For example, mesh data 1142 for an exemplary teapot mesh 1144 may be generated using a marching cube meshing algorithm technique that identifies lines connecting points associated with voxels in each hash table and interpolates to identify vertices along those lines corresponding to surfaces. Such a meshing technique is disclosed herein with reference to FIG. 8.

[0091] Numerous specific details are set forth herein to provide a comprehensive understanding of the claimed subject matter. However, those skilled in the art will appreciate that the claimed subject matter may be practiced without these specific details. In other instances, methods, devices, or systems known to those of ordinary skill are not described in detail so as not to obscure the claimed subject matter.

[0092] Unless otherwise specifically noted, it should be understood that throughout the specification, discussions utilizing terms such as "process," "compute," "calculate," "determine," and "identify" refer to the actions or processes of a computing device, such as one or more computers or similar electronic computing devices, that manipulate or transform data represented as physical electronic or magnetic quantities within a memory, register, or other information storage device, transmission device, or display device of a computing platform.

[0093] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include multi-purpose microprocessor-based computer systems that access stored software that programs or configures the computing system from a general-purpose computing device to a dedicated computing device that implements one or more specific implementations of the subject matter of the present invention. Any suitable programming, scripting, or other type of language or combination of languages ​​may be used to implement the teachings contained herein in software for programming or configuring a computing device.

[0094] The specific implementation of the method disclosed herein can be performed in the operation of such a computing device. The order of the blocks presented in the above examples can be changed, for example, the blocks can be reordered, combined and / or divided into sub-blocks. Some blocks or processes can be executed in parallel.

[0095] The use of "suitable for" or "configured to" herein is meant to be open and inclusive language that does not exclude devices that are suitable for or configured to perform additional tasks or steps. In addition, the use of "based on" is meant to be open and inclusive, as a process, step, calculation, or other action "based on" one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated. The headings, lists, and numbers included herein are for ease of explanation only and are not intended to be limiting.

[0096] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0097] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "a" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their grouping.

[0098] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.

[0099] The foregoing description and summary of the present invention should be understood to be illustrative and exemplary in every aspect, rather than restrictive, and the scope of the present invention disclosed herein is determined not only by the detailed description of the exemplary specific implementations, but by the full breadth allowed by the patent law. It should be understood that the specific implementations shown and described herein are only illustrative of the principles of the present invention, and that various modifications can be implemented by those skilled in the art without departing from the scope and essence of the present invention.

Claims

1. A method, comprising: At an electronic device having a processor and a sensor: obtaining depth data of a physical environment at a sensor location using the sensor, the physical environment comprising a surface; determining a plurality of resolutions associated with the obtained depth data based on the depth data and the determined distance to the surface of the physical environment; Based on the determined plurality of resolutions, determining multi-resolution voxels based on a first group of voxels being positioned at a distance within a first distance range from the sensor location and a second group of voxels being positioned at a distance within a second distance range from the sensor location, the multi-resolution voxels comprising at least the first group of voxels comprising a first resolution and the second group of voxels comprising a second resolution different from the first resolution; A single mesh having different resolutions is generated by positioning vertices along lines connecting first voxels in the first set of voxels corresponding to the first resolution and second voxels in the second set of voxels corresponding to the second resolution, wherein the single mesh is generated directly from the multi-resolution voxels. 2 . The method of claim 1 , wherein the single grid is generated based on a first hash table and a second hash table.

3. The method according to claim 2, wherein: The first hash table is generated based on storing three-dimensional 3D positions of the first set of voxels, the three-dimensional 3D positions of the first set of voxels having signed distance values ​​representing distances from the surface of the physical environment based on the depth data; The second hash table is generated based on storing 3D positions of the second set of voxels, the 3D positions of the second set of voxels having signed distance values ​​representing distances from the surface of the physical environment based on the depth data. 4 . The method of claim 3 , wherein the signed distance values ​​comprise truncated signed distance field (TSDF) values ​​representing the voxel distance of each voxel from a closest one of the surfaces of the physical environment corresponding to the depth data. 5 . The method of claim 3 , wherein the first hash table and the second table use the 3D position as a key to generate a memory address for storing voxel information.

6. The method of claim 2, wherein the first hash table and the second hash table include memory addresses storing signed distance values.

7. The method of claim 2, wherein generating the single grid comprises: generating a line connecting points associated with the first set of voxels and the second set of voxels in both the first hash table and the second hash table; as well as Interpolation is performed along the lines to identify vertices of the mesh corresponding to the surface.

8. The method according to claim 1, further comprising: A determination is made whether to represent the 3D position as the first set of voxels or to represent the 3D position as the second set of voxels.

9. The method according to claim 1, further comprising: Whether to represent a 3D position as the first set of voxels or to represent a 3D position as the second set of voxels is determined based on determining noise in the depth data.

10. The method according to claim 1, further comprising: Whether to represent the 3D position as the first set of voxels or the 3D position as the second set of voxels is determined based on a distance of a surface closest to each voxel from a source of the depth data.

11. The method according to claim 1, further comprising: Whether to represent the 3D position as the first set of voxels or the 3D position as the second set of voxels is determined based on semantic labels identified for surfaces closest to each voxel. 12 . The method of claim 1 , wherein voxels of the first set of voxels have a first size and voxels of the second set of voxels have a second size, wherein the first size is greater than the second size.

13. The method of claim 1, wherein the depth data is obtained using one or more depth cameras.

14. The method of claim 1, wherein the depth data comprises pixel depth values ​​from a viewpoint and a sensor position.

15. A device, comprising: a non-transitory computer-readable storage medium; and one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium includes program instructions that, when executed on the one or more processors, cause the device to perform operations including: obtaining, using a sensor, depth data of a physical environment at a sensor location, the physical environment comprising a surface; determining a plurality of resolutions associated with the obtained depth data based on the depth data and the determined distance to the surface of the physical environment; Based on the determined plurality of resolutions, determining multi-resolution voxels based on a first group of voxels being positioned at a distance within a first distance range from the sensor location and a second group of voxels being positioned at a distance within a second distance range from the sensor location, the multi-resolution voxels comprising at least the first group of voxels comprising a first resolution and the second group of voxels comprising a second resolution different from the first resolution; A single mesh having different resolutions is generated by positioning vertices along lines connecting first voxels in the first set of voxels corresponding to the first resolution and second voxels in the second set of voxels corresponding to the second resolution, wherein the single mesh is generated directly from the multi-resolution voxels. The apparatus of claim 15 , wherein the single grid is generated based on a first hash table and a second hash table.

17. The apparatus of claim 16, wherein: The first hash table is generated based on storing three-dimensional 3D positions of the first set of voxels, the three-dimensional 3D positions of the first set of voxels having signed distance values ​​representing distances from the surface of the physical environment based on the depth data; The second hash table is generated based on storing 3D positions of the second set of voxels, the 3D positions of the second set of voxels having signed distance values ​​representing distances from the surface of the physical environment based on the depth data.

18. The apparatus of claim 17, wherein the signed distance values ​​comprise truncated signed distance field (TSDF) values ​​representing the voxel distance of each voxel from a closest one of the surfaces of the physical environment corresponding to the depth data.

19. The apparatus of claim 17, wherein the first hash table and the second table use the 3D position as a key to generate a memory address storing voxel information.

20. A non-transitory computer-readable storage medium storing program instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: obtaining, using a sensor, depth data of a physical environment at a sensor location, the physical environment comprising a surface; determining a plurality of resolutions associated with the obtained depth data based on the depth data and the determined distance to the surface of the physical environment; Based on the determined plurality of resolutions, determining multi-resolution voxels based on a first group of voxels being positioned at a distance within a first distance range from the sensor location and a second group of voxels being positioned at a distance within a second distance range from the sensor location, the multi-resolution voxels comprising at least the first group of voxels comprising a first resolution and the second group of voxels comprising a second resolution different from the first resolution; A single mesh having different resolutions is generated by positioning vertices along lines connecting first voxels in the first set of voxels corresponding to the first resolution and second voxels in the second set of voxels corresponding to the second resolution, wherein the single mesh is generated directly from the multi-resolution voxels.

Images