Multi-level block grid simplification

Abstract

A method for operating a computing system to generate a model of an environment represented by a mesh is provided. The method allows the 3D mesh to be updated to a client application in real time with low latency to support flight environment changes. The method provides a 3D mesh that adapts to different levels of simplification requested by various client applications. The method provides local updates, for example, updating the portion of the mesh that has changed since the last update. The method also provides a 3D mesh with a planarized surface to support robust physical simulation. The method includes segmenting the 3D mesh into mesh blocks. The method also includes performing multi-level simplification on selected mesh blocks. The multi-level simplification includes a pre-simplification operation, a planarization operation, and a post-simplification operation.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of U.S. Provisional Patent Application Serial No. 62 / 610,000, filed on December 22, 2017, entitled “VIEWPOINT DEPENDENT BRICK SELECTION FOR FAST VOLUMMETRIC RESTRUCTION,” the entire contents of which are hereby incorporated by reference herein. This application also claims priority to and the benefit of U.S. Provisional Patent Application Serial No. 62 / 609,990, filed on December 22, 2017, entitled “MULTI-STAGE BLOCK MESH SIMPLIFICATION FOR MULTIPLE Targets,” the entire contents of which are hereby incorporated by reference herein. This application also claims priority to and the benefit of U.S. Provisional Patent Application Serial No. 62 / 609,962, filed on December 22, 2017, entitled “CACHING AND UPDATING OF DENSE 3D RECONSTRUCTION DATA ON MOBILE DEVICES,” the entire contents of which are hereby incorporated by reference herein. This application also claims priority to and the benefit of U.S. Provisional Patent Application Serial No. 62 / 609,979, filed on December 22, 2017, entitled “METHOD OF OCCLUSION RENDERING USING RAYCAST AND LIVE DEPTH,” the entire contents of which are hereby incorporated by reference herein. This application also claims priority to and the benefit of U.S. Provisional Patent Application Serial No. 62 / 702,761, filed on July 24, 2018, entitled “VIEWPOINT DEPENDENT BRICK SELECTION FOR FASTVOLUMMETRIC RESTRUCTION,” the entire contents of which are hereby incorporated by reference herein. This application also claims priority to and the benefit of U.S. Provisional Patent Application Serial No. 62 / 702,740, filed on July 24, 2018, entitled “CACHING AND UPDATING OF DENSE 3D RECONSTRUCTION DATA,” the entire contents of which are hereby incorporated by reference herein.This application also claims priority to and the benefit of U.S. Provisional Patent Application Serial No. 62 / 702,736, filed on July 24, 2018, entitled “METHOD OF OCCLUSION RENDERING USING RAYCAST AND LIVE DEPTH,” the entire contents of which are hereby incorporated by reference. Technical Field

[0003] The present application generally relates to a cross-reality system for rendering a scene using 3D world reconstruction. Background Art

[0004] A computer can control a human user interface to create an X-reality (XR or cross-reality) environment in which part or all of the XR environment perceived by the user is generated by the computer. These XR environments can be virtual reality (VR), augmented reality (AR), and mixed reality (MR) environments, where part or all of the XR environment can be generated by a computer in part using data describing the environment. The data can describe, for example, a virtual object that can be rendered in a way that the user senses or perceives it as part of the physical world and can interact with the virtual object. Since the data is rendered and presented by a user interface device (e.g., a head-mounted display device), the user may experience these virtual objects. The data can be displayed to the user, or the audio played to the user can be controlled, or a tactile (or haptic) interface can be controlled, so that the user can experience the touch sensation of the user sensing or perceiving the virtual object.

[0005] XR systems can be useful for many applications, spanning areas such as scientific visualization, medical training, engineering design and prototyping, telemanipulation and telepresence, and personal entertainment. Compared to VR, AR and MR include one or more virtual objects that are related to real objects in the physical world. The experience of virtual objects interacting with real objects greatly enhances the user's enjoyment of using XR systems and opens the door to a variety of applications that present realistic and easy-to-understand information about how to change the physical world. Summary of the invention

[0006] Aspects of the present application relate to methods and apparatus for rapidly generating environments containing computer-generated objects.The techniques described herein may be used together, separately, or in any suitable combination.

[0007] Some embodiments relate to a method of operating a computing system to generate a model of an environment represented by a grid. The method includes: segmenting the grid into a plurality of first grid blocks; performing a first simplification operation on each of the plurality of selected first grid blocks to generate a plurality of second grid blocks; performing an area-based operation on each of the second grid blocks to generate a plurality of third grid blocks; and performing a second simplification operation on each of the third grid blocks to generate a fourth grid block.

[0008] In some embodiments, the area-based operation comprises a planarization operation.

[0009] In some embodiments, the first grid tile comprises a portion of a representation of the environment having a complexity metric of a first value.The fourth grid tile comprises the portion of the representation of the environment having the complexity metric of a fourth value, the fourth value being less than the first value.

[0010] In some embodiments, the complexity metric indicates the number of triangles in the corresponding mesh block.

[0011] In some embodiments, the second grid tile includes the portion of the representation of the environment for which the complexity metric is a second value. The third grid tile includes the portion of the representation of the environment for which the complexity metric is a third value. The percentage decrease between the third value and the fourth value is greater than the percentage decrease between the first value and the second value.

[0012] In some embodiments, the percentage reduction between the third value and the fourth value is at least twice as great as the percentage reduction between the first value and the second value.

[0013] In some embodiments, the fourth mesh block is generated by a meshing service using an XR platform. The method also includes: receiving a value representing mesh complexity as input to the meshing service; and controlling the percentage reduction between the third value and the fourth value and the percentage reduction between the first value and the second value based on the value representing mesh complexity.

[0014] In some embodiments, the value is provided by an application requesting a mesh from the meshing service.

[0015] In some embodiments, the first simplification operation and the second simplification operation are performed using the same simplification algorithm.

[0016] In some embodiments, the first simplification operation and the second simplification operation are performed using a triangle reduction algorithm.

[0017] In some embodiments, performing the first simplification operation on the plurality of selected first grid blocks includes: processing the selected first grid blocks in parallel.

[0018] In some embodiments, the plurality of fourth grid blocks are combined with a plurality of unselected second grid blocks into a new grid of the environment.

[0019] In some embodiments, performing the region-based operation includes: detecting plane regions in the selected plurality of second grid blocks by a region growing algorithm; and simplifying the detected plane regions in the selected plurality of second grid blocks.

[0020] In some embodiments, the detected planar regions in the selected plurality of second mesh tiles are simplified based on the target triangle count.

[0021] In some embodiments, the region-based operation further comprises: planarizing the selected plurality of second grid blocks by projecting the detected triangles of the planar region in the selected plurality of second grid blocks onto corresponding planes.

[0022] In some embodiments, the method further comprises adjusting plane normals of the detected plane regions in the selected plurality of second grid blocks to be substantially perpendicular and parallel to the corresponding plane before simplifying the detected plane regions in the selected plurality of second grid blocks.

[0023] In some embodiments, the model of the environment is three-dimensional (3D). Each first grid block corresponds to a cube in the 3D model of the environment.

[0024] In some embodiments, the first simplification operation is performed in parallel on each of the plurality of selected first grid blocks.

[0025] In some embodiments, the region-based operation is performed in parallel on each of the second grid blocks.

[0026] In some embodiments, the second simplification operation is performed in parallel on each of the third grid blocks.

[0027] The foregoing summary is provided by way of illustration and is not intended to be limiting. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component shown in various figures is represented by a like numeral. For clarity, not every component will be labeled in every figure. In the drawings:

[0029] Figure 1is a schematic diagram illustrating an example of a simplified augmented reality (AR) scene according to some embodiments.

[0030] Figure 2 is a sketch of an exemplary simplified AR scene illustrating an exemplary world reconstruction use case including visual occlusion, physics-based interaction, and environmental reasoning in accordance with some embodiments.

[0031] Figure 3 is a schematic diagram illustrating data flow in an AR system configured to provide an experience of AR content interacting with the physical world according to some embodiments.

[0032] Figure 4 is a schematic diagram illustrating an example of an AR display system according to some embodiments.

[0033] Figure 5A is a diagram illustrating an AR display system that renders AR content while worn by a user as the user moves through a physical world environment, according to some embodiments.

[0034] Figure 5B is a schematic diagram illustrating a viewing optics assembly and accompanying components according to some embodiments.

[0035] Figure 6 is a schematic diagram illustrating an AR system using a world reconstruction system in accordance with some embodiments.

[0036] Fig. 7A is a schematic diagram illustrating a 3D space discretized into voxels according to some embodiments.

[0037] Figure 7B is a schematic diagram illustrating a reconstruction range relative to a single viewpoint according to some embodiments.

[0038] Figure 7C is a schematic diagram illustrating the perception range relative to the reconstruction range at a single location according to some embodiments.

[0039] Figures 8A to 8F is a schematic diagram showing reconstruction of a surface in the physical world as a voxel model by an image sensor viewing the surface from multiple positions and viewpoints in accordance with some embodiments.

[0040] Fig. 9 is a schematic diagram showing a scene represented by bricks comprising voxels, a surface in the scene, and a depth sensor capturing the surface in a depth image in accordance with some embodiments.

[0041] Fig. 10A is a schematic diagram showing a 3D space represented by eight bricks.

[0042] Fig. 10B It is shown Fig. 10A Schematic diagram of the voxel grid in the brick.

[0043] Fig.11 is a schematic diagram illustrating a volume representation hierarchy according to some embodiments.

[0044] Fig.12 is a flow chart illustrating a method of operating a computing system to generate a 3D reconstruction of a scene in accordance with some embodiments.

[0045] Fig.13 is a diagram showing a method for Fig.12 Flowchart of a method for picking a portion of a plurality of bricks using a camera frustum of a depth sensor.

[0046] Fig.14 is a diagram showing the process according to some embodiments Fig.13 Flowchart of the method in which the camera frustum is tested.

[0047] Fig.15 is a diagram showing a method for Fig.12 Flowchart of a method for selecting a portion of a first plurality of bricks from a depth image.

[0048] Fig.16 is a diagram showing the process according to some embodiments Fig.15 Flowchart of a method in which a first depth image in is subjected to testing.

[0049] Fig.17 is a diagram showing the process according to some embodiments Fig.15 Flow chart of a method for accepting a second depth image for testing.

[0050] Fig.18 According to some embodiments, Fig.17 The table used by the method to classify all pixels in the rectangle using the minimum brick value (bmin) and the maximum brick value (bmax).

[0051] Figures 19A to 19F is a schematic diagram illustrating picking bricks for a camera frustum in accordance with some embodiments.

[0052] Figures 20A to 20B is a schematic diagram illustrating picking bricks for a depth image including a surface according to some embodiments.

[0053] Fig.21 is a schematic diagram illustrating a plane extraction system according to some embodiments.

[0054] Fig. 22 is a diagram showing details about plane extraction according to some embodiments Fig.21Schematic diagram of the various parts of the plane extraction system.

[0055] Fig.23 is a schematic diagram showing a scene represented by bricks comprising voxels and exemplary plane data in the scene according to some embodiments.

[0056] Fig.24 is a diagram showing some embodiments of the present invention. Fig.21 Schematic diagram of a flat data repository.

[0057] Fig.25 is an illustration of how, according to some embodiments, a plane query is sent to Fig.21 Schematic diagram of plane geometry extraction when using a plane data repository.

[0058] Fig.26A is a diagram showing the generation of Fig.25 Schematic diagram of the plane covering points.

[0059] Fig.26B is a schematic diagram illustrating various exemplary planar geometric representations that may be extracted from an exemplary rasterized planar mask according to some embodiments.

[0060] Fig. 27 A grid for a scene is shown in accordance with some embodiments.

[0061] Fig.28A An external rectangular plane is shown according to some embodiments. Fig. 27 scene.

[0062] Fig.28B An example of a schematic diagram of a circuit diagram represented by an inner rectangular plane according to some embodiments Fig. 27 scene.

[0063] Fig.28C FIG. 1 shows a polygonal plane representation according to some embodiments. Fig. 27 scene.

[0064] Fig.29 According to some embodiments, Fig. 27 The mesh shown in the figure is flattened with a denoised mesh. Fig. 27 scene.

[0065] Fig.30 is a flow chart illustrating a method of generating a model of an environment represented by a mesh according to some embodiments.

[0066] Fig.31 is a schematic diagram showing a 2D representation of a portion of the physical world through four tiles, according to some embodiments.

[0067] Figures 32A-32D is a schematic diagram illustrating mesh evolution of an exemplary mesh block during multi-level simplification according to some embodiments.

[0068] Fig.33A and 33B Representations of the same environment without simplification and after simplification by triangle reduction are shown, respectively.

[0069] Fig.34A and 34B Close-up representations of the same environment without simplification and after simplification by triangle reduction are shown, respectively.

[0070] Fig.35A and 35B Representations of the same environment without planarization and with planarization are shown respectively.

[0071] Fig.36A and 36B Representations of the same environment without simplification and simplified by removing disconnected components are shown respectively.

[0072] Fig.37 is a schematic diagram illustrating an electronic system that enables an interactive X-reality environment for multiple users in accordance with some embodiments.

[0073] Fig.38 is a diagram showing some embodiments of the present invention. Fig.37 Schematic diagram of the interaction of the components of an electronic system.

[0074] Fig.39 is a diagram illustrating the operation according to some embodiments Fig.37 A flow chart of a method for an electronic system in FIG.

[0075] Fig.40 is a diagram illustrating capturing 3D information about objects in the physical world and representing the physical world as Fig.39 Flowchart of a method for 3D reconstruction of a block in FIG.

[0076] Fig.41 is a diagram showing a selection representation according to some embodiments Fig.39 Flow chart of a method for retrieving versions of blocks of a subset of blocks in a block.

[0077] Fig.42 is a diagram illustrating the operation according to some embodiments Fig.37 A flow chart of a method for an electronic system in FIG.

[0078] Fig.43A is a simplified schematic diagram illustrating updates detected in a portion of the physical world represented by a grid of tiles, according to some embodiments.

[0079] Fig.43B is a simplified schematic diagram illustrating a grid block according to some embodiments.

[0080] Fig.43C is a simplified schematic diagram illustrating a crack at the edge of two adjacent mesh blocks according to some embodiments.

[0081] Fig.43D FIG. 2 is a diagram showing a method of masking a grid area by implementing a grid side edge that overlaps with adjacent grid areas according to some embodiments. Fig.43C Simplified schematic diagram of the crack in .

[0082] Fig.44 is a schematic diagram showing a 2D representation of a portion of the physical world through four tiles, according to some embodiments.

[0083] Fig.45 is a schematic diagram showing a 3D representation of a portion of the physical world through eight tiles, according to some embodiments.

[0084] Fig.46 is an illustration of the updating of Fig.45 A schematic diagram of a 3D representation of a portion of the physical world obtained by a 3D representation.

[0085] Fig.47 is a diagram illustrating an example of an augmented world viewable by first and second users wearing an AR display system according to some embodiments.

[0086] Fig.48 is an illustration of updating the Fig.47 Schematic diagram of an example of an augmented world obtained by an augmented world.

[0087] Fig.49 is a schematic diagram illustrating an occlusion rendering system according to some embodiments.

[0088] Fig.50 is a schematic diagram showing a depth image with holes.

[0089] Fig.51 is a flow chart illustrating a method of performing occlusion rendering in an augmented reality environment according to some embodiments.

[0090] Fig.52 is a diagram showing the Fig.51 A flowchart of details of generating surface information from depth information captured by a depth sensor worn by a user.

[0091] Fig.53 is a diagram illustrating filtering depth information to generate Fig.52 Flowchart of the details of the depth map in .

[0092] Fig.54A It is a sketch of the area imaged from a first-person perspective (point of view) using a depth camera to identify voxels occupied by the surface and empty voxels.

[0093] Fig.54B It is a sketch imaged from multiple views with a depth camera to identify voxels occupied by the surface and empty and to indicate areas of "holes" for which no volume information is available since the voxels in the area of ​​the "holes" were not imaged with the depth camera. DETAILED DESCRIPTION

[0094] Methods and apparatus for creating and using three-dimensional (3D) world reconstructions in augmented reality (AR), mixed reality (MR), or virtual reality (VR) systems are described herein. In order to provide a realistic AR / MR / VR experience to a user, the AR / MR / VR system must understand the user's physical environment in order to correctly associate the positions of virtual objects with real objects. A world reconstruction can be constructed based on image and depth information about that physical environment, which is collected by sensors that are part of the AR / MR / VR system. The world reconstruction can then be used by any of a number of components of such a system. For example, the world reconstruction can be used by components that perform visual occlusion processing, compute physics-based interactions, or perform environmental reasoning.

[0095] Occlusion processing identifies portions of a virtual object that should not be rendered and / or displayed to a user because there is an object in the physical world that blocks the user from viewing the location where the virtual object would be perceived by the user. Physics-based interactions are calculated to determine where or how virtual objects appear to the user. For example, a virtual object may be rendered to appear to be resting on a physical object, moving through empty space, or colliding with the surface of a physical object. This world reconstruction provides a model from which information about objects in the physical world can be obtained to make such calculations.

[0096] Environmental reasoning can also use world reconstruction in the process of generating information that can be used to calculate how to render virtual objects. For example, environmental reasoning can involve identifying transparent surfaces by recognizing whether the transparent surface is a window pane or a glass tabletop. Based on this identification, areas containing physical objects can be classified as not occluding virtual objects, but can be classified as interacting with virtual objects. Environmental reasoning can also generate information used in other ways, such as identifying fixed objects that can be tracked relative to the user's field of view to calculate the movement of the user's field of view.

[0097] However, there are significant challenges in providing such a system. Computing the world reconstruction may require a lot of processing. In addition, the AR / MR / VR system must correctly know how to position virtual objects relative to the user's head, body, etc. As the user's position relative to the physical environment changes, relevant parts of the physical world may also change, which may require further processing. Moreover, it is often necessary to update the 3D reconstruction data as objects move in the physical world (for example, a cup moves on a table). Updates to the data representing the environment that the user is experiencing must be performed quickly without using a lot of the computing resources of the computer generating the AR / MR / VR environment, because other functions cannot be performed while the world reconstruction is being performed. In addition, processing the reconstruction data through components that "consume" the data will increase the demand on computer resources.

[0098] Known AR / MR / VR systems require high computing power (e.g., GPU) to run real-time world reconstruction only within a predefined reconstruction volume (e.g., a predefined voxel grid). The inventors have recognized and appreciated techniques for operating AR / MR / VR systems to provide accurate 3D reconstruction data in real-time with little use of computing resources (e.g., computing power (e.g., a single ARM core), memory (e.g., less than 1GB), and network bandwidth (e.g., less than 100Mbps)). These techniques relate to reducing the processing required to generate and maintain world reconstructions, and providing and consuming data with low computing overhead.

[0099] These techniques may include, for example, reducing the amount of data processed when updating a world reconstruction by identifying portions of sensor data that are readily available for use in creating or updating a world reconstruction. For example, sensor data may be selected based on whether the sensor data represents a portion of the physical world that is likely to be close to a surface of an object represented in the world reconstruction.

[0100] In some embodiments, computing resources can be reduced by simplifying the data representing the world reconstruction. A simpler representation can reduce resources used for processing, storage, and / or management of the data and its use.

[0101] In some embodiments, the use of computing resources can be reduced by representing the physical world in tiles that can be stored and retrieved individually, but combined in a way that provides a realistic representation of the physical world. These tiles can be managed in memory to limit computing resources, and in some embodiments, tiles can be shared between AR / MR / VR systems operating in the same physical space, so that each AR / MR / VR system performs less processing to build a world reconstruction.

[0102] In some embodiments, the use of computing resources can be reduced by selecting from different representations of the physical world when accessing information about the physical world. World reconstruction may, for example, include information about the physical world captured from different sensors and / or stored in different formats. The simplest data consumed or provided may be provided to components that use world reconstruction to render virtual objects. If simpler data is not available, data acquired using other sensors may be accessed, which may generate a higher computational load. As an example, world reconstruction may include a depth map collected using a depth sensor and a more cumbersome representation of the 3D world, such as a grid that may be stored as a calculation based on image information. Information about the physical world may be provided to components that perform occlusion processing based on a depth map (where available). In the case of holes in the depth map, information to fill these holes can be extracted from the grid. In some embodiments, the depth map may be "live," representing the physical world captured by the depth sensor when the data is accessed.

[0103] The techniques described herein may be used with or on their own in conjunction with various types of devices and in various types of scenarios, including wearable or portable devices with limited computing resources that provide augmented reality scenarios.

[0104] AR System Overview

[0105] Figure 1-2 Such a scenario is shown.For illustration purposes, an AR system is used as an example of an XR system. Figure 3 -8 shows an exemplary AR system that includes one or more processors, memory, sensors, and a user interface that can operate according to the techniques described herein.

[0106] refer to Figure 1 , depicting an AR scene 4 in which a user of the AR technology sees a physical-world park-like setting 6, characterized by people, trees, buildings in the background, and a concrete platform 8. In addition to these items, the user of the AR technology also believes that they "see" a robotic statue 10 standing on the physical-world concrete platform 8, and an anthropomorphic, cartoon-like avatar character 2 that looks like a bumblebee flying, even though these elements (e.g., avatar character 2 and robotic statue 10) do not exist in the physical world. Due to the extreme complexity of human visual perception and the nervous system, it is challenging to produce an AR technology that promotes a comfortable, natural-feeling, rich presentation of virtual image elements among other virtual or physical-world image elements.

[0107] Such an AR scene can be implemented by a system including a world reconstruction component that can build and update a representation of the physical world surface around the user. This representation can be used for occlusion rendering, placement of virtual objects in physics-based interactions, and for virtual character path planning and navigation, or for other operations in which information about the physical world is used. Figure 2 Another example of an AR scene 200 is depicted, illustrating an exemplary world reconstruction use case, including visual occlusion 202 , physics-based interaction 204 , and environmental reasoning 206 , in accordance with some embodiments.

[0108] The exemplary scene 200 is a living room with a wall, a bookshelf on one side of the wall, a floor lamp at the corner of the room, a floor, a sofa, and a coffee table on the floor. In addition to these physical items, the user of the AR technology also perceives virtual objects, such as an image on the wall behind the sofa, a bird flying over the door, a deer peeking from the bookshelf, and a decoration in the form of a windmill placed on the coffee table. For the image on the wall, the AR technology requires not only information about the surface of the wall, but also information about objects and surfaces in the room that are blocking the image (such as the shape of the lamp) to correctly render the virtual object. For the flying bird, the AR technology needs information about all objects and surfaces around the room to render the bird with realistic physical effects to avoid objects and surfaces or bounce when the bird collides. For the deer, the AR technology needs information about the surface (e.g., the floor or coffee table) to calculate the placement of the deer. For the windmill, the system can identify it as an object separated from the table, and it can be inferred that it is movable, while the corner of the bookshelf or the corner of the wall can be inferred to be fixed. Such a distinction can be used to reason about which parts of the scene are used or updated in each of the various operations.

[0109] The scene can be presented to the user through a system including multiple components, including a user interface that can stimulate one or more user senses (including visual sound and / or touch). In addition, the system may include one or more sensors that can measure parameters of the physical part of the scene (including the position and / or movement of the user within the physical part of the scene). In addition, the system may include one or more computing devices with associated computer hardware (such as memory). These components can be integrated into a single device or distributed among multiple interconnected devices. In some embodiments, some or all of these components can be integrated into a wearable device.

[0110] Figure 3An AR system 302 is depicted that is configured to provide an experience in which AR content interacts with a physical world 306 in accordance with some embodiments. The AR system 302 may include a display 308. In the illustrated embodiment, the display 308 may be worn by a user as part of a headset so that the user may wear the display over their eyes like a pair of goggles or glasses. At least a portion of the display may be transparent so that the user may observe a see-through reality 310. The see-through reality 310 may correspond to a portion of the physical world 306 that is within a current viewpoint of the AR system 302, which may correspond to a viewpoint of the user if the user is wearing a headset that incorporates a display and sensors of the AR system to obtain information about the physical world.

[0111] AR content may also be presented on display 308, overlaid on see-through reality 310. To provide accurate interaction between AR content and see-through reality 310 on display 308, AR system 302 may include sensors 322 configured to capture information about physical world 306.

[0112] Sensors 322 may include one or more depth sensors that output depth maps 312. Each depth map 312 may have a plurality of pixels, each of which may represent a distance to a surface in the physical world 306 in a particular direction relative to the depth sensor. Raw depth data may come from the depth sensor to create a depth map. Such a depth map may be updated as fast as the depth sensor can form a new image, which may be hundreds or thousands of times per second. However, the data may be noisy and incomplete, and have holes in the depth map shown, which are displayed as black pixels.

[0113] The system may include other sensors, such as image sensors. Image sensors may acquire information that may be processed in other ways to represent the physical world. For example, images may be processed in world reconstruction component 316 to create a mesh that represents connected parts of objects in the physical world. Metadata about such objects, including, for example, color and surface texture, may similarly be acquired by sensors and stored as part of the world reconstruction.

[0114] The system may also obtain information about the user's head pose relative to the physical world. In some embodiments, the sensor 310 may include an inertial measurement unit that can be used to calculate and / or determine a head pose 314. The head pose 314 for the depth map may indicate the current viewpoint at which the sensor is capturing the depth map with, for example, six degrees of freedom (6DoF), but the head pose 314 may be used for other purposes, such as correlating image information with a specific part of the physical world or correlating the position of a display worn on the user's head with the physical world. In some embodiments, the head pose information may be derived in other ways than an IMU (e.g., analyzing objects in the image).

[0115] A world reconstruction component 316 may receive the depth map 312 and head pose 314 from the sensor, as well as any other data, and integrate the data into a reconstruction 318, which may at least appear to be a single, combined reconstruction. The reconstruction 318 may be more complete and less noisy than the sensor data. The world reconstruction component 316 may update the reconstruction 318 using spatial and temporal averaging of sensor data from multiple viewpoints over time.

[0116] Reconstruction 318 may include a representation of the physical world in one or more data formats including, for example, voxels, meshes, planes, etc. Different formats may represent alternative representations of the same portion of the physical world, or may represent different portions of the physical world. In the example shown, on the left side of reconstruction 318, the portion of the physical world is presented as a global surface; on the right side of reconstruction 318, the portion of the physical world is presented as a mesh.

[0117] Reconstruction 318 can be used for AR functions, such as generating a surface representation of the physical world for occlusion handling or physics-based processing. The surface representation can change as the user moves or objects in the physical world change. Aspects of reconstruction 318 can be used, for example, by a component 320 that generates a changing global surface representation in world coordinates, which can be used by other components.

[0118] AR content can be generated based on this information, for example, by an AR application 304. The AR application 304 can be, for example, a game program that performs one or more functions based on information about the physical world (e.g., visual occlusion, physics-based interaction, and environmental reasoning). It can perform these functions by querying data in different formats from the reconstruction 318 generated by the world reconstruction component 316. In some embodiments, component 320 can be configured to output updates when the representation in the area of ​​interest of the physical world changes. The area of ​​interest can be set to, for example, a portion of the physical world that is close to the user of the system, such as a portion within the user's field of view, or is projected (predicted / determined) as entering the user's field of view.

[0119] The AR application 304 can use this information to generate and update the AR content. The virtual portion of the AR content can be presented on the display 308 in combination with the see-through reality 310, thereby creating a realistic user experience.

[0120] In some embodiments, an AR experience may be provided to a user via a wearable display system. Figure 4 An example of a wearable display system 80 (hereinafter "system 80") is shown. System 80 includes a head mounted display device 62 (hereinafter "display device 62") and various mechanical and electronic modules and systems to support the functionality of display device 62. Display device 62 may be coupled to a frame 64 that may be worn by a display system user or viewer 60 (hereinafter "user 60") and configured to position display device 62 in front of the eyes of user 60. According to various embodiments, display device 62 may be a sequential display. Display device 62 may be monocular or binocular. In some embodiments, display device 62 may be Figure 3 An example of display 308 in FIG.

[0121] In some embodiments, a speaker 66 is coupled to the frame 64 and positioned near the ear canal of the user 60. In some embodiments, another speaker, not shown, is positioned near the other ear canal of the user 60 to provide stereo / shapeable sound control. The display device 62 is operably coupled to a local data processing module 70, such as by a wired lead or a wireless connection 68, which can be mounted in a variety of configurations, such as fixedly attached to the frame 64, fixedly attached to a helmet or hat worn by the user 60, embedded in headphones, or otherwise removably connected to the user 60 (e.g., in a backpack configuration, in a belt-coupled configuration).

[0122] The local data processing module 70 may include a processor and digital memory, such as non-volatile memory (e.g., flash memory), both of which may be used to assist in the processing, caching, and storage of data. The data includes a) captured from a sensor (which may, for example, be operably coupled to the frame 64) or otherwise attached to the user 60 (e.g., an image capture device (e.g., a camera), a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a radio device, and / or a gyroscope), and / or b) data acquired using a remote processing module 72 and / or a remote data repository 74, possibly for delivery to the display device 62 after such processing or retrieval. The local data processing module 70 may be operably coupled to the remote processing module 72 and the remote data repository 74, respectively, via communication links 76, 78 (e.g., via a wired or wireless communication link), such that these remote modules 72, 74 are operably coupled to each other and may be used as resources for the local processing and data module 70. In some embodiments, Figure 3 The world reconstruction component 316 in may be implemented at least in part in the local data processing module 70. For example, the local data processing module 70 may be configured to execute computer executable instructions to generate a physical world representation based at least in part on at least a portion of the data.

[0123] In some embodiments, the local data processing module 70 may include one or more processors (e.g., a graphics processing unit (GPU)) configured to analyze and process data and / or image information. In some embodiments, the local data processing module 70 may include a single processor (e.g., a single-core or multi-core ARM processor), which will limit the computational budget of the module 70 but enable smaller devices. In some embodiments, the world reconstruction component 316 may use a computational budget less than a single ARM core to generate a physical world representation in real time over a non-predefined space, so that the remaining computational budget of the single ARM core can be accessed for other purposes, such as extracting a mesh.

[0124] In some embodiments, the remote data repository 74 may include a digital data storage facility that may be used via the Internet or other network configuration in a "cloud" resource configuration. In some embodiments, all data is stored and all computations are performed in the local data processing module 70, allowing for fully autonomous use from the remote module. World reconstructions, for example, may be stored in whole or in part in this repository 74.

[0125] In some embodiments, the local data processing module 70 is operably coupled to a battery 82. In some embodiments, the battery 82 is a removable power source, such as above a counter battery. In other embodiments, the battery 82 is a lithium-ion battery. In some embodiments, the battery 82 includes both an internal lithium-ion battery that can be charged by the user 60 during the non-operating time of the system 80 and a removable battery, so that the user 60 can operate the system 80 for a longer period of time without having to connect to a power source to charge the lithium-ion battery or having to shut down the system 80 to replace the battery.

[0126] Figure 5A A user 30 is shown wearing an AR display system that renders AR content as the user 30 moves in a physical world environment 32 (hereinafter referred to as "environment 32"). The user 30 places the AR display system at a location 34, and the AR display system records passable world environment information relative to the location 34 (e.g., digital representations of real objects in the physical world that can be stored and updated as the real objects in the physical world change), such as gestures associated with mapped features or directional audio input. The location 34 is aggregated to a data input 36 and processed by at least a passable world module 38, which can be performed, for example, by Figure 3 The traversable world module 38 may include a world reconstruction component 316.

[0127] The traversable world module 38 determines where and how the AR content 40 can be placed in the physical world as determined from the data input 36. By presenting a representation of the physical world and the AR content via a user interface, the AR content is "placed" in the physical world, where the AR content is rendered as if it is interacting with objects in the physical world, and the objects in the physical world are rendered as if the AR content is blocking the user's view of these objects when appropriate. In some embodiments, the AR content can be placed by appropriately selecting a portion of a fixed element 42 (e.g., a table) from a reconstruction (e.g., reconstruction 318) to determine the shape and position of the AR content 40. As an example, the fixed element can be a table, and the virtual content can be placed so that it appears to be on the table. In some embodiments, the AR content can be placed within a structure in the field of view 44, which can be the current field of view or an estimated future field of view. In some embodiments, the AR content can be placed relative to a mapped mesh model 46 of the physical world.

[0128] As depicted, fixed element 42 serves as a proxy for any fixed element within the physical world, which can be stored in traversable world module 38, so that user 30 can perceive content on fixed element 42 without the system having to map to fixed element 42 every time user 30 sees it. Fixed element 42 can therefore be a mapped mesh model from a previous modeling session, or determined based on an individual user, but all stored on traversable world module 38 for future reference by multiple users. Thus, traversable world module 38 can recognize environment 32 from a previously mapped environment and display AR content without requiring the user 30's device to first map environment 32, thereby saving computational processes and cycles and avoiding delays in any rendered AR content.

[0129] A mapped mesh model 46 of the physical world may be created by the AR display system, and appropriate surfaces and metrics to be used for interaction and display of AR content 40 may be mapped and stored in the traversable world module 38 for future retrieval by the user 30 or other users without the need for remapping or modeling. In some embodiments, the data input 36 is an input such as geographic location, user identification, and current activity to indicate to the traversable world module 38 which fixed element 42 of one or more fixed elements is available, which AR content 40 was last placed on the fixed element 42, and whether to display the same content (the AR content is "persistent" content regardless of whether the user is viewing a particular traversable world model or not).

[0130] Figure 5B A schematic diagram of the viewing optical assembly 48 and accompanying components is shown. In some embodiments, facing the user's eyes 49, two eye-tracking cameras 50 detect metrics of the user's eyes 49, such as eye shape, eyelid occlusion, pupil direction, and glint on the user's eyes 49. In some embodiments, a depth sensor 51 (e.g., a time-of-flight sensor) transmits a relay signal to the world to determine the distance to a given object. In some embodiments, a world camera 52 records a greater-than-peripheral view to map the environment 32 and detect inputs that can affect AR content. Camera 53 can further capture specific timestamps of images of the physical world within the user's field of view. Each of the world camera 52, camera 53, and depth sensor 51 has a corresponding field of view 54, 55, and 56 to obtain information from, for example, the user's surrounding environment 32. Figure 5A The physical world scene of the physical world environment 32 is shown collecting data and recording the physical world scene.

[0131] Inertial measurement unit 57 can determine the motion and orientation of viewing optical assembly 48. In some embodiments, each component is operably coupled to at least one other component. For example, depth sensor 51 is operably coupled to eye tracking camera 50 as a confirmation of the measured accommodation relative to the actual distance at which the user's eye 49 is looking.

[0132] Information from these sensors in the viewing optical assembly 48 can be coupled to one or more processors in the system. The processor can generate data that can be rendered so that the user perceives virtual content that interacts with objects in the physical world. The rendering can be implemented in any suitable manner, including generating image data that depicts physical and virtual objects. In other embodiments, physical and virtual content can be depicted in a scene by modulating the opacity of a display device through which the user views the physical world. The opacity can be controlled to create the appearance of virtual objects and also prevent the user from seeing objects in the physical world that are obscured by virtual objects. Regardless of how the content is presented to the user, a model of the physical world is required so that the characteristics of virtual objects that can be affected by physical objects, including the shape, position, motion, and visibility of virtual objects, can be correctly calculated. In some embodiments, the model may include a reconstruction of the physical world, such as reconstruction 318.

[0133] The model may be created from data collected from sensors on the user's wearable device. However, in some embodiments, the model may be created from data collected from multiple users, which may be aggregated in a computing device remote from all users (and may be in the "cloud").

[0134] can be at least partially achieved through a world reconstruction system (e.g. Figure 6 Described in more detail in Figure 3 The world reconstruction component 316 of the sensor may be used to create a model. The world reconstruction component 316 may include a perception module 160, which may generate, update, and store a representation of a portion of the physical world. In some embodiments, the perception module 160 may represent the portion of the physical world within the reconstruction range of the sensor as a plurality of voxels. Each voxel may correspond to a 3D cube of a predetermined volume in the physical world and include surface information indicating whether a surface exists in the volume represented by the voxel. Voxels may be assigned values ​​indicating whether their corresponding volumes have been determined to include the surface of a physical object, determined to be empty, or have not yet been measured with a sensor and therefore their values ​​are unknown. It should be understood that there is no need to explicitly store values ​​indicating voxels determined to be empty or unknown, as the values ​​of voxels may be stored in computer memory in any suitable manner, including not storing information about voxels determined to be empty or unknown.

[0135] Fig. 7A An example of a 3D space 100 discretized into voxels 102 is depicted. In some embodiments, perception module 160 may determine an object of interest and set the volume of the voxel so as to capture features of the object of interest and avoid redundant information. For example, perception module 160 may be configured to identify larger objects and surfaces, such as walls, ceilings, floors, and large furniture. Thus, the volume of the voxel may be set to a relatively large size, such as 4 cm 3 cube.

[0136] The reconstruction of the physical world including voxels may be referred to as a volume model. Information used to create the volume model may be created over time as the sensor moves through the physical world. This movement may occur when a user of a wearable device including the sensor moves around. Fig. 8A -F depicts an example of reconstructing the physical world as a volume model. In the example shown, the physical world consists of Fig. 8A A portion 180 of the surface is shown in FIG. Fig. 8A In the first position, the sensor 182 may have a field of view 184 within which the portion 180 of the surface is visible.

[0137] Sensor 182 may be of any suitable type, such as a depth sensor. However, depth data may be obtained from an image sensor or otherwise. Perception module 160 may receive data from sensor 182 and then, as described in Figure 8B The values ​​of a plurality of voxels 186 are shown set to represent a portion 180 of a surface visible in a field of view 184 by a sensor 182 .

[0138] exist Figure 8C In the embodiment, the sensor 182 can be moved to a second position and has a field of view 188. Fig.8D As shown, another set of voxels becomes visible, and the values ​​of these voxels can be set to indicate the location of the portion of the surface that has entered the field of view 188 of the sensor 182. The values ​​of these voxels can be added to the volume model for the surface.

[0139] exist Fig. 8E In the example shown, the sensor 182 can be further moved to a third position and has a field of view 190. In the example shown, an additional portion of the surface becomes visible in the field of view 190. Figure 8F As shown, another set of voxels may become visible, and the values ​​of these voxels may be set to indicate the location of the portion of the surface that has entered the field of view 190 of the sensor 182. The values ​​of these voxels may be added to the volume model for the surface. Figure 6As shown, this information can be stored as part of the retained world as volume information 162a. Information about the surface, such as color or texture, can also be stored. Such information can be stored, for example, as volume metadata 162b.

[0140] In addition to generating information for the retained world representation, the perception module 160 can also identify and output indications of changes in the area surrounding the user of the AR system. Such indications of changes can trigger updates to the volume data stored as part of the retained world, or trigger other functions, such as triggering the trigger component 304 that generates AR content to update the AR content.

[0141] In some embodiments, the perception module 160 can identify changes based on a signed distance function (SDF) model. The perception module 160 can be configured to receive sensor data such as a depth map 160a and a head pose 160b, and then fuse the sensor data into an SDF model 160c. The depth map 160a can directly provide SDF information, and the image can be processed to obtain the SDF information. The SDF information represents the distance from the sensor used to capture the information. Since those sensors can be part of a wearable unit, the SDF information can represent the physical world from the perspective of the wearable unit and therefore from the perspective of the user. The head pose 160b can enable the SDF information to be associated with voxels in the physical world.

[0142] Back to Figure 6 In some embodiments, the perception module 160 may generate, update, and store a representation of a portion of the physical world within a perception range. The perception range may be determined based at least in part on a reconstruction range of the sensor, which may be determined based at least in part on a limit of an observation range of the sensor. As a specific example, an active depth sensor operating using active IR pulses may reliably operate over a range of distances, thereby creating an observation range of the sensor, which may be from a few centimeters or tens of centimeters to several meters.

[0143] Figure 7B A reconstruction range is depicted relative to a sensor 104 having a viewpoint 106. A reconstruction of the 3D space within the viewpoint 106 may be constructed based on data captured by the sensor 104. In the example shown, the observation range of the sensor 104 is 40 cm to 5 m. In some embodiments, the reconstruction range of the sensor may be determined to be less than the observation range of the sensor because the sensor output near its observation limit may be noisier, incomplete, and inaccurate. For example, in the example of 40 cm to 5 m shown, the corresponding reconstruction range may be set to from 1 to 3 m, and data collected using the sensor indicating surfaces outside of this range may not be employed.

[0144] In some embodiments, the perception range may be larger than the reconstruction range of the sensor. If the component 164 using data about the physical world requires data about areas within the perception range that are outside the portion of the physical world that is within the current reconstruction range, this information can be provided from the retained world 162. Accordingly, information about the physical world can be easily accessed through queries. In some embodiments, an API can be provided to respond to such queries, thereby providing information about the user's current perception range. Such techniques can reduce the time required to access existing reconstructions and provide an improved user experience.

[0145] In some embodiments, the perception range can be a 3D space corresponding to a bounding box centered around the user's location. As the user moves, the portion of the physical world that is within the perception range and can be queried by component 164 can move with the user. Figure 7C A bounding box 110 is depicted centered at location 112. It should be appreciated that the size of bounding box 110 may be set to encompass the sensor's viewing range with reasonable extension, since the user cannot move at unreasonable speeds. In the example shown, the viewing limit of the sensor worn by the user is 5m. Bounding box 110 is set to 20m 3 cube.

[0146] return Figure 6 , world reconstruction component 316 may include additional modules that may interact with perception module 160. In some embodiments, preserved world module 162 may receive a representation of the physical world based on data acquired by perception module 160. Preserved world module 162 may also include representations of the physical world in various formats. For example, volume metadata 162b such as voxels may be stored, as well as meshes 162c and planes 162d. In some embodiments, other information may be saved, such as a depth map.

[0147] In some embodiments, the perception module 160 may include modules that generate representations for the physical world in various formats, including, for example, grids 160d, planes, and semantics 160e. These modules may generate representations based on data within the perception range of one or more sensors at the time of generating the representation, as well as data captured at previous times and information in the retained world 162. In some embodiments, these components may operate on depth information captured using a depth sensor. However, the AR system may include a visual sensor and may generate such a representation by analyzing monocular or binocular visual information.

[0148] In some embodiments, as described below, these modules may operate on an area of ​​the physical world, such as an area represented by a block or tile. When perception module 160 detects a change in the physical world in other sub-areas, those modules may be triggered to update the block or tile of the physical world or that sub-area. For example, such a change may be detected by detecting a new surface or other criteria in SDF model 160c (e.g., changing the value of a sufficient number of voxels representing the sub-area).

[0149] The world reconstruction component 316 may include a component 164 that may receive a representation of the physical world from the perception module 160. Information about the physical world may be pulled by these components based on, for example, a usage request from an application. In some embodiments, information may be pushed to the usage component, for example, via an indication of a change in a pre-identified area or a change in the representation of the physical world within the perception range. The component 164 may include, for example, a game program and other components that perform processing for visual occlusion, physics-based interaction, and environmental reasoning.

[0150] In response to a query from component 164, perception module 160 may send a representation of the physical world in one or more formats. For example, when component 164 indicates that the use is for visual occlusion or physics-based interaction, perception module 160 may send a representation of a surface. When component 164 indicates that the use is for environmental reasoning, perception module 160 may send meshes, planes, and semantics of the physical world.

[0151] In some embodiments, perception module 160 may include a component that formats information to provide component 164. An example of such a component may be ray casting component 160f. Using a component (e.g., component 164), for example, information about the physical world may be queried from a particular viewpoint. Ray casting component 160f may select from one or more representations of physical world data within the field of view from that viewpoint.

[0152] Viewpoint-dependent brick selection for fast volume reconstruction

[0153] It should be understood from the foregoing description that the perception module 160 or another component of the AR system can process data to create a 3D representation of a portion of the physical world. The data to be processed can be reduced by picking portions of the 3D reconstruction volume based at least in part on the camera frustum and / or the depth image, extracting and retaining plane data, capturing, retaining, and updating 3D reconstruction data in tiles that allow local updates while maintaining neighborhood consistency, providing occlusion data to applications that generate such scenes (wherein the occlusion data is derived from a combination of one or more depth data sources), and / or performing multi-level mesh simplification.

[0154] A world reconstruction system can integrate sensor data over time from multiple viewpoints of the physical world. As a device including a sensor moves, the pose (e.g., position and orientation) of the sensor can be tracked. Since the sensor's frame pose is known and how it relates to other poses, each of these multiple viewpoints of the physical world can be fused in a single combined reconstruction. By using spatial and temporal averaging (i.e., averaging the data from multiple viewpoints over time), the reconstruction can be more complete and less noisy than the raw sensor data.

[0155] The reconstruction may contain data of varying levels of complexity, including, for example, raw data (eg, real-time depth data), fused volumetric data (eg, voxels), and computed data (eg, meshes).

[0156] In some embodiments, AR and MR systems represent 3D scenes with a regular grid of voxels, where each voxel may contain a signed distance field (SDF) value. The SDF value describes whether the voxel is inside or outside a surface in the scene to be reconstructed and the distance from the voxel to the surface. Calculating the 3D reconstruction data required to represent the volume of the scene requires a large amount of memory and processing power. Since the number of variables required for 3D reconstruction grows cubically with the number of depth images processed, these requirements increase for scenes representing larger spaces.

[0157] Described herein is an efficient way to reduce processing. According to some embodiments, a scene may be represented by one or more bricks. Each brick may include multiple voxels. The bricks processed to generate a 3D reconstruction of the scene may be selected by picking a set of bricks representing the scene based on a frustum derived from a field of view (FOV) of an image sensor and / or a depth image (or "depth map") of the scene created using a depth sensor.

[0158] The depth image may have one or more pixels, each pixel representing the distance to a surface in the scene. These distances may be related to the position relative to the image sensor so that the data output from the image sensor may be selectively processed. The image data may be processed for those bricks that represent portions of the 3D scene that contain surfaces visible from the perspective (or "viewpoint") of the image sensor. Processing of some or all of the remaining bricks may be omitted. In this way, the bricks selected may be bricks that are likely to contain new information that can be obtained by picking out bricks about which the output of the image sensor is unlikely to provide useful information. The data output from the image sensor is unlikely to provide useful information about bricks that are closer to or farther away from the image sensor than the surface indicated by the depth map, because these bricks are empty space or behind the surface and are therefore not depicted in the image from the image sensor.

[0159] In some embodiments, one or more criteria may be applied to efficiently select a set of bricks for processing. The initial set of bricks may be limited to those bricks within the frustum of the image sensor. A larger number of bricks outside the frustum may then be selected. More computer resource intensive processing may then be performed on the subset of bricks accepted for processing after selection to update the 3D reconstruction. Thus, the 3D representation of the scene to be updated may be computed more efficiently using processing of a reduced number of voxels.

[0160] By picking bricks based on the depth image, a greater reduction in processing can be achieved. According to some embodiments, the picking and / or acceptance of bricks can be performed by projecting the outline of each brick in the initial set into the depth image. Such picking can be based on whether the brick corresponds to a portion of the scene indicated by the depth image near the surface. Bricks that can be simply identified as being completely in front of the surface or completely behind the surface can be picked. In some embodiments, such determinations can be made efficiently. For example, a bounding box surrounding the projection of the brick into the depth map can be used to determine the maximum brick value and the minimum brick value along the z-coordinate direction, which can be substantially perpendicular to the 2D plane of the depth image. By comparing these maximum brick values ​​and minimum brick values ​​with the distance represented by the pixel in the depth map, the brick can be picked and / or accepted for further processing. Such processing can result in the selection of a brick for initial processing that intersects and / or is in front of the surface reflected in the depth image. In some embodiments, such processing can distinguish between bricks in front of a solid surface and bricks in front of a perforated surface (ie, bricks representing areas for which the depth sensor cannot reliably measure the distance to the surface).

[0161] In some embodiments, the selection / acceptance criteria can result in the classification of some or all bricks that are accepted for further processing so that the processing algorithms used to compute the volume reconstruction can be customized to the characteristics of the bricks. In some embodiments, different processing can be selected based on whether the brick is classified as intersecting a surface, in front of a solid surface, or in front of a surface with holes.

[0162] Fig. 9 A cross-sectional view of scene 400 along a plane parallel to the y-coordinate and the z-coordinate is shown. The XR system can represent scene 400 by a voxel grid 504. Conventional XR systems can update each voxel of the voxel grid based on each new depth image captured by sensor 406, which can be an image sensor or a depth sensor, so that a 3D reconstruction generated from the voxel grid can reflect changes in the scene. Updating in this manner can consume a large amount of computing resources and also cause artifacts at the output of the XR system due to, for example, time delays caused by heavy calculations.

[0163] Described herein are techniques for providing accurate 3D reconstruction data with low computational resource utilization, for example, by picking portions of a voxel grid 504 based at least in part on a camera frustum 404 of an image sensor 406 and / or a depth image captured by the image sensor.

[0164] In the example shown, image sensor 406 captures a depth image (not shown) of surface 402 that includes scene 400. The depth image may be stored in computer memory in any convenient manner that captures the distance between some reference point and the surface in scene 400. In some embodiments, the depth image may be represented as values ​​in a plane parallel to the x-axis and the y-axis, such as Fig. 9 As shown, where the reference point is the origin of the coordinate system. The positions in the XY plane can correspond to directions relative to the reference point, and the values ​​at those pixel positions can indicate the distance from the reference point to the nearest surface in the direction indicated by the coordinates in the plane. Such a depth image may include a grid of pixels in a plane parallel to the x-axis and y-axis (not shown). Each pixel may indicate the distance from the image sensor 406 to the surface 402 in a particular direction. In some embodiments, the depth sensor may not be able to measure the distance to the surface in a particular direction. For example, this situation may occur if the surface is beyond the range of the image sensor 406. In some embodiments, the depth sensor may be an active depth sensor that measures distance based on reflected energy, but the surface may not reflect enough energy to make accurate measurements. Therefore, in some embodiments, the depth image may have "holes" where pixels at these holes are not assigned values.

[0165] In some embodiments, the reference point of the depth image can vary. Such a configuration may allow the depth image to represent surfaces in the entire 3D scene, rather than being limited to portions having a predetermined and limited angular range relative to a particular reference point. In such embodiments, the depth image may indicate the distance to the surface as the image sensor 406 moves through six degrees of freedom (6DOF). In these embodiments, the depth image may include a set of pixels for each of a plurality of reference points. In these embodiments, a portion of the depth image may be selected based on a "camera pose," which represents the direction and / or orientation to which the image sensor 406 was pointing when the image data was captured.

[0166] The image sensor 406 may have a field of view (FOV), which may be represented by the camera frustum 404. In some embodiments, the depicted infinite camera frustum may be reduced to a finite 3D trapezoidal prism 408 by assuming a maximum depth 410 that the image sensor 406 may provide and / or a minimum depth 412 that the image sensor 406 may provide. The 3D trapezoidal prism 408 may be a convex polyhedron defined at six planes.

[0167] In some embodiments, one or more voxels 504 may be grouped into bricks 502 . Fig. 10A A portion 500 of the scene 400 is shown, comprising eight bricks 502 . Fig. 10B Shown includes 8 3 An exemplary brick 502 of voxels 504. Fig. 9 , scene 400 may include one or more bricks, Figure 4 Sixteen of these are shown in the illustrated view. Each brick can be identified by a brick identifier such as

[0000] -

[0015] .

[0168] Fig.11 Depicted is a volume representation hierarchy that may be implemented in some embodiments. In some embodiments, such a volume representation hierarchy may reduce latency in data transmission. In some embodiments, a voxel grid of the physical world may be mapped to conform to the structure of a storage architecture of a processor used to compute AR content (such as a processor on which component 304 is executed). One or more voxels may be grouped into "bricks." One or more bricks may be grouped into "tiles." The size of a tile may correspond to a storage page of a storage medium local to the processor. Tiles may be moved between local memory and remote memory, such as over a wireless connection, according to a storage management algorithm based on usage or expected usage.

[0169] In some embodiments, uploads and / or downloads between perception module 160 and retained world module 162 may be performed on multiple tiles in one operation. One or more tiles may be grouped into a "RAM tile set". The size of a RAM tile set may correspond to an area within the reconstruction range of a sensor worn by a user. One or more RAM tile sets may be grouped into a "global tile set". The size of a global tile set may correspond to the perception range of a world reconstruction system (e.g., the perception range of perception module 160).

[0170] Fig.12 6 is a flow chart illustrating a method 600 of operating a computing system to generate a 3D reconstruction of a scene in accordance with some embodiments. Method 600 may begin by representing a scene (e.g., scene 400) with one or more bricks (e.g., brick 502), each brick comprising one or more voxels (e.g., voxel 504). Each brick may represent a portion of the scene. The bricks may be identifiable with respect to a preserved coordinate system such that the same brick represents the same volume in the scene even if the pose of an image sensor (e.g., image sensor 406) changes.

[0171] At act 604, method 600 can capture a depth image (e.g., a depth image including surface 402) from a depth sensor (e.g., depth sensor 406). The depth sensor can be an active depth sensor that sends, for example, IR radiation for reflection and measures the time of flight. Each such measurement represents the distance from the depth sensor to the surface in a particular direction. The depth information can represent the same volume as that represented by the brick.

[0172] At act 606, method 600 may select a portion of one or more bricks for a camera frustum (e.g., finite 3D trapezoidal prism 408 derived from camera frustum 404) to produce a first one or more bricks that is a reduced set of bricks from the one or more bricks. Such selection may eliminate bricks that represent portions of the scene that were outside the field of view of the image sensor when the image data being processed was acquired. The image data is therefore less likely to contain information useful for creating or updating the bricks.

[0173] At act 608, method 600 may select portions of the first one or more tiles for the depth image to produce a second one or more tiles, the second one or more tiles being a reduced set of tiles from the first one or more tiles. At act 610, method 600 may generate a 3D reconstruction of the scene based on the second one or more tiles.

[0174] return Fig. 9 , given the surface 402 and the corresponding camera pose captured by the depth image, the voxels between the image sensor 406 and the surface 402 may be empty. The farther the voxel is from the image sensor 406 behind the surface 402, the less certain it is that the voxel represents the interior of an object or empty space. The degree of certainty may be represented by a weight function that weights voxel updates based on the distance to the surface 402. When the weight function of a voxel located behind the surface 402 (further from the image sensor 402) is above a threshold, the voxel may not have been updated or have a zero update (e.g., an update with zero change). Moreover, all voxels that do not fall within the camera frustum 404 may not be updated or investigated for the depth image.

[0175] The method 600 can not only improve the processing speed of volumetric depth image fusion, but also consume less memory storage, which enables the method 600 to run on wearable hardware. 3A small reconstruction volume of 5m*5m*3m with a voxel size and 8 bytes per voxel (4 bytes for distance value and 4 bytes for weight value) will already require approximately 600MB. Method 600 can sort bricks according to their distance to the surface relative to a truncated threshold. For example, method 600 can identify empty bricks (e.g., picked bricks, or bricks that are more than a truncated threshold away from the surface) so that memory space is not allocated for empty bricks. Method 600 can also identify bricks far from the surface by the truncated threshold so that these bricks are stored with a constant distance value of negative truncated threshold and weight 1. Method 600 can also identify bricks with a distance to the surface between zero and the truncated threshold so that these bricks are stored with a constant SDF value with a positive truncated threshold but varying weights. Storing distance or weight values ​​that are constant for bricks with a single value can be an entropy-based compression for a zero entropy field.

[0176] Method 600 may allow tiles to be marked as "not containing any part of a surface" during voxel updates, which may significantly speed up the processing of tiles. The processing may include, for example, converting an image of the portion of the scene represented by the tile into a mesh.

[0177] Fig.13 An exemplary method 606 of picking a portion of one or more bricks for a camera frustum 404 of an image sensor 406 is shown in accordance with some embodiments. The method 606 may begin by finding a cube axis aligned bounding box (AABB) for containing the camera frustum 404. The AABB may contain one or more bricks in a scene. The method 606 may include dividing (act 704) the AABB into one or more sub-AABBs, and performing (act 706) a camera frustum acceptance test. If the method 606 determines at act 708 that the sub-AABB reaches the size of a brick, the method 606 may generate (act 710) a first one or more bricks. If the method 606 determines at act 708 that the sub-AABB is larger than the size of a brick, the method 606 may repeat acts 704-708 until the sub-AABB reaches the size of a brick.

[0178] For example, given a 3D trapezoidal prism 408 corresponding to the camera frustum 404, an AABB may be found in constant time whose side length is a power of 2 and contains the 3D trapezoidal prism 408. The AABB may be divided into eight sub-AABBs. Each of the eight sub-AABBs may be tested for intersection with the camera frustum 404. When it is determined that a sub-AABB does not intersect the camera frustum 404, a brick corresponding to the sub-AABB may be picked. The picked brick may be rejected for further processing. When it is determined that a sub-AABB intersects the camera frustum 404, the sub-AABB may be further divided into eight sub-AABBs of the sub-AABB. Each of the eight sub-AABBs of the sub-AABB may then be tested for intersection with the camera frustum 404. Iterations of division and testing continue until a sub-AABB corresponds to a single brick. To determine whether the camera frustum 404 intersects the AABB, a two-step test may be performed. First, it may be tested whether at least one corner point of the AABB lies within each plane defining the camera frustum 404. Second, it may be tested whether each corner point of the camera frustum 404 lies within the AABB so that some cases of AABBs that do not intersect the camera frustum 404 but are incorrectly classified as being partially inside (e.g., with only one corner point on the edge of the frustum) may be captured.

[0179] A desirable byproduct of this two-step test is that for each brick that intersects the camera frustum 404, it can be known whether it is completely within the camera frustum 404 or only partially within the camera frustum 404. For bricks that are completely within the camera frustum 404, the test of whether it is inside the camera frustum 404 can be skipped for each voxel later when each voxel is updated.

[0180] Fig.14 An exemplary method 706 for performing a camera frustum acceptance test is shown in accordance with some embodiments. The method 706 may begin by testing (act 802) each of one or more sub-AABBs relative to each plane that bounds the camera frustum 404. At act 804, the method 706 may determine whether the tested sub-AABB is completely outside the camera frustum 404. At act 806, if it is determined that the tested sub-AABB is completely outside the camera frustum 404, the method 706 may select all bricks contained by the tested sub-AABB. At act 808, if it is determined that the tested sub-AABB is not completely outside the camera frustum 404, the method 706 may determine whether the tested sub-AABB is completely inside the camera frustum 404.

[0181] At act 810, if it is determined that the tested sub-AABB is completely inside the camera frustum 404, the method 706 may add all bricks contained by the tested sub-AABB to the first one or more bricks. At act 708, if it is determined that the tested sub-AABB is not completely inside the camera frustum 404, which may indicate that the tested sub-AABB intersects the camera frustum 404, the method 706 may determine whether the tested sub-AABB reaches the size of one brick.

[0182] At act 814, if it is determined that the tested sub-AABB is equal to the size of the brick, method 706 may further determine whether each corner point of camera frustum 404 is located inside the tested sub-AABB. If it is determined that each corner point of camera frustum 404 is located inside the brick of the tested sub-AABB, method 706 may select (act 806) the brick of the tested sub-AABB. If it is determined that not every corner point of the camera frustum is located inside the brick of the tested sub-AABB, method 706 may add (act 810) the brick of the tested sub-AABB to the first one or more bricks.

[0183] Fig.15 An exemplary method 608 is shown for selecting a portion of a first one or more bricks for a depth image in accordance with some embodiments. The method 608 may begin by performing (act 902) a first depth image acceptance test on each of the first one or more bricks. At act 904, the method 808 may determine whether the tested brick was accepted by the first depth image acceptance test. If it is determined that the tested brick is accepted by the first depth image acceptance test, which may indicate that the tested brick intersects a surface in the scene, the method 608 may apply (act 906) a delta of the change to the selected voxels and add (act 914) the tested brick to the second one or more bricks.

[0184] At act 908, if it is determined that the first depth image acceptance test did not accept the tested brick, the method 608 may perform (act 908) a second depth image acceptance test for the tested brick. At act 910, it is determined whether the second depth image acceptance test accepted the tested brick. If it is determined that the second depth image acceptance test accepted the tested brick, which may indicate that the tested brick is located in front of a solid or perforated background in the scene, the method 608 may apply (act 912) a constant increment to all voxels or selected voxels, and then add (act 914) the tested brick to the second one or more bricks. If it is determined that the second depth image acceptance test also did not accept the tested brick, the method 608 may select (act 916) the tested brick.

[0185] Fig.16An exemplary method 902 for performing a first depth image acceptance test in accordance with some embodiments is shown. For each brick to be tested, the method 902 may begin by determining (act 1002) a minimum brick value (bmin) and a maximum brick value (bmax) along a direction parallel to the z-coordinate. The bmin value and the bmax value may be padded to account for an integration threshold, beyond which depth values ​​indicate a continuous update of voxels in the brick. In act 1004, the method 902 may calculate the 2D pixel locations of the corners of the tested brick by projecting the corners of the brick into the depth image. In act 1006, the method 902 may calculate a rectangle by building a convex hull of the 2D pixel locations of the corners of the brick. In act 1008, the method 902 may test each pixel in the rectangle for the bmin value and the bmax value. In act 1010, the method 902 may determine whether all pixels in the rectangle have depth values ​​between the bmin value and the bmax value. If it is determined that all pixels in the rectangle have depth values ​​between the bmin value and the bmax value, then method 902 may accept (act 1012) the tile. If it is determined that not all pixels in the rectangle have depth values ​​between the bmin value and the bmax value, then method 902 may perform (act 908) a second depth image acceptance test for the tile.

[0186] Fig.17 An exemplary method 908 of performing a second depth image acceptance test according to some embodiments is shown. The method 908 may begin at act 1102 by classifying all pixels in a rectangle with respect to a bmin value and a bmax value. At act 1104, the method 908 may be performed, for example, by using Fig.18 1106). If the tested brick is determined to be in front of a solid or perforated background, method 908 may accept (act 1106) the brick. If the tested brick is determined not to be in front of a solid or perforated background, method 908 may select the brick in act 916.

[0187] Fig.19A -F depicts an example of selecting bricks representing the scene 190 for the camera frustum 192. Fig.19A In , the scene 190 is represented by a single AABB 194a, which in the example shown includes 16×16 bricks. Fig.19B, a single AABB 194a is divided into four sub-AABBs 194b, each of which includes 8×8 bricks. After performing the camera frustum acceptance test (e.g., method 706), one of the four sub-AABBs 194b fails the camera frustum acceptance test, so the 8×8 bricks in the failed sub-AABB 194b are picked out and illustrated as white bricks. Fig.19C , each of the three sub-AABBs 194 that pass the camera frustum acceptance test is further divided into four sub-AABBs 194c, each of which includes 4×4 bricks. After performing the camera frustum acceptance test (e.g., method 706), eight of the sixteen sub-AABBs 194c fail the camera frustum acceptance test, so the bricks in the failed sub-AABBs 194c are picked out and illustrated as white bricks. Similarly, in Fig.19D , sub-AABB 194d includes 2×2 bricks. Fig.19E , sub-AABB 194e includes a single brick, and thus sub-AABB 194e and the corresponding brick that pass the camera frustum test are generated as a first plurality of bricks and are Fig.19F 196f. In the example shown, if no picking is performed, the world reconstruction component will calculate all 256 bricks. With brick picking for the camera frustum, the world reconstruction component only needs to calculate the first plurality of bricks, i.e., 34 bricks, and can therefore render the result faster.

[0188] Fig. 20A An example is depicted of further selecting 34 tiles 196f for a depth image including surface 220 by performing method 608, for example. Fig. 20B The results of the selection for the depth image are depicted, showing that 12 bricks 222a out of 34 bricks 196f pass the first depth image acceptance test (e.g., method 904), 9 bricks 222b out of 34 bricks 196f pass the second depth image acceptance test (e.g., method 910), and finally, after the selection is performed for the depth image including the surface 220, 13 bricks 222c out of 34 bricks 196f are selected. As a result, in the example shown, the number of bricks calculated by the world reconstruction component is further reduced to 21 bricks by brick selection for the depth image. It should also be understood that due to the results of the first and second depth image acceptance tests, the calculation speed of the world reconstruction component can be accelerated not only by reducing the number of bricks but also by classifying the bricks. For example, as described with respect to Fig.15As discussed, a constant increment may be applied to the 9 bricks 222b that fail the first depth image acceptance test but pass the second depth image acceptance test. Batch application of constant increments may further increase computational speed compared to applying a variable increment to each voxel. Geometry (e.g., planes) in a scene may be obtained in an XR system to support applications such as walls for placing virtual screens and / or floors for navigating virtual robots. A common representation of the geometry of a scene is a mesh, which may include a group of connected triangles with vertices and edges. Conventionally, the geometry in a scene is obtained by generating a mesh for the scene and searching for the geometry in the mesh, which takes some time (e.g., a few seconds) to process and does not indicate the relationship between the geometry requested by different queries. For example, a first query may be for a table plane. In response to the first query, the system may search for a table plane and leave a watch on the table plane. Then a second query may be for a watch. In response to the second query, the system may search for all possible table planes and check whether a watch exists on each table plane until the table is found, because there is no indication from the response to the first query whether the table plane is the table plane.

[0189] A geometry extraction system is described herein. In some embodiments, the geometry extraction system can extract geometry while scanning a scene with a camera and / or sensor, which allows for fast and efficient extraction that can adapt to dynamic environmental changes. In some embodiments, the geometry extraction system can retain the extracted geometry in local and / or remote memory. The retained geometry can have a unique identifier so that, for example, different queries at different timestamps and / or from different applications can share the retained geometry. In some embodiments, the geometry extraction system can support different representations of geometry based on individual queries. In the following Figure 21-29 In the description of FIG, a plane is used as an exemplary geometric figure. It should be understood that the geometry extraction system can detect other geometric figures instead of or in addition to the plane for subsequent processing, including, for example, cylinders, cubes, lines, corners, or semantics such as glass surfaces or holes. In some embodiments, the principles described herein for geometry extraction can be applied to object extraction, etc.

[0190] Fig.21 A plane extraction system 1300 is shown according to some embodiments. The plane extraction system 1300 may include a depth fusion 1304, which may receive a plurality of depth maps 1302. The plurality of depth maps 1302 may be created by one or more users wearing a depth sensor, and / or downloaded from a local / remote memory. The plurality of depth maps 1302 may represent multiple views of the same surface. There may be differences between the plurality of depth maps, which may be reconciled by the depth fusion 1304.

[0191] In some embodiments, deep fusion 1304 may generate SDF 1306 based at least in part on method 600. Grid bricks 1308 may be generated by, for example, placing blocks on corresponding bricks (e.g., Fig.23

[0000] to

[0015] in the mesh bricks 1308). Plane extraction 1310 can detect plane planes in the mesh bricks 1308 and extract planes based at least in part on the mesh bricks 1308. Plane extraction 1310 can also extract facets for each brick based at least in part on the corresponding mesh bricks. The facet mesh may include vertices in the mesh, but not edges connecting adjacent vertices, so that storing facets consumes less storage space than a mesh. A plane data repository 1312 can retain the extracted planes and facets.

[0192] In some embodiments, the XR application can request and obtain a plane from the plane data repository 1312 via a plane query 1314, which can be sent by an application program interface (API). For example, an application can send information about its location to the plane extraction system 1300 and request all planes near it (e.g., within a five-meter radius). The plane extraction system 1300 can then search its plane data repository 1312 and send the selected plane to the application. The plane query 1314 may include information such as where the application needs a plane, what kind of plane the application needs, and / or how the plane should look (e.g., horizontal, vertical, or angled, which can be determined by examining the primitive normal of the plane in the plane data repository).

[0193] Fig. 22 A portion 1400 of the plane extraction system 1300 is shown showing details regarding plane extraction 1310 in accordance with some embodiments. Plane extraction 1310 may include dividing each mesh tile 1308 into sub-tiles 1402. Plane detection 1404 may be performed on each sub-tile 1402. For example, plane detection 1404 may: compare the raw normals of each mesh triangle in the sub-tile; merge those mesh triangles having raw normal differences less than a predetermined threshold into one mesh triangle; and identify mesh triangles having an area greater than a predetermined area value as planes.

[0194] Fig.23 is a schematic diagram showing a scene 1500 represented by tiles

[0000] to

[0015] comprising voxels and exemplary plane data including tile planes 1502, global planes 1504, and surfels 1506 in the scene according to some embodiments. Fig.23A tile

[0011] is shown divided into four sub-tiles 1508. It will be appreciated that a grid tile may be divided into any suitable number of sub-tiles. The granularity of the planes detected by plane detection 1404 may be determined by the size of the sub-tiles, which in turn may be determined by the granularity of the local / remote memory in which the volumetric 3D reconstruction data is stored.

[0195] Back to Fig. 22 , plane detection 1404 can determine a tile plane (e.g., tile plane 1502) for each grid tile based at least in part on the detected planes for each sub-tile in the grid tile. Plane detection 1404 can also determine a global plane (e.g., global plane 1504) that extends over more than one tile.

[0196] In some embodiments, plane extraction 1310 may include plane updates 1406, which may update existing brick planes and / or global planes stored in plane data store 1312 based at least in part on planes detected by plane detection 1404. Plane updates 1406 may include adding additional brick planes, removing some existing brick planes, and / or replacing some existing brick planes with brick planes detected by plane detection 1404 and corresponding to the same bricks, so that real-time changes in the scene are preserved in plane data store 1312. Plane updates 1406 may also include aggregating brick planes detected by plane detection 1404 to existing global planes, for example, when a brick plane is detected to be adjacent to an existing global plane.

[0197] In some embodiments, plane extraction 1310 may further include plane merging and splitting 1408. For example, when a brick plane is added and connects two global planes, plane merging may merge multiple global planes into a large global plane. Plane splitting may split a global plane into multiple global planes, for example, when a brick plane in the middle of a global plane is removed.

[0198] Fig.24 16 shows a data structure in a plane data repository 1312 according to some embodiments. Global planes 1614 indexed by plane ID 1612 can be at the highest level of the data structure. Each global plane 1614 can include multiple brick planes and facets of bricks adjacent to the corresponding global plane, so that one brick plane can be retained for each brick, and the global plane can be accurately rendered when the edge of the global plane does not qualify as a brick plane for the corresponding brick. In some embodiments, facets of bricks adjacent to the global plane are retained instead of facets of all bricks in the scene, because this is sufficient to accurately render the global plane. For example, Fig.23As shown, global plane 1504 extends across bricks

[0008] to

[0010] and

[0006] . Brick

[0006] has brick plane 1502, which is not part of global plane 1504. Using the data structures in plane data store 1312, when a plane query requests global plane 1504, the facets of bricks

[0006] and

[0012] are checked to determine whether global plane 1504 extends into bricks

[0006] and

[0012] . In the example shown, facet 1506 indicates that global plane 1504 extends into brick

[0006] .

[0199] Back to Fig.24 , a global plane 1614 can be bi-directionally associated with a corresponding brick plane 1610. A brick can be identified by a brick ID 1602. Bricks can be divided into planar bricks 1604 that include at least one plane and non-planar bricks 1606 that do not include a plane. Facets of both planar bricks and non-planar bricks can be retained, depending on whether the brick is adjacent to a global plane and not on whether the brick includes a plane. It should be understood that while the XR system is viewing the scene, the planes can be continuously retained in the plane data repository 1312 regardless of whether there is a plane query 1314.

[0200] Fig.25 A plane geometry extraction 1702 is shown that can extract a plane for use by an application when an application sends a plane query 1314 to a plane data repository 1312 according to some embodiments. The plane geometry extraction 1702 can be implemented as an API. The plane query 1314 can indicate a requested plane geometry representation, such as an outer rectangular plane, an inner rectangular plane, or a polygonal plane. Based on the plane query 1314, the plane search 1704 can search for and obtain plane data in the plane data repository 1312.

[0201] In some embodiments, rasterization from planar coverage points 1706 may generate planar coverage points. Fig.26A An example is shown in . There are four tiles

[0000] -

[0003] , each with a tile plane 1802. By projecting the boundary points of the tile plane onto the global plane 1804, plane coverage points 1806 (or "rasterized points") are generated.

[0202] Back to Fig.25, the rasterization from the plane coverage point 1706 can also generate a rasterized plane mask from the plane coverage point. According to the plane geometry representation requested by the plane query 1314, the inner rectangle plane representation, the outer rectangle plane representation, and the polygon plane representation can be extracted by inner rectangle extraction 1708, outer rectangle extraction 1710, and polygon extraction 1712, respectively. In some embodiments, the application can receive the requested plane geometry representation within a few milliseconds after sending the plane query.

[0203] Fig.26B An exemplary rasterized plane mask 1814 is shown in FIG. Based on the rasterized plane mask, various plane geometric representations can be generated. In the example shown, a polygon 1812 is generated by connecting some plane coverage points of the rasterized plane mask so that none of the plane coverage points in the mask are outside the polygon. An outer rectangle 1808 is generated so that the outer rectangle 1808 is the smallest rectangle surrounding the rasterized plane mask 1814. The inner rectangle 1810 is generated by the following operations: assigning "1" to bricks with two plane coverage points and assigning "0" to bricks without two plane coverage points to form a rasterized grid, determining groups of bricks marked as "1" and aligned on lines parallel to the edges of the bricks (for example, bricks

[0001] ,

[0005] ,

[0009] and

[00013] as a group, bricks

[0013] -

[0015] as a group), and generating an inner rectangle for each determined group so that the inner rectangle is the minimum rectangle surrounding the corresponding group.

[0204] Fig. 27 A grid for scene 1900 is shown in accordance with some embodiments. Fig.28A -C shows a scene 1900 represented by an outer rectangular plane, an inner rectangular plane, and a polygonal plane, respectively, according to some embodiments.

[0205] Fig.29 A less noisy 3D representation of scene 1900 is shown, which is obtained by extracting plane data (e.g. Fig.28A -The plane shown in C) Fig. 27 The mesh shown is obtained by flattening.

[0206] Multi-level block grid simplification

[0207] In some embodiments, before a representation of an XR environment is stored or used for rendering functions such as occlusion handling or computing physical interactions between objects in the XR environment, processing may be employed to reduce the complexity of the representation. For example, mesh component 160d may simplify a mesh or a portion of a mesh before storing the mesh or a portion of the mesh in retained world 162 as mesh 162c.

[0208] Such processing may require performing operations in a hierarchical manner on the representation of the XR environment. These levels may include simplified operations before and after the region-based operations. Like simplified operations, region-based operations may reduce the complexity of the representation of the XR environment. By grading the operations in this manner, the overall processing to produce a simplified representation of the XR environment may be reduced while maintaining the quality of the representation of the XR environment. As a result, the simplified high-quality representation may be updated frequently, so that the XR environment may be updated frequently, thereby improving the performance of the XR system, such as by presenting a more realistic environment to the user.

[0209] An XR environment may represent the physical world, and data representing the XR environment may be captured by one or more sensors. However, the techniques described herein may be applied to an XR environment regardless of the data source representing the environment. In some embodiments, the XR environment may be represented by a mesh including one or more points and polygons (e.g., triangles) defined by a subset of the points. A first simplification operation prior to the region-based operation may reduce the complexity of the representation of the environment. For example, the mesh may be simplified by reducing the number of such polygons in the mesh. As a specific example, the first simplification operation may employ a triangle reduction algorithm that may reduce the number of triangles used to represent the XR environment.

[0210] Region-based operations can be shape detection operations that can detect one or more shapes. A common shape detection operation is a planarization operation, in which the detected shape is a plane. The detected plane can represent an object or part of an object. The detection of a plane can simplify the process of rendering an XR environment. For example, a moving object rendered in an XR environment can move in an easy-to-compute manner when it collides with a plane. Therefore, identifying a plane can simplify the subsequent rendering of a moving object compared to performing calculations based on multiple polygons representing the same part of the environment. Instead of or in addition to a plane, other shapes can be detected and used in subsequent processing, including cylinders, cubes, lines, corners, or semantics such as glass surfaces or holes. Such an operation can group polygons representing the surfaces of the detected shapes.

[0211] A second simplification operation following the region-based operation may further simplify the representation of the environment, for example by further reducing the number of polygons in the representation. The second simplification operation may focus on reducing the number of polygons within each region detected by the region-based operation.

[0212] Such processing can enable a meshing service that processes sensor data collected in the physical environment and provides a mesh for an application that generates content. In some embodiments, the processing can provide a simplified representation of a virtual object in a virtual environment.

[0213] In XR systems (e.g., virtual reality (VR), augmented reality (AR), and mixed reality (MR) systems), three-dimensional (3D) mesh data is often used for a variety of purposes, including, for example, occluding virtual content in a graphics / game engine based on physical objects in the environment, or calculating rigid body collision effects for virtual objects in a game engine's physics engine. In some embodiments, the requirements for the mesh may be different for different uses of the mesh, and a simplified mesh may be suitable for many such uses, with some simplification techniques being more suitable for certain uses than others.

[0214] Therefore, the processing described herein can be implemented with any of a plurality of simplification techniques and / or simplification techniques that can be configured based on the intended use of the simplified mesh. The processing described herein can be used to improve the practicality of a mesh partitioning service that provides a simplified mesh to multiple client applications, which can use the mesh in different ways. Each client application may need a mesh with different simplification levels. In some embodiments, the application accessing the mesh partitioning service can specify a target simplification or a mesh to be provided to it. The mesh simplification method described herein can be used for multiple client applications, including, for example, those client applications that perform virtual content occlusion, physical simulation, or environmental geometry visualization. The mesh processing described herein can have low latency and can be flexible because it can optimize / bias operations for different purposes (e.g., flat surfaces, varying triangle counts).

[0215] The mesh simplification methods described herein can provide real-time performance (e.g., low latency to support on-the-fly (real-time) environmental changes), local update capabilities (e.g., restarting (renw) the parts of the mesh that have changed since the last update), and flattened surfaces (e.g., flattened flat surfaces to support robust physics simulations).

[0216] In some embodiments, the representation of the XR environment may be segmented into multiple tiles, some or all of which may be processed in parallel. In some embodiments, the resulting tiles may then be reassembled. In some embodiments, tiles may be defined by "skirts" that overlap adjacent tiles. Skirts enable tiles to be reassembled with fewer and / or less noticeable discontinuities at the butts of the reassembled tiles.

[0217] Thus, in some embodiments, the mesh simplification method may include mesh block segmentation, pre-simplification, mesh planarization, and post-simplification. To speed up the process, the global mesh may first be segmented into blocks of component meshes so that the mesh blocks may be addressed (e.g., processed) in parallel. The mesh blocks may then be extended with side edges on the boundaries between adjacent blocks. For mesh blocks with side edges, individual mesh blocks may be simplified, and the global mesh may be visually seamless, although topologically disconnected.

[0218] In some embodiments, the mesh simplification method may be suitable for use by applications that use simplified meshes to represent interactions with objects in an XR environment, for example by making the simplification process plane-aware. To simplify the mesh, a three-step simplification process may be implemented. The mesh may first be moderately pre-simplified using a relatively high target triangle count. Planar regions may then be detected by a region growing algorithm. The mesh may be planarized by projecting corresponding triangles onto the detected plane. In some embodiments, the mesh may be normalized by adjusting the plane (or original) normals to be substantially perpendicular and parallel to the detected plane. Thereafter, a post-simplification process may be run on the planarized mesh. The post-simplification process may focus more on the detected planar regions, for example, simplifying the mesh of each detected planar region to achieve a desired level of complexity (e.g., metric complexity), such as may be indicated by a target value of one or more metrics.

[0219] Fig.30 A method 3000 of generating a model of an environment represented by a mesh is shown in accordance with some embodiments. In some embodiments, the method 3000 may be performed on a meshing service on an XR platform. The method 3000 may begin at action 3002 by inputting a mesh representing an environment. In some embodiments, the input mesh may have a high resolution, which may be indicated by the number of triangles. The input mesh may be generated by a reconstruction system (e.g., a volumetric 3D reconstruction system), and the input mesh may include 3D reconstruction data.

[0220] In some embodiments, the reconstruction system may generate a volumetric 3D representation of the environment, which may create a data hierarchy of 3D information of the environment captured by one or more sensors. For example, the sensor may be a depth camera, which may capture 3D information of the environment, such as a stream of depth images with a corresponding pose (i.e., camera pose) of the depth camera. The 3D information of the environment may be processed into a grid of voxels. Each voxel may contain one or more signed distance functions (SDFs) that describe whether the voxel is inside or outside the geometry of an object in the environment. Voxels may be grouped into "bricks". Each brick may include, for example, a plurality of voxels of a cubic volume, such as 8 3 Voxels. Bricks can be further grouped into "tiles". Each tile can contain multiple bricks.

[0221] The size of the tiles may be selected to facilitate memory operations in a computing device. For example, the size may be selected based on the amount of information about the environment maintained in active memory of a device processing such data. For example, the system may transfer tiles between active memory that is typically local to the device and other memory with greater latency (e.g., non-volatile memory or remote memory in the cloud). One or more complete or partial tiles may contain information representing a "block" in a grid or other representation of the environment.

[0222] In some embodiments, the volumetric 3D reconstruction system may generate the input mesh 3002 as a topologically connected global mesh. In some embodiments, the volumetric 3D reconstruction system may generate the input mesh 3002 as a global mesh that is topologically disconnected but visually seamless. For example, the topologically disconnected global mesh may be composed of a plurality of mesh blocks, each of which is generated by a block.

[0223] The reconstruction system may be configured to capture substantial details of the environment, which enables the system to distinguish between adjacent portions of the representation, which have relatively small characteristic differences. Adjacent regions with different properties may be identified as different surfaces, resulting in the system identifying a large number of surfaces in the environment. However, such a system may capture unnecessary but still processed details for many applications. For example, for a client application requesting a mesh from a meshing service, when two triangles made into a rectangle would be a sufficient representation of a wall, the reconstruction system may unnecessarily display a bump on the wall with many triangles. In some embodiments, when requesting a mesh from a meshing service, the application may specify a target simplification level for the requested mesh. The target simplification level may be expressed as a degree of compression, a number of triangles per unit area, or in any other suitable manner.

[0224] Method 3000 can efficiently generate a model of an environment sufficient for a client application from an input mesh.At act 3004, the input mesh can be segmented into one or more first mesh blocks, each of which can correspond to a block in a data hierarchy of a volumetric 3D representation of the environment.

[0225] Each first grid tile may represent a portion of the environment, and the complexity metric (e.g., grid resolution) may be a first value. In some embodiments, the complexity metric of a grid tile indicates the number of triangles in the grid tile. In some embodiments, the processing may be performed sequentially and / or in parallel on the grid tiles. However, the simplification processing described herein may be applied to the entire grid or any suitable portion (e.g., one or more grid tiles).

[0226] Action 3006 represents a sub-process performed on each of the plurality of grid tiles. The sub-process may be performed independently on the plurality of grid tiles, such that the process may be easily performed in parallel on some or all of the grid tiles. The sub-process may be performed on all of the grid tiles or on a subset of the grid tiles selected for further processing. The subset of grid tiles may be selected based at least in part on the field of view of the device on which the application requesting the simplification of the grid is executing.

[0227] In action 3006, some first grid tiles may be selected based on, for example, objects described in the first grid tiles or locations of the first grid tiles. For each selected first grid tile, multiple levels of simplification may be performed. In some embodiments, the multiple levels of simplification on the selected first grid tiles may be performed in parallel, and as a result, the simplification on the selected first grid tiles may be completed at approximately the same point in time, although this may depend on the complexity metric of each of the selected first grid tiles.

[0228] The multi-level simplification may include a pre-simplification operation, an area-based operation (e.g., a planarization operation), and a post-simplification operation. In some embodiments, the multi-level simplification may be performed based on an input value from a client application. The input value may indicate the mesh complexity (e.g., mesh resolution) required by the client application. For each selected first mesh block, the input value from the client application may be the same or different.

[0229] In action 3012, a pre-simplification operation may be performed on the selected first mesh block to generate a second mesh block. The pre-simplification operation may reduce the complexity of the block. For a mesh block, pre-simplification may reduce the number of polygons in the mesh block. In some embodiments, the amount of pre-simplification at action 3012 may be configurable. For example, a target value may be provided by, for example, a client application as an input to the processing at action 3012. The target value may be a single value or multiple values ​​of one or more specified or predetermined metrics. One or more metrics may include, for example, an absolute triangle count, a percentage of an initial triangle count, and / or a quadratic error metric, which may measure the average squared distance between a simplified mesh and an original mesh (e.g., input mesh 3002).

[0230] The target value may be provided in any suitable manner. For example, an instance of method 3000 may be pre-configured with a target value. In some embodiments, the target value may be provided by an application requesting a mesh from a mesh partitioning service executing method 3000 via an API. For example, the target value of action 3012 may be the final target requested by a rendering function (e.g., the requesting application). In some embodiments, the target value provided as input may be adjusted or overridden to ensure that enough data is retained in the mesh for subsequent processing. For example, the processing in action 3014 may require a minimum number of triangles, and if the target value is lower than the minimum number of triangles, the target value provided by the application may be replaced by the minimum value.

[0231] In such an embodiment, the pre-simplified mesh may have values ​​of one or more metrics such that the pre-simplified mesh may be processed faster than the original block-partitioned input mesh during region-based operations while still containing all or most of the regions of the original block-partitioned input mesh.

[0232] Because the values ​​of one or more metrics are not controlled, the simplified mesh may be too coarse, unevenly distributed, and / or miss many regions of the original block-partitioned input mesh required for the following region-based operations.

[0233] The complexity measure of the second mesh block generated in act 3012 may be a second value, which may be less than the first value of the measure of complexity. In some embodiments, the pre-simplification operation of act 3012 may be performed using a triangle reduction algorithm.

[0234] In action 3014, a shape detection operation may be performed on the second mesh block to generate a third mesh block. Take the planarization operation as an example. The complexity metric of the third mesh block may be a third value. In some embodiments, the third value of the metric complexity may be the same as the second value of the metric complexity. In some embodiments, the third value of the metric complexity may be less than the second value of the metric complexity. The planarization operation may include: for example, using a region growing algorithm, detecting a planar region in the second mesh block, projecting the mesh of the detected planar region to a corresponding plane, adjusting the plane normal of the detected planar region to be substantially perpendicular to the corresponding plane, and simplifying the projected mesh on each corresponding plane based on, for example, a target triangle count. In some embodiments, the plane normal of the detected planar region may be adjusted before the mesh of the detected planar region is projected to the corresponding plane.

[0235] At act 3016, a post-simplification operation may be performed on the third mesh tile to generate a fourth mesh tile. In some embodiments, the processing at act 3014 may desirably be performed on the mesh at a higher resolution than would be required in the simplified mesh to be output from method 3000. In some embodiments, the processing at act 3016 may simplify the entire mesh tile to achieve a desired level of complexity (e.g., metric complexity), such as may be indicated by a target value for one or more metrics, which may be the same or different than the target provided to act 3012. In some embodiments, the post-simplification operation at act 3016 may focus on reducing the number of polygons within each plane detected by the planarization operation at act 3014.

[0236] The complexity metric of the fourth grid block may be a fourth value, which may be less than the third value of the metric complexity. In some embodiments, the percentage reduction between the third value of the metric complexity and the fourth value of the metric complexity may be greater than the percentage reduction between the first value of the metric complexity and the second value of the metric complexity. In some embodiments, the percentage reduction between the third value of the metric complexity and the fourth value of the metric complexity may be at least twice as large as the percentage reduction between the first value of the metric complexity and the second value of the metric complexity. In some embodiments, the post-simplification operation at action 3016 may be performed using a triangle reduction algorithm. In some embodiments, the post-simplification operation at action 3016 may be performed using the same simplification algorithm as the pre-simplification operation at action 3012.

[0237] In action 3008, the simplified selected tiles may be combined with other selected mesh tiles similarly processed in action 3006, and / or may be combined with unselected tiles into a new mesh of the environment. In action 3010, the new mesh of the environment may be provided to the client application. In some embodiments, the new mesh of the environment may be referred to as a simplified mesh.

[0238] In some embodiments, act 3008 may be skipped.The simplified grid tiles may be sent directly to a client application where the grid tiles may be visually seamless despite being topologically disconnected.

[0239] Fig.31An example of segmenting a grid representation 3100 of an environment into grid tiles according to some embodiments is shown. The grid representation 3100 may be segmented into four grid tiles: grid tiles AD. In some embodiments, the grid tiles may correspond to areas of the same volume in the physical world belonging to the environment. In some embodiments, the grid tiles may correspond to areas of different volumes in the physical world belonging to the environment. For example, when the physical world is an office, the office may be divided into multiple areas, each of which may be one cubic foot. A tile may include a 3D representation of an area of ​​the office.

[0240] Although the grid representation 3100 of the environment is shown in two dimensions (2D), it should be understood that the environment can be three-dimensional and correspondingly represented by a 3D grid representation. Although the grid representation 3100 of the environment is illustrated as a combination of four grid tiles, it should be understood that the environment can be represented by any suitable number (e.g., two, three, five, six, or more) of grid tiles.

[0241] Representation 3100 may be partitioned into four portions: portions 3102, 3104, 3106, and 3108, such as indicated by solid line 3110. In some embodiments, portions 3102, 3104, 3106, and 3108 may be designated as grid blocks AD, respectively.

[0242] As a grid block is updated, it may continue to dock with adjacent blocks that have not been updated. As a result, discontinuities may appear at the boundaries between grid blocks. If the area represented by adjacent blocks has discontinuities, the fused grid may be interpreted in subsequent processing as indicating that there is a crack between adjacent blocks. In some embodiments, such a crack in the representation of the physical world space may be interpreted as a space with infinite depth. In this way, the space may be an artifact of the representation of the physical world, rather than an actual feature. Any application that uses such a fused grid to generate a representation of an object in the physical world may not generate output correctly. For example, an application that renders a virtual character on a surface in the physical world may render the character as if it fell into a crack, which does not create the desired appearance of the object.

[0243] To reduce the appearance of such cracks, in some embodiments, a portion of adjacent blocks may represent the same area of ​​the physical world. For example, the docking area between adjacent blocks may be represented by a portion of each adjacent block, which can achieve easy independent update and / or rendering considering the level of detail (LOD) (e.g., reducing the complexity of the 3D reconstruction of a portion of the physical world because the portion moves out of the user's field of view). Even if a block is updated and its adjacent blocks are not updated, the fused mesh can be represented by combining data representing the docking area from the two blocks. As a specific example, when an updated block is fused with an adjacent block, the physics engine can determine the overlapping area of ​​the adjacent blocks based on, for example, which of the adjacent blocks can be observed in its overlapping area. The block-based data structure can use a side edge, a zipper, or any other suitable method to represent the docking area between adjacent blocks, so that when a block is updated, it will continue to dock with the adjacent block that has not been updated. The appearance of this method can be "paper over" on the cracks between adjacent blocks. Therefore, blocks can be updated independently of adjacent blocks.

[0244] exist Fig.31 In the example shown, areas at the borders of portions 3102, 3104, 3106, and 3108 may be designated as side edges, as shown by dashed line 3112. In some embodiments, each mesh block AD may include one of portions 3102, 3104, 3106, and 3108 and a corresponding side edge. For example, mesh block B may include portion 3104 and side edge 3114, which overlaps with the borders of adjacent mesh blocks A, C, and D of mesh block B so that cracks between mesh blocks may be masked when the blocks are connected into a single mesh. Mesh blocks A, C, and D may also include corresponding side edges. Thus, the processor may mask on any cracks between mesh blocks before returning a single connected 3D mesh representation to the application.

[0245] In some embodiments, the tile mesh including the side edges can be sent directly to the application without being combined into a topologically connected global mesh. The application can have a global mesh composed of tile meshes that is visually seamless despite being topologically disconnected.

[0246] Figures 32A-32D 32 shows the mesh evolution of an exemplary mesh block 3201 during multi-level simplification according to some embodiments. The mesh block 3201 may include vertices 3206, edges 3208, and faces 3210. Each face may have a normal, which may be composed of multiple coordinates (e.g., Fig.32A denoted by x, y, and z).

[0247] A pre-simplification operation may be performed on mesh block 3201 to generate mesh block 3202. An edge collapse transformation may be used. In the example shown, mesh block 3202 reduces the number of faces of mesh block 3201 from ten to eight. The resulting faces of mesh block 3202 may each have a corresponding set of normals (e.g., x1, y1, z1; x2, y2, z2; ...; x8, y8, z8).

[0248] A planarization operation may be performed on the mesh block 3202 to generate the mesh block 3203. The planarization operation may include detecting a plane region in the mesh 3202 based on, for example, a plane (or original) normal of a face. The values ​​of the plane normals x1, y1, z1 of the first face 3212 and the plane normals x2, y2, z2 of the second face 3214 may be compared. The comparison result of the plane normals of the first face and the second face may indicate an angle between the plane normals (e.g., an angle between x1 and x2). When the comparison result is within a threshold, it may be determined that the first plane and the second plane are on the same plane region. In the example shown, planes 3212, 3214, 3216, and 3218 may be determined to be on a first plane region corresponding to plane 3228; planes 3220, 3222, 3224, and 3226 may be determined to be on a second same plane region corresponding to plane 3230.

[0249] The planarization operation may further include: projecting a triangle formed by the edges of planes 3212, 3214, 3216, and 3218 onto plane 3228 as indicated by dashed line 3232; and projecting a triangle formed by the edges of planes 3220, 3222, 3224, and 3226 onto plane 3230 as indicated by dashed line 3234. The planarization operation may further include: adjusting the plane normals of planes 3212, 3214, 3216, and 3218 to be the same as the plane normal (x_a, y_a, z_a) of plane 3228; and adjusting the plane normals of planes 3220, 3222, 3224, and 3226 to be the same as the plane normal (x_b, y_b, z_b) of plane 3230.

[0250] A post-simplification operation may be performed on mesh tile 3203 to generate mesh tile 3204. In the example shown, mesh tile 3204 reduces the number of faces of mesh tile 3203 from eight to four.

[0251] Fig.33A and 33B 36A and 36B illustrate the effect of simplification, showing side by side the same portion of the physical world with and without simplification applied. These figures provide a graphical illustration of how simplification can provide usable information to operate an AR system while providing less data to process.

[0252] Fig.33A and 33B Representations of the same environment without simplification and after simplification by triangle reduction are shown. Fig.30 An example of the processing performed at the pre-simplification box 3012 and the post-simplification box 3016 in.

[0253] Fig.34A and 34B Close-up representations of the same environment without and with simplification by triangle reduction are shown, respectively.

[0254] Fig.35A and 35B A representation of the same environment without and with planarization is shown. Fig.30 An example of the processing performed at the planarization block 3014 in FIG.

[0255] Fig.36A and 36B 1 and 2 show representations of the same environment without simplification and by removing disconnected components. Fig.30 An example of an alternative embodiment of the region-based operation performed at box 3014 of .

[0256] Caching and updating of dense 3D reconstruction data

[0257] In some embodiments, 3D reconstruction data may be captured, retained, and updated in the form of tiles, which may allow local updates while maintaining neighbor consistency. The tile-based 3D reconstruction data representation may be used in conjunction with a multi-layer caching mechanism that efficiently retrieves, pre-fetches, and stores 3D data for AR and MR applications (including single-device and multi-device applications). For example, volume information 162a and / or mesh 162c ( Figure 6 ) can be stored in a block. The using component 164 can use the representation based on the block to receive information about the physical world. Similarly, the sensing component 160 can store and retrieve such information in a block.

[0258] These technologies extend the capabilities of portable devices with limited computing resources to present AR and MR content in a highly realistic manner. Such technologies can be used, for example, to effectively update and manage real-time or offline reconstruction and scanned outputs in mobile devices with limited resources and connected to the Internet (continuously or discontinuously). These technologies can provide the latest, accurate and comprehensive 3D reconstruction data for various mobile AR and MR applications in single-device applications or multi-device applications that share and update the same 3D reconstruction data. These 3D reconstruction data can be in any suitable format, including meshes, point clouds, voxels, etc.

[0259] Some AR and MR systems have attempted to simplify the presentation of MR and AR scenes by limiting the amount of 3D reconstruction data processed at any time. A sensor used to capture 3D information may have a maximum reconstruction range that may limit a bounding volume around the sensor's field of view. To reduce the amount of 3D reconstruction data, some reconstruction systems only keep areas near the sensor's field of view in active working memory and store other data in auxiliary storage devices. For example, areas near the sensor's field of view are stored in CPU memory, while other data is retained in a local cache (e.g., disk) or retained over a network to a remote storage device (e.g., in the cloud).

[0260] The computational cost of generating the information stored in CPU memory, while limited, can still be relatively high. Some AR and MR systems continually recompute a global representation of the environment that the reconstruction system has seen in order to select information to store in CPU memory, which can be prohibitively expensive for interactive applications. Other AR and MR systems use methods that compute only local updates to connected representations that can be equally expensive, especially for simplified meshes, because it requires decomposing the existing mesh, computing another mesh with the same boundaries, and then reconnecting the mesh parts.

[0261] In some embodiments, the 3D reconstruction data may be segmented into blocks. The 3D reconstruction data may be sent between storage media based on blocks. For example, blocks may be paged out from active memory and retained in a local or remote cache. The system may implement a paging algorithm in which an active memory associated with a wearable device (e.g., a head mounted display device) stores blocks representing a portion of a 3D reconstruction of the physical world in the field of view of a user of the wearable device. The wearable device may capture data about a portion of the physical world commensurate with the field of view of the user of the wearable device. As the physical world changes in the user's field of view, the blocks representing that area of ​​the physical world may be in the active memory, from which it may be easily updated. As the user's field of view changes, blocks representing an area of ​​the physical world that moves out of the user's field of view may be moved to a cache so that blocks representing an area of ​​the physical world that enters the user's field of view may be loaded into the active memory.

[0262] In some embodiments, a coordinate system may be created for a portion of the physical world to be reconstructed in 3D. Each tile in the 3D representation of the portion of the physical world may correspond to a different region of the physical world identifiable using the coordinate system.

[0263] In some embodiments, when a block is updated, the updated block may continue to dock with adjacent blocks that may not have been updated. If the areas represented by adjacent blocks do not overlap, there may be cracks in the fused mesh of the adjacent blocks. In some embodiments, such cracks in the representation of the physical world space may be interpreted as a space with infinite depth. In this way, the space may be an artificial trace of the representation of the physical world, rather than an actual feature. Any application that uses such a fused mesh to generate a representation of an object in the physical world may not be able to generate output correctly. For example, an application that renders a virtual character on a surface in the physical world may render the character as if it fell into a crack, which does not create the desired appearance of the object. Therefore, in some embodiments, a portion of adjacent blocks may represent the same area of ​​the physical world, for example, docking between adjacent blocks may represent the same area of ​​the physical world, which can achieve easy independent updating and / or rendering considering level of detail (LOD) (for example, reducing the complexity of 3D reconstruction of a portion of the physical world because the portion moves out of the user's field of view). For example, when a block is updated, its adjacent blocks may not be updated. When merging an updated block with an adjacent block, the physics engine can determine the overlapping area of ​​the adjacent blocks based on, for example, which of the adjacent blocks is observable in its overlapping area. The block-based data structure can use side edges, zippers, or any other suitable method to represent the docking area between adjacent blocks, so that when a block is updated, it will continue to dock with adjacent blocks that have not been updated. The appearance of this method can be "covering" over the cracks between adjacent blocks. Therefore, the block that has changed can be updated independently of the adjacent blocks.

[0264] In some embodiments, these techniques may be used in an AR and / or MR "platform" that receives and processes data from one or more sensors worn by a user. The sensor data may be used to create and update 3D reconstruction data representing a portion of the physical world encountered by the user. As the sensor is capturing and updating data, a reconstruction service may continuously reconstruct a 3D representation of the physical world. One or more techniques may be used to determine the blocks affected by changes in the physical world, and those blocks may be updated. The 3D reconstruction data may then be provided to an application that uses the 3D reconstruction data to render a scene to depict virtual reality objects placed in the physical world or interacting with objects in the physical world. The data may be provided to the application via an application programming interface (API). The API may be a push or pull interface that pushes data to the application when the relevant portion changes or in response to a request from the application for the latest information.

[0265] In the example of a pull interface, when an application requests 3D reconstruction data of the physical world, the reconstruction service can determine the appropriate version of each tile that needs to be provided to the application, so that the reconstruction service can start from the latest tile. The reconstruction service can, for example, search for previously reserved tiles. A single-device system can enable a single device to contribute 3D reconstruction data about the physical world. In a single-device system, if the requested area of ​​the physical world is in an active area (e.g., an area in the current field of view of the device) or extends outside the active area, the reserved tiles can be used directly as the latest tiles, because these reserved tiles will not be updated because they are retained when the area moves out of the field of view of the device. On the other hand, a multi-device system can enable multiple devices to provide 3D reconstruction data about the physical world, such as using cloud persistence or peer-to-peer local cache sharing of tiles. Each device can update the area that can be retained within its active area. A multi-device system can create a coordinate system so that tiles generated by different devices can be identified using the coordinate system. Thus, tiles requested by the application for data generated by the first device may be based on updates from other devices if those updates were made after any version was made by the first device. Tiles built using data from the first device and other devices may be merged using a coordinate system.

[0266] The selected blocks can be used to provide 3D reconstruction data about the physical world in any suitable format, but the grid is used as an example of a suitable representation here. The grid can be created by processing image data to identify points of interest in the environment (e.g., the edges of an object). These points can be connected to form the grid. The group of points (usually three points) associated with the same object or a portion thereof in the grid defines the surface of the object or a portion thereof. The information stored with the group of points describes the surface in the environment. This information can then be used in various ways to render and / or display information about the environment. The selected blocks can be used to provide 3D reconstruction data in any suitable manner. In some embodiments, the latest blocks can be provided. In some embodiments, the latest blocks can be used to determine whether the blocks need to be updated.

[0267] For example, in some embodiments, in a multi-device system, when a tile requested by an application has been identified, the reconstruction service can check the tiles retained by other devices to determine whether there have been any significant updates (e.g., by a geometry change threshold or timestamp), re-run meshing on the tiles that have changed, and then retain these updated mesh tiles.

[0268] In some embodiments, when a tileset requested by an application has been identified, if the application requested a connected mesh, the tileset may be processed as a global mesh that may be topologically connected, or visually seamless although topologically disconnected using any suitable technique (e.g., side edges and zippers).

[0269] In some embodiments, an application (e.g., a graphics / game engine) may update its internal tiles (e.g., tiles stored in active memory and / or local cache) when a tile change occurs. The reconstruction service may know which tiles the application has, and therefore may calculate which other (e.g., adjacent) tiles need to be updated in the engine to maintain the correct overlap with the side edge / zipper when the tiles in the field of view are updated.

[0270] In some embodiments, the AR and / or MR platform may be implemented to support, for example, the execution of AR and / or MR applications on a mobile device. An application that executes or generates data for presentation on a user interface may request 3D reconstruction data representing the physical world. The 3D reconstruction data may be provided from an active memory on the device, and the active memory may be updated with 3D reconstruction data representing the physical world in the field of view of the device as the user changes its field of view. The 3D reconstruction data in the active memory may represent the active area of ​​the mobile device. In some embodiments, the 3D reconstruction data outside the active area of ​​the device may be stored in other memories, such as in a local cache on the device, or coupled to the device via a low-latency connection. In some embodiments, the 3D reconstruction data outside the active area of ​​the device may also be stored in a remote cache, such as in the cloud, which may be accessed by the device via a higher latency connection. When the user changes his field of view, the platform may access / load the 3D reconstruction data from the cache to add to the active memory to represent the area moving into the user's field of view. The platform may move other data representing the area moving out of the user's field of view to the cache.

[0271] Predictions of movement of the device (which may cause regions to move into the field of view of the device and other regions to move out of the field of view of the device) may be used to initiate transfer of 3D reconstruction data between active memory and a cache. Predictions of movement may, for example, be used to select 3D reconstruction data for transfer into and / or out of active memory. In some embodiments, the predicted movement may be used to transfer 3D reconstruction data into and / or out of a local cache by retrieving or transferring 3D reconstruction data from or to a remote cache. Exchanging 3D reconstruction data between the local cache and the remote cache based on the predicted movement of the user may ensure that the 3D reconstruction data is available with low latency for transfer into the active memory.

[0272] In embodiments where tiles represent regions of the physical world, initiating transmission of tiles may require prior requesting tiles representing regions predicted to enter the user's field of view. For example, if the platform determines based on sensor data or other data that the user is walking in a particular direction at a particular speed, it may identify regions that may enter the user's field of view and transmit tiles representing those regions to a local cache on the mobile device. If the mobile device is a wearable device, such as a set of glasses, predicting motion may require receiving sensor data indicating the position, orientation, and / or rotation of the user's head.

[0273] Fig.37 A system 3700 for enabling an interactive X-reality environment for multiple users is shown in accordance with some embodiments. The system 3700 may include a computing network 3705 including one or more computer servers 3710 connected via one or more high-bandwidth interfaces 3715. The servers in the computing network need not be co-located. The one or more servers 3710 may each include one or more processors for executing program instructions. The servers also include memory for storing program instructions and data used and / or generated by processes performed by the servers under the direction of the program instructions. The system 3700 may include one or more devices 3720 including, for example, an AR display system 80 (e.g., Figure 5B viewing optical component 48).

[0274] Computing network 3705 passes data between servers 3710 and between servers and devices 3720 via one or more data network connections 3730. Examples of such data networks include, but are not limited to, any and all types of public and private data networks, both mobile and wired, including, for example, the interconnection of many such networks, commonly referred to as the Internet. This diagram is not intended to imply any particular media, topology, or protocol.

[0275] In some embodiments, the device may be configured to communicate directly with the computing network 3705 or any server 3710. In some embodiments, the device 3720 may communicate with a remote server 3710 and, optionally, communicate locally with other devices and the AR display system through a local gateway 3740 for processing data and / or passing data between the network 3705 and one or more devices 3720.

[0276] As shown, the gateway 3740 is implemented as a separate hardware component, which includes a processor for executing software instructions and a memory for storing software instructions and data. The gateway has its own wired and / or wireless connection to the data network for communicating with a server 3710 including a computing network 3705. In some embodiments, the gateway 3740 can be integrated with a device 3720 worn or carried by a user. For example, the gateway 3740 can be implemented as a downloadable software application installed and run on a processor included in the device 3720. In one embodiment, the gateway 3740 provides access to the computing network 3705 through the data network 3730 for one or more users. In some embodiments, the gateway 3740 may include communication links 76 and 78.

[0277] The servers 3710 each include, for example, working memory and storage devices for storing data and software programs, microprocessors for executing program instructions, graphics processors and other special processors for rendering and generating graphics, images, video, audio and multimedia files. The computing network 3705 may also include devices for storing data accessed, used or created by the servers 3710. In some embodiments, the computing network 3705 may include a remote processing module 72 and a remote data repository 74.

[0278] The software program running on the server and the optional device 3720 and gateway 3740 is used to generate a digital world (also referred to herein as a virtual world) through which the user interacts with the device 3720. The digital world is represented by data and processes that describe and / or define virtual, non-existent entities, environments, and conditions that can be presented to the user by the device 3720 for user experience and interaction. For example, when instantiated in a scene that the user is watching or experiencing, an object, entity, or item of a certain type that appears to be physically present may include a description of its appearance, its behavior, how to allow the user to interact with it, and other characteristics. The data used to create the environment of the virtual world (including virtual objects) may include, for example, atmospheric data, terrain data, weather data, temperature data, location data, and other data used to define and / or describe the virtual environment. In addition, the data defining various conditions for the operation of the virtual world may include, for example, physical laws, time, spatial relationships, and other data that can be used to define and / or create various conditions for the operation of the virtual world (including virtual objects).

[0279] Unless the context indicates otherwise, this article will generally refer to entities, objects, conditions, characteristics, behaviors or other features of the digital world as objects (e.g., digital objects, virtual objects, rendered physical objects, etc.). Objects can be any type of animate or inanimate object, including but not limited to buildings, plants, vehicles, people, animals, organisms, machines, data, videos, texts, pictures and other users. Objects can also be defined in the digital world to store information about items, behaviors or conditions that actually exist in the physical world. Data that describes or defines an entity, object or item or stores its current state is generally referred to as object data in this article. The data is processed by the server 3710, or by the gateway 3740 or device 3720 according to the implementation, to instantiate an instance of the object and render the object in an appropriate manner for the user to experience through the device 3720.

[0280] Programmers who develop and / or curate digital worlds can create or define objects and the conditions under which objects are instantiated. However, digital worlds can allow others to create or modify objects. Once an object is instantiated, one or more users experiencing the digital world can be allowed to change, control, or manipulate the state of the object.

[0281] For example, in one embodiment, the development, generation, and management of digital worlds is typically provided by one or more system administration programmers. In some embodiments, this may include the development, design, and / or execution of storylines, themes, and events in the digital worlds, as well as the distribution of narratives through various forms of events and media (e.g., film, digital, web, mobile, augmented reality, and live entertainment). System administration programmers may also handle technical management, moderation, and curation of digital worlds and user communities associated therewith, as well as other tasks typically performed by network administrators.

[0282] The user interacts with one or more digital worlds using some type of local computing device, generally referred to as device 3720. Examples of such devices include, but are not limited to, smartphones, tablet devices, heads-up displays (HUDs), game consoles, or any other device that can deliver data and provide an interface or display to the user, and combinations of these devices. In some embodiments, device 3720 may include or communicate with local peripherals or input / output components (e.g., keyboards, mice, joysticks, game controllers, tactile interface devices, motion capture controllers, audio devices, voice devices, projector systems, 3D displays, and holographic 3D contact lenses).

[0283] Fig.38 3800 is a schematic diagram showing an electronic system 3800 according to some embodiments. In some embodiments, the system 3800 may be Fig.37The system 3800 may include a first device 3810 (eg, a first portable device of a first user) and a second device 3820 (eg, a second portable device of a second user). The devices 3810 and 3820 may be, for example, Fig.37 Device 3720 and AR display system 80. Devices 3810 and 3820 can communicate with cloud cache 3802 via networks 3804a and 3804b, respectively. In some embodiments, Fig.37 The cloud cache 3802 is implemented in the memory of one or more servers 3710. The networks 3804a and 3804b can be Fig.37 Examples of data network 3730 and / or local gateway 3740.

[0284] Devices 3810 and 3820 may be separate AR systems (e.g., device 3720). In some embodiments, devices 3810 and 3820 may include AR display systems worn by their respective users. In some embodiments, one of devices 3810 and 3820 may be an AR display system worn by the user; the other may be a smartphone held by the user. Although two devices 3810 and 3820 are shown in this example, it should be understood that system 3800 may include one or more devices, and the one or more devices may run the same type of AR system or different types of AR systems.

[0285] Devices 3810 and 3820 may be portable computing devices. The first device 3810 may include, for example, a processor 3812, a local cache 3814, and one or more AR applications 3816. The processor 3812 may include a computing portion 3812a configured to calculate a location based at least in part on one or more sensors (e.g., Figure 5B The computer-executable instructions are executed based on data collected by the depth sensor 51, the world camera 52, and / or the inertial measurement unit 57 to provide a 3D representation of a portion of the physical world (e.g., 3D reconstruction data).

[0286] The computing portion 3812a may represent the physical world as one or more blocks. Each block may represent an object in a different area of ​​the physical world. Each area may have a corresponding volume. In some embodiments, blocks may represent areas with the same volume. In some embodiments, blocks may represent areas with different volumes. For example, when the physical world is an office, the office may be divided into cubes, each of which may be one cubic foot. A block may include a 3D representation (e.g., 3D reconstruction data) of a cube of the office. In some embodiments, the office may be divided into areas with various volumes, and each volume may include a similar amount of 3D information (e.g., 3D reconstruction data) so that the data size of the 3D representation of each area may be similar. The representation may be formatted to facilitate further processing such as occlusion processing to determine whether a virtual object is occluded by a physical object or physical processing to determine how a virtual object should move or deform when interacting with a physical object in the physical world. Blocks may be formatted, for example, as grid blocks, where features (e.g., corners) of objects in the physical world become points in the grid blocks, or are used as points to create grid blocks. Connections between points in the grid may indicate groups of points on the same surface of a physical object.

[0287] Each tile may have one or more versions, each version containing data representing its corresponding area based on data at a point in time (e.g., volumetric 3D reconstruction data such as voxels, and / or a mesh that may represent a surface in the area represented by the corresponding tile). The computing portion 3812a may create a new version of a tile as additional data becomes available, such as data indicating that an object in the physical world has changed or additional data from which a more accurate representation of the physical world can be created. The additional data may come from sensors on a device (e.g., device 3810 and / or 3820). In some embodiments, the additional data may come from remote sensors and may be obtained, for example, over a network connection.

[0288] The processor 3812 may also include an active memory 3812b that may be configured to store blocks in the field of view of the device. In some embodiments, the active memory 3812b may store blocks outside the field of view of the device. In some embodiments, the active memory 3812b may store blocks adjacent to blocks in the field of view of the device. In some embodiments, the active memory 3812b may store blocks predicted to be in the field of view of the device. In some embodiments, if the block is within the field of view of the device at the time, the processor 3812 maintains the block in the active memory 3812b. The field of view may be determined by the imaging area of ​​one or more sensors. In some embodiments, the field of view may be determined by the amount of the physical world presented to the user of the device or that can be perceived by an ordinary user without using an AR system. Therefore, the field of view may depend on the user's position in the physical world and the orientation of the wearable component of the device.

[0289] If a tile becomes outside the field of view of the device 3810 as the user moves, the processor 3812 may consider the tile inactive. Inactive tiles may be paged out from active memory to a cache. The cache may be a local cache or a remote cache. Fig.38 In the embodiment of the present invention, the block is first paged out to the local cache 3814 through the local gateway 3818b. In some embodiments, the local cache 3814 can be the only available cache.

[0290] In some embodiments, there may be a remote cache accessible via a network. In the illustrated embodiment, a cloud cache (e.g., a remote cache) 3802 accessed via a network 3804a is an example of a remote cache. The processor 3812 can manage when to move blocks between the local cache 3814 and the cloud cache 3802. For example, when the local cache 3814 is full, the processor 3812 can page out blocks to the cloud cache 3802 via the network 3804a. Since the blocks in the local cache 3814 are accessible to render the scene with lower latency than the blocks in the cloud cache, the processor 3812 can select blocks to be paged out from the local cache 3814 using an algorithm designed to retain the blocks that are most likely to become active in the local cache 3814. Such an algorithm can be based on the time of access. In some embodiments, the algorithm can be based on a prediction of the motion of the device that will change the field of view of the device.

[0291] An application (e.g., a computer game) that renders a scene can obtain information representing the portion of the physical world that affects the scene to be rendered. Application 3816 can obtain active blocks from active memory 3812b through local gateway 3818a. In some embodiments, local gateway 3818a can be implemented as an application programming interface (API) so that processor 3812 implements a "service" for application 3816. In an embodiment where data of the physical world is represented as a grid, the service can be a "grid partitioning service." The API can be implemented as a push or pull interface, or can have attributes of both. In a pull interface, for example, application 3816 can indicate the portions of the physical world for which it needs data, and the service can provide data for those portions. For example, in a push system, a service can provide data about portions of the physical world when such data changes or becomes available.

[0292] The portion of the physical world for which relevant data is provided can be limited to the relevant portion indicated by the application 3816, for example, data within the field of view of the device or data representing a portion of the physical world within a threshold distance of the field of view of the device. In a pull / push system, an application 3816 can request data for a portion of the physical world, and the service can provide data about the requested portion plus any adjacent portions where the data has changed. In order to limit the information to that which has changed, in addition to maintaining blocks that describe the physical world, the service can also track which versions of the blocks have been provided to each application 3816. The operation of determining which portion of the representation of the physical world will be updated and where the update occurs can be partitioned between the application 3816 and the service in any suitable manner. Similarly, the location where updated data is incorporated into the representation of the physical world can be partitioned in any suitable manner.

[0293] In some embodiments, the reconstruction service may continuously reconstruct a 3D representation of the physical world while the sensors are capturing and updating data. The data may then be provided to an application 3816 that uses the 3D reconstruction data to render a scene depicting the physical world and virtual reality objects that are located in or interact with objects in the physical world. The data may be provided to the application 3816 via an API that may be implemented as a push interface that pushes data to the application 3816 when relevant parts change, or a pull interface that responds to requests from the application 3816 for the latest information, or both.

[0294] For example, application 3816 may operate on a mesh representation of a portion of the physical world that constitutes a 45 degree viewing angle at a distance of 10 meters relative to an origin defined by the current position of the device and the direction the device is facing. When the area changes or data indicating physical changes within the area becomes available, a mesh may be computed to represent the area. The mesh may be computed in application 3816 based on data provided by the service, or may be computed in the service and provided to application 3816. In both cases, the service may store information in the physical world to simplify the computation of the mesh. As described herein, blocks having zippers, side edges, or implemented using other techniques to facilitate "masking" over cracks between adjacent blocks may be used to enable processing of only the changed portion of the representation of the physical world. The changed portion of the representation of the physical world may then replace the corresponding portion in the previous representation of the physical world.

[0295] Efficiently accessing representations of portions of the physical world for generating meshes to be used by application 3816 to render a scene to a user can reduce computer resources, thereby making the XR system easier to implement on portable devices or other devices with limited computing resources, and can produce a more realistic user experience because the XR scene better matches the physical world. Accordingly, instead of or in addition to using tiles with side edges, zippers, or other techniques to facilitate masking over cracks between tiles, as described elsewhere herein, algorithms for paging tiles into and out of active memory and / or local caches can be selected to reduce access time to tiles needed to compute the mesh at any given time.

[0296] exist Fig.38 In an exemplary embodiment of the invention, the gateway 3818a is a pull interface. When the AR application 3816 requests information about an area of ​​the physical world, but a tile representing the area is not in the active memory 3812b, the processor 3812 may search for the tile held in the local cache 3814. If the processor 3812 cannot find the tile in both the active memory 3812b and the local cache 3814, the processor 3812 may search for the tile held in the cloud cache 3802. Since access to the active memory 3812b has lower latency than access to data in the local cache 3814, and access to data in the local cache 3814 has lower latency than access to data in the cloud cache 3802, the overall speed of generating the mesh may be improved by implementing the services of a paging algorithm that loads tiles into the active memory before they are requested, or moves tiles from the cloud cache 3802 to the local cache 3814 before they are requested.

[0297] Similar to the first device 3810, the second device 3820 may include a processor 3822 having a computing portion 3822a and an active memory 3822b, a local cache 3824, and one or more AR applications 3826. The AR application 3826 may communicate with the processor 3822 via a local gateway 3828a. The local cache 3824 may communicate with the processor 3822 via a local gateway 3828b.

[0298] Therefore, the cloud cache 3802 can retain the blocks sent from both devices 3810 and 3820. The first device 3810 can access the blocks captured and sent from the second device 3820 in the cloud cache 3802; similarly, the second device 3820 can access the blocks captured and sent from the first device 3810 in the cloud cache 3802.

[0299] Devices 3801 and 3802 are provided as examples of portable AR devices. Any suitable device, such as a smartphone, may be similarly used and implemented.

[0300] Fig.39 3900 is a flow chart illustrating a method 3900 of operating a system (e.g., system 3700) according to some embodiments. In action 3902, a device may capture 3D information about a physical world including objects in the physical world, and represent the physical world as a tile including 3D reconstruction data. In some embodiments, the 3D reconstruction data may be captured by a single system and used only for rendering information on that system. In some embodiments, the 3D reconstruction data may be captured by multiple systems and may be used for rendering information on any one of the multiple systems or any other system. In these embodiments, the 3D reconstruction data from multiple systems may be combined and accessible by the multiple systems or any other system.

[0301] For example, several users, each wearing an AR system, can set their devices to augmented mode while exploring a warehouse. The sensor of each device may be capturing 3D information (e.g., 3D reconstruction data including depth maps, images, etc.) of the warehouse in the sensor's viewpoint file, which includes objects in the warehouse (e.g., tables, windows, doors, floors, ceilings, walls). Each device can divide the warehouse into regions with corresponding volumes and represent each region as a block. These blocks can have versions. Each version of a block can have a value representing an object in the region of the physical world at a certain point in time.

[0302] When an application requires information about the physical world, a version of the tile representing that portion of the physical world may be selected and used to generate that information. Although this selection process may be performed by or distributed across any suitable processor, according to some embodiments, the processing may be performed locally on the device on which the application requesting the data is executing.

[0303] Thus, in action 3904, the processor (e.g., processor 3812 or 3822) may respond to a request for 3D reconstruction data from an application (e.g., AR application 3816 or 3826). In some embodiments, regardless of whether the application requests 3D reconstruction data, the device may continue to capture 3D information including 3D reconstruction data about the physical world and represent the physical world as a tile of 3D reconstruction data. The 3D reconstruction data may be used to create a new version of the tile.

[0304] If the application requests 3D reconstruction data, processing can proceed to action 3906, in which the processor can identify a subset of tiles corresponding to a portion of the physical world required to deliver 3D reconstruction data based on the request. The identification of the tiles can be based on data collected, for example, by a sensor (e.g., a depth sensor 51, a world camera 52, an inertial measurement unit 57, a global positioning system, etc.). The multi-device system can create a common coordinate system so that tiles generated by different devices associated with corresponding parts of the physical world can be created using the common coordinate system without regard to which device provides 3D reconstruction data to reconstruct the portion of the physical world represented by the tile. As an example of how a common coordinate system can be created, data from devices in approximately the same vicinity can be routed to the same server or one or more servers for processing. There, the data from each device can initially be represented in a device-specific coordinate system. Once enough data is collected from each device to identify features in a common portion of the physical world, these features can be associated to provide a conversion from one device-specific coordinate system to other device-specific coordinate systems. One of these device-specific coordinate systems may be designated as the common coordinate system, and the other coordinate systems may be converted between and used to convert data from the device-specific coordinate system to the coordinate system designated as the common coordinate system. Regardless of the specific mechanism for creating the common coordinate system, once the common coordinate system is created, 3D reconstruction data requested by an application that generates data for a first device may be based on updates from other devices (if such updates are made after any version made by the first device). Blocks from a first device and other devices may be merged by using, for example, a common coordinate system.

[0305] The specific processing in act 3906 may depend on the nature of the request. In some embodiments, if the application requesting the 3D reconstruction data maintains its own information about the tiles and requests specific tiles, the request for 3D reconstruction data at act 3904 may include a reference to a specific subset of tiles, and identifying the subset of tiles at act 3906 may include determining the subset of tiles that corresponds to the specific subset of tiles. In some embodiments, the request for 3D reconstruction data at act 3904 may include a reference to a field of view of a device on which the application is executing, and identifying the subset of tiles at act 3906 may include determining the subset of tiles that corresponds to a reference field of view of the device.

[0306] Regardless of the manner in which the tiles are identified / determined, in action 3908, the processor may select a version of the tiles for the subset of tiles. The selection may be based on one or more criteria. The criteria may, for example, be based on the latest version of the tile from an available source. In the illustrated embodiment, the versions of the tiles may be stored in active memory, a local cache, or a remote cache. Action 3908 may, for example, include selecting a version in active memory (if available), or if not available, selecting a version in a local cache (if available), or selecting a version from a remote cache (if available). If no tile version is available, the selection may require generating the tile, for example, based on data (e.g., 3D reconstruction data) collected using a sensor (e.g., a depth sensor 51, a world camera 52, and / or an inertial measurement unit 57). This algorithm for tile selection may be used in a system that manages tile versions stored in each possible location, such as a background process. The following is combined with Fig.41 An exemplary management process is described.

[0307] At act 3910, the processor may provide information based on the selected tile version to the application. Processing at act 3910 may entail simply providing the tile to the application, which may be appropriate when the application uses the tile directly. In the case where the application receives a mesh, processing at act 3910 may entail generating a mesh from the tile and / or a subset of the tile and providing the mesh or any appropriate portion of the mesh to the application.

[0308] Fig.40 is a flow chart illustrating details of capturing 3D information about objects in the physical world and representing the physical world as tiles of 3D reconstruction data according to some embodiments. Fig.40 It shows Fig.39 Flowchart of details of action 3902. In action 4002, one or more sensors (e.g., depth sensor 51, world camera 52, inertial measurement unit 57, etc.) of a system (e.g., system 3700) captures 3D information about objects in the physical world, including objects in the physical world.

[0309] In action 4004, a processor of the system (e.g., processor 3812 or 3822) can create a version of the tile including 3D reconstruction data of the physical world based on the 3D information captured by one or more sensors. In some embodiments, each tile can be formatted as one or more parts of a grid. In some embodiments, other representations of the physical world can be used.

[0310] The tile may be versioned such that each time information about an area of ​​the physical world is captured by any device, a new version of the tile may be stored. Each version of the tile may have 3D reconstruction data including values ​​representing an object in the area of ​​the physical world at a point in time. In some embodiments, this processing may be performed locally on the device, resulting in a new version of the tile being stored in active memory. In some embodiments, in a multi-device system, similar processing may be performed on a server (e.g., Fig.37 The server 3710) is executed in the server, which can manage the versions of the blocks so that the latest version of each block available in its remote cache is provided whenever it is requested by any device.

[0311] Because these tiles represent the physical world, most of which will remain unchanged, new versions of the tiles will not necessarily be created when new 3D reconstruction data representing corresponding areas of the physical world is available. Instead, managing versions of the tiles may require processing the 3D reconstruction data representing the physical world to determine whether there have been enough changes since the last version of the tiles representing those areas of the physical world to warrant a change. In some embodiments, sufficient changes may be indicated by the size of a tile metric becoming above a threshold since the last version was stored.

[0312] In some embodiments, when a chunk is requested, other criteria may be applied to determine which version of the chunk is provided as the current version, such as the version with the smallest value of a metric indicating the integrity or accuracy of the data in the chunk. Similar processing may be performed on each device, resulting in a chunk version stored in a local cache on the device.

[0313] One or more techniques may be used to manage the versions of the blocks available to the service on each device. For example, if there is an acceptable version of the block that has been calculated, instead of creating a new version of the block based on the sensor data, the processor may access the previously stored block. Such access may be performed efficiently by managing the storage of versions of the block. In action 4006, the processor of the device may page out a version of the block of 3D reconstruction data of the physical world from active memory (e.g., active memory 3812b or 3822b). Paging may include the processor accessing the sensor data so as to continuously update the blocks in the active memory / local cache / cloud cache, for example, based on the field of view of the device. When the field of view of the device changes, the blocks corresponding to the new field of view may be transferred (e.g., paged) from the local cache and / or cloud cache to the active memory, and the blocks corresponding to the area just outside the new field of view (e.g., blocks adjacent to the blocks in the new field of view) may be transferred (e.g., paged) from the active memory and / or cloud cache to the local cache. For example, in action 4008, versions of the chunks paged out by the processor may be retained in local memory (e.g., local cache 3814 or 3824) and / or remote memory (e.g., cloud cache 3802). In some embodiments, as each new version of a chunk is created, for example on a device, the version may be sent to remote memory so that other users may access it.

[0314] Fig.41 is a flow chart illustrating an exemplary process for performing selection of a version of a block representing a subset of blocks according to some embodiments. Fig.41 It is shown Fig.39 4102. To select a version of each tile in the subset of tiles, in action 4102, a processor (e.g., processor 3812 or 3822) may query whether a latest version is stored in active memory (e.g., active memory 3812b or 3822b). In some embodiments, a determination may be made as to whether a version is the latest by comparing a value attached to the version (e.g., geometry change size, timestamp, etc.) with data collected with a sensor (e.g., depth sensor 51, world camera 52, and / or inertial measurement unit 57). In some embodiments, a comparison may be made between current sensor data and a version of a tile stored in active memory. Based on the extent of the difference, which may represent a change in the physical world or, for example, the quality of the version in active memory, the version in active memory may be considered the latest.

[0315] If the latest version is stored in the active memory, processing proceeds to act 4104, where the latest version is selected. If the latest version is not stored in the active memory, processing proceeds to act 4106, where the processor may query whether the latest version is stored in the local memory (e.g., local cache 3814 or 3824). The query may be performed using the criteria described above in conjunction with act 4102 or any other suitable criteria. If the latest version is stored in the local memory, then in act 4108, the latest version is selected.

[0316] If the latest version is not stored in the local memory, then in action 4110, the processor can query whether the latest version is stored in the remote memory (e.g., cloud cache 3802). The query can also be performed using the criteria described above in conjunction with action 4102 or any other suitable criteria. If the latest version is stored in the remote memory, then in action 4112, the latest version is selected.

[0317] If the latest version is not stored in the remote memory, the process may proceed to action 4114, where the processor of the device may generate a new version of the tile based on the 3D information (e.g., 3D reconstruction data) captured by the sensor. In some embodiments, in action 4116, the processor may identify neighboring tiles of the tile with the new version and update the identified neighboring tiles according to the new version of the tile.

[0318] Fig.42 is a flow diagram illustrating a method 4200 of an operating system according to some embodiments. In method 4200, rather than pulling tiles into active memory and / or local cache when the device needs those tiles, paging can be managed based on a projection of the device's field of view based on device motion.

[0319] Similar to act 3902 , in act 4202 , sensors on the device may capture 3D information about the physical world including objects in the physical world and represent the physical world as a block including 3D reconstruction data.

[0320] In action 4204, a processor (e.g., processor 3812 or 3822) may calculate an area of ​​the physical world, based at least in part on the output of the sensor, at which the portable pointable component (e.g., depth sensor 51, world camera 52, and / or inertial measurement unit 57) will be pointed at some time in the future. In some embodiments, the processor may perform calculations based on analysis results of motion data from an inertial sensor or a captured image. In simple calculations, for example, to obtain quick results, the processor may perform calculations based on translation and rotation of the user's head. When applying a more comprehensive algorithm, the processor may perform calculations based on objects in the scene. For example, the algorithm may consider that a user walking toward a wall or table is unlikely to walk through the wall or table.

[0321] In act 4206, the processor may select a block based on the calculated area. In act 4208, the processor may update the active memory (e.g., active memory 3812b or 3822b) using the selected block. In some embodiments, the processor may select a block based on the calculated area. Fig.41 In action 4210, the processor may select blocks from active memory to provide to an application (eg, application 3816 or 3826), for example, via an API, based on changes to each block since a version of the block was last provided to the application.

[0322] In some embodiments, in action 4206, the processor may request the selected block from the remote memory (e.g., cloud cache 3802) and update the information stored in the local cache (e.g., 3814 or 3824) so ​​that the local cache stores the selected block. Action 4206 may be similar to Fig.39 Action 3908 described in .

[0323] Tile-based processing as described above can be based on tiles that allow portions of a 3D representation to be processed separately and then combined with other tiles. According to some embodiments, tiles can be formatted so that when a tile changes, the changed representation largely or completely maintains the value of the tile at the interface with other tiles. Such processing enables a changed version of a tile to be used with versions of unchanged neighboring tiles without creating unacceptable artifacts in a scene rendered based on the changed and unchanged tiles. Figures 43A-48 Such a block is shown.

[0324] A 3D representation of the physical world can be provided by volumetric 3D reconstruction, which can create a 3D reconstruction data hierarchy of 3D information of the physical world captured by a sensor. For example, the sensor can be a depth camera, which can capture 3D information of the physical world, such as a stream of depth images with various poses of the depth camera (i.e., camera poses). The 3D information of the physical world can be processed into a grid of voxels. Each voxel can contain one or more signed distance functions (SDFs) that describe whether the voxel is inside or outside the geometry of an object in the physical world. Voxels can be grouped into "bricks". Each brick can include, for example, a plurality of voxels of a cubic volume, such as 8 3 Voxels. Bricks can be further grouped into "tiles". Each tile can contain multiple bricks.

[0325] In some embodiments, the voxel grid may be mapped to conform to a memory structure. A tile may correspond to a storage page of a storage medium. The size of a tile may be variable, for example, depending on the size of the storage page of the storage medium used. Thus, 3D reconstruction data may be sent between storage media (e.g., active memory and / or local memory of a device, and / or remote memory in the cloud) based on a tile. In some embodiments, one or more tiles may be processed to generate a block. A block may be updated, for example, when at least one voxel in one or more tiles changes.

[0326] A block may not necessarily be limited to corresponding to a tile. In some embodiments, a block may be generated from one or more bricks, one or more voxels, or one or more SDF samples, etc. A block may be any suitable partition of the physical world. A block need not necessarily be limited to the format of a grid. A block may be any suitable format for 3D reconstruction data.

[0327] Fig.43A -D shows an exemplary physical world 4300 represented by grid blocks 4302. Each grid block 4302 can be extracted from voxels 4304 corresponding to a predetermined volume of the grid block. In the example shown, each block can be a cubic area of ​​voxels in the low-level reconstruction model (e.g., 1m 3 ) output. Each grid tile 4302 may contain a portion of the world grid and may be processed independently. Since some tiles change when exploring new areas or things moving in the environment, scalability can be achieved through fast local updates. In the example shown, none of the grid tiles have changed except for grid tile 4306 (which has a new object 4308 placed in front of an existing surface 4310). In this case, the AR system only needs to update grid tile 4306, which can save a lot of computing power compared to arbitrarily updating the entire grid of the world.

[0328] Fig.43B is a simplified schematic diagram illustrating a mesh block according to some embodiments. In the example shown, the mesh block may have a fully connected mesh inside, meaning that vertices are shared by multiple triangles.

[0329] On the other hand, individual grid blocks may be independent grids that are not connected. Fig.43C is a simplified schematic diagram illustrating a crack according to some embodiments, which may exist at the edge of two adjacent mesh blocks. Fig.43D is a diagram showing that according to some embodiments, the side edges of the grid that overlap with adjacent grid blocks are covered. Fig.43C Simplified schematic diagram of the crack in .

[0330] Fig.44 4400 is a schematic diagram showing a representation 4400 of a portion of a physical world segmented in 2D according to some embodiments. The 2D representation 4400 can be obtained by connecting four sets of tiles (tiles AD). The representation 4400 can be segmented into four tiles: tiles 4402, 4404, 4406, and 4408, for example, as shown by solid lines 4410. In some embodiments, tiles 4402, 4404, 4406, and 4408 can be designated as tiles AD, respectively. An application may require 3D reconstruction data in a mesh format for further processing, such as occlusion testing, and generating physical effects in a physics engine. In some embodiments, the tile set can be in the format of a mesh, which can be generated by a device (e.g., device 3810, 3820), a network (e.g., a cloud including a cloud cache 3802), or a discrete application (e.g., application 3816, 3826).

[0331] In some embodiments, the area at the border of blocks 4402, 4404, 4406, and 4408 may be, for example, a side edge shown by dashed line 4412. In some embodiments, each block in blocks AD may include a block and a corresponding side edge. For example, block B may include block 4404 and a side edge 4414 that partially overlaps the border of neighboring blocks A, C, and D of block B so that cracks between blocks can be covered when the blocks are connected into a global grid. Blocks A, C, and D may also include corresponding side edges. Therefore, before returning the block including 3D reconstruction data to the application, the processor can cover any cracks between blocks.

[0332] In some embodiments, the global mesh may be a topologically connected global mesh. For example, adjacent tiles in a tile set may share mesh vertices at a tile boundary such as line 4410. In some embodiments, the global mesh may be visually seamless despite being topologically disconnected using any suitable technique (e.g., skirting and zippers).

[0333] Although an approach using side edges is shown, other approaches may be used to enable a changed block to be combined with an unchanged adjacent block, such as a zipper. Although in the example shown, a portion of the physical world is represented by four 2D blocks, it should be appreciated that a portion of the physical world may be represented by any suitable number (e.g., two, three, five, six, or more) of 2D and / or 3D blocks. Each block may correspond to a space in the physical world. In some embodiments, blocks in a 2D and / or 3D representation of a portion of the physical world may correspond to spaces of the same size (e.g., area / volume) in the physical world. In some embodiments, blocks in a 2D and / or 3D representation of a portion of the physical world may correspond to spaces of different sizes in the physical world.

[0334] Fig.45 4500 is a schematic diagram showing a 3D representation 4500 of a portion of the physical world according to some embodiments. Similar to the 2D representation 4400, the 3D representation 4500 can be obtained by connecting eight blocks (blocks AH). In some embodiments, blocks AH can be exclusive of each other, for example, without overlapping areas. In some embodiments, blocks AH may have an area (e.g., side edge 4516) that overlaps with adjacent blocks. In some embodiments, each block in blocks AH may have a version. Each version of a block may have a value representing an object in an area of ​​the physical world at a certain point in time. In the example shown, the 3D representation 4500 includes versions of blocks AH: version 4502 of block A, version 4504 of block B, version 4514 of block C, version 4512 of block D, version 4534 of block E, version 4506 of block F, version 4508 of block G, and version 4510 of block H. Version 4502 of block A may include value 4518; version 4504 of block B may include value 4522; version 4514 of block C may include value 4528; version 4512 of block D may include value 4532; version 4534 of block E may include value 4520; version 4506 may include value 4524; version 4508 may include value 4526; version 4510 may include value 4530.

[0335] Fig.464 is a schematic diagram showing a 3D representation 4600 of a portion of the physical world obtained by updating the 3D representation 4500 according to some embodiments. Compared to the 3D representation 4500, the 3D representation 4600 may have a new version 4610 of the tile H including information 4630. The information 4630 may be different from the information 4530. For example, the first device may retain the version 4510 of the tile H in the remote memory. The version 4510 of the tile H may include information 4530 corresponding to a table with an empty surface. After the first device leaves the area (e.g., the field of view of the first device no longer includes the tile H), the second device may place a virtual and / or physical box on the surface of the table and then retain the version 4610 of the tile H in the remote memory. The version 4610 of the tile H may include information 4630 corresponding to the table with the virtual and / or physical box. If the first device returns, the first device can select version 4610 of block H to view from the available versions of block H (including versions 4610 and 4510 of block H).

[0336] Fig.47 4702A is a schematic diagram showing an augmented world 4700 that can be viewed by a first device 4702 (e.g., device 3810) and a second device 4712 (e.g., device 3820). The first and second devices may include AR display systems 4704 and 4714 (e.g., AR display system 80) operating in augmented mode. The augmented world 4700 can be obtained by connecting four tiles (tiles AD). In the example shown, the augmented world 4700 includes versions of tiles AD: version 4702A of tile A, version 4702B of tile B, version 4702C of tile C, and version 4702D of tile D. The first device 4702 may be looking at a first direction 4706 and have a first field of view (FOV) 4708. In the example shown, the first FOV includes version 4702B of tile B and version 4702D of tile D. The processor (e.g., 3812) of the second device 4704 may include computer-executable instructions for identifying tiles B and D corresponding to the first FOV and selecting version 4702B of tile B and version 4702D of tile D. The second device 4714 may be looking in a second direction 4716 and have a second FOV 4718. In the example shown, the second FOV includes version 4702C of tile C and version 4702D of tile D. The processor (e.g., 3822) of the second device 4714 may include computer-executable instructions for identifying tiles C and D corresponding to the second FOV and selecting version 4702C of tile C and version 4702D of tile D.

[0337] Fig.484800 is a schematic diagram illustrating an augmented world 4800 obtained by updating the augmented world 4700 with a new version of a tile according to some embodiments. Compared to augmented world 4700, augmented world 4800 may include a version 4802C of tile C that is different from version 4702C, and a version 4802D of tile D that is different from version 4702D. First device 4702 may be looking in a third direction 4806 and have a third FOV 4808. In the example shown, the third FOV includes version 4802C of tile C and version 4802D of tile D. The processor of first device 4702 may include computer executable instructions for determining which version of versions 4702C, 4802C, 4702D, and 4802D to provide to an application based on, for example, a change in the FOV and / or information collected with a sensor of first device 4702. In some embodiments, the processor of the first device 4702 may include computer executable instructions for generating a version 4802C of chunk C and a version 4802D of chunk D when there are no corresponding latest versions of chunks C and D in a local memory (e.g., local cache 3814, 3824) or a remote memory (e.g., cloud cache 3802). In some embodiments, the first device 4702 is capable of estimating a change in its FOV (e.g., from the first FOV 4708 to the third FOV 4808), selecting chunk C based on the estimation, and storing the version of chunk C in a memory closer to the processor (e.g., moving the version of chunk C from the remote memory to the local cache, or from the local cache to the active memory).

[0338] A method for occlusion rendering using ray casting and real-time depth

[0339] The realism of AR and MR scenes presented to users may be enhanced by providing occlusion data to applications generating AR and MR scenes, where the occlusion data is derived from a combination of one or more depth data sources. The occlusion data may represent surfaces of physical objects in the scene and may be formatted in any suitable manner, such as by depth data indicating distances from a viewpoint and surfaces for rendering the scene. For example, usage component 164 may retrieve information from perception module 160 ( Figure 6 ) receives occlusion data.

[0340] However, in some embodiments, one data source may be one or more depth cameras that directly sense and capture the position between the depth camera and real objects in the physical world. Data from the depth camera may be provided directly to the use component 164, or may be provided indirectly, for example, through the perception module 160. One or more depth cameras may provide an instant view of the physical world at a frame rate that is high enough to capture changes in the physical world but low enough not to increase the processing burden. In some embodiments, the frame rate may be 5 frames per second, 10 frames per second, 12 frames per second, 15 frames per second, 20 frames per second, 24 frames per second, 30 frames per second, and so on. In some embodiments, the frame rate may be less than 5 frames per second. In some embodiments, the frame rate may be greater than 30 frames per second. Therefore, in some embodiments, the frame rate may be in the range of 1-5 frames per second, 5-10 frames per second, 10-15 frames per second, 15-20 frames per second, or 20-30 frames per second, and so on.

[0341] The second data source may be a stereoscopic vision camera that can capture a visual representation of the physical world. Depth data from a depth camera and / or image data from a visual camera may be processed to extract points representing real objects in the physical world. Images from a visual camera such as a stereo camera may be processed to calculate a three-dimensional (3D) reconstruction of the physical world. In some embodiments, the depth data may be generated, for example, using deep learning techniques based on images from a visual camera. Some or all of the 3D reconstruction may be calculated before the occlusion data and stored in a memory. In some embodiments, the 3D reconstruction may be maintained in a computer memory by a process unrelated to any process for generating depth information for occlusion processing, and the occlusion processing may access the stored 3D reconstruction as needed. In some embodiments, the 3D reconstruction may be maintained in a memory, and portions thereof may be updated in response to the following indications, such as changes in the physical world calculated based on the depth information corresponding to portions of the 3D reconstruction. In some embodiments, the second data source may be implemented by raycasting into a 3D reconstruction of the physical world to obtain low-level 3D reconstruction data (e.g., a raycast point cloud). Through raycasting, data from the second data source may be selected to fill any holes in the occlusion data, enabling integration of data from two (or more) sources.

[0342] According to some embodiments, the depth data and / or image data and / or low-level data of the 3D reconstruction may be oriented relative to a user of the AR or MR system. For example, such orientation may be achieved by using data from a sensor worn by the user. The sensor may be worn, for example, on a head mounted display device / unit.

[0343] In a system in which occlusion data can be generated from multiple depth data sources, the system can include a filter that identifies which portions of the 3D region are represented by data from each of the multiple depth data sources. The filter can apply one or more criteria to identify portions of the region for which data from a second data source is to be collected. The criteria can be an indication of the reliability of the depth data. Another criterion can be a change detected in a portion of the region as a result of collecting the depth data.

[0344] Selecting between multiple depth data sources to provide data for different parts of the representation of a 3D area can reduce processing time. For example, when the processing required to derive occlusion data from data collected with the first data source is less than the processing required with the second data source, the selection may favor data from the first data source, but use data from the second source when data from the first data source is not available or acceptable. As a specific example, the first data source can be a depth camera and the second data source can be a stereo vision camera. The data from the stereo camera can be formatted as a 3D reconstruction of the physical world. In some embodiments, the 3D reconstruction can be calculated before the occlusion data is needed. Alternatively or additionally, the 3D reconstruction can be recalculated when the occlusion data is needed. In some embodiments, criteria can be applied to determine whether the 3D reconstruction should be recalculated.

[0345] In some embodiments, the occlusion data is calculated by the service, thereby providing the occlusion data to an application executing on a computing device that will render the XR scene through an application programming interface (API). The service can be executed on the same computing device as the application, or can be executed on a remote computer. The service may include one or more of the components discussed herein, such as a filter for data from a first data source, and / or an engine for selectively obtaining data from a second data source based on the filtered data from the first data source. The service may also include a component that combines the filtered data from the first data source with the selected data from the second data source to generate the occlusion data.

[0346] The occlusion data may be formatted in any suitable manner to represent a surface in the physical world. For example, the occlusion data may be formatted as a depth buffer for a surface, storing data identifying the location of the surface in the physical world. The occlusion data may then be used in any suitable manner. In some embodiments, the occlusion data may be provided to one or more applications that want to cause a virtual object to be occluded by a real object. In some embodiments, the occlusion data may be formatted as a depth filter created by the system for an application to request occlusion data from an occlusion service to render a virtual object at one or more locations. The depth filter may identify locations for which an application should not render image information for a virtual object because the virtual objects in these locations will be occluded by surfaces in the physical world. It should be understood that "occlusion data" may have an appropriate format to provide information about a surface in the physical world and need not be used for occlusion processing. In some embodiments, the occlusion data may be used in any application that performs processing based on a representation of a surface in a scene in the physical world.

[0347] Compared to conventional AR and MR systems in which applications use mesh data to perform occlusion processing, the methods described herein provide occlusion data with less latency and / or using lower computational resources. The mesh data may be obtained by processing geometric data extracted by an image sensor using multiple time or cost intensive steps, including a marching cube algorithm, mesh simplification, and applying a triangle count limit. The calculation of the mesh data may take hundreds of milliseconds to several seconds, and when the environment changes dynamically and the application uses an outdated mesh to render the scene, the delay with the latest mesh may cause visible artifacts. These artifacts manifest, for example, as virtual content appears to be superimposed on top of real objects when it is assumed to be rendered behind real objects, which destroys the perception / feeling of immersion for users of such applications and provides users with incorrect cues for 3D depth perception.

[0348] For applications that use a mesh for occlusion processing to have an up-to-date mesh, the application must either continuously query the mesh (resulting in a lot of continuous processing) or utilize a mechanism to determine if there has been a change and then query the new mesh (which reduces overall processing, but still has higher latency between changes in the physical world and when the mesh reflecting those changes reaches the application).

[0349] By directly using low-level data from 3D reconstruction data (such as point clouds) and real-time depth data instead of meshes for occlusion, the delay between changes occurring in the environment and being reflected in the occlusion data is reduced, allowing for closer synchronization with the physical world and, therefore, higher perceived visual quality.

[0350] In some embodiments, a real-time depth map of the physical environment can be obtained from a depth sensor (e.g., a depth camera). Each pixel in the depth map can correspond to a discrete distance measurement captured from a 3D point in the environment. In some embodiments, these depth cameras can provide a depth map including a set of points at a real-time rate. However, the depth map may have holes, which may be caused by the depth camera being unable to obtain sensor data representing the area or obtaining incorrect or unreliable data representing the area. In some embodiments, if the depth sensor uses infrared (IR) light, these holes may be generated, for example, because materials or structures in the physical environment do not reflect IR light well or do not reflect IR light at all. In some embodiments, these holes may be generated, for example, by very thin structures or surfaces at grazing angles of incidence that do not reflect light toward the depth sensor. The depth sensor may also encounter motion blur when moving quickly, which may also result in lost data. In addition, a "hole" in the depth map represents an area of ​​the depth map that is not suitable for use in occlusion processing for any other reason. Such holes can be detected using any appropriate processing, such as processing the depth map to detect insufficient connectivity between points or areas in the depth map. As another example, holes can be detected using a process that computes a quality metric for regions of a depth map and treats regions with a low quality metric as holes. One such metric can be the image-to-image variation of pixels in the depth map that represent the same location in the physical world. Pixels with such variation that exceeds a threshold can be classified as holes. In some embodiments, holes can be identified by pixels that meet a predetermined statistical criterion for a cluster of pixels where the quality metric is below a threshold.

[0351] In some embodiments, the depth map may first be "filtered" to identify holes. Then, rays from the perspective from which the scene will be rendered to the holes may be determined. These rays may be "cast" into a 3D representation of the physical world created using multiple sensors rather than a single depth sensor to identify data representing the area of ​​the hole. The 3D representation of the physical world may be, for example, a 3D reconstruction created from data from a stereo vision camera. Data from the 3D reconstruction identified by such ray casting may be added to the depth map, thereby filling in the hole.

[0352] When the holes are identified, a 3D reconstruction may be calculated based on the image sensor data. Alternatively, some or all of the 3D reconstruction may be pre-calculated and stored in memory. For example, the 3D reconstruction may be maintained in computer memory by a process independent of any process that generates depth information for occlusion handling, and the process may access the stored 3D reconstruction as needed. As a further alternative, the 3D reconstruction may be maintained in memory, but portions of it may be updated in response to an indication calculated based on the depth information that there has been a change in the physical world that corresponds to portions of the 3D reconstruction.

[0353] In an XR system, the rays may have the same pose as the user's eye gaze. In an exemplary system described below, a depth map may similarly be acquired with the same eye gaze as the user because the depth sensor may be worn by the user and may be mounted on the user's head near the eyes. The user may similarly wear a visual camera for forming 3D reconstruction data such that images and data derived from those images may be associated with a coordinate system that enables rays defined relative to the depth map to be projected into a 3D reconstruction calculated from the visual images. An inertial measurement unit and / or other sensors similarly worn by the user and / or associated with the sensor may provide data to perform coordinate transformations to add data to the 3D representation regardless of the pose of the visual camera, as well as to associate rays defined relative to the depth map with the 3D reconstruction.

[0354] In some embodiments, the user's focus or related virtual content placement information can guide ray casting to make ray casting adaptive in image space by casting denser rays at depth discontinuities to obtain high-quality occlusion at object boundaries and casting sparse rays in the center of the object to reduce processing requirements. Ray casting can additionally provide local 3D surface information (such as normals and positions) that can be used to improve time warping processing with depth information and alleviate the loss of visible pixels that need to be rendered or ray traced in a typical rendering engine. Time warping is a technique in XR that modifies the rendered image before sending it to the display to correct for calculated head movement that would occur between rendering and display. In some embodiments, time warping can be used to synchronize data from a depth map with a 3D representation of the physical world, which can be used to generate data to fill holes in the depth map. Data from both data sources can be time warped to represent poses calculated at the time of display. In some embodiments, data from the 3D representation can be time warped to represent poses calculated when the data was captured with the depth map.

[0355] In some embodiments, advanced features such as time warping can utilize 3D local surface information from ray casting. When rendering a content frame without physical world occlusion or depth imaging with erosion, the time warp can fill in all the missing visible pixels that were previously occluded. Therefore, the rendering engine may not be required to fill in pixels, enabling more loosely decoupled rendering applications (or more independent time warps).

[0356] The processing described above may be performed in many suitable forms of hardware processors on data acquired through many suitable sensors and presented on many suitable interfaces. An example of a suitable system including sensors, processing, and user interfaces is presented below. In the illustrated embodiment, the "service" may be implemented as part of an XR system with computer executable instructions. The execution of these instructions may control one or more processors to access sensor data and then generate depth information and provide it to an application executed on the XR system. These instructions may be executed on the same processor or the same device that executes the application that presents the XR scene to the user, or may be executed on a remote device that the user device accesses over a computer network.

[0357] Fig.49 An occlusion rendering system 4900 is shown in accordance with some embodiments. The occlusion rendering system 4900 may include a reconstruction filter 4902. The reconstruction filter 4902 may receive depth information 4904. In some embodiments, the depth information 4904 may be a sequence of depth images captured by a depth camera. In some embodiments, the depth information 4904 may be derived from a sequence of images captured by a visual camera, for example using structure based on motion of a single camera and / or using stereo calculations based on two cameras. Fig.50 A depth image 5000 according to some embodiments is shown. In some embodiments, surface information can be generated from the depth information. The surface information may indicate the distance to a physical object in the field of view (FOV) of a head mounted display device including a depth camera and / or a visual camera. The surface information may be updated in real time as the scene and FOV change.

[0358] The second source of depth information is shown as 3D reconstruction 4908. 3D reconstruction 4908 may include a 3D representation of the physical world. A 3D representation of the physical world may be created and / or maintained in a computer memory. In some embodiments, the 3D representation may be generated from an image captured by a visual camera using, for example, a structure from motion based on a single camera and / or using stereo calculations based on two cameras. In some embodiments, the 3D representation may be generated from a depth image captured by a depth camera. For example, 3D reconstruction 4908 may be created and / or updated using depth information 4904 in conjunction with the posture of the depth camera relative to the world origin. The representation may be established and modified over time, such as when a user wearing the camera looks around in the physical world. In some embodiments, depth information 4904 may also be used to generate a 3D representation of the physical world. 3D reconstruction 4908 may be a volumetric reconstruction including 3D voxels. In some embodiments, each 3D voxel may represent a cubic space (e.g., 0.5 meters by 0.5 meters by 0.5 meters), and each 3D voxel may include data related to and / or describing a surface in the real world in the cubic space.

[0359] The 3D reconstruction 4908 of the world may be stored in any suitable manner. In some embodiments, the 3D reconstruction 4908 may be stored as a "cloud" of points representing features of objects in the physical world. In some embodiments, the 3D reconstruction 4908 may be stored as a mesh in which groups of points define vertices of triangles representing surfaces. In some embodiments, other techniques such as a room layout detection system and / or object detection may be used to generate the 3D reconstruction 4908. In some embodiments, multiple techniques may be used together to generate the 3D reconstruction 4908. For example, object detection may be used for known physical objects in the physical world, 3D modeling may be used for unknown physical objects in the physical world, and a room layout detection system may also be used to identify boundaries in the physical world, such as walls and floors.

[0360] The reconstruction filter 4902 may include computer executable instructions for generating a depth map based on the depth information 4904. The depth map may include one or more pixels. Each pixel may indicate a distance to a point on a surface in the physical world. In some embodiments, the reconstruction filter 4902 may synthesize the depth information 4904 and data from the ray casting engine 4906. In some embodiments, the reconstruction filter 4902 may reduce or remove noise from the depth information 4904 based at least in part on data from the ray casting engine 4902 and / or from a synthesis of the depth information 4904 and data from the ray casting engine 4906. In some embodiments, the reconstruction filter 4902 may upsample the depth information 4904 using deep learning techniques.

[0361] Reconstruction filter 4902 may identify regions of the depth map based on a quality metric. For example, when a pixel's quality metric is above a threshold, the pixel may be determined to be incorrect or noisy. Regions of the depth map containing incorrect or noisy pixels may be referred to as holes (e.g., hole 5002).

[0362] Fig.54A and 54B An alternative example of how holes may be created in a depth map in embodiments where the depth map is composed of multiple depth images is provided. Fig.54A is a schematic diagram of an area imaged from a first-person perspective using a depth camera to identify voxels occupied by a surface and areas of empty voxels. Fig.54B is a schematic diagram of imaging with a depth camera from multiple viewpoints to identify regions of voxels occupied by a surface and empty voxels. Fig.54BA plurality of voxels are shown that have been determined to be occupied by a surface or empty by fusing data from multiple camera images. However, voxels in region 5420 have not yet been imaged. Region 5420 may have been imaged at position 5422 using a depth camera, but the camera has not yet moved to position. Thus, region 5420 is an observed space for which no volumetric information is available. The AR system may guide a user wearing it to scan the observed space.

[0363] return Fig.49 , the reconstruction filter 4902 can inform the ray casting engine 4906 about the location of the hole. The ray casting engine 4906 can generate a view of the physical world given the user's pose and can remove the hole from the depth map. The data may represent a portion of the user's current view of the physical world at the current time or, if time warping is used, at a time when the data will be time warped. The ray casting engine 4906 can generate one or more 2D images, such as one image for each eye. In some embodiments, the reconstruction filter 4902 can remove areas of the depth map that are separated from the location of the virtual object by more than a threshold distance, because these areas may not be relevant to the occlusion test of the virtual object.

[0364] The ray casting engine 4906 can be implemented by any appropriate technology, and the ray casting engine 4906 generates a view of the physical world given the user's posture. In some embodiments, the ray casting engine 4906 can implement a ray casting algorithm on the 3D reconstruction 4908 to extract data from it. The ray casting algorithm can take the user's posture as input. The ray casting engine 4906 can cast rays from the virtual camera to the 3D reconstruction 4908 of the physical world to obtain surface information missing in the depth map (such as holes). The ray casting engine 4906 can cast dense rays at the boundaries of physical objects in the physical world to obtain high-quality occlusion at the boundaries of the objects, and cast sparse rays in the central area of ​​the physical objects to reduce processing requirements. Then, the ray casting engine 4906 can provide the ray casting point cloud to the reconstruction filter 4902. The ray casting engine 4906 is shown as an example. In some embodiments, the ray casting engine 4906 can be a meshing engine. The meshing engine can implement a meshing algorithm on the 3D reconstruction 4908 to extract data from it, such as including triangles and the connectivity of these triangles. The meshing algorithm can take the user's posture as input.

[0365] The reconstruction filter 4902 may synthesize the depth information 4904 and the data from the ray casting engine 4906, thereby compensating for holes in the depth map from the depth information 4904 with data from the ray casting point cloud from the ray casting engine 4906. In some embodiments, the resolution of the depth map may be improved. This approach may be used to generate a high resolution depth image from a sparse or low resolution depth image.

[0366] The reconstruction filter 4902 may provide the updated depth map to an occlusion service 4910. The occlusion service 4910 may calculate occlusion data based on the updated depth map and information about the positions of virtual objects in the scene. The occlusion data may be a depth buffer of a surface in the physical world. The depth buffer may store the depth of a pixel. In some embodiments, the occlusion service 4910 may be an interface with an application 4912. In some embodiments, the occlusion service 4910 may interface with a graphics system. In these embodiments, the graphics system may expose a depth buffer to the application 4912, wherein the depth buffer is pre-filled with occlusion data.

[0367] The occlusion service 4910 may provide occlusion data to one or more applications 4912. In some embodiments, the occlusion data may correspond to a user's gesture. In some embodiments, the occlusion data may be a pixel-by-pixel representation. In some embodiments, the occlusion data may be a grid representation. The application 4912 may be configured to execute computer executable instructions based on the occlusion data to render a virtual object in a scene. In some embodiments, occlusion rendering may be performed by a separate graphics system rather than the application 4912. The separate graphics system may use a time warp technique.

[0368] In some embodiments, the reconstruction filter 4902, the ray casting engine 4906, and the occlusion service 4910 may be remote services, such as the remote processing module 72; or the 3D reconstruction 4908 may be stored in a remote memory, such as the remote data repository 74; and the application 4912 may be on the AR display system 80.

[0369] Fig.51A method 5100 for occlusion rendering in an augmented reality (AR) environment in accordance with some embodiments is shown. In action 5102, depth information can be captured from, for example, a depth sensor (e.g., depth sensor 51) on a head mounted display device. The depth information can indicate a distance between the head mounted display device and a physical object. In action 5104, surface information can be generated from the depth information. The surface information can indicate a distance to a physical object in a field of view (FOV) of the head mounted display device and / or a user of the head mounted display device. The surface information can be updated in real time as the scene and FOV change. In action 5106, a portion of a virtual object to be rendered can be calculated based on the surface information and information about the position of the virtual object in the scene.

[0370] Fig.52 Details of action 5104 are shown according to some embodiments. In action 5202, depth information may be filtered to generate a depth map. The depth map may include one or more pixels. Each pixel may indicate a distance to a point of the physical object. In action 5204, low-level data for a 3D reconstruction of the physical object may be selectively acquired, for example, from 3D reconstruction 4908. In action 5206, surface information may be generated based on the depth map and the selectively acquired low-level data for the 3D reconstruction of the physical object.

[0371] Fig.53 Details of act 5202 are shown in accordance with some embodiments. In act 5302, a quality metric for a region of a depth map can be determined. The quality metric can indicate whether a region of the depth map is incorrect or noisy. In act 5304, holes in the depth map can be identified based on the quality metric, for example, by comparing to a threshold. In act 5306, the identified holes can be removed from the depth map.

[0372] in conclusion

[0373] Having thus described several aspects of some embodiments, it is to be appreciated various alterations, modifications, and improvements will readily occur to those skilled in the art.

[0374] As an example, embodiments are described in conjunction with an augmented (AR) environment. It should be understood that some or all of the techniques described herein may be applied in an MR environment or more generally in other XR environments and VR environments.

[0375] As another example, embodiments are described in conjunction with devices such as wearable devices. It should be appreciated that some or all of the techniques described herein may be implemented via a network (eg, cloud), discrete applications, and / or any suitable combination of devices, networks, and discrete applications.

[0376] Such changes, modifications and improvements are intended to be part of the present disclosure and are intended to fall within the spirit and scope of the present disclosure. In addition, although the advantages of the present disclosure are indicated, it should be understood that not every embodiment of the present disclosure will include every described advantage. Some embodiments may not implement any of the features described herein and in some cases as being advantageous. Therefore, the foregoing description and accompanying drawings are intended to be examples only.

[0377] The above-mentioned embodiments of the present disclosure may be implemented in any of a variety of ways. For example, the embodiments may be implemented using hardware, software, or a combination thereof. When implemented in software, the software code may be executed on any suitable processor or processor set, whether provided in a single computer or distributed in multiple computers. Such a processor may be implemented as an integrated circuit, with one or more processors in an integrated circuit component, including commercially available integrated circuit components such as CPU chips, GPU chips, microprocessors, microcontrollers, or coprocessors known in the art. In some embodiments, the processor may be implemented in a custom circuit such as an ASIC or a semi-custom circuit generated by configuring a programmable logic device. As another alternative, the processor may be part of a larger circuit or semiconductor device, whether commercially available, semi-custom, or custom. As a specific example, some commercially available microprocessors have multiple cores, so that one or a subset of these cores may constitute a processor. However, a circuit of any appropriate format may be used to implement the processor.

[0378] Furthermore, it should be understood that a computer may be embodied in any of a variety of forms, such as a rack-mounted computer, a desktop computer, a laptop computer, or a tablet computer. Additionally, a computer may be embedded in a device not generally considered a computer but having appropriate processing capabilities, including a personal digital assistant (PDA), a smart phone, or any other appropriate portable or fixed electronic device.

[0379] In addition, the computer may have one or more input and output devices. These devices can be used in particular for presenting a user interface. Examples of output devices that can be used to provide a user interface include a printer or display screen for visual presentation output, and a speaker or other sound generating device for auditory presentation output. Examples of input devices that can be used for a user interface include a keyboard and a pointing device, such as a mouse, a touch pad, and a digitized tablet computer. As another example, a computer can receive input information by voice recognition or other audible formats. In the illustrated embodiment, the input / output device is shown as being physically separated from the computing device. However, in some embodiments, the input and / or output device can be physically integrated into the same unit as other elements of a processor or computing device. For example, a keyboard can be implemented as a soft keyboard on a touch screen. In some embodiments, the input / output device can be completely disconnected from the computing device and functionally integrated by a wireless connection.

[0380] Such computers may be interconnected by one or more networks of any suitable form, including as a local area network or a wide area network such as an enterprise network or the Internet. Such a network may be based on any suitable technology and may operate according to any suitable protocol, and may include wireless networks, wired networks, or fiber optic networks.

[0381] In addition, the various methods or processes outlined herein may be encoded as software that can be executed on one or more processors using any of a variety of operating systems or platforms. In addition, such software may be written using any of a variety of suitable programming languages ​​and / or programming or scripting tools, and may also be compiled into executable machine language code or intermediate code that is executed on a framework or virtual machine.

[0382] In this regard, the present disclosure may be embodied as a computer-readable storage medium (or multiple computer-readable media) encoded with one or more programs (e.g., computer memory, one or more floppy disks, compact discs (CDs), optical disks, digital video discs (DVDs), tapes, flash memory, circuit configurations in field programmable gate arrays or other semiconductor devices, or other tangible computer storage media), which, when executed on one or more computers or other processors, perform methods for implementing the various embodiments of the present disclosure discussed above. It is apparent from the foregoing examples that the computer-readable storage medium can retain information for a sufficient time to provide computer-executable instructions in a non-temporary form. Such one or more computer-readable storage media may be removable so that one or more programs stored thereon may be loaded onto one or more different computers or other processors to implement various aspects of the present disclosure as described above. As used herein, the term "computer-readable storage medium" includes only computer-readable media that can be considered as manufacturing (i.e., manufactured goods) or machines. In some embodiments, the present disclosure may be embodied as a computer-readable medium other than a computer-readable storage medium, such as a propagation signal.

[0383] The term "program" or "software" is used herein in a general sense to refer to any type of computer code or computer executable instruction set that can be used to program a computer or other processor to implement various aspects of the present disclosure as described above. In addition, it should be understood that according to one aspect of this embodiment, one or more computer programs that perform the methods of the present disclosure when executed do not need to reside on a single computer or processor, but can be distributed in a modular manner among multiple different computers or processors to implement various aspects of the present disclosure.

[0384] Computer executable instructions may take a variety of forms, such as program modules that are executed by one or more computers or other devices. Typically, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Typically, in various embodiments, the functionality of program modules may be combined or distributed as desired.

[0385] In addition, the data structure can be stored in a computer-readable medium in any suitable form. To simplify the description, the data structure can be shown as having fields that are related by position in the data structure. Such a relationship can also be achieved by allocating storage devices with position fields in a computer-readable medium that conveys the relationship between the fields. However, any suitable mechanism can be used to establish the relationship between the information in the fields of the data structure, including by using pointers, tags, or other mechanisms that establish relationships between data elements.

[0386] The various aspects of the present disclosure may be used individually, in combination, or in various arrangements not specifically discussed in the foregoing embodiments, and are therefore not limited to the details and arrangements of the components described in the foregoing description or shown in the drawings. For example, aspects described in one embodiment may be combined in any manner with aspects described in other embodiments.

[0387] In addition, the present disclosure may be embodied as a method, examples of which have been provided. The actions performed as part of the method may be ordered in any suitable manner. Thus, embodiments may be constructed in which the actions are performed in an order different from that shown, which may include performing some actions simultaneously even though they are shown as sequential actions in the illustrative embodiments.

[0388] The use of ordinal terms (e.g., "first," "second," "third," etc.) in the claims to modify claim elements does not in itself indicate that one claim element has any priority, precedence, or sequence over another claim or a temporal order of acts of performing a method, but serves merely as a label to distinguish one claim element having a certain name from another element having the same name (but used in ordinal terms) to distinguish these claim elements.

[0389] Furthermore, the phraseology and terminology used herein are for the purpose of description and should not be regarded as limiting. The use of "including" or "having", "comprising", "involving" and variations thereof herein is intended to encompass the items listed thereafter and equivalents thereof as well as additional items.

Claims

1. A method of operating a computing system including one or more processors to generate a model of an environment represented by a grid, the method comprising: dividing the grid into a plurality of first grid blocks using a processor of the one or more processors; performing, with a processor of the one or more processors, a first simplification operation to reduce the complexity of each of the plurality of selected first grid blocks to generate a plurality of second grid blocks; performing, with a processor of the one or more processors, an area-based operation to detect a shape of each of the second grid blocks to generate a plurality of third grid blocks, wherein the area-based operation comprises a planarization operation, wherein the detected shape is a planar surface; and A second simplification operation is performed with a processor of the one or more processors to simplify each of the third grid blocks to a desired complexity level to generate a fourth grid block.

2. The method according to claim 1, wherein: The first grid tile includes a portion of a representation of the environment having a complexity metric of a first value, and the fourth grid tile includes the portion of the representation of the environment having the complexity metric of a fourth value, the fourth value being less than the first value.

3. The method according to claim 2, wherein: The complexity metric indicates the number of triangles in the corresponding mesh block.

4. The method according to claim 2, wherein: the second grid region comprising the portion of the representation of the environment for which the complexity metric is a second value; the third grid tile comprising the portion of the representation of the environment for which the complexity metric is a third value; as well as The percentage decrease between the third value and the fourth value is greater than the percentage decrease between the first value and the second value.

5. The method according to claim 4, wherein: The percentage reduction between the third value and the fourth value is at least twice as great as the percentage reduction between the first value and the second value.

6. The method according to claim 4, wherein: The fourth grid block is generated by the grid partitioning service using the cross-reality platform; And the method further comprises: receiving, as input to the meshing service, a value representing mesh complexity; as well as The percentage reduction between the third value and the fourth value and the percentage reduction between the first value and the second value are controlled based on the value representing mesh complexity.

7. The method according to claim 6, wherein: The value is provided by an application requesting meshing from the meshing service.

8. The method according to claim 1, wherein: The first simplification operation and the second simplification operation are performed using the same simplification algorithm.

9. The method according to claim 1, wherein: The first simplification operation and the second simplification operation are performed using a triangle reduction algorithm.

10. The method according to claim 1, comprising: The plurality of fourth grid blocks are combined with a plurality of unselected second grid blocks into a new grid of the environment.

11. The method according to claim 1, wherein: Performing the region-based operation includes: detecting a plane region in the selected plurality of second grid blocks by a region growing algorithm; and The detected plane regions in the selected plurality of second grid blocks are simplified.

12. The method according to claim 11, wherein: The detected planar regions in the selected plurality of second mesh blocks are simplified based on the target triangle count.

13. The method according to claim 11, wherein: The area-based operation further includes planarizing the selected plurality of second grid blocks by projecting triangles of the detected planar areas in the selected plurality of second grid blocks onto corresponding planes.

14. The method according to claim 13, further comprising: Before simplifying the detected plane regions in the selected plurality of second grid blocks, plane normals of the detected plane regions in the selected plurality of second grid blocks are adjusted to be substantially perpendicular and parallel to the corresponding plane.

15. The method according to claim 1, wherein: The first simplification operation is performed in parallel on each of the plurality of selected first grid blocks.

16. The method according to claim 1, wherein: The region-based operation is performed in parallel on each of the second grid blocks.

17. The method according to claim 1, wherein: The second simplification operation is performed in parallel on each of the third grid blocks.

18. A method of operating a computing system including one or more processors to generate a model of an environment represented by a grid, the method comprising: dividing the grid into a plurality of first grid blocks using a processor of the one or more processors; performing, with a processor of the one or more processors, a first simplification operation to reduce the complexity of each of the plurality of selected first grid blocks to generate a plurality of second grid blocks; performing, with a processor of the one or more processors, an area-based operation to detect a shape of each of the second grid blocks to generate a plurality of third grid blocks, wherein the area-based operation comprises a planarization operation, wherein the detected shape is a planar surface; and Performing a second simplification operation with a processor of the one or more processors to simplify each of the third grid blocks to achieve a desired complexity level to generate a fourth grid block, wherein performing the first simplification operation on the plurality of selected first grid blocks includes: processing the selected first grid blocks in parallel.

19. A method of operating a computing system including one or more processors to generate a model of an environment represented by a grid, the method comprising: dividing the grid into a plurality of first grid blocks using a processor of the one or more processors; performing, with a processor of the one or more processors, a first simplification operation to reduce the complexity of each of the plurality of selected first grid blocks to generate a plurality of second grid blocks; performing, with a processor of the one or more processors, an area-based operation to detect a shape of each of the second grid blocks to generate a plurality of third grid blocks, wherein the area-based operation comprises a planarization operation, wherein the detected shape is a planar surface; and Performing a second simplification operation with a processor of the one or more processors to simplify each of the third grid blocks to a desired complexity level to generate a fourth grid block, wherein the model of the environment is a three-dimensional 3D model and each first grid block corresponds to a cube in the 3D model of the environment.

Images