Caching and updating dense 3D reconstruction data

Abstract

A method is provided for efficiently updating and managing real-time or offline 3D reconstruction and scan outputs on mobile devices with limited resources and internet connectivity. The method makes new, accurate, and comprehensive 3D reconstruction data available to a variety of available mobile XR applications, either in single-user applications or in multi-user applications that share and update the same 3D reconstruction data. The method includes a block-based 3D data representation that enables local updates and simultaneously maintains neighborhood consistency, and a multi-tiered cache mechanism that efficiently reads, pre-fetches, and stores 3D data for XR applications.

Description

[Technical field]

[0001] This application relates generally to cross-reality systems that use 3D world reconstruction to render scenes. [Background technology]

[0002] A computer may control a human user interface and create an X-reality (XR or cross-reality) environment in which some or all of the XR environment is generated by the computer as it is perceived by the user. These XR environments may be virtual reality (VR), augmented reality (AR), and mixed reality (MR) environments in which some or all of the XR environment may be generated by the computer, in part, using data that describes the environment. This data may describe virtual objects, for example, that may be rendered so that a user can sense or perceive and interact with the virtual objects as part of the physical world. The user may experience these virtual objects as a result of data being rendered and presented through a user interface device, such as a head-mounted display device. The data may be displayed for the user to see, or may control audio played for the user to hear, or may control a tactile (or haptic) interface, allowing the user to experience touch sensations that the user senses or perceives as they feel the virtual objects.

[0003] XR systems can be useful for many applications ranging from areas of scientific visualization, medical training, engineering design and prototyping, remote manipulation and telepresence, and personal entertainment. AR and MR, in contrast to VR, involve one or more virtual objects in association with real objects in the physical world. The experience of virtual objects interacting with real objects greatly enhances the user's enjoyment when using XR systems and also opens up possibilities for a variety of applications that present realistic and easily understandable information about how the physical world can be altered. Summary of the Invention [Means for solving the problem]

[0004] Aspects of the present application relate to methods and apparatus for caching and updating 3D reconstruction data. The inventors have recognized and understood techniques for caching and updating dense 3D reconstruction data in real-time on devices with limited computational resources, such as mobile devices. These techniques may be used together, separately, or in any suitable combination.

[0005] Some embodiments relate to a portable electronic system. The portable electronic system comprises a sensor configured to capture three-dimensional (3D) information about objects in a physical world, an active memory, a local memory, a transceiver configured for communication with the remote memory via a computer network, and a processor communicatively coupled to the sensor, the active memory, the local memory, and the transceiver. The processor is configured to execute computer-executable instructions to provide a 3D representation of a portion of the physical world based at least in part on the 3D information about the objects in the physical world. The 3D representation of the portion of the physical world comprises a plurality of blocks, each block having a value that represents an object in an area of ​​the portion of the physical world at a time. The computer-executable instructions comprise instructions for persisting the plurality of blocks in at least one of the active memory, the local memory, and the remote memory when the portable electronic system changes to an inactive operating status.

[0006] In some embodiments, the inactive operating status is shut down or sleep.

[0007] In some embodiments, the step of persisting the plurality of blocks in at least one of the active memory, the local memory, and the remote memory includes persisting the plurality of blocks in a separate memory in which the plurality of blocks currently reside such that, when the inactive operating status is sleep, the plurality of blocks are accessible by the portable electronic system upon waking up of the portable electronic system.

[0008] In some embodiments, the step of persisting the plurality of blocks in at least one of the active memory, the local memory, and the remote memory includes persisting at least a portion of the blocks in the local memory in the remote memory and persisting the blocks in the active memory in the local memory and / or the remote memory when the inactive operational status is shut down.

[0009] In some embodiments, the computer-executable instructions comprise instructions for calculating one or more new blocks based, at least in part, on incoming 3D information about objects in the physical world, and selecting blocks from the persisted blocks that at least partially overlap with the one or more new blocks such that the persisted blocks around the portable electronic system can be retrieved.

[0010] In some embodiments, selecting blocks from the survived blocks that at least partially overlap with the one or more new blocks comprises comparing the one or more new blocks with the survived blocks.

[0011] In some embodiments, comparing the one or more new blocks to the blocks that are persisted in the active memory includes comparing the one or more new blocks to the blocks that are persisted in the active memory.

[0012] In some embodiments, comparing the one or more new blocks to the blocks that are retained includes comparing the one or more new blocks to the blocks that are retained in the local memory when the active memory does not have blocks that at least partially overlap with the one or more new blocks.

[0013] In some embodiments, comparing the one or more new blocks to the persisted blocks includes comparing the one or more new blocks to the persisted blocks in the remote memory when the local memory does not have blocks that at least partially overlap with the one or more new blocks.

[0014] In some embodiments, the computer-executable instructions comprise instructions for updating a plurality of blocks with one or more new blocks.

[0015] Some embodiments relate to a method of operating a portable electronic system in an augmented reality system, the method including obtaining, with at least one processor, a plurality of blocks comprising three-dimensional (3D) information about a surface in a physical world, storing at least one of the plurality of blocks in an active memory, using a block from the active memory to render virtual content to a user of the portable electronic system, persisting the block in the active memory in response to a transition to an inactive operating status at a first time, and determining whether to use the persisted block to render virtual content to a user of the portable electronic system in response to a transition to an active operating status at a second time after the first time.

[0016] In some embodiments, the block is a mesh block.

[0017] In some embodiments, the method includes paging blocks between the active memory and the local memory based on a current or projected field of view of a user of the portable electronic system.

[0018] In some embodiments, the method includes persisting the block in local memory in response to a transition to an inactive operating status.

[0019] In some embodiments, using the blocks from the active memory to render virtual content to a user includes using the blocks for visual occlusion processing and computing physics-based interactions and / or environmental inference.

[0020] Some embodiments relate to a non-transitory computer-readable medium with instructions stored thereon that, when executed on a processor, perform acts including obtaining a plurality of blocks comprising three-dimensional (3D) information about a surface within the physical world; storing at least one of the plurality of blocks in an active memory; using a block from the active memory to render virtual content to a user of the portable electronic system; persisting the block in the active memory in response to a transition to an inactive operating status at a first time; and determining whether to use the persisted block to render virtual content to a user of the portable electronic system in response to a transition to an active operating status at a second time after the first time.

[0021] The foregoing summary is provided by way of illustration and is not intended to be limiting. The present invention provides, for example, the following: (Item 1) 1. A portable electronic system comprising: a sensor configured to capture three-dimensional (3D) information about objects in a physical world; and An active memory; A local memory; a transceiver configured for communication with a remote memory over a computer network; a processor, the processor being communicatively coupled to the sensor, the active memory, the local memory, and the transceiver, and configured to execute computer-executable instructions and provide a 3D representation of a portion of the physical world based, at least in part, on 3D information about objects in the physical world; Equipped with the 3D representation of the portion of the physical world comprising a plurality of blocks, each block having a value representing an object within an area of ​​the portion of the physical world at a point in time; The computer executable instructions include: When the portable electronic system changes to an inactive operating status, the plurality of blocks are persisted in at least one of the active memory, the local memory, and the remote memory. 23. A portable electronic system comprising: (Item 2) 2. The portable electronic system of claim 1, wherein the inactive operating status is shut down or sleep. (Item 3) Persisting the plurality of blocks in at least one of the active memory, the local memory, and the remote memory includes: and persisting the plurality of blocks in a separate memory in which the plurality of blocks currently resides such that, when the inactive operating status is sleep, the plurality of blocks are accessible by the portable electronic system upon waking up of the portable electronic system. 3. The portable electronic system of claim 2, comprising: (Item 4) Persisting the plurality of blocks in at least one of the active memory, the local memory, and the remote memory includes: persisting at least a portion of blocks in the local memory in the remote memory and persisting blocks in the active memory in the local memory and / or the remote memory when the inactive operational status is shutdown. 3. The portable electronic system of claim 2, comprising: (Item 5) The computer executable instructions include: calculating one or more new blocks based, at least in part, on incoming 3D information about objects in the physical world; and selecting from the persisted blocks surrounding the portable electronic system blocks that at least partially overlap with the one or more new blocks such that the persisted blocks may be read; 2. The portable electronic system of claim 1, comprising instructions for performing the steps of: (Item 6) 6. The portable electronic system of claim 5, wherein selecting blocks from the retained blocks that at least partially overlap with the one or more new blocks includes comparing the one or more new blocks to the retained blocks. (Item 7) Comparing the one or more new blocks with the surviving block includes: Comparing the one or more new blocks with the blocks persisted in the active memory. 7. The portable electronic system of claim 6, comprising: (Item 8) Comparing the one or more new blocks with the surviving block includes: comparing the one or more new blocks with blocks persisted in the local memory when the active memory does not have blocks that at least partially overlap with the one or more new blocks. 8. The portable electronic system of item 7, comprising: (Item 9) Comparing the one or more new blocks with the surviving block includes: comparing the one or more new blocks with blocks persisted in the remote memory when the local memory does not have blocks that at least partially overlap with the one or more new blocks. 9. The portable electronic system of claim 8, comprising: (Item 10) The computer executable instructions include: updating said plurality of blocks with said one or more new blocks; 6. The portable electronic system of claim 5, comprising instructions for: (Item 11) 1. A method of operating a portable electronic system in an augmented reality system, the method comprising: Using at least one processor, obtaining a plurality of blocks comprising three-dimensional (3D) information about a surface in the physical world; storing at least one of said plurality of blocks in an active memory; using blocks from the active memory to render virtual content to a user of the portable electronic system; responsive to a transition to an inactive operating status at a first time, persisting the block in active memory; determining whether to use the persisted blocks to render virtual content to a user of the portable electronic system in response to a transition to an active operating status at a second time after the first time; A method comprising: (Item 12) Item 12. The method according to item 11, wherein the block is a mesh block. (Item 13) 12. The method of claim 11, comprising paging blocks between the active memory and a local memory based on a current or projected field of view of a user of the portable electronic system. (Item 14) 12. The method of claim 11, further comprising, in response to a transition to an inactive operating status, causing the block to persist in the local memory. (Item 15) Using blocks from the active memory to render virtual content to the user includes: using said block for visual occlusion processing; Calculating physics-based interactions and / or environmental inferences; Item 12. The method according to item 11, comprising: (Item 16) A non-transitory computer readable medium having instructions stored thereon that, when executed on a processor, obtaining a plurality of blocks comprising three-dimensional (3D) information about a surface in the physical world; storing at least one of said plurality of blocks in an active memory; using the blocks from the active memory to render virtual content to a user of the portable electronic system; responsive to a transition to an inactive operating status at a first time, persisting the block in active memory; determining whether to use the persisted blocks to render virtual content to a user of the portable electronic system in response to a transition to an active operating status at a second time after the first time; A non-transitory computer readable medium performing acts including: (Item 17) Item 17. The non-transitory computer-readable medium of item 16, wherein the block is a mesh block. (Item 18) The said act is paging blocks between the active memory and a local memory based on a current or projected field of view of a user of the portable electronic system; 17. The non-transitory computer-readable medium of item 16, comprising: (Item 19) The said act is and persisting the block in the local memory in response to a transition to an inactive operating status. 17. The non-transitory computer-readable medium of item 16, comprising: (Item 20) Using blocks from the active memory to render virtual content to the user includes: using said block for visual occlusion processing; Calculating physics-based interactions and / or environmental inferences; 17. The non-transitory computer-readable medium of item 16, comprising: [Brief description of the drawings]

[0022] The accompanying drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component that is illustrated in various figures is represented by a like numeral. For purposes of clarity, not every component is labeled in every figure.

[0023] [Figure 1] FIG. 1 is a sketch illustrating an example of a simplified augmented reality (AR) scene, according to some embodiments.

[0024] [Diagram 2] FIG. 2 is a sketch of an example simplified AR scene illustrating an example world reconstruction use case including visual occlusion, physics-based interaction, and environmental inference, according to some embodiments.

[0025] [Diagram 3] FIG. 3 is a schematic diagram illustrating data flow within an AR system configured to provide an experience of AR content that interacts with the physical world, according to some embodiments.

[0026] [Figure 4] FIG. 4 is a schematic diagram illustrating an example of an AR display system, according to some embodiments.

[0027] [Figure 5A]FIG. 5A is a schematic diagram illustrating a user wearing an AR display system that renders AR content as the user moves through a physical world environment, according to some embodiments.

[0028] [Figure 5B] FIG. 5B is a schematic diagram illustrating a viewing optics assembly and associated components, according to some embodiments.

[0029] [Figure 6] FIG. 6 is a schematic diagram illustrating an AR system using a world reconstruction system, according to some embodiments.

[0030] [Figure 7A] FIG. 7A is a schematic diagram illustrating a 3D space discretized into voxels, according to some embodiments.

[0031] [Figure 7B] FIG. 7B is a schematic diagram illustrating the reconstruction range for a single viewpoint, according to some embodiments.

[0032] [Figure 7C] FIG. 7C is a schematic diagram illustrating the perception range versus the reconstruction range at a single point, according to some embodiments.

[0033] [Figure 8] 8A-F are schematic diagrams illustrating the reconstruction of a surface in the physical world into a voxel model by an image sensor viewing the surface from multiple positions and viewpoints, according to some embodiments.

[0034] [Figure 9] FIG. 9 is a schematic diagram illustrating a scene represented by bricks containing voxels, surfaces within the scene, and a depth sensor that captures the surfaces in a depth image, according to some embodiments.

[0035] [Figure 10A] FIG. 10A is a schematic diagram illustrating a 3D space represented by eight bricks.

[0036] [Figure 10B] FIG. 10B is a schematic diagram illustrating a voxel grid within the brick of FIG. 10A.

[0037] [Figure 11] FIG. 11 is a schematic diagram illustrating a stereoscopic representation hierarchy, according to some embodiments.

[0038] [Figure 12] FIG. 12 is a flowchart illustrating a method of operating a computing system to generate a 3D reconstruction of a scene, according to some embodiments.

[0039] [Figure 13] FIG. 13 is a flowchart illustrating a method of thinning a portion of a plurality of bricks for the camera frustum of the depth sensor in FIG. 12 according to some embodiments.

[0040] [Figure 14] FIG. 14 is a flow chart illustrating a method for performing the camera frustum tolerance test in FIG. 13 according to some embodiments.

[0041] [Figure 15] FIG. 15 is a flowchart illustrating a method of thinning out a portion of the first plurality of bricks for the depth image in FIG. 12 according to some embodiments.

[0042] [Figure 16] FIG. 16 is a flow chart illustrating a method for performing the first depth image acceptance test in FIG. 15 according to some embodiments.

[0043] [Figure 17]FIG. 17 is a flow chart illustrating a method for performing the second depth image acceptance test in FIG. 15 according to some embodiments.

[0044] [Figure 18] FIG. 18 shows a table used by the method to categorize all pixels within the rectangle in FIG. 17 against a minimum brick value (bmin) and a maximum brick value (bmax) according to some embodiments.

[0045] [Figure 19] 19A-F are schematic diagrams illustrating thinning bricks relative to the camera frustum, according to some embodiments.

[0046] [Figure 20] 20A-B are schematic diagrams illustrating brick thinning for a depth image that includes a surface, according to some embodiments.

[0047] [Figure 21] FIG. 21 is a schematic diagram illustrating a planar extraction system, according to some embodiments.

[0048] [Figure 22] FIG. 22 is a schematic diagram illustrating a portion of the plane extraction system of FIG. 21 along with details regarding the plane extraction, in accordance with some embodiments.

[0049] [Diagram 23] FIG. 23 is a schematic diagram illustrating a scene represented by bricks containing voxels and example planar data within the scene, according to some embodiments.

[0050] [Figure 24] FIG. 24 is a schematic diagram illustrating the planar data store of FIG. 21 according to some embodiments.

[0051] [Diagram 25]FIG. 25 is a schematic diagram illustrating planar geometry extraction when a planar query is sent to the planar data store of FIG. 21 in accordance with some embodiments.

[0052] [Figure 26A] FIG. 26A is a schematic diagram illustrating steps of generating the cover points of FIG. 25 according to some embodiments.

[0053] [Figure 26B] FIG. 26B is a schematic diagram illustrating various planar geometric surfaces that may be extracted from an example rasterized planar mask, according to some embodiments.

[0054] [Figure 27] FIG. 27 illustrates a mesh for a scene, according to some embodiments.

[0055] [Figure 28A] FIG. 28A shows the scene of FIG. 27 represented by an outer rectangular plane, according to some embodiments.

[0056] [Figure 28B] FIG. 28B shows the scene of FIG. 27 represented by an inner rectangular plane, according to some embodiments.

[0057] [Figure 28C] FIG. 28C shows the scene of FIG. 27 represented by a polygonal plane according to some embodiments.

[0058] [Figure 29] FIG. 29 shows the scene of FIG. 27 with the mesh denoised by flattening the mesh shown in FIG. 27 in accordance with some embodiments.

[0059] [Diagram 30] FIG. 30 is a flowchart illustrating a method of generating a model of an environment represented by a mesh, according to some embodiments.

[0060] [Diagram 31] FIG. 31 is a schematic diagram illustrating a 2D representation of a portion of the physical world by four blocks according to some embodiments.

[0061] [Diagram 32] 32A-32D are schematic diagrams illustrating mesh evolution of an example mesh block during multi-stage simplification, according to some embodiments.

[0062] [Diagram 33] 33A and 33B show representations of the same environment with and without simplification through triangle reduction, respectively.

[0063] [Diagram 34] 34A and 34B show zoomed-in representations of the same environment with and without simplification through triangle reduction, respectively.

[0064] [Diagram 35] 35A and 35B show representations of the same environment with and without planarization, respectively.

[0065] [Diagram 36] 36A and 36B show representations of the same environment with and without simplification through removal of disconnected components, respectively.

[0066] [Figure 37] FIG. 37 is a schematic diagram illustrating an electronic system that enables an interactive XReality environment for multiple users, according to some embodiments.

[0067] [Figure 38] FIG. 38 is a schematic diagram illustrating the interaction of components of the electronic system in FIG. 37 according to some embodiments.

[0068] [Figure 39]FIG. 39 is a flow chart illustrating a method of operating the electronic system in FIG. 37 according to some embodiments.

[0069] [Diagram 40] FIG. 40 is a flowchart illustrating a method of capturing 3D information about objects in the physical world and representing the physical world as blocks of the 3D reconstruction in FIG. 39 according to some embodiments.

[0070] [Diagram 41] FIG. 41 is a flowchart illustrating a method of selecting a version of a block that represents a subset of the blocks in FIG. 39 according to some embodiments.

[0071] [Diagram 42] FIG. 42 is a flow chart illustrating a method of operating the electronic system in FIG. 37 according to some embodiments.

[0072] [Figure 43A] FIG. 43A is a simplified schematic diagram illustrating updates being detected in a portion of a physical world represented by a mesh block, according to some embodiments.

[0073] [Figure 43B] FIG. 43B is a simplified schematic diagram illustrating a mesh block, according to some embodiments.

[0074] [Figure 43C] FIG. 43C is a simplified schematic diagram illustrating a crack at the edge of two adjacent mesh blocks, according to some embodiments.

[0075] [Fig. 43D]FIG. 43D is a simplified schematic diagram illustrating the crack in FIG. 43C being concealed by implementing an overlapping mesh skirt on adjacent mesh blocks, according to some embodiments.

[0076] [Diagram 44] FIG. 44 is a schematic diagram illustrating a 2D representation of a portion of the physical world by four blocks according to some embodiments.

[0077] [Diagram 45] FIG. 45 is a schematic diagram illustrating a 3D representation of a portion of the physical world by eight blocks, according to some embodiments.

[0078] [Figure 46] FIG. 46 is a schematic diagram illustrating a 3D representation of a portion of the physical world obtained by updating the 3D representation in FIG. 45 according to some embodiments.

[0079] [Figure 47] FIG. 47 is a schematic diagram illustrating an example of an augmented world viewable by a first and a second user wearing an AR display system, according to some embodiments.

[0080] [Figure 48] FIG. 48 is a schematic diagram illustrating an example of an augmented world obtained by updating the augmented world of FIG. 47 with a new version of a block, according to some embodiments.

[0081] [Figure 49] FIG. 49 is a schematic diagram illustrating an occlusion rendering system according to some embodiments.

[0082] [Figure 50] FIG. 50 is a schematic diagram illustrating a depth image with a hole.

[0083] [Figure 51] FIG. 51 is a flowchart illustrating a method of occlusion rendering in an augmented reality environment according to some embodiments.

[0084] [Figure 52] FIG. 52 is a flowchart illustrating details of the steps of generating surface information from depth information captured by a depth sensor worn by a user in FIG. 51 in accordance with some embodiments.

[0085] [Figure 53] FIG. 53 is a flowchart illustrating details of the step of filtering depth information and generating the depth map in FIG. 52 according to some embodiments.

[0086] [Figure 54A] FIG. 54A is a sketch of an area being imaged with a depth camera from a first viewpoint to identify voxels that are occupied by surfaces and those that are empty.

[0087] [Figure 54B] FIG. 54B is a sketch of an area that has been imaged using depth cameras from multiple viewpoints to identify voxels that are occupied by surfaces and those that are empty, showing a "hole" for which no stereo information is available as a result of voxels within the region of the "hole" that were not imaged using the depth cameras. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0088] Described herein are methods and apparatus for creating and using three-dimensional (3D) world reconstructions in an augmented reality (AR), mixed reality (MR), or virtual reality (VR) system. To provide a user with a realistic AR / MR / VR experience, the AR / MR / VR system must understand the user's physical surroundings to properly correlate the location of virtual objects in relation to real objects. The world reconstruction may be built from images and depth information about those physical surroundings collected with sensors that are part of the AR / MR / VR system. The world reconstruction may then be used by any of a number of components of such a system. For example, the world reconstruction may be used by components that perform visual occlusion handling, calculate physics-based interactions, or perform environmental inference.

[0089] Occlusion processing identifies portions of a virtual object that should not be rendered and / or displayed to the user due to the presence of objects in the physical world that block the user's view of where the virtual object should be perceived by the user. Physics-based interactions are calculated to determine where or how a virtual object will appear to the user. For example, a virtual object may be rendered to appear as if it is resting on a physical object, moving through void, or colliding with the surface of a physical object. World reconstruction provides a model from which information about objects in the physical world can be obtained for such calculations.

[0090] Environmental inference may also use world reconstruction in the process of generating information that can be used in calculating how to render virtual objects. For example, environmental inference may involve identifying clear surfaces by recognizing that they are window panes or glass table tops. From such identification, areas containing physical objects may be classified as not occluding virtual objects, but may be classified as interacting with the virtual object. Environmental inference may also generate information used in other ways, such as identifying stationary objects that can be tracked relative to the user's field of view and calculating the motion of the user's field of view.

[0091] However, there are significant challenges in providing such a system. Substantial processing may be required to compute the world reconstruction. Furthermore, the AR / MR / VR system must correctly figure out how to position the virtual objects relative to the user's head, body, etc. As the user's position relative to the physical environment changes, the relevant parts of the physical world may also change, which may require further processing. Furthermore, the 3D reconstruction data is often required to be updated as objects move within the physical world (e.g., a cup moves across a table). Updates to the data representing the environment the user is experiencing must be performed quickly, without using too many of the computing resources of the computer generating the AR / MR / VR environment, since it is not possible to perform other functions while performing the world reconstruction. Furthermore, processing of the reconstruction data by components that "consume" that data may exacerbate demands on computer resources.

[0092] Known AR / MR / VR systems require high computational computing power (e.g., GPU) to drive real-time world reconstruction only within a predefined reconstruction volume (e.g., a predefined voxel grid). The present inventors have recognized and understood techniques for operating AR / MR / VR systems to provide accurate 3D reconstruction data in real-time using a small amount of computational resources, such as computational power (e.g., a single ARM core), memory (e.g., less than 1 GB), and network bandwidth (e.g., less than 100 Mbps). These techniques relate to reducing the processing required to generate and maintain the world reconstruction, and providing and consuming data with low computational overhead.

[0093] These techniques may include reducing the amount of data processed when updating the world reconstruction, such as by identifying portions of sensor data that are available at any time and using them in creating or updating the world reconstruction. The sensor data may be selected, for example, based on whether it represents a portion of the physical world that is likely to be in the vicinity of a surface of an object to be represented in the world reconstruction.

[0094] In some embodiments, computational resources may be reduced by simplifying the data representing the world reconstruction: a simpler representation may reduce resources for processing, storing, and / or managing that data and its use.

[0095] In some embodiments, computational resource usage may be reduced by representing the physical world in blocks that are stored and retrieved separately but may be combined to provide a realistic representation of the physical world. Blocks may be managed in memory to limit computational resources and in some embodiments may enable sharing of blocks across AR / MR / VR systems operating in the same physical space such that each AR / MR / VR system does little processing to construct the world reconstruction. Alternatively or in addition, blocks may be persisted across sessions of a portable user system within an AR / MR / VR system. Upon initiation of a new session for a portable user system, blocks persisted from a previous session may be selectively used to reduce the time for the user system to have an adequate representation of the physical world while still enabling the system to render the virtual content to the user in a realistic manner that reflects the interaction of the virtual content with the physical world.

[0096] In some embodiments, the use of computational resources may be reduced by selecting among different representations of the physical world when accessing information about the physical world. The world reconstruction may include information about the physical world, for example, captured from different sensors and / or stored in different formats. The simplest data to consume or provide may be provided to a component that uses the world reconstruction to render virtual objects. If the simpler data is not available, data obtained with different sensors that may generate a higher computational load may be accessed. As an example, the world reconstruction may include a depth map collected with a depth sensor and a more comprehensive representation of the 3D world, such as may be stored as a mesh calculated from image information. The information about the physical world, when available, may be provided to a component that performs occlusion processing based on the depth map. If holes exist in the depth map, information to fill those holes may be extracted from the mesh. In some embodiments, the depth map may be "live," representing the physical world as it is captured by the depth sensor at the time the data is accessed.

[0097] The techniques as described herein may be used with or separately from many types of devices, including wearable or portable devices with limited computing resources, that provide augmented reality scenes, and for many types of scenes.

[0098] AR System Overview

[0099] Figures 1-2 illustrate such a scene. For illustrative purposes, an AR system is used as an example of an XR system. Figures 3-8 illustrate an example AR system including one or more processors, memory, sensors, and a user interface that may operate in accordance with the techniques described herein.

[0100] With reference to FIG. 1, an AR scene 4 is depicted in which a user of the AR technology sees a physical world park-like setting 6 featuring people, trees, buildings in the background, and a concrete platform 8. In addition to these items, the user of the AR technology also perceives that they "see" a robot figure 10 standing on the physical world concrete platform 8, and a flying cartoon-like avatar character 2 that appears to be an anthropomorphic bumblebee, although these elements (e.g., the avatar character 2 and the robot figure 10) do not exist within the physical world. Due to the extreme complexity of the human visual perception and nervous system, it is difficult to produce AR technology that facilitates a comfortable, natural-feeling, and rich presentation of virtual image elements among other virtual or physical world image elements.

[0101] Such AR scenes can be achieved using a system that includes a world reconstruction component that can build and update a representation of the physical world surfaces around the user. This representation can be used for occluding rendering, placing virtual objects into physics-based interaction states, and for virtual character path planning and navigation, or other operations where information about the physical world is used. Figure 2 depicts another example of an AR scene 200 showing an example world reconstruction use case including visual occlusion 202, physics-based interaction 204, and environmental inference 206, according to some embodiments.

[0102] The example scene 200 is a living room with a wall, a bookcase on one side of the wall, a floor lamp in the corner of the room, a floor, a sofa, and a coffee table on the floor. In addition to these physical items, the user of the AR technology also perceives virtual objects such as an image on the wall behind the sofa, a bird that flew in through the door, a deer peeking out of the bookcase, and an ornament in the form of a windmill placed on the coffee table. For the image on the wall, the AR technology requires information not only about the surface of the wall, but also about the objects and surfaces in the room, such as the lamp shape, which occludes the image and renders the virtual object correctly. For the flying bird, the AR technology requires information about all objects and surfaces around the room to render the bird with realistic physics and avoid bouncing off objects and surfaces or the bird in case of collision. For the deer, the AR technology requires information about surfaces such as the floor or the coffee table and calculates where the deer should be placed. For the windmill, the system may identify it as a separate object from the table and may infer that it is movable, while the corner of the shelf or the corner of the wall may be inferred to be stationary. Such specificity can be used in inferences regarding the parts of the scene that are used or updated in each of the various operations.

[0103] The scene may be presented to the user via a system that includes multiple components, including a user interface that may stimulate one or more user senses, including sight, sound, and / or touch. In addition, the system may include one or more sensors that may measure parameters of the physical portion of the scene, including the user's position and / or movement within the physical portion of the scene. Furthermore, the system may include one or more computing devices with associated computer hardware, such as memory. These components may be integrated into a single device or distributed across multiple interconnected devices. In some embodiments, some or all of these components may be integrated into a wearable device.

[0104] 3 depicts an AR system 302 configured to provide an experience of AR content that interacts with a physical world 306, according to some embodiments. The AR system 302 may include a display 308. In the illustrated embodiment, the display 308 may be worn by a user as part of a headset such that the user may wear the display over their eyes like a pair of goggles or glasses. At least a portion of the display may be transparent such that the user may observe a see-through reality 310. The see-through reality 310 may correspond to a portion of the physical world 306 within a current perspective of the AR system 302, which may correspond to the user's perspective when the user is wearing a headset that incorporates both the display and sensors of the AR system and obtains information about the physical world.

[0105] The AR content may also be presented on the display 308, overlaid on the see-through reality 310. To provide accurate interaction between the AR content and the see-through reality 310 on the display 308, the AR system 302 may include a sensor 322 configured to capture information about the physical world 306.

[0106] The sensors 322 may include one or more depth sensors that output a depth map 312. Each depth map 312 may have a number of pixels, each of which may represent a distance to a surface in the physical world 306 in a particular direction relative to the depth sensor. Raw depth data may originate from the depth sensors to create a depth map. Such a depth map may be updated as fast as the depth sensors can form new images, which may be hundreds or thousands of times per second. However, the data may be noisy, incomplete, and have holes, which are shown on the depth map illustrated as black pixels.

[0107] The system may include other sensors, such as image sensors. The image sensors may obtain information that may be processed to represent the physical world in other ways. For example, images may be processed in the world reconstruction component 316 to create a mesh that represents connected portions of objects in the physical world. Metadata about such objects, including, for example, color and surface texture, may also be obtained using the sensors and stored as part of the world reconstruction.

[0108] The system may also obtain information about the user's head pose relative to the physical world. In some embodiments, the sensor 310 may include an inertial measurement unit, which may be used to calculate and / or determine the head pose 314. The head pose 314 for a depth map may indicate the current viewpoint of the sensor capturing the depth map, for example with six degrees of freedom (6DoF), but the head pose 314 may be used for other purposes, such as to relate image information to a particular part of the physical world or to relate the position of a display worn on the user's head to the physical world. In some embodiments, the head pose information may be derived in a manner other than from an IMU, such as from analyzing objects in an image.

[0109] The world reconstruction component 316 may receive the depth map 312 and head pose 314 and any other data from the sensors and integrate the data into a reconstruction 318, which may at least appear to be a single combined reconstruction. The reconstruction 318 may be more complete and less noisy than the sensor data. The world reconstruction component 316 may update the reconstruction 318 using spatial and temporal averages over time of the sensor data from multiple viewpoints.

[0110] Reconstruction 318 may include a representation of the physical world in one or more data formats, including, for example, voxels, meshes, planes, etc. Different formats may represent alternative representations of the same portion of the physical world or may represent different portions of the physical world. In the illustrated example, on the left side of reconstruction 318, a portion of the physical world is presented as a global surface, and on the right side of reconstruction 318, a portion of the physical world is presented as a mesh.

[0111] Reconstruction 318 may be used for AR functions, such as producing a surface representation of the physical world for occlusion handling or physics-based processing. This surface representation may change as the user moves or objects in the physical world change. Aspects of reconstruction 318 may be used by component 320, for example, to produce a changing global surface representation in world coordinates, which may be used by other components.

[0112] AR content may be generated, such as by an AR application 304, based on this information. The AR application 304 may be, for example, a game program that performs one or more functions based on information about the physical world, such as visual occlusion, physics-based interactions, and environmental inference. These functions may be performed by querying data in different formats from the reconstruction 318 produced by the world reconstruction component 316. In some embodiments, the component 320 may be configured to output updates as the representation in the region of interest of the physical world changes. The region of interest may be set to approximate a portion of the physical world in the vicinity of the user of the system, such as a portion in the user's field of view, or projected (predicted / determined) to occur in the user's field of view.

[0113] The AR application 304 uses this information to generate and update AR content, the virtual portions of which may be combined with a see-through reality 310 and presented on the display 308 to create a realistic user experience.

[0114] In some embodiments, the AR experience may be provided to the user through a wearable display system. FIG. 4 illustrates an example of a wearable display system 80 (hereinafter referred to as "system 80"). System 80 includes a head-mounted display device 62 (hereinafter referred to as "display device 62") and various mechanical and electronic modules and systems to support the functionality of display device 62. Display device 62 may be coupled to a frame 64, which is wearable by a display system user or viewer 60 (hereinafter referred to as "user 60") and configured to position display device 62 in front of the eyes of user 60. According to various embodiments, display device 62 may be a sequential display. Display device 62 may be monocular or binocular. In some embodiments, display device 62 may be an example of display 308 in FIG. 3.

[0115] In some embodiments, a speaker 66 is coupled to the frame 64 and positioned proximate the ear canal of the user 60. In some embodiments, another speaker, not shown, is positioned adjacent another ear canal of the user 60 to provide stereo / adjustable sound control. The display device 62 is operatively coupled, such as by wired leads or wireless connectivity 68, to a local data processing module 70, which may be mounted in a variety of configurations, such as fixedly attached to the frame 64, fixedly attached to a helmet or hat worn by the user 60, built into headphones, or otherwise removably attached to the user 60 (e.g., in a backpack configuration, in a belt-coupled configuration).

[0116] The local data processing module 70 may include a processor and digital memory, such as non-volatile memory (e.g., flash memory), both of which may be utilized to aid in processing, caching, and storing data. The data includes a) data captured from sensors (e.g., which may be operatively coupled to the frame 64 or otherwise attached to the user 60), such as image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, and / or gyroscopes, and / or b) data obtained and / or processed using the remote processing module 72 and / or remote data repository 74, possibly for passing to the display device 62 after processing or reading. The local data processing module 70 may be operatively coupled to the remote processing module 72 and the remote data repository 74 by communication links 76, 78, via wired or wireless communication links, etc., respectively, such that these remote modules 72, 74 are operatively coupled to each other and available as resources to the local processing and data module 70. 3 may be implemented, at least in part, within the local data processing module 70. For example, the local data processing module 70 may be configured to execute computer-executable instructions to generate a physical world representation based, at least in part, on at least a portion of the data.

[0117] In some embodiments, the local data processing module 70 may include one or more processors (e.g., a graphics processing unit (GPU)) configured to analyze and process the data and / or image information. In some embodiments, the local data processing module 70 may include a single processor (e.g., a single-core or multi-core ARM processor), which would limit the computational budget of the module 70 but allow for smaller devices. In some embodiments, the world reconstruction component 316 may generate the physical world representation in real time over a non-predetermined space using a computational budget of less than a single ARM core, such that the remaining computational budget of the single ARM core may be accessed for other uses, such as, for example, extracting meshes.

[0118] In some embodiments, the remote data repository 74 may include a digital data storage facility, which may be available through the Internet or other networking configurations in a "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed within the local data processing module 70, allowing for fully autonomous use from remote modules. World reconstructions may, for example, be stored in whole or in part within this repository 74.

[0119] In some embodiments, the local data processing module 70 is operably coupled to a battery 82. In some embodiments, the battery 82 is a removable power source, such as a commercially available battery. In other embodiments, the battery 82 is a lithium ion battery. In some embodiments, the battery 82 includes both an internal lithium ion battery that is rechargeable by the user 60 during periods of non-operation of the system 80, and a removable battery, so that the user 60 may operate the system 80 for longer periods of time without having to plug in a power source and charge the lithium ion battery, or without having to shut off the system 80 and replace the battery.

[0120] 5A illustrates a user 30 wearing an AR display system that renders AR content as the user 30 moves through a physical world environment 32 (hereafter referred to as "environment 32"). The user 30 positions the AR display system at a location 34, and the AR display system records ambient information of the passable world (e.g., digital representations of real objects in the physical world that may be stored and updated with changes to the real objects in the physical world) for the location 34, such as pose relationships to mapped features or directional audio input. The location 34 is aggregated into a data input 36 and processed by at least a passable world module 38, which may be implemented by, for example, processing on the remote processing module 72 of FIG. 3. In some embodiments, the passable world module 38 may include a world reconstruction component 316.

[0121] The passable world module 38 determines where and how the AR content 40 may be placed in the physical world as determined from the data input 36. The AR content is "placed" in the physical world by presenting, via the user interface, both a representation of the physical world and the AR content, the AR content is rendered as if it were interacting with objects in the physical world, and the objects in the physical world are presented as if the AR content were obscuring the user's view of those objects, when appropriate. In some embodiments, the AR content may be placed by appropriately selecting a portion of a fixed element 42 (e.g., a table) from the reconstruction (e.g., reconstruction 318) and determining the shape and position of the AR content 40. As an example, the fixed element may be a table, and the virtual content may be positioned to appear as if it were on the table. In some embodiments, the AR content may be placed within a structure in the field of view 44, which may be the current field of view or an estimated future field of view. In some embodiments, the AR content may be placed relative to a mapped mesh model 46 of the physical world.

[0122] As depicted, the fixed elements 42 serve as a substitute for any fixed elements in the physical world, which may be stored in the passable world module 38 such that the user 30 may perceive content on the fixed elements 42 without the system having to map to the fixed elements 42 each time it is visible to the user 30. The fixed elements 42 may thus be mapped mesh models from a previous modeling session or determined from a separate user, but stored on the passable world module 38 for future reference by multiple users. Thus, the passable world module 38 may allow the device of the user 30 to recognize the environment 32 from the previously mapped environment and display AR content without first mapping the environment 32, saving computational processes and cycles and avoiding latency of any rendered AR content.

[0123] A mapped mesh model 46 of the physical world may be created by the AR display system, and appropriate surfaces and metrics for interacting with and displaying the AR content 40 can be mapped and stored in the passable world module 38 for future retrieval by the user 30 or other users without the need for remapping or modeling. In some embodiments, the data inputs 36 are inputs such as geographic location, user identification, and current activity that indicate to the passable world module 38 the fixed elements 42 of one or more available fixed elements, the AR content 40 last placed on the fixed elements 42, and whether that same content should be displayed (such AR content is "persistent" content regardless of whether the user is viewing a particular passable world model).

[0124] FIG. 5B illustrates a schematic of the viewing optics assembly 48 and associated components. When directed at the user's eye 49, in some embodiments, two eye tracking cameras 50 detect metrics of the user's eye 49, such as eye shape, eyelid occlusion, pupil direction, and phosphenes on the user's eye 49. In some embodiments, a depth sensor 51, such as a time-of-flight sensor, emits relay signals into the world and determines the distance to a given object. In some embodiments, a world camera 52 records a view beyond the peripheral field of view, maps the environment 32, and detects inputs that may affect the AR content. Camera 53 may also capture specific timestamps of physical world images within the user's field of view. World camera 52, camera 53, and depth sensor 51 each have separate fields of view 54, 55, and 56, and collect and record data from a physical world scene, such as the physical world environment 32 depicted in FIG. 3A.

[0125] The inertial measurement unit 57 may determine the movement and orientation of the viewing optics assembly 48. In some embodiments, each component is operably coupled to at least one other component. For example, the depth sensor 51 is operably coupled to the eye tracking camera 50 as a check of accommodation measured against the actual distance the user's eyes 49 are looking at.

[0126] Information from these sensors in the viewing optics assembly 48 may be coupled to one or more of the processors in the system. The processor may generate data that may be rendered to cause the user to perceive virtual content interacting with objects in the physical world. The rendering may be implemented in any suitable manner, including generating image data that depicts both physical and virtual objects. In other embodiments, physical and virtual content may be depicted in a scene by modulating the opacity of a display device through which the user sees the physical world. The opacity may be controlled to create the appearance of virtual objects and also to block the user from seeing objects in the physical world that are occluded by the virtual objects. Regardless of how content is presented to the user, a model of the physical world is required so that properties of virtual objects that may be affected by physical objects, including the shape, position, motion, and visibility of the virtual objects, may be correctly calculated. In some embodiments, the model may include a reconstruction of the physical world, e.g., reconstruction 318.

[0127] The model may be created from data collected from sensors on the user's wearable device, although in some embodiments the model may be created from data collected by multiple users, which may be aggregated in a computing device remote from all users (and may be "in the cloud").

[0128] The model may be created, at least in part, by a world reconstruction system, such as the world reconstruction component 316 of FIG. 3, which is depicted in further detail in FIG. 6. The world reconstruction component 316 may include a perception module 160, which may generate, update, and store representations for portions of the physical world. In some embodiments, the perception module 160 may represent a portion of the physical world within the reconstruction range of a sensor as a plurality of voxels. Each voxel may correspond to a 3D cube of a given volume within the physical world and may include surface information indicating whether a surface exists within the volume represented by the voxel. A voxel may be assigned a value indicating whether its corresponding volume has been determined to include a surface of a physical object, has been determined to be empty, or has not yet been measured with a sensor and therefore its value is unknown. It should be understood that the value indicates that voxels determined to be empty or unknown need not be explicitly stored, and that the value of a voxel may be stored in computer memory in any suitable manner, including not storing information regarding voxels determined to be empty or unknown.

[0129] 7A depicts an example of a 3D space 100 that is discretized into voxels 102. In some embodiments, the perception module 160 may set the volume of voxels to determine objects of interest, capture features of the objects of interest, and avoid redundant information. For example, the perception module 160 may be configured to identify larger objects and surfaces, such as walls, ceilings, floors, and large furniture. Thus, the volume of voxels may be relatively large in size, e.g., 4 cm 3 may be set to a cube.

[0130] The reconstruction of the physical world, including the voxels, may be referred to as a solid model. Information for creating the solid model may be created over time as the sensor moves about the physical world. Such motion may occur as a user of a wearable device, including the sensor, moves around. Figures 8A-F depict an example of reconstructing the physical world into a solid model. In the illustrated example, the physical world includes a portion of a surface 180, which is shown in Figure 8A. In Figure 8A, a sensor 182 at a first location may have a field of view 184 in which the portion of the surface 180 is visible.

[0131] The sensor 182 may be of any suitable type, such as a depth sensor. However, the depth data may be derived from an image sensor or in other ways. The perception module 160 may receive the data from the sensor 182 and then set values ​​of a number of voxels 186, as shown in FIG. 8B, to represent a portion 180 of a surface visible by the sensor 182 within a field of view 184.

[0132] In Figure 8C, the sensor 182 may move to a second location and have a field of view 188. As shown in Figure 8D, an additional group of voxels becomes visible, and the values ​​of these voxels may be set to indicate the location of the portion of the surface that has entered the field of view 188 of the sensor 182. The values ​​of these voxels may be added to a solid model for the surface.

[0133] In FIG. 8E, the sensor 182 may further move to a third location and have a field of view 190. In the illustrated embodiment, an additional portion of the surface becomes visible within the field of view 190. As shown in FIG. 8F, an additional group of voxels may become visible and the values ​​of these voxels may be set to indicate the location of the portion of the surface that entered the field of view 190 of the sensor 182. The values ​​of these voxels may be added to a solid model for the surface. As shown in FIG. 6, this information may be stored as solid information 162a as part of the persisted world. Information about the surface, such as color or texture, may also be stored. Such information may be stored, for example, as solid metadata 162b.

[0134] In addition to generating information for the persisted world representation, perception module 160 may identify and output indications of changes in the area surrounding a user of the AR system. Such indications of changes may trigger updates to volumetric data stored as part of the persisted world, or trigger other functionality, such as triggering generate AR content, update AR content, component 304.

[0135] In some embodiments, the perception module 160 may identify changes based on a signed distance function (SDF) model. The perception module 160 may be configured to receive sensor data, such as, for example, a depth map 160a and a head pose 160b, and then fuse the sensor data into an SDF model 160c. The depth map 160a may provide SDF information directly, or images may be processed to become SDF information. The SDF information represents distances from sensors used to capture that information. Because those sensors may be part of the wearable unit, the SDF information may represent the physical world from the line of sight of the wearable unit, and therefore the user. The head pose 160b may enable the SDF information to be related to voxels in the physical world.

[0136] Referring back to FIG. 6, in some embodiments, perception module 160 may generate, update, and store representations for portions of the physical world within a perception range. The perception range may be determined, at least in part, based on the reconstruction range of the sensor, which may be determined, at least in part, based on the limits of the observation range of the sensor. As a specific example, an active depth sensor operating using active IR pulses may operate reliably over a range of distances, creating an observation range for the sensor, which may be a few centimeters or tens of centimeters to several meters.

[0137] FIG. 7B depicts a reconstruction range for a sensor 104 having a viewpoint 106. A reconstruction of a 3D space within the viewpoint 106 may be constructed based on data captured by the sensor 104. In the illustrated example, the sensor 104 has an observation range of 40 cm to 5 m. In some embodiments, the reconstruction range of the sensor may be determined to be smaller than the observation range of the sensor, since the sensor output closer to its observation limit may be noisier, incomplete, and inaccurate. For example, in the illustrated example of 40 cm to 5 m, the corresponding reconstruction range may be set to 1 to 3 m, and data collected with the sensor showing surfaces outside this range may not be used.

[0138] In some embodiments, the perception range may be larger than the reconstruction range of the sensor. If the component 164 that uses data about the physical world requests data about an area in the perception range outside the portion of the physical world in the current reconstruction range, that information may be provided from the persisted world 162. Thus, information about the physical world may be easily accessible by query. In some embodiments, an API may be provided to respond to such queries and provide information about the user's current perception range. Such techniques may reduce the time required to access an existing reconstruction and provide an improved user experience.

[0139] In some embodiments, the perception range may be a 3D space corresponding to a bounding box centered around the user location. As the user moves, the portion of the physical world within the perception range that may be queriable by component 164 may also move with the user. FIG. 7C depicts a bounding box 110 centered around location 112. It should be understood that the size of the bounding box 110 may be set to enclose the observation range of the sensor with a reasonable extension since a user may not move at an unreasonable speed. In the illustrated example, the sensor worn by the user has an observation limit of 5 m. The bounding box 110 is sized to 20 m. 3 It is set as a cube of .

[0140] 6, the world reconstruction component 316 may include additional modules that may interact with the perception module 160. In some embodiments, the persisted world module 162 may receive a representation for the physical world based on data obtained by the perception module 160. The persisted world module 162 may also include various formats of the representation of the physical world. For example, volumetric metadata 162b, such as voxels, may be stored along with the meshes 162c and planes 162d. In some embodiments, other information, such as depth maps, may also be saved.

[0141] In some embodiments, the perception module 160 may include modules that generate representations for the physical world in various formats, including, for example, meshes 160d, planes, and semantics 160e. These modules may generate the representations based on data within the perception range of one or more sensors at the time the representations are generated and data captured at previous times and information within the persisted world 162. In some embodiments, these components may operate on depth information captured using a depth sensor. However, the AR system may also include a vision sensor and generate such representations by analyzing monocular or binocular visual information.

[0142] In some embodiments, these modules may operate on a region of the physical world, such as a region represented by a block or tile, as described below. They may be triggered to update the block or tile or other sub-region of the physical world when perception module 160 detects a change in the physical world in that sub-region. Such a change may be detected by other criteria, such as, for example, detecting a new surface in SDF model 160c, or by a change in value of a sufficient number of voxels representing the sub-region.

[0143] The world reconstruction components 316 may include components 164 that may receive a representation of the physical world from the perception module 160. Information about the physical world may be pulled by these components, for example, according to usage requests from applications. In some embodiments, information may be pushed to the usage components, such as via indications of changes in pre-identified areas or changes in the physical world representation within a perception range. The components 164 may include, for example, game programs and other components that implement processing for visual occlusion, physics-based interactions, and environmental inference.

[0144] In response to a query from component 164, perception module 160 may transmit a representation for the physical world in one or more formats. For example, when component 164 indicates that the use is for visual occlusion or physics-based interaction, perception module 160 may transmit a representation of a surface. When component 164 indicates that the use is for environmental inference, perception module 160 may transmit meshes, planes, and semantics of the physical world.

[0145] In some embodiments, perception module 160 may include a component that formats information and provides it to component 164. An example of such a component may be ray casting component 160f. A usage component (e.g., component 164) may, for example, query information about the physical world from a particular viewpoint. Ray casting component 160f may select from one or more representations of the physical world data within a field of view from that viewpoint.

[0146] Viewpoint-Dependent Brick Selection for Fast Volume Reconstruction.

[0147] As should be understood from the foregoing description, the perception module 160 or another component of the AR system may process data and create a 3D representation of a portion of the physical world. The data to be processed may be reduced, at least in part, by thinning portions of the 3D reconstruction volume based on the camera frustum and / or depth image, extracting and persisting planar data, capturing, persisting, and updating the 3D reconstruction data on a block-by-block basis to enable local updates while maintaining neighborhood consistency, providing occlusion data to an application, generating such scenes in which the occlusion data is derived from a combination of one or more depth data sources, and / or performing multi-stage mesh simplification.

[0148] The world reconstruction system may integrate sensor data from multiple viewpoints of the physical world over time. The pose (e.g., position and orientation) of the sensor may be tracked as the device containing the sensor is moved. As the frame pose of the sensor and how it relates to other poses are understood, each of these multiple viewpoints of the physical world may be fused together into a single combined reconstruction. The reconstruction may be more complete and less noisy than the original sensor data by using spatial and temporal averaging (i.e., averaging data from multiple viewpoints over time).

[0149] The reconstruction may contain data of different levels of sophistication, including, for example, raw data such as live depth data, fused volumetric data such as voxels, and calculated data such as meshes.

[0150] In some embodiments, the AR and MR systems represent 3D scenes with a regular voxel grid, where each voxel may contain a signed distance field (SDF) value. The SDF value describes whether the voxel is inside or outside a surface in the scene to be reconstructed, and the distance from the voxel to the surface. Computing 3D reconstruction data representing the desired volume of the scene requires large amounts of memory and processing power. These requirements increase for scenes representing larger spaces, since the number of variables required for 3D reconstruction grows cube-wise with the number of depth images to be processed.

[0151] Described herein are efficient methods of reducing processing. According to some embodiments, a scene may be represented by one or more bricks. Each brick may include multiple voxels. The bricks processed to generate a 3D reconstruction of the scene may be selected by thinning out the set of bricks representing the scene based on a frustum derived from the field of view (FOV) of an image sensor and / or a depth image (or "depth map") of the scene created using a depth sensor.

[0152] The depth image may have one or more pixels, each representing a distance to a surface in the scene. These distances may be related to a position relative to the image sensor so that data output from the image sensor may be selectively processed. Image data may be processed for those bricks that represent a portion of the 3D scene that contains a surface that would be visible from the viewpoint (or "viewpoint") of the image sensor. Processing of some or all of the remaining bricks may be omitted. Using such an approach, selected bricks may be those that are more likely to contain new information, which may be achieved by thinning out bricks around which the output of the image sensor is less likely to provide useful information. Data output from the image sensor is less likely to provide useful information for bricks that are either closer to or farther from the image sensor than the surface indicated by the depth map, because those bricks are either in void or behind a surface and therefore not depicted in the image from the image sensor.

[0153] In some embodiments, one or more criteria may be applied to efficiently select a set of bricks for processing. The initial set of bricks may be limited to those within the frustum of the image sensor. Many bricks outside the frustum may then be thinned out. More computationally intensive processing to update the 3D reconstruction may then be performed on the subset of bricks accepted for processing following the thinning. Thus, the 3D representation of the scene to be updated is more efficiently calculated using processing on a reduced number of voxels.

[0154] Even more reduction in processing may be achieved by thinning out bricks based on the depth image. According to some embodiments, thinning and / or acceptance of bricks may be performed by projecting a silhouette of each brick in the initial set into the depth image. Such thinning may be based on whether the brick corresponds to a portion of the scene that the depth image shows in the vicinity of the surface. Bricks that may be identified as simply being entirely in front of the surface or entirely behind it may be thinned out. In some embodiments, such a determination may be made efficiently. For example, a bounding box around the projection of the brick into the depth map may be used to determine maximum and minimum brick values ​​along a z-coordinate direction, which may be approximately perpendicular to the 2D plane of the depth image. By comparing these maximum and minimum brick values ​​against the distance represented by a pixel in the depth map, the brick may be thinned out and / or accepted for further processing. Such processing may result in the selection of bricks for initial processing that intersect the surface as reflected in the depth image and / or are in front of the surface as reflected in the depth image. In some embodiments, such processing may distinguish between bricks in front of a solid surface and bricks in front of a perforated surface (i.e., bricks representing areas for which the depth sensor was unable to reliably measure the distance to the surface).

[0155] In some embodiments, the thinning / acceptance criteria may result in classification of some or all of the bricks that are accepted for further processing, such that processing algorithms for computational solid reconstruction may be tuned for the characteristics of the bricks. In some embodiments, different treatments may be selected based on whether the brick is classified as intersecting a surface, in front of a solid surface, or in front of a surface with a hole.

[0156] 9 shows a cross-sectional view of a scene 400 along a plane parallel to the y- and z-coordinates. The XR system may represent the scene 400 by a grid of voxels 504. A conventional XR system may update each voxel of the grid of voxels based on every new depth image captured by a sensor 406, which may be an image sensor or a depth sensor, so that a 3D reconstruction generated from the grid of voxels may reflect changes in the scene. Updating in this manner may consume significant computational resources and may also introduce artifacts into the output of the XR system, for example due to latency caused by heavy computation.

[0157] Described herein are techniques for providing accurate 3D reconstruction data using low usage computational resources, for example, by thinning out a portion of a grid of voxels 504 based, at least in part, on the camera frustum 404 of an image sensor 406 and / or a depth image captured by the image sensor.

[0158] In the illustrated example, the image sensor 406 captures a depth image (not shown) that includes the surface 402 of the scene 400. The depth image may be stored in computer memory in any convenient manner that captures the distance between a certain reference point in the scene 400 and the surface. In some embodiments, the depth image may be represented as values ​​in a plane parallel to the x- and y-axes, as illustrated in FIG. 9, where the reference point is the origin of the coordinate system. Locations in the XY plane may correspond to directions relative to the reference point, and values ​​at those pixel locations may indicate the distance from the reference point to the nearest neighbor of the surface in the direction indicated by the coordinates in the plane. Such a depth image may include a grid of pixels (not shown) in a plane parallel to the x- and y-axes. Each pixel may indicate the distance in a particular direction from the image sensor 406 to the surface 402. In some embodiments, the depth sensor may be unable to measure the distance to the surface in a particular direction. Such a condition may occur, for example, when the surface is out of range of the image sensor 406. In some embodiments, the depth sensor may be an active depth sensor that measures distance based on reflected energy, but a surface may not reflect enough energy for an accurate measurement, and thus in some embodiments the depth image may have "holes" - there are pixels for which no value is assigned.

[0159] In some embodiments, the reference point of the depth image may vary. Such a configuration allows the depth image to represent surfaces throughout the entire 3D scene and may not be limited to portions having a predetermined and limited range of angles relative to a particular reference point. In such embodiments, the depth image may indicate the distance to the surface as the image sensor 406 moves through six degrees of freedom (6DOF). In these embodiments, the depth image may include a set of pixels for each of a number of reference points. In these embodiments, the portion of the depth image may be selected based on a "camera pose," which represents the direction and / or orientation in which the image sensor 406 is facing at the time the image data is captured.

[0160] The image sensor 406 may have a field of view (FOV), which may be represented by a camera frustum 404. In some embodiments, the depicted infinite camera frustum may be reduced to a finite 3D trapezoidal prism 408 by assuming a maximum depth 410 that the image sensor 406 can provide and / or a minimum depth 412 that the image sensor 406 can provide. The 3D trapezoidal prism 408 may be a convex polyhedron bounded by six planes.

[0161] In some embodiments, one or more voxels 504 may be grouped into bricks 502. Figure 10A illustrates a portion 500 of a scene 400 that includes eight bricks 502. Figure 10B illustrates a portion 500 of a scene 400 that includes eight bricks 502. 3 Illustrated is an example brick 502 that includes voxels 504. Referring back to Figure 9, the scene 400 may include one or more bricks, 16 of which are illustrated in the diagram depicted in Figure 4. Each brick may be identifiable by a brick identification, for example,

[0000] -

[0015] .

[0162] FIG. 11 depicts a volumetric representation hierarchy implemented in some embodiments. In some embodiments, such a volumetric representation hierarchy may reduce latency for data transfer. In some embodiments, a voxel grid of the physical world may be mapped to conform to a structure of a memory architecture for a processor that computes the AR content, such as the processor on which component 304 executes. One or more voxels may be grouped into a "brick." One or more bricks may be grouped into a "tile." The size of a tile may correspond to a memory page of a storage medium local to the processor. Tiles may be moved between local and remote memory, such as via a wireless connection, based on usage or expected usage according to a memory management algorithm.

[0163] In some embodiments, uploads and / or downloads between perception module 160 and persisted world module 162 may be performed on multiple tiles in one operation. One or more tiles may be grouped into a "RAM tileset." The size of the RAM tileset may correspond to an area within the reconstruction range of a sensor worn by a user. One or more RAM tileset may be grouped into a "global tileset." The size of the global tileset may correspond to the perception range of the world reconstruction system (e.g., the perception range for perception module 160).

[0164] 12 is a flow chart illustrating a method 600 of operating a computing system to generate a 3D reconstruction of a scene, according to some embodiments. The method 600 may begin with representing a scene (e.g., scene 400) with one or more bricks (e.g., brick 502), each of which includes one or more voxels (e.g., voxel 504). Each brick may represent a portion of the scene. The bricks may be identifiable with respect to a persistent coordinate system such that the same brick represents the same volume in the scene even if the pose of the image sensor (e.g., image sensor 406) changes.

[0165] At act 604, the method 600 may capture a depth image (e.g., a depth image including the surface 402) from a depth sensor (e.g., depth sensor 406). The depth sensor may be, for example, an active depth sensor that transmits IR radiation and measures the time of flight for reflection. Each such measurement represents the distance from the depth sensor to the surface in a particular direction. That depth information may represent the same volume as represented by the brick.

[0166] At act 606, the method 600 may decimate a portion of one or more bricks relative to the camera frustum (e.g., the finite 3D trapezoidal prism 408 derived from the camera frustum 404) to produce a first one or more bricks that are a reduced set of bricks from the one or more bricks. Such decimation may eliminate bricks that represent portions of the scene outside the field of view of the image sensor at the time the image data being processed was acquired. That image data is therefore unlikely to contain information useful in creating or updating the bricks.

[0167] At act 608, method 600 may thin out a portion of the first one or more bricks from the depth image to produce a second one or more bricks that are a reduced set of bricks from the first one or more bricks. At act 610, method 600 may generate a 3D reconstruction of the scene based on the second one or more bricks.

[0168] 9, given the surface 402 and corresponding camera pose captured by the depth image, voxels between the image sensor 406 and the surface 402 may be empty. The further a voxel is behind the surface 402 from the image sensor 406, the less certain it may be that it represents the inside of an object or void. The degree of certainty may be represented by a weighting function, which weights the voxel update based on the distance to the surface 402. When a voxel behind the surface 402 (farther away from the image sensor 402) has a weighting function above a threshold, the voxel may receive no update or zero update (e.g., an update with zero change). Also, all voxels not within the camera frustum 404 may not be updated or examined for this depth image.

[0169] Method 600 not only improves the processing speed of stereoscopic depth image fusion, but also consumes little memory storage, which allows method 600 to run on wearable hardware. 3A small reconstruction volume of 5m x 5m x 3m with voxel size and 8 bytes per voxel (4 bytes for distance value and 4 bytes for weight value) would already require about 600MB. Method 600 can categorize bricks by their distance to the surface relative to a truncation threshold. For example, method 600 can identify empty bricks (e.g., bricks that are thinned out or bricks that are far from the surface above the truncation threshold) so as not to allocate memory space for empty bricks. Method 600 can also identify bricks that are far from the surface by the truncation threshold, so as to store these bricks with a constant distance value of a negative truncation threshold and a weight of 1. Method 600 can also identify bricks with a distance to the surface between zero and the truncation threshold, so as to store these bricks with a constant SDF value of a positive truncation threshold but with a variable weight. Storing a distance or weight value that is constant for bricks with a single value can be an entropy-based compression for a zero entropy field.

[0170] Method 600 may allow for flagging a brick as "not containing any part of a surface" during voxel updates, which may significantly speed up processing of the brick, which may include, for example, converting an image of the portion of the scene represented by the brick into a mesh.

[0171] FIG. 13 illustrates an example method 606 of thinning a portion of one or more bricks for the camera frustum 404 of the image sensor 406, according to some embodiments. The method 606 may begin with finding a cubic axis aligned bounding box (AABB) containing the camera frustum 404. The AABB may contain one or more bricks within the scene. The method 606 may include splitting the AABB into one or more sub-AABBs (act 704) and performing a camera frustum acceptance test (act 706). If the method 606 determines in act 708 that the sub-AABB reaches the size of a brick, the method 606 may produce a first one or more bricks (act 710). If the method 606 determines in act 708 that the sub-AABB is larger than the size of a brick, the method 606 may repeat acts 704-708 until the sub-AABB reaches the size of a brick.

[0172] For example, given a 3D trapezoidal prism 408 corresponding to the camera frustum 404, an AABB may be found in constant time using the square of the side length containing the 3D trapezoidal prism 408. The AABB may be divided into eight sub-AABBs. Each of the eight sub-AABBs may be tested for intersection with the camera frustum 404. If it is determined that a sub-AABB does not intersect with the camera frustum 404, the brick corresponding to this sub-AABB may be thinned out. The thinned brick may be rejected for further processing. If it is determined that a sub-AABB does intersect with the camera frustum 404, the sub-AABB may be further divided into eight sub-AABBs of the sub-AABB. Each of the eight sub-AABBs of the sub-AABB may then be tested for intersection with the camera frustum 404. The iteration of dividing and testing continues until the sub-AABB corresponds to a single brick. To determine whether the camera frustum 404 intersects with the AABB, a two-step test may be performed. First, it may be tested whether at least one corner point of the AABB is inside each of the planes that bound the camera frustum 404. Second, it may be tested whether all corner points of the camera frustum 404 are inside the AABB, so that some cases of AABBs that do not intersect the camera frustum 404 but are incorrectly classified as partially inside (e.g., having only one corner point on the edge of the frustum) may be captured.

[0173] A desirable by-product of this two-step test is that for each brick that intersects with the camera frustum 404, it can be known whether it is fully or only partially inside the camera frustum 404. For those bricks that are fully inside the camera frustum 404, in subsequent individual voxel updates, the per-voxel test for inside the camera frustum 404 can be skipped.

[0174] 14 illustrates an example method 706 for performing a camera frustum acceptance test, according to some embodiments. The method 706 may begin by testing each of one or more sub-AABBs against each of the planes that bound the camera frustum 404 (act 802). In act 804, the method 706 may determine whether the sub-AABB being tested is entirely outside the camera frustum 404. In act 806, if it is determined that the sub-AABB being tested is entirely outside the camera frustum 404, the method 706 may cull all bricks contained by the sub-AABB being tested. In act 808, if it is determined that the sub-AABB being tested is not entirely outside the camera frustum 404, the method 706 may determine whether the sub-AABB being tested is entirely inside the camera frustum 404.

[0175] In act 810, if it is determined that the sub-AABB being tested is entirely inside the camera frustum 404, the method 706 may add all bricks contained by the sub-AABB being tested to the first one or more bricks. In act 708, if it is determined that the sub-AABB being tested is not entirely inside the camera frustum 404 (which may indicate that the sub-AABB being tested intersects with the camera frustum 404), the method 706 may determine whether the sub-AABB being tested reaches the size of a brick.

[0176] In act 814, if it is determined that the sub-AABB being tested is equal to the size of the brick, the method 706 may further determine whether all corner points of the camera frustum 404 are inside the sub-AABB being tested. If it is determined that all corner points of the camera frustum are inside the sub-AABB being tested, the method 706 may thin out the bricks of the sub-AABB being tested (act 806). If it is determined that all corner points of the camera frustum are not inside the bricks of the sub-AABB being tested, the method 706 may add the bricks of the sub-AABB being tested to the first one or more bricks (act 810).

[0177] 15 illustrates an example method 608 for thinning a portion of a first one or more bricks for a depth image, according to some embodiments. The method 608 may begin by performing a first depth image acceptance test for each of the first one or more bricks (act 902). In act 904, the method 808 may determine whether the tested brick is accepted by the first depth image acceptance test. If the tested brick is determined to be accepted by the first depth image acceptance test (which may indicate that the tested brick intersects with a surface in the scene), the method 608 may apply a variable increment to the selected voxel (act 906) and add the tested brick to the second one or more bricks (act 914).

[0178] In act 908, if it is determined that the tested brick is not approved by the first depth image approval test, the method 608 may perform a second depth image approval test on the tested brick (act 908). In act 910, it is determined whether the tested brick is approved by the second depth image approval test. If it is determined that the tested brick is approved by the second depth image approval test (which may indicate that the tested brick is in front of a solid or perforated background in the scene), the method 608 may apply a constant increment to all or selected voxels (act 912) and add the tested brick to the second one or more bricks (act 914). If it is determined that the tested brick is also not approved by the second depth image approval test, the method 608 may thin out the tested brick (act 916).

[0179] FIG. 16 illustrates an example method 902 for performing a first depth image acceptance test, according to some embodiments. For each brick to be tested, the method 902 may begin by determining a minimum brick value (bmin) and a maximum brick value (bmax) along a direction parallel to the z-coordinate (act 1002). The bmin and bmax values ​​may be padded to account for an integral threshold, beyond which depth values ​​indicate a constant update to voxels within the brick. In act 1004, the method 902 may calculate 2D pixel locations of corners of the tested brick by projecting the corners of the brick into the depth image. In act 1006, the method 902 may calculate a rectangle by building a convex hull of the 2D pixel locations of the corners of the brick. In act 1008, the method 902 may test each pixel within the rectangle against the bmin and bmax values. In act 1010, the method 902 may determine whether all pixels in the rectangle have depth values ​​between the bmin and bmax values. If it is determined that all pixels in the rectangle have depth values ​​between the bmin and bmax values, the method 902 may approve the brick (act 1012). If it is determined that all pixels in the rectangle do not have depth values ​​between the bmin and bmax values, the method 902 may perform a second depth image approval test for the brick (act 908).

[0180] FIG. 17 illustrates an example method 908 for performing a second depth image acceptance test, according to some embodiments. The method 908 may begin in act 1102 by categorizing all pixels within a rectangle for bmin and bmax values. In act 1104, the method 908 may determine whether the tested brick is in front of a solid or perforated background, for example, by using a table shown in FIG. 18. If it is determined that the tested brick is in front of a solid or perforated background, the method 908 may accept the brick (act 1106). If it is determined that the tested brick is not in front of a solid or perforated background, the method 908 may thin out the brick in act 916.

[0181] 19A-F depict an example of thinning bricks representing a scene 190 relative to a camera frustum 192. In FIG. 19A, the scene 190 is represented by a single AABB 194a, which in the illustrated example includes a 16×16 brick. In FIG. 19B, the single AABB 194a is split into four sub-AABBs 194b, each including an 8×8 brick. After performing a camera frustum acceptance test (e.g., method 706), one of the four sub-AABBs 194b fails the camera frustum acceptance test, and thus the 8×8 bricks in the failing sub-AABB 194b are thinned out and illustrated as white bricks. In FIG. 19C, each of the three sub-AABBs 194 that pass the camera frustum acceptance test are further split into four sub-AABBs 194c, each including a 4×4 brick. After performing the camera frustum acceptance test (e.g., method 706), 8 of the 16 sub-AABBs 194c fail the camera frustum acceptance test, and therefore the bricks in the failing sub-AABBs 194c are thinned out and illustrated as white bricks. Similarly, in FIG. 19D, sub-AABB 194d includes a 2×2 brick. In FIG. 19E, sub-AABB 194e includes a single brick, and therefore sub-AABB 194e and the individual bricks that passed the camera frustum test are produced as a first plurality of bricks, and in FIG. 19F, illustrated as gray bricks 196f. In the illustrated embodiment, if thinning was not performed, the world reconstruction component would calculate all 256 bricks. With brick thinning for the camera frustum, the world reconstruction component would only need to calculate the first plurality of bricks, i.e., 34 bricks, and would therefore render the result much faster.

[0182] 20A illustrates an example of further thinning 34 bricks 196f for a depth image including surface 220, for example, by performing method 608. FIG. 20B illustrates the results of thinning for the depth image, showing that 12 bricks 222a of the 34 bricks 196f pass a first depth image acceptance test (e.g., method 904), 9 bricks 222b of the 34 bricks 196f pass a second depth image acceptance test (e.g., method 910), and finally, after thinning for the depth image with surface 220, 13 bricks 222c of the 34 bricks 196f are thinned. As a result, in the illustrated example, using brick thinning for the depth image further reduces the number of bricks calculated by the world reconstruction component to 21 bricks. It should also be appreciated that the computation speed of the world reconstruction component may also be accelerated not only by the reduced number of bricks, but also by classification of the bricks as a result of the first and second depth image acceptance tests. For example, as discussed with respect to FIG. 15, a fixed increment may be applied to the 9 bricks 222b that failed the first depth image acceptance test but passed the second depth image acceptance test. Applying a fixed increment collectively may also improve computation speed compared to applying a variable increment to each voxel. Geometry (e.g., planes) within a scene, such as walls for placing a virtual screen and / or floors for navigating a virtual robot, may be captured in an XR system to support applications. A common representation of the geometry of a scene is a mesh, which may comprise a group of connected triangles with vertices and edges. Conventionally, geometry in a scene is obtained by generating a mesh for the scene and searching for geometry in the mesh, which takes time to process, e.g., several seconds, and does not show the relationships between the geometries required by different queries. For example, a first query may be for a table plane. In response to the first query, the system may find the table plane and leave the watch on the table plane. Then, a second query may be for the watch.In response to the second query, the system may find all possible table planes and check whether a watch is present on each table plane until it finds the watch, since there is no indication from the response to the first query that the table plane is a table plane or not.

[0183] Described herein is a geometry extraction system. In some embodiments, the geometry extraction system may extract geometry while scanning a scene using a camera and / or sensor, which allows for fast efficient extraction and can adapt to dynamic environmental changes. In some embodiments, the geometry extraction system may persist the extracted geometry in local and / or remote memory. The persisted geometry may be accompanied by a unique identification, such that different queries, e.g., at different timestamps and / or from different applications, may share the persisted geometry. In some embodiments, the geometry extraction system may support different representations of the geometry according to individual queries. A plane is used as an exemplary geometry in the following description of Figures 21-29. It should be understood that instead of or in addition to a plane, the geometry extraction system may detect other geometries, including semantics such as, for example, cylinders, cubes, lines, corners, or glass surfaces or holes, and use them in subsequent processing. In some embodiments, the principles described herein with respect to geometry extraction may be applicable to object extraction and the like.

[0184] 21 illustrates a plane extraction system 1300 according to some embodiments. The plane extraction system 1300 may include a depth fusion 1304, which may receive multiple depth maps 1302. The multiple depth maps 1302 may be created by one or more users wearing depth sensors and / or downloaded from local / remote memory. The multiple depth maps 1302 may represent multiple viewpoints of the same surface. Differences may exist between the multiple depth maps, which may be resolved by the depth fusion 1304.

[0185] In some embodiments, depth fusion 1304 may generate SDF 1306 based at least in part on method 600. Mesh bricks 1308 may be extracted from SDF 1306 by, for example, applying a marching cubes algorithm over the corresponding bricks (e.g., bricks

[0000] -

[0015] in FIG. 23). Plane extraction 1310 may detect planar surfaces in mesh bricks 1308 and extract planes based at least in part on mesh bricks 1308. Plane extraction 1310 may also extract surfels for each brick based at least in part on the corresponding mesh bricks. Surfels may include vertices in a mesh but not edges connecting to neighboring vertices, such that storing surfels consumes less memory space than meshes. Plane data store 1312 may persist the extracted planes and surfels.

[0186] In some embodiments, an XR application may request and obtain a plane from the plane data store 1312 by a plane query 1314, which may be sent by an application programming interface (API). For example, an application may send information about its location to the plane extraction system 1300 and ask about all planes in its vicinity (e.g., within a 5 meter radius). The plane extraction system 1300 may then search its plane data store 1312 and send the selected plane to the application. The plane query 1314 may include information such as where the application needs the plane, the type of plane the application needs, and / or how the plane should look (e.g., whether horizontal or vertical or angled, which may be determined by checking the primitive normals of the plane in the plane data store).

[0187] 22 shows a portion 1400 of the plane extraction system 1300 illustrating details regarding plane extraction 1310, according to some embodiments. Plane extraction 1310 may include dividing each of the mesh bricks 1308 into sub-bricks 1402. Plane detection 1404 may be performed for each sub-brick 1402. For example, plane detection 1404 may compare primitive normals of each mesh triangle within a sub-brick, merge those mesh triangles with primitive normal difference less than a predetermined threshold into one mesh triangle, and identify mesh triangles with area greater than a predetermined area value as planes.

[0188] 23 is a schematic diagram illustrating a scene 1500, represented by bricks 0000-0015, including voxels, and example plane data, including a brick plane 1502, a global plane 1504, and surfels 1506 in the scene, according to some embodiments. FIG. 23 illustrates a brick 0011, which is divided into four sub-bricks 1508. It should be understood that a mesh brick may be divided into any suitable number of sub-bricks. The granularity of the planes detected by plane detection 1404 may be determined by the size of the sub-bricks, while the size of the bricks may be determined by the granularity of the local / remote memory that stores the volumetric 3D reconstruction data.

[0189] 22, plane detection 1404 may determine a brick plane (e.g., brick plane 1502) for each mesh brick based, at least in part, on the planes detected for each sub-brick within the mesh brick. Plane detection 1404 may also determine a global plane (e.g., global plane 1504) that extends across more than one brick.

[0190] In some embodiments, plane extraction 1310 may include plane updating 1406, which may update existing brick planes and / or global planes stored in plane data store 1312 based at least in part on planes detected by plane detection 1404. Plane updating 1406 may include adding additional brick planes, removing some of the existing brick planes, and / or replacing some of the existing brick planes with brick planes detected by plane detection 1404 and corresponding to the same brick, such that real-time changes in the scene are persisted in plane data store 1312. Plane updating 1406 may also include aggregating brick planes detected by plane detection 1404 into the existing global plane, for example, when a brick plane is detected adjacent to an existing global plane.

[0191] In some embodiments, plane extraction 1310 may further include plane merging and splitting 1408. Plane merging may merge multiple global planes into one larger global plane, for example, when a brick plane is added to connect two global planes. Plane splitting may split one global plane into multiple global planes, for example, when a brick plane in the middle of a global plane is removed.

[0192] FIG. 24 illustrates a data structure in the plane data store 1312, according to some embodiments. A global plane 1614, indexed by plane ID 1612, may be at the highest level of the data structure. While one brick plane may be persisted per brick, each global plane 1614 may contain multiple brick planes and surfels of bricks adjacent to the corresponding global plane, so that the global plane may be rendered correctly when an edge of the global plane does not qualify as a brick plane for the corresponding brick. In some embodiments, surfels of bricks adjacent to the global plane are persisted instead of surfels of all bricks in the scene, since these will be sufficient to render the global plane correctly. For example, as illustrated in FIG. 23, a global plane 1504 extends across bricks

[0008] -

[0010] and

[0006] . Brick

[0006] has a brick plane 1502 that is not part of the global plane 1504. Using the data structures in the plane data store 1312, when a plane query requests the global plane 1504, surfels for bricks

[0006] and

[0012] are checked to determine whether the global plane 1504 extends into bricks

[0006] and

[0012] . In the illustrated example, surfel 1506 indicates that the global plane 1504 extends into brick

[0006] .

[0193] 24 , the global plane 1614 may be bidirectionally associated with the corresponding brick plane 1610. The bricks may be identifiable by a brick ID 1602. The bricks may be divided into planar bricks 1604, which contain at least one plane, and non-planar bricks 1606, which do not contain a plane. Surfels for both planar and non-planar bricks may be persisted depending on whether the brick is adjacent to the global plane, not whether the brick contains a plane. It should be understood that the planes may be persisted continuously in the plane data store 1312 while the XR system observes the scene, regardless of whether a plane query 1314 is present.

[0194] 25 illustrates a plane geometry extraction 1702 that may extract planes for use by an application when the application sends a plane query 1314 to the plane data store 1312, according to some embodiments. The plane geometry extraction 1702 may be implemented as an API. The plane query 1314 may indicate a requested planar geometry representation, for example, an outer rectangular plane, an inner rectangular plane, or a polygonal plane. In accordance with the plane query 1314, the plane search 1704 may search and retrieve plane data in the plane data store 1312.

[0195] In some embodiments, rasterization from the planar cover points 1706 may generate planar cover points. An example is illustrated in Figure 26A. There are four bricks, 0000-0003, each with a brick plane 1802. The planar cover points 1806 (or "rasterized points") are generated by projecting the boundary points of the brick planes onto a global plane 1804.

[0196] 25, rasterization from the planar cover points 1706 may also generate a rasterized planar mask from the planar cover points. According to the planar geometry representation requested by the plane query 1314, an inner rectangle planar representation, an outer rectangle planar representation, and a polygon planar representation may be extracted by inner rectangle extraction 1708, outer rectangle extraction 1710, and polygon extraction 1712, respectively. In some embodiments, the application may receive the requested planar geometry representation within milliseconds of sending the plane query.

[0197] An exemplary rasterized planar mask 1814 is illustrated in FIG. 26B. From the rasterized planar mask, various planar geometric shape representations can be generated. In the illustrated example, a polygon 1812 is generated by connecting some of the planar coverage points of the rasterized planar mask such that none of the planar coverage points in the mask are outside the polygon. An outer rectangle 1808 is generated such that the outer rectangle 1808 is the smallest rectangle that encloses the rasterized planar mask 1814. The inner rectangle 1810 is generated by assigning a "1" to bricks with two planar coverage points and a "0" to bricks without two planar coverage points, forming a rasterization grid, determining groups of bricks that are marked as "1" and aligned within lines parallel to the edges of the bricks (e.g., bricks

[0001] ,

[0005] ,

[0009] , and

[0013] as a group, bricks

[0013] -

[0015] as a group), such that the inner rectangle is the smallest rectangle that encloses the respective group, and generating one inner rectangle for each determined group.

[0198] Figure 27 shows a mesh for a scene 1900, according to some embodiments. Figures 28A-C show the scene 1900, represented by an outer rectangular plane, an inner rectangular plane, and a polygonal plane, respectively, according to some embodiments.

[0199] FIG. 29 illustrates a less noisy 3D representation of a scene 1900, which is obtained by flattening the mesh shown in FIG. 27 based on extracted plane data, e.g., the planes illustrated in FIGS. 28A-C.

[0200] Multi-step block mesh simplification

[0201] In some embodiments, processing may be employed to reduce the complexity of the representation of the XR environment before the representation is stored or used in rendering functions such as handling occlusion or calculating the physics of interactions between objects in the XR environment. For example, meshing component 160d may simplify a mesh or part of a partial mesh before storing it as mesh 162c in persisted world 162.

[0202] Such processing may involve performing operations on the representation of the XR environment in stages. The stages may include simplification operations preceding and following region-based operations. The region-based operations, like the simplification operations, may reduce the complexity of the representation of the XR environment. By staging the operations in this manner, the total processing to produce the simplified representation of the XR environment may be reduced while maintaining the quality of the representation of the XR environment. As a result, the simplified high-quality representation may be updated frequently such that the XR environment may be updated frequently, such as by presenting a more realistic environment to the user, improving the performance of the XR system.

[0203] The XR environment may represent the physical world, and data representing the XR environment may be captured by one or more sensors. However, techniques as described herein may be applied to the XR environment regardless of the source of the data representing the environment. In some embodiments, the XR environment may be represented by a mesh, including one or more points and polygons, such as triangles, defined by a subset of points. A first simplification operation, preceding the region-based operation, may reduce the complexity of the representation of the environment. The mesh may be simplified, for example, by reducing the number of such polygons in the mesh. As a specific example, the first simplification operation may employ a triangle reduction algorithm, which may reduce the number of triangles used to represent the XR environment.

[0204] The region-based operation may be a shape detection operation, which may detect one or more shapes. A common shape detection operation is a planarization operation, where the detected shapes are planes. The detected planes may represent objects or portions of objects. Detecting planes may simplify processing for rendering the XR environment. Moving objects being rendered in the XR environment may move in an easily calculable manner, for example, when colliding with a plane. Thus, identifying a plane may simplify subsequent rendering of the moving object, compared to performing calculations based on multiple polygons representing the same portion of the environment. Other shapes, including semantics such as cylinders, cubes, lines, corners, or glass surfaces or holes, may also be detected and used in subsequent processing instead of or in addition to a plane. Such an operation may group polygons that represent the surfaces of the detected shapes.

[0205] A second simplification operation, following the region-based operation, may also further simplify the representation of the environment, such as by further reducing the number of polygons in the representation. The second simplification operation may focus on reducing the number of polygons in each region detected by the region-based operation.

[0206] Such processing may enable a meshing service that processes sensor data collected in the physical environment and provides a mesh to content generating applications. In some embodiments, the processing may provide simplified representations of virtual objects in the virtual environment.

[0207] In XR systems, such as virtual reality (VR), augmented reality (AR), and mixed reality (MR) systems, three-dimensional (3D) mesh data is often used for multiple purposes, including, for example, to occlude virtual content in a graphics / game engine based on physical objects in the environment, or to calculate effective rigid-body collisions for virtual objects in a game engine's physics engine. In some embodiments, the requirements for a mesh may be different for different uses of the mesh, and simplified meshes may be suitable for many such uses, with some simplification techniques being more suitable for some uses than others.

[0208] Thus, the processing as described herein may be implemented using any of a number of simplification techniques and / or simplification techniques that may be configured based on the intended use of the simplified mesh. The utility of a meshing service that provides simplified meshes to multiple client applications that may use the mesh in different ways may be improved using processing as described herein. Each client application may request a mesh with a different level of simplification. In some embodiments, an application that accesses the meshing service may specify a target simplification or mesh to be provided to it. Mesh simplification methods as described herein may be useful for multiple client applications, including, for example, those that perform virtual content occlusion, physics simulation, or environment geometry visualization. The mesh processing described herein may have low latency and may be flexible in that it may optimize / bias operation for different uses (e.g., flattened surfaces, variable triangle counts).

[0209] Mesh simplification methods as described herein may provide real-time performance (e.g., low latency to support on-the-fly (real-time) environmental changes), local update capabilities (e.g., refreshing portions of the mesh that have changed since the last update), and flattened surfaces (e.g., planar surfaces that are flattened to support robust physics simulations).

[0210] In some embodiments, the representation of the XR environment may be segmented into multiple blocks, some or all of which may be processed in parallel. In some embodiments, the resulting blocks may then be recombined. In some embodiments, blocks may be defined with "skirts" that overlap adjacent blocks. Skirts allow for the recombination of blocks with less and / or less noticeable disruption to the interfaces of the blocks being reassembled.

[0211] Thus, in some embodiments, mesh simplification methods may include mesh block segmentation, pre-simplification, mesh flattening, and post-simplification. To speed up the process, the global mesh may first be segmented into blocks of component meshes so that the mesh blocks can be handled (e.g., processed) in parallel. The mesh blocks may then be extended with skirts that span the boundaries between adjacent blocks. With skirted mesh blocks, simplification may be performed on individual mesh blocks while the global mesh may be topologically disconnected but visually seamless.

[0212] In some embodiments, the mesh simplification method may be suitable for use by applications that use simplified meshes to represent object interactions in an XR environment, such as by having the simplification process recognize planes. To simplify the mesh, a three-step simplification process may be implemented. The mesh may first be moderately pre-simplified using a relatively high target triangle count. Planar areas may then be detected by a region growing algorithm. The mesh may be flattened by projecting corresponding triangles onto the detected plane. In some embodiments, the mesh may be regularized by adjusting the plane (or primitive) normals to be approximately perpendicular and parallel to the detected plane. A post-simplification process may then be launched over the flattened mesh. The post-simplification process may focus more on the detected planar areas, for example, simplifying the mesh of each detected planar area to reach a desired level of complexity (e.g., a complexity metric) as may be indicated by a target value of one or more metrics.

[0213] FIG. 30 illustrates a method 3000 of generating a model of an environment represented by a mesh, according to some embodiments. In some embodiments, the method 3000 may be implemented on a meshing service on an XR platform. The method 3000 may begin, at act 3002, with an input mesh representing the environment. In some embodiments, the input mesh may have a high resolution, which may be indicated by the number of triangles. The input mesh may be generated by a reconstruction system (e.g., a stereoscopic 3D reconstruction system), and the input mesh may include 3D reconstruction data.

[0214] In some embodiments, the reconstruction system may generate a stereoscopic 3D representation of the environment, which may create a data hierarchy of 3D information of the environment captured by one or more sensors. For example, the sensor may be a depth camera, which may capture a stream of 3D information of the environment, e.g., depth images, using a distinct pose (i.e., camera pose) of the depth camera. The 3D information of the environment may be processed into a voxel grid. Each voxel may contain one or more signed distance functions (SDFs) that describe whether the voxel is inside or outside the geometry of an object in the environment. The voxels may be grouped into "bricks." Each brick may contain, for example, 8 3 A brick may contain multiple voxels within a cubic volume, such as a voxel. Bricks may be further grouped into "tiles." Each tile may contain multiple bricks.

[0215] The size of the tiles may be selected to facilitate memory operations within a computing device. The size may be selected, for example, based on the amount of information about the environment that is maintained in the active memory of the device that processes such data. The system may, for example, transfer tiles between the active memory, which is typically local to the device, and other memories that have more latency, such as non-volatile memory or remote memory in the cloud. One or more whole or partial tiles may contain information that represents a "block" in a mesh or other representation of the environment.

[0216] In some embodiments, the volumetric 3D reconstruction system may generate the input mesh 3002 as a topologically connected global mesh. In some embodiments, the volumetric 3D reconstruction system may generate the input mesh 3002 as a visually seamless, but topologically disconnected, global mesh. For example, a topologically disconnected global mesh may be composed of multiple mesh blocks, each generated from a block.

[0217] The reconstruction system may be configured to capture substantial detail of the environment, which allows the system to distinguish adjacent portions of the representation that have relatively small differences in their characteristics. Adjacent regions of different nature may be identified as different surfaces, resulting in the system identifying multiple surfaces in the environment. However, such a system may capture detail that is unnecessary for many applications, but is processed nonetheless. For example, the reconstruction system may unnecessarily present irregularities with many triangles on a wall, when two triangles that make a rectangle would be a sufficient representation of the wall for a client application that requests the mesh from the meshing service. In some embodiments, when an application requests a mesh from the meshing service, it may specify a target simplification level of the requested mesh. The target simplification level may be expressed in degree of compression, number of triangles per unit area, or any other suitable manner.

[0218] From the input mesh, the method 3000 may efficiently generate a model of the environment that is sufficient for the client application. At act 3004, the input mesh may be segmented into one or more first mesh blocks, each of which may correspond to a block in the data hierarchy of the stereoscopic 3D representation of the environment.

[0219] Each first mesh block may represent a portion of the environment and may have a first value of a complexity metric (e.g., mesh resolution). In some embodiments, the complexity metric of a mesh block indicates the number of triangles in the mesh block. In some embodiments, processing may be performed on mesh blocks in a sequential and / or parallel manner. However, simplification processing as described herein may be applied to the entire mesh or any suitable portion (e.g., one or more mesh blocks).

[0220] Act 3006 represents a sub-process that is performed on each of the multiple mesh blocks. The sub-process may be performed on multiple mesh blocks independently, such that processing may be easily performed in parallel for some or all of the mesh blocks. The sub-process may be performed on all mesh blocks or a subset of the mesh blocks selected for further processing. The subset of mesh blocks may be selected based, at least in part, on the field of view of the device on which the application requesting the simplified mesh is executed.

[0221] In act 3006, some of the first mesh blocks may be selected, for example, based on the object described in the first mesh block or the location of the first mesh block. For each selected first mesh block, a multi-stage simplification may be performed. In some embodiments, the multi-stage simplification on the selected first mesh blocks may be performed in parallel, such that the simplification on the selected first mesh blocks may be completed at approximately the same time, which may depend on the complexity metric of each mesh block of the selected first mesh blocks.

[0222] The multi-stage simplification may include a pre-simplification operation, a region-based operation (e.g., a flattening operation), and a post-simplification operation. In some embodiments, the multi-stage simplification may be performed based on input values ​​from a client application. The input values ​​may indicate a mesh complexity (e.g., mesh resolution) required by the client application. The input values ​​from the client application may be the same or different for each selected first mesh block.

[0223] In act 3012, a pre-simplification operation may be performed on the selected first mesh block to generate a second mesh block. The pre-simplification operation may reduce the complexity of the block. For mesh blocks, the pre-simplification may reduce the number of polygons in the mesh block. In some embodiments, the amount of pre-simplification in act 3012 may be configurable. A target value may be provided as an input to the process, for example, in act 3012, for example, by a client application. The target value may be a single value or multiple values ​​of one or more prescribed or predefined metrics. The one or more metrics may include, for example, an absolute triangle count, a percentage of the initial triangle count, and / or a quadratic error metric, which may measure the mean squared distance between the simplified mesh and the original mesh (e.g., input mesh 3002).

[0224] The target values ​​may be provided in any suitable manner. An instance of method 3000 may, for example, be pre-configured with the target values. In some embodiments, the target values ​​may be supplied through an API by an application that requests a mesh from a meshing service that implements method 3000. For example, the target values ​​for act 3012 may be the final targets requested by a rendering function (e.g., a requesting application). In some embodiments, the target values ​​provided as input may be adjusted or overwritten to ensure that enough data remains in the mesh for subsequent processing. For example, processing in act 3014 may require a minimum number of triangles, and the target value provided by the application may be replaced by the minimum value if the target value falls below the minimum number of triangles.

[0225] In such embodiments, the pre-simplified mesh may have values ​​of one or more metrics such that the pre-simplified mesh may be processed faster than the segmented input mesh of the original block during region-based operations while still containing all or most of the region of the segmented input mesh of the original block.

[0226] If the values ​​of one or more metrics are not controlled, the simplified mesh may be too coarse, non-uniformly distributed, and / or miss many regions of the segmented input mesh of the original block that are required in subsequent region-based operations.

[0227] The second mesh block produced in act 3012 may have a second value of the complexity metric, which may be less than the first value of the complexity metric. In some embodiments, the pre-simplification operation of act 3012 may be performed using a triangle reduction algorithm.

[0228] In act 3014, a shape detection operation may be performed on the second mesh block to generate a third mesh block. A flattening operation is illustrated as an example. The third mesh block may have a third value of a complexity metric. In some embodiments, the third value of the complexity metric may be the same as the second value of the complexity metric. In some embodiments, the third value of the complexity metric may be less than the second value of the complexity metric. The flattening operation may include detecting planar areas in the second mesh block, using, for example, a region growing algorithm, projecting meshes of the detected planar areas onto corresponding planes, adjusting plane normals of the detected planar areas to be approximately perpendicular to the corresponding planes, and simplifying the projected mesh on each of the corresponding planes based, for example, on a target triangle count. In some embodiments, the plane normals of the detected planar areas may be adjusted prior to projecting the meshes of the detected planar areas onto the corresponding planes.

[0229] In act 3016, a post simplification operation may be performed on the third mesh block to generate a fourth mesh block. In some embodiments, the processing in act 3014 may desirably be performed on a mesh with a resolution greater than that required in the simplified mesh that will be output from method 3000. In some embodiments, the processing in act 3016 may simplify the entire mesh block to reach a desired level of complexity (e.g., a complexity metric) as may be indicated by target values ​​of one or more metrics, which may be the same or different targets as those provided to act 3012. In some embodiments, the post simplification operation in act 3016 may focus on reducing the number of polygons in each plane detected by the planarization operation in act 3014.

[0230] The fourth mesh block may have a fourth value of the complexity metric, which may be less than the third value of the complexity metric. In some embodiments, the percentage reduction between the third value of the complexity metric and the fourth value of the complexity metric may be greater than the percentage reduction between the first value of the complexity metric and the second value of the complexity metric. In some embodiments, the percentage reduction between the third value of the complexity metric and the fourth value of the complexity metric may be at least two times greater than the percentage reduction between the first value of the complexity metric and the second value of the complexity metric. In some embodiments, the post-simplification operation of act 3016 may be performed using a triangle reduction algorithm. In some embodiments, the post-simplification operation of act 3016 may be performed using the same simplification algorithm as the pre-simplification operation of act 3012.

[0231] In act 3008, the simplified selected blocks may be combined with other selected mesh blocks similarly processed as in act 3006, and / or may be combined with unselected blocks into a new mesh of the environment. In act 3010, the new mesh of the environment may be provided to the client application. In some embodiments, the new mesh of the environment may be referred to as a simplified mesh.

[0232] In some embodiments, act 3008 may be skipped. The simplified mesh blocks may be sent directly to the client application, where the mesh blocks may be visually seamless, but topologically disconnected.

[0233] FIG. 31 illustrates an example of segmenting a mesh representation 3100 of an environment into mesh blocks, according to some embodiments. The mesh representation 3100 may be segmented into four mesh blocks, namely mesh blocks AD. In some embodiments, the mesh blocks may correspond to regions with the same volume in the physical world that belongs to the environment. In some embodiments, the mesh blocks may correspond to regions with different volumes in the physical world that belongs to the environment. For example, when the physical world is an office room, the office room may be divided into regions, each of which may be one cubic foot. One block may contain a 3D representation of one area of ​​the office room.

[0234] Although the mesh representation 3100 of the environment is illustrated in two dimensions (2D), it should be understood that the environment may be three-dimensional, correspondingly represented by a 3D mesh representation. Although the mesh representation 3100 of the environment is illustrated as a combination of four mesh blocks, it should be understood that the environment may be represented by any suitable number of mesh blocks, for example, two, three, five, six, or more.

[0235] Representation 3100 may be segmented, for example, into four portions, namely portions 3102, 3104, 3106, and 3108, illustrated by solid lines 3110. In some embodiments, portions 3102, 3104, 3106, and 3108 may each be designated as a mesh block AD.

[0236] Even if a mesh block is updated, it may continue to interface with neighboring blocks that were not updated. As a result, discontinuities at the boundaries between mesh blocks may occur. If the area represented by the neighboring blocks has a discontinuity, the fused mesh may be interpreted in subsequent processing as indicating that a crack exists between the neighboring blocks. In some embodiments, such a crack in the representation of the physical world space may be interpreted as a space with infinite depth. Thus, the space may not actually be a feature, but an artifact of the representation of the physical world. Any application that uses such a fused mesh to generate a representation of an object in the physical world may not correctly generate output. For example, an application that renders a virtual character on a surface in the physical world may render the character to appear as if it were falling through the crack, which would not create the desired appearance of the object.

[0237] To reduce the appearance of such cracks, in some embodiments, portions of adjacent blocks may represent the same region of the physical world. For example, the interface region between adjacent blocks may be represented by a portion of each of the adjacent blocks, which may allow for easy independent updating and / or rendering that takes into account the level of detail (LOD) (e.g., reducing the complexity of 3D reconstruction of a portion of the physical world as that portion moves out of the user's field of view). Even if one block is updated and its adjacent block is not updated, the fused mesh may be represented by combining data representing the interface region from both of the blocks. As a specific example, when fusing an updated block with an adjacent block, the physics engine may determine the overlap region of the adjacent blocks, for example, based on which of the adjacent blocks should be observable within that overlap region. A block-based data structure may be employed along with a skirt, zipper, or any other suitable method for representing the interface region between adjacent blocks, such that when a block is updated, it will continue to interface with an adjacent block that was not updated. Such an approach may have the appearance of "hiding" the crack between adjacent blocks. Thus, blocks may be updated independently of their adjacent blocks.

[0238] In the example illustrated in FIG. 31 , the area at the boundary of portions 3102, 3104, 3106, and 3108 may be designated as a skirt illustrated by dashed line 3112. In some embodiments, mesh block A-D may each include one of portions 3102, 3104, 3106, and 3108 and a corresponding skirt. For example, mesh block B may include portion 3104 and skirt 3114 that overlaps with boundary portions of mesh block B near mesh blocks A, C, and D such that cracks between the mesh blocks may be veiled when the blocks are connected into a single mesh. Mesh blocks A, C, and D may also include corresponding skirts. Thus, the processor may veil any cracks between the mesh blocks before returning a single connected 3D mesh representation to the application.

[0239] In some embodiments, the block mesh, including the skirt, may be sent directly to the application without being combined so as to be topologically connected to the global mesh. The application may have a global mesh, made up of the block meshes, that is visually seamless, but may be topologically disconnected.

[0240] 32A-32D illustrate the mesh evolution of an example mesh block 3201 during multi-stage simplification, according to some embodiments. Mesh block 3201 may include vertices 3206, edges 3208, and faces 3210. Each face may have a normal, which may be represented by multiple coordinates, for example, shown in FIG. 32A as x, y, z.

[0241] A pre-simplification operation may be performed on mesh block 3201 to generate mesh block 3202. An edge collapse transformation may be used. In the illustrated example, mesh block 3202 reduces the number of faces of mesh block 3201 from 10 to 8. Each of the resulting faces of mesh block 3202 may have a corresponding set of normals (e.g., x1, y1, z1; x2, y2, z2; ... x8, y8, z8).

[0242] A planarization operation may be performed on the mesh block 3202 to generate a mesh block 3203. The planarization operation may include, for example, detecting planar areas within the mesh 3202 based on the plane (or primitive) normals of the faces. Values ​​of the plane normals x1, y1, z1 of the first face 3212 and the plane normals x2, y2, z2 of the second face 3214 may be compared. A comparison result of the plane normals of the first and second faces may indicate an angle between the plane normals (e.g., the angle between x1 and x2). When the comparison result is within a threshold, it may be determined that the first and second planes are coplanar. In the illustrated example, planes 3212, 3214, 3216, and 3218 may be determined to be on a first planar area corresponding to plane 3228, and planes 3220, 3222, 3224, and 3226 may be determined to be on a second, coplanar area corresponding to plane 3230.

[0243] The planarization operation may also include projecting the triangles formed by the edges of planes 3212, 3214, 3216, and 3218 onto plane 3228 as illustrated by dashed line 3232, and projecting the triangles formed by the edges of planes 3220, 3222, 3224, and 3226 onto plane 3230 as illustrated by dashed line 3234. The planarization operation may also include adjusting the plane normals for planes 3212, 3214, 3216, and 3218 to be identical to the plane normal (x_a, y_a, z_a) for plane 3228, and adjusting the plane normals for planes 3220, 3222, 3224, and 3226 to be identical to the plane normal (x_b, y_b, z_b) for plane 3230.

[0244] A post-simplification operation may be performed on mesh block 3203 to produce mesh block 3204. In the illustrated example, mesh block 3204 reduces the number of faces of mesh block 3203 from eight to four.

[0245] 33A and 33B-36A and 36B illustrate the effect of simplification, showing side-by-side the same portion of the physical world with and without simplification applied. These figures provide a graphical illustration of how simplification can provide usable information for operating an AR system while providing less data that needs to be processed.

[0246] Figures 33A and 33B show representations of the same environment with and without simplification through triangle reduction, respectively. Such processing is an example of processing that may be implemented in pre-simplification and post-simplification blocks 3012 and 3016 in Figure 30.

[0247] 34A and 34B show zoomed-in representations of the same environment with and without simplification through triangle reduction, respectively.

[0248] 35A and 35B show representations of the same environment with and without flattening, respectively. Such processing is an example of processing that may be performed in flattening block 3014 in FIG.

[0249] 36A and 36B show representations of the same environment with and without simplification through removal of disconnected components, respectively. Such processing is an example of an alternative embodiment of the region-based operation that may be implemented in block 3014 in FIG.

[0250] Caching and updating dense 3D reconstruction data

[0251] In some embodiments, the 3D reconstruction data may be captured, persisted, and updated on a block-by-block basis, which may allow for local updates while maintaining neighborhood consistency. The block-based 3D reconstruction data representation may be used in conjunction with a multi-layer cache mechanism to efficiently retrieve, pre-fetch, and store 3D data for AR and MR applications, including both single-device and multi-device applications. For example, the stereoscopic information 162a and / or mesh 162c (FIG. 6) may be stored on a block-by-block basis. The usage component 164 may use this block-based representation to receive information about the physical world. Similarly, the perception component 160 may store and retrieve such information on a block-by-block basis.

[0252] These techniques extend the capabilities of portable devices with limited computing resources to present AR and MR content with high realism. Such techniques may be used, for example, to efficiently update and manage real-time or offline reconstruction and scanning outputs in mobile devices with limited resources and a connection (continuous or non-continuous) to the Internet. The techniques make up-to-date, accurate, and comprehensive 3D reconstruction data available to a variety of mobile AR and MR applications, either in single-device applications or multi-device applications that share and update the same 3D reconstruction data. These 3D reconstruction data may be in any suitable format, including meshes, point clouds, voxels, and the like.

[0253] Some AR and MR systems attempt to simplify the presentation of MR and AR scenes by limiting the amount of 3D reconstruction data being processed at any given time. The sensor used to capture 3D information may have a maximum reconstruction range that may define a bounding volume around the sensor's field of view. To reduce the amount of 3D reconstruction data, some reconstruction systems keep only the area near the sensor's field of view in active working memory and store other data in secondary storage. For example, the area near the sensor's field of view is stored in CPU memory, while other data is persisted in a local cache (e.g., disk) or remote storage (e.g., in the cloud) via a network.

[0254] The computational cost of generating the information stored in the CPU memory, even if limited, can still be relatively high. Some AR and MR systems continually recalculate a global representation of the environment previously visible to the reconstruction system in order to select information for storage in the CPU memory, which can be prohibitively expensive for interactive applications. Other AR and MR systems that use some approaches to compute only local updates to the connected representation can also be expensive, especially for simplified meshes, since they require decomposing an existing mesh, computing another mesh with the same boundary, and reconnecting the mesh parts.

[0255] In some embodiments, the 3D reconstruction data may be segmented into blocks. The 3D reconstruction data may be transmitted between storage media based on blocks. For example, blocks may be paged out of active memory and persisted to a local or remote cache. The system may implement a paging algorithm in which an active memory associated with a wearable device (e.g., a head mounted display device) stores blocks representing portions of the 3D reconstruction of the physical world within a field of view of a user of the wearable device. The wearable device may capture data about portions of the physical world in conjunction with a field of view of a user of the wearable device. As the physical world changes within the field of view of the user, the blocks representing that region of the physical world may be in active memory, from which they can be easily updated. As the field of view of the user changes, blocks representing regions of the physical world that move out of the user's field of view may be moved to a cache such that blocks representing regions of the physical world that have come into the field of view of the user may be loaded into the active memory.

[0256] In some embodiments, a coordinate frame may be created for the portion of the physical world to be 3D reconstructed, and each block in the 3D representation of the portion of the physical world may correspond to a different region of the physical world identifiable using the coordinate frame.

[0257] In some embodiments, when a block is updated, the updated block may continue to interface with adjacent blocks that may not be updated. If the areas represented by adjacent blocks do not overlap, cracks may exist in the fused meshes of adjacent blocks. In some embodiments, such cracks in the representation of the physical world space may be interpreted as spaces with infinite depth. Thus, the spaces may not actually be features, but rather artifacts of the representation of the physical world. Any application that uses such fused meshes to generate a representation of an object in the physical world may not generate the output correctly. For example, an application that renders a virtual character on a surface in the physical world may render the character to appear as if it were falling through a crack, which would not create the desired appearance of the object. Thus, in some embodiments, some of the adjacent blocks may represent the same region of the physical world, e.g., the interface between adjacent blocks, which may allow for easy independent updating and / or rendering that takes into account level of detail (LOD) (e.g., reducing the complexity of 3D reconstruction of a portion of the physical world as that portion moves out of the user's field of view). For example, when a block is updated, its adjacent blocks may not be updated. When blending an updated block with an adjacent block, the physics engine may determine the overlapping area of ​​the adjacent blocks, for example, based on which of the adjacent blocks should be observable in that overlapping area. A block-based data structure may be employed with a skirt, zipper, or any other suitable method for representing the interface area between adjacent blocks, such that when a block is updated, it will continue to interface with an adjacent block that was not updated. Such an approach may have the appearance of "unmasking" the cracks between adjacent blocks. Thus, the changed block may be updated independently of the adjacent blocks.

[0258] In some embodiments, these techniques may be used within an AR and / or MR “platform” that receives and processes data from sensors worn by one or more users. This sensor data may be used to create and update 3D reconstruction data to represent the portion of the physical world encountered by the user. A reconstruction service may continually reconstruct a 3D representation of the physical world as the sensors capture and update data. One or more techniques may be used to determine blocks affected by changes in the physical world, which may be updated. This 3D reconstruction data may then be provided to an application that uses the 3D reconstruction data to render the scene and represent virtual reality objects that are positioned in or interact with objects in the physical world. This data may be provided to the application through an application programming interface (API). The API may be a push or pull interface that either pushes data to the application when relevant parts change, or in response to requests from the application for up-to-date information.

[0259] In an embodiment of a pull delivery interface, when an application requests 3D reconstruction data of the physical world, the reconstruction service may determine the appropriate version of each block that needs to be provided to the application, allowing the reconstruction service to start from the newest block. The reconstruction service may, for example, retrieve blocks that were previously persisted. A single device system may allow a single device to contribute to the 3D reconstruction data for the physical world. In a single device system, if the requested area of ​​the physical world is outside or extends outside the active area (e.g., the area within the device's current field of view), the persisted block may be used directly as the newest block because the persisted block would not have been updated since it was persisted when that area moved outside the device's field of view. On the other hand, a multi-device system may allow multiple devices to contribute to the 3D reconstruction data for the physical world, such as by using cloud persistence or peer-to-peer local cache sharing of blocks. Each device may update the areas within its active area that may be persisted. A multi-device system may create a coordinate frame such that blocks generated by different devices are identifiable using the coordinate frame. Thus, blocks requested by an application generating data for a first device may be based on updates from other devices if those updates were made after any version made by the first device. Blocks constructed using data from the first device and other devices may be merged by using coordinate frames.

[0260] The selected block may be used to provide 3D reconstruction data for the physical world in any suitable format, although a mesh is used as an example of a suitable representation herein. The mesh may be created by processing the image data and identifying points of interest in the environment, such as edges of an object. These points may be connected to form a mesh. Points in the mesh associated with the same object or part thereof, conventionally a group of three points, define a surface of the object or part thereof. Information stored in conjunction with the group of points describes the surface in the environment. This information may then be used in a variety of ways to render and / or display information for the environment. The selected block may be used to provide 3D reconstruction data in any suitable manner. In some embodiments, a newest block may be provided. In some embodiments, the newest block may be used to determine whether an update of the block is required.

[0261] For example, in some embodiments, in a multi-device system, when a block required by an application is identified, the reconstruction service may check the blocks persisted by other devices, determine whether there are any significant updates (e.g., via a threshold or timestamp of the magnitude of the geometry change), restart meshing on the blocks that have changed, and then persist these updated mesh blocks.

[0262] In some embodiments, when the set of blocks required by an application is identified, if the application requires a connected mesh, the set of blocks may be processed into a global mesh that is topologically connected using any suitable technique, such as skirts and zippers, or that is visually seamless but may be topologically disconnected.

[0263] In some embodiments, an application (e.g., a graphics / game engine) may update its internal blocks (e.g., blocks stored in active memory and / or local cache) when a block change occurs. The reconstruction service may know what blocks the application has and therefore can calculate other (e.g., nearby) blocks that need to be updated in the engine to maintain the correct overlap with the skirt / zipper as an in-view block update.

[0264] In some embodiments, an AR and / or MR platform may be implemented, for example, on a mobile device, to support the execution of AR and / or MR applications. An application running on or generating data for presentation through a user interface may request 3D reconstruction data representing the physical world. The 3D reconstruction data may be sourced from an active memory on the device, which may be updated with 3D reconstruction data representing the physical world within the field of view of the device as the user changes its field of view. The 3D reconstruction data in the active memory may represent an active area for the mobile device. In some embodiments, the 3D reconstruction data outside the active area of ​​the device may be stored in other memory, such as in a local cache on the device or coupled to the device with a low latency connection. In some embodiments, the 3D reconstruction data outside the active area of ​​the device may also be stored in a remote cache, such as in the cloud, accessible to the device via a longer latency connection. As the user changes its field of view, the platform may access / load the 3D reconstruction data from the cache and add it to the active memory, representing areas that have moved into the user's field of view. The platform may move other data representing areas that have moved out of the user's field of view to the cache.

[0265] Predictions of device movement, which may result in areas moving into the device's field of view and other areas moving out of the device's field of view, may be used to initiate transfers of 3D reconstruction data between the active memory and the cache. Predictions of movement may be used, for example, to select 3D reconstruction data for transfer into and / or out of the active memory. In some embodiments, predicted movement may be used to transfer 3D reconstruction data into and / or out of the local cache by retrieving or transferring the 3D reconstruction data from a remote cache. Exchange of 3D reconstruction data between the local cache and a remote cache based on predicted movements of the user may ensure that the 3D reconstruction data is available with low latency for moving into the active memory.

[0266] In embodiments where regions of the physical world are represented with blocks, initiating transfer of blocks may involve pre-requesting blocks representing areas predicted to be within the user's field of view. For example, if the platform determines, based on sensor or other data, that a user is walking in a particular direction at a particular speed, it may identify areas likely to be within the user's field of view and transfer blocks representing these areas to a local cache on the mobile device. If the mobile device is a wearable device, such as a set of glasses, predicting movement may involve receiving sensor data indicating the location, orientation, and / or rotation of the user's head.

[0267] FIG. 37 illustrates a system 3700 that enables an interactive X-reality environment for multiple users, according to some embodiments. The system 3700 may include a computing network 3705 including one or more computer servers 3710 connected through one or more high-bandwidth interfaces 3715. The servers in the computing network need not be co-located. The one or more servers 3710 may each include one or more processors for executing program instructions. The servers also include memory for storing program instructions and data used and / or generated by processes being performed by the servers under the direction of the program instructions. The system 3700 may include one or more devices 3720, including, for example, an AR display system 80 (e.g., the viewing optics assembly 48 in FIG. 3B).

[0268] The computing network 3705 communicates data between the servers 3710 and between the servers and the devices 3720 via one or more data network connections 3730. Examples of such data networks include, but are not limited to, all types of public and private data networks, both mobile and wired, including, for example, the many interconnections of such networks commonly referred to as the Internet. No particular medium, topology, or protocol is intended to be implied by the illustration.

[0269] In some embodiments, the devices may be configured to communicate directly with either the computing network 3705 or the server 3710. In some embodiments, the devices 3720 may communicate locally with the remote server 3710, and optionally with other devices and AR display systems, through a local gateway 3740 to process data and / or communicate data between the network 3705 and one or more devices 3720.

[0270] As shown, the gateway 3740 is implemented as a separate hardware component, including a processor for executing software instructions and memory for storing software instructions and data. The gateway has its own wired and / or wireless connection with a data network to communicate with a server 3710 comprising the computing network 3705. In some embodiments, the gateway 3740 can be integrated with a device 3720 worn or carried by a user. For example, the gateway 3740 may be implemented as a downloadable software application that is installed and runs on a processor included within the device 3720. The gateway 3740, in one embodiment, provides one or more users with access to the computing network 3705 via the data network 3730. In some embodiments, the gateway 3740 may include communication links 76 and 78.

[0271] The servers 3710 each include, for example, working memory and storage devices for storing data and software programs, a microprocessor for executing program instructions, a graphics processor, and other specialized processors for rendering and generating graphic, image, video, audio, and multimedia files. The computing network 3705 may also include devices for storing data that is accessed, used, or created by the servers 3710. In some embodiments, the computing network 3705 may include a remote processing module 72 and a remote data repository 74.

[0272] Software programs running on the server, and optionally on the device 3720 and gateway 3740, are used to generate a digital world (also referred to herein as a virtual world) with which the user interacts with the device 3720. The digital world is represented by data and processes that describe and / or define virtual non-existent entities, environments, and conditions that may be presented to the user through the device 3720 for the user to experience and interact with. For example, some types of objects, entities, or items that would appear to be physically present when instantiated within a scene viewed or experienced by the user may include descriptions of their appearance, their behavior, the way the user is allowed to interact with them, and other characteristics. Data used to create the environment of the virtual world (including virtual objects) may include, for example, atmospheric data, terrain data, weather data, temperature data, location data, and other data used to define and / or describe the virtual environment. In addition, data defining various conditions that govern the operation of the virtual world may include, for example, physical laws, time, spatial relationships, and other data that may be used to define and / or create various conditions that govern the operation of the virtual world (including virtual objects).

[0273] Entities, objects, conditions, properties, behaviors, or other features of the digital world will generally be referred to herein as objects (e.g., digital objects, virtual objects, rendered physical objects, etc.) unless otherwise indicated by context. Objects may be any type of animate or inanimate object, including, but not limited to, buildings, plants, vehicles, people, animals, living things, machines, data, videos, text, photos, and other users. Objects may also be defined in the digital world to store information about items, behaviors, or conditions that actually exist in the physical world. Data that describes or defines an entity, object, or item, or that stores its current state, is generally referred to herein as object data. This data is processed by the server 3710, or, depending on the implementation, by the gateway 3740 or the device 3720, to instantiate instances of the object and render the object in an appropriate manner for a user to experience through the device 3720.

[0274] Programmers who develop and / or curate a digital world create or define objects and the conditions under which they are instantiated. However, a digital world may allow others to create or modify objects. Once an object is instantiated, the state of the object may be allowed to be altered, controlled, or manipulated by one or more users experiencing the digital world.

[0275] For example, in one embodiment, the development, production, and management of the digital world is generally provided by one or more system administration programmers. In some embodiments, this may include the development, design, and / or execution of storylines, themes, and events within the digital world, and the distribution of storylines through various forms of events and media, such as, for example, film, digital, networked, mobile, augmented reality, and live entertainment. System administration programmers may also handle the technical management, coordination, and curation of the digital world and its associated user community, as well as other tasks typically performed by network administrators.

[0276] A user interacts with one or more digital worlds using some type of local computing device, generally designated as device 3720. Examples of such devices include, but are not limited to, smartphones, tablet devices, heads-up displays (HUDs), gaming consoles, or any other device capable of communicating data and providing an interface or display to a user, and combinations of such devices. In some embodiments, device 3720 may include or communicate with local peripherals or input / output components, such as, for example, a keyboard, a mouse, a joystick, a gaming controller, a haptic interface device, a motion capture controller, audio equipment, voice equipment, a projector system, a 3D display, and holographic 3D contact lenses.

[0277] FIG. 38 is a schematic diagram illustrating an electronic system 3800, according to some embodiments. In some embodiments, the system 3800 may be part of the system 3700 of FIG. 37. The system 3800 may include a first device 3810 (e.g., a first portable device of a first user) and a second device 3820 (e.g., a second portable device of a second user). The devices 3810 and 3820 may be, for example, the device 3720 and the AR display system 80 of FIG. 37. The devices 3810 and 3820 may communicate with a cloud cache 3802 through networks 3804a and 3804b, respectively. In some embodiments, the cloud cache 3802 may be implemented in memory of one or more servers 3710 of FIG. 37. The networks 3804a and 3804b may be examples of the data network 3730 and / or the local gateway 3740 of FIG. 37.

[0278] The devices 3810 and 3820 may be individual AR systems (e.g., device 3720). In some embodiments, the devices 3810 and 3820 may include an AR display system worn by its respective user. In some embodiments, one of the devices 3810 and 3820 may be an AR display system worn by the user, and the other may be a smartphone held by the user. Although two devices 3810 and 3820 are illustrated in the example, it should be understood that the system 3800 may include one or more devices, and one or more devices may operate the same type of AR system or different types of AR systems.

[0279] The devices 3810 and 3820 may be portable computing devices. The first device 3810 may include, for example, a processor 3812, a local cache 3814, and one or more AR applications 3816. The processor 3812 may include a computing portion 3812a configured to execute computer-executable instructions and provide a 3D representation (e.g., 3D reconstruction data) of a portion of the physical world based, at least in part, on data collected using one or more sensors (e.g., the depth sensor 51, the world camera 52, and / or the inertial measurement unit 57 of FIG. 3B).

[0280] The computing portion 3812a may represent the physical world as one or more blocks. Each block may represent an object in a different region of the physical world. Each region may have a corresponding volume. In some embodiments, the blocks may represent regions with the same volume. In some embodiments, the blocks may represent regions with different volumes. For example, when the physical world is an office room, the office room may be divided into cubes, which may be one cubic foot. One block may contain a 3D representation (e.g., 3D reconstruction data) of one cubic foot of the office room. In some embodiments, the office room may be divided into regions with various volumes, and each volume may contain a similar amount of 3D information (e.g., 3D reconstruction data) such that the data size of the 3D representation of each region may be similar. The representation may be formatted to facilitate further processing, such as occlusion processing to determine if a virtual object is occluded by a physical object, or physics processing to determine how a virtual object should move or deform when interacting with a physical object in the physical world. Blocks may be formatted as mesh blocks, where features of objects in the physical world, such as corners, become points in the mesh blocks or are used as points to create mesh blocks. The connections between points in the mesh may represent groups of points on the same surface of a physical object.

[0281] Each block may have one or more versions, each version containing data (e.g., volumetric 3D reconstruction data such as voxels and / or meshes that may represent surfaces within the region represented by the individual block) that represents its corresponding region based on data from a point in time. Computing portion 3812a may create new versions of blocks as additional data becomes available, such as data indicating that objects in the physical world have changed or additional data from which a more accurate representation of the physical world may be created. The additional data may come from sensors on the device (e.g., devices 3810 and / or 3820). In some embodiments, the additional data may come from remote sensors, e.g., obtained via a network connection.

[0282] The processor 3812 may also include an active memory 3812b that may be configured to store blocks within the field of view of the device. In some embodiments, the active memory 3812b may store blocks outside the field of view of the device. In some embodiments, the active memory 3812b may store blocks adjacent to blocks within the field of view of the device. In some embodiments, the active memory 3812b may store blocks that are predicted to be within the field of view of the device. In some embodiments, the processor 3812 maintains a block in the active memory 3812b if it is within the field of view of the device at that time. The field of view may be determined by imaging an area of ​​one or more of the sensors. In some embodiments, the field of view may be determined by the amount of the physical world that is presented to a user of the device or that an average user may perceive without the use of the AR system. The field of view may thus depend on the location of the user within the physical world and the orientation of the wearable components of the device.

[0283] The processor 3812 may treat a block as inactive if it is out of view of the device 3810 as the user moves. Inactive blocks may be paged from active memory to a cache. The cache may be a local cache or a remote cache. In the embodiment of Figure 38, the block is first paged through the local gateway 3818b to the local cache 3814. In some embodiments, the local cache 3814 may be the only available cache.

[0284] In some embodiments, there may be a remote cache accessible via a network. In the illustrated embodiment, a cloud cache (e.g., remote cache) 3802, accessed through a network 3804a, is an example of a remote cache. The processor 3812 may manage when blocks are moved between the local cache 3814 and the cloud cache 3802. For example, when the local cache 3814 is full, the processor 3812 may page blocks to the cloud cache 3802 through the network 3804a. Because blocks in the local cache 3814 are accessible for rendering a scene with less latency than blocks in the cloud cache, the processor 3812 may select blocks to be paged out of the local cache 3814 using an algorithm designed to keep blocks in the local cache 3814 that are most likely to be active. Such an algorithm may be based on time of access. In some embodiments, the algorithm may be based on predictions of device movement that will change the device's field of view.

[0285] An application (e.g., a computer game) that renders a scene may obtain information that represents parts of the physical world that affect the scene to be rendered. The application 3816 may obtain active blocks from the active memory 3812b through the local gateway 3818a. In some embodiments, the local gateway 3818a may be implemented as an application programming interface (API) such that the processor 3812 implements a "service" for the application 3816. In embodiments where the data of the physical world is represented as a mesh, the service may be a "meshed service." The API may be implemented as a push or pull interface, or may have attributes of both. In a pull interface, for example, the application 3816 may indicate parts of the physical world for which it requests data, and the service may provide the data for those parts. In a push system, for example, the service may provide data about parts of the physical world as such data changes or becomes available.

[0286] The portions of the physical world for which data is provided may be limited to those portions that the application 3816 has indicated are relevant, such as data within the field of view of the device or data representing portions of the physical world within a threshold distance of the field of view of the device. In a pull / push delivery system, an application 3816 may request data about a portion of the physical world, and the service may provide data about the requested portion, plus any adjacent portions where the data has changed. In order to limit the information to those that have changed, the service may keep track of versions of the blocks that have been provided to each of the applications 3816, in addition to maintaining blocks that describe the physical world. The operations for determining portions of the representation of the physical world to be updated and where the updates have occurred may be partitioned between the applications 3816 and the service in any suitable manner. Similarly, where updated data is incorporated into the representation of the physical world may also be partitioned in any suitable manner.

[0287] In some embodiments, the reconstruction service may continually reconstruct a 3D representation of the physical world as sensors capture and update data. This data may then be provided to applications 3816, which use the 3D reconstruction data to render the scene and depict both the physical world and virtual reality objects that are positioned within or interact with objects in the physical world. This data may be provided to applications 3816 through an API, which may be implemented as a push interface that pushes data to applications 3816 as relevant parts change, or as a pull interface that responds to requests from applications 3816 for up-to-date information, or both.

[0288] For example, the application 3816 may operate on a mesh representation of a portion of the physical world that constitutes a 45 degree viewing angle over a distance of 10 meters relative to an origin defined by the device's current location and the direction the device is facing within it. As this area changes or data indicating physical changes within this area becomes available, a mesh may be calculated to represent this area. The mesh may be calculated within the application 3816 based on data provided by the service, or may be calculated within the service and provided to the application 3816. In either case, the service may store information within the physical world that simplifies the calculation of the mesh. As described herein, blocks implemented with zippers, skirts, or with other techniques to facilitate "unmasking" of cracks between adjacent blocks may be used to enable processing of only the changed portions of the representation of the physical world. The changed portions of the representation of the physical world may then replace the corresponding portions in the previous representation of the physical world.

[0289] Efficient access to representations of portions of the physical world used to generate meshes to be used by applications 3816 to render the scene to the user reduces computer resources, may allow XR systems to be easily implemented on portable or other devices with limited computing resources, and may result in a more realistic user experience because the XR scene better matches the physical world. Thus, instead of or in addition to using blocks with skirts, zippers, or other techniques to facilitate concealment of cracks between blocks, as described anywhere herein, algorithms used to page blocks in and out of active memory and / or local caches may be selected to reduce the access time to blocks needed to compute a mesh at any given time.

[0290] 38, the gateway 3818a is a pull delivery interface. When an AR application 3816 requests information about an area of ​​the physical world, but the block representing the area is not in active memory 3812b, the processor 3812 may search for a block persisted in local cache 3814. If the processor 3812 cannot find the block in both active memory 3812b and local cache 3814, the processor 3812 may search for a block persisted in cloud cache 3802. Because access to active memory 3812b has lower latency than access to data in local cache 3814 (which has lower latency than access to data in cloud cache 3802), the overall speed in generating the mesh may be increased by the service implementing a paging algorithm that loads blocks into active memory before they are requested, or moves them from cloud cache 3802 to local cache 3814 before they are requested.

[0291] Similar to the first device 3810, the second device 3820 may include a processor 3822 having a computing portion 3822a and an active memory 3822b, a local cache 3824, and one or more AR applications 3826. The AR applications 3826 may communicate with the processor 3822 through a local gateway 3828a. The local cache 3824 may communicate with the processor 3822 through a local gateway 3828b.

[0292] Thus, the cloud cache 3802 may persist blocks transmitted from both devices 3810 and 3820. The first device 3810 may access blocks in the cloud cache 3802 that were captured and transmitted from the second device 3820. Similarly, the second device 3820 may access blocks in the cloud cache 3802 that were captured and transmitted from the first device 3810.

[0293] Devices 3801 and 3802 are provided as examples of portable AR devices, any suitable device may be used and implemented as well, for example a smartphone.

[0294] FIG. 39 is a flow chart illustrating a method 3900 of operating a system (e.g., system 3700) according to some embodiments. In act 3902, the device may capture 3D information about the physical world, including objects within the physical world, and represent the physical world as blocks that include 3D reconstruction data. In some embodiments, the 3D reconstruction data may be captured by a single system and used to render information about only that system. In some embodiments, the 3D reconstruction data may be captured by multiple systems and used to render information about any one of the multiple systems or any other system. In these embodiments, the 3D reconstruction data from the multiple systems may be combined and made accessible to the multiple systems or any other system.

[0295] For example, several users, each wearing an AR system, may set their devices to an augmented mode while exploring a warehouse. A sensor on each device may capture 3D information (e.g., 3D reconstruction data, including depth maps, images, and the like) of the warehouse within the sensor's field of view, including objects (e.g., tables, windows, doors, floors, ceilings, walls) in the warehouse. Each device may segment the warehouse into regions with corresponding volumes and represent individual regions as blocks. The blocks may have versions. Each version of a block may have a value that represents an object in a region of the physical world at a point in time.

[0296] When an application requests information about the physical world, a version of the block to represent that portion of the physical world may be selected and used to generate that information. Such selection processing may be performed by or distributed across any suitable processor, although according to some embodiments the processing may occur locally at the device on which the application requesting the data is running.

[0297] Thus, in act 3904, a processor (e.g., processor 3812 or 3822) may respond to a request from an application (e.g., AR application 3816 or 3826) for 3D reconstruction data. In some embodiments, regardless of whether an application requests 3D reconstruction data, the device may continue to capture 3D information, including 3D reconstruction data about the physical world, and represent the physical world as blocks of 3D reconstruction data. That 3D reconstruction data may be used to create new versions of the blocks.

[0298] If the application requests 3D reconstruction data, processing may proceed to act 3906, where the processor may identify a subset of blocks corresponding to the portion of the physical world required to deliver 3D reconstruction data in accordance with the request. Identification of the blocks may be based, for example, on data collected by sensors (e.g., depth sensor 51, world camera 52, inertial measurement unit 57, global positioning system, and / or the like). The multi-device system may create a common coordinate frame such that blocks generated by different devices that are associated with corresponding portions of the physical world may be created using a common coordinate frame regardless of which device provided the 3D reconstruction data for reconstructing the portion of the physical world represented by that block. As one example of how a common coordinate frame may be created, data from devices within the same general vicinity may be routed to the same server or one or more servers for processing. There, data from each device may first be represented in a device-specific coordinate frame. Once sufficient data from each of the devices has been collected to identify features within the common portion of the physical world, those features may be correlated to provide a transformation from one device-specific coordinate frame to the other. One of these device-specific coordinate frames may be designated as the transform between the common coordinate frame and the other coordinate frame, and that coordinate frame may be used to transform data from the device-specific coordinate frame to the coordinate frame designated as the common coordinate frame. Further examples of creating and localizing relative to a tracking map are provided in U.S. Patent Application Serial No. 62 / 928,833, which is incorporated herein by reference.

[0299] Regardless of the specific mechanism by which the common coordinate frame is created, once created, 3D reconstruction data requested by an application generating data for the first device may be based on updates from the other device if those updates were made after any version made by the first device. Blocks from both the first device and the other device may be merged, for example, by using a common coordinate frame.

[0300] The specific processing in act 3906 may depend on the nature of the request. In some embodiments, if the application requesting the 3D reconstruction data maintains its own information about the blocks and requests specific blocks, the request for the 3D reconstruction data in act 3904 may include a reference to the specific subset of blocks, and identifying the subset of blocks in act 3906 may include determining the subset of blocks that corresponds to the specific subset of blocks. In some embodiments, the request for the 3D reconstruction data in act 3904 may include a reference to a field of view of the device on which the application is running, and identifying the subset of blocks in act 3906 may include determining the subset of blocks that corresponds to the referenced field of view of the device.

[0301] Regardless of the manner in which the blocks are identified / determined, in act 3908, the processor may select a version of the block of the subset of blocks. This selection may be based on one or more criteria. The criteria may be based, for example, on the most recent version of the block from an available source. In the illustrated embodiment, the version of the block may be stored in active memory, local cache, or remote cache. Act 3908 may include, for example, selecting the version in active memory if available, or if not, selecting the version in local cache if available, or selecting the version from remote cache if available. If no version of the block is available, the selection may involve generating the block, for example, from data (e.g., 3D reconstruction data) collected using sensors (e.g., depth sensor 51, world camera 52, and / or inertial measurement unit 57), etc. An algorithm for block selection may be used within the system to manage the background process, block versions stored within each of the possible locations, etc. An example management process is described below in connection with FIG. 41.

[0302] In act 3910, the processor may provide information to the application based on the version of the selected block. Processing at act 3910 may simply involve providing the block to the application, which may be appropriate when the application uses the block directly. If the application receives a mesh, processing at act 3910 may involve generating a mesh from the block and / or a subset of the block and providing the mesh or any suitable portion of the mesh to the application.

[0303] Figure 40 is a flowchart illustrating details of capturing 3D information about objects in the physical world and representing the physical world as blocks of 3D reconstruction data, according to some embodiments. In some embodiments, Figure 40 is a flowchart illustrating details of act 3902 of Figure 39. In act 4002, one or more sensors (e.g., depth sensor 51, world camera 52, inertial measurement unit 57, and / or the like) of a system (e.g., system 3700) capture 3D information about objects in the physical world, including objects in the physical world.

[0304] In act 4004, a processor of the system (e.g., processor 3812 or 3822) may create a version of the block that includes a 3D reconstruction of the physical world based on the 3D information captured by one or more sensors. In some embodiments, each block may be formatted as one or more portions of a mesh. In some embodiments, other representations of the physical world may be used.

[0305] A block may have versions such that each time information about an area of ​​the physical world is captured by any device, a new version of the block may be stored. Each version of a block may have 3D reconstruction data that includes values ​​that represent objects in a certain area of ​​the physical world at a certain point in time. In some embodiments, such processing may be performed locally on the device, resulting in a new version of the block being stored in active memory. In some embodiments, in a multi-device system, a similar process may be performed in a server (e.g., server 3710 of FIG. 37) that may manage versions of blocks such that the most recent version of each block available in its remote cache is provided when requested by any device.

[0306] Because blocks represent a physical world, much of which will likely remain unchanged, new versions of blocks are not necessarily created when new 3D reconstruction data representing corresponding regions of the physical world becomes available. Rather, managing block versions may involve processing the 3D reconstruction data representing the physical world to determine whether there has been sufficient change since the last version of the block representing those regions of the physical world to justify a change. In some embodiments, sufficient change may be indicated by the magnitude of a block metric exceeding a threshold since the last version was stored.

[0307] In some embodiments, other criteria may be applied to determine which version of a block is served as the current version when the block is requested, such as the version having the lowest value of a metric indicating the completeness or correctness of the data in the block. Similar processes may be performed on each device, resulting in versions of the block being stored in a local cache on the device.

[0308] One or more techniques may be used to manage the versions of blocks available to the service on each device. For example, if an acceptable version of a block already calculated exists, rather than creating a new version of the block from the sensor data, the processor may access the previously stored block. Such access may be implemented by efficiently managing the storage of the versions of the block. In act 4006, the processor of the device may page a version of the block of the 3D reconstruction data of the physical world from an active memory (e.g., active memory 3812b or 3822b). Paging may include the processor accessing the sensor data to continually update the blocks in the active memory / local cache / cloud cache, for example, as a function of the field of view of the device. As the field of view of the device changes, blocks corresponding to the new field of view may be transferred (e.g., paged) from the local cache and / or cloud cache into the active memory, and blocks corresponding to areas just outside the new field of view (e.g., blocks adjacent to blocks in the new field of view) may be transferred (e.g., paged) from the active memory and / or cloud cache to the local cache. For example, in act 4008, the version of the block that is paged externally by the processor may be persisted in local memory (e.g., local cache 3814 or 3824) and / or remote memory (e.g., cloud cache 3802). In some embodiments, as each new version of the block is created, e.g., on the device, that version may be sent to remote memory so that other users may access it.

[0309] FIG. 41 is a flowchart illustrating an exemplary process for performing a selection of a version of a block representing a subset of blocks, according to some embodiments. In some embodiments, FIG. 41 is a flowchart illustrating details of act 3908 of FIG. 39. To select a version of each block in the subset of blocks, in act 4102, a processor (e.g., processor 3812 or 3822) may query whether a latest version is stored in the active memory (e.g., active memory 3812b or 3822b). In some embodiments, whether a version is latest may be determined by comparing values ​​tied to the version (e.g., magnitude of geometric change, timestamp, and the like) with data collected using a sensor (e.g., depth sensor 51, world camera 52, and / or inertial measurement unit 57). In some embodiments, a comparison may be made between the current sensor data and the version of the block stored in the active memory. Based on the degree of difference, which may represent, for example, a change in the physical world or the quality of the version in the active memory, the version in the active memory may be considered latest.

[0310] If the latest version is stored in active memory, processing continues to act 4104, where the latest version is selected. If the latest version is not stored in active memory, processing continues to act 4106, where the processor may query whether the latest version is stored in local memory (e.g., local cache 3814 or 3824). The query may be performed using the criteria described above in connection with act 4102 or any other suitable criteria. If the latest version is stored in local memory, then in act 4108, the latest version is selected.

[0311] If the latest version is not stored in the local memory, then in act 4110 the processor may query whether the latest version is stored in a remote memory (e.g., cloud cache 3802). The query may also be performed using the criteria described above in connection with act 4102 or any other suitable criteria. If the latest version is stored in the remote memory, then in act 4112 the latest version is selected.

[0312] If the latest version is not stored in the remote memory, processing may proceed to act 4114, where the device's processor may generate a new version of the block based on 3D information (e.g., 3D reconstruction data) captured by the sensor. In some embodiments, in act 4116, the processor may identify neighboring blocks of the block with the new version and update the identified neighboring blocks according to the new version of the block.

[0313] 42 is a flow chart illustrating a method 4200 of operating the system according to some embodiments, in which paging may be managed based on a projection of the device's field of view based on device motion, rather than pulling blocks into active memory and / or local cache as the device requests them.

[0314] Similar to act 3902, in act 4202, sensors on the device may capture 3D information about the physical world, including objects in the physical world, and represent the physical world as blocks containing 3D reconstruction data.

[0315] In act 4204, a processor (e.g., processor 3812 or 3822) may calculate, based at least in part on the output of the sensors, an area of ​​the physical world to which the portable pointable component (e.g., depth sensor 51, world camera 52, and / or inertial measurement unit 57) will be pointed at a future time. In some embodiments, the processor may calculate based on motion data from the inertial sensors or analysis of captured images. In a simple calculation, for example, to obtain fast results, the processor may calculate based on the translation and rotation of the user's head. When applying a more comprehensive algorithm, the processor may calculate based on objects in the scene. For example, the algorithm may take into account that a user walking towards a wall or table is unlikely to walk through the wall or table.

[0316] In act 4206, the processor may select a block based on the calculated region. In act 4208, the processor may update the active memory (e.g., active memory 3812b or 3822b) with the selected block. In some embodiments, the processor may select the block based on the flowchart of Figure 41. In act 4210, the processor may select a block from the active memory and provide it to the application (e.g., application 3816 or 3826), for example through an API, based on changes to each of the blocks since a version of the block was last provided to the application.

[0317] In some embodiments, in act 4206, the processor may request the selected block from a remote memory (e.g., cloud cache 3802) and update information stored in the local cache (e.g., 3814 or 3824) so ​​that the local cache stores the selected block. Act 4206 may be similar to act 3908 illustrated in FIG.

[0318] In some embodiments, blocks may persist across shutdown / start-up cycles, sleep / wake-up cycles, and various other scenarios. Blocks may be persisted by ensuring that upon transition of a portable user device used within an XR system to an inactive operating status, the blocks are stored in a memory that will reserve the blocks until the user initiates a new session with the device. When a new session begins and the device transitions to an active operating status, the persisted information may be selectively used. For example, blocks persisted in various memories (e.g., active memory, local memory, remote memory) at shutdown / sleep (e.g., the time the device, including active memory / local memory, is shut down / powered down / sleep) may remain in separate memories and thus be available at startup / wake-up (e.g., the time the device, including active memory / local memory, is started / powered on / wake-up). Such an approach may be used, for example, when the memory that stores the blocks is non-volatile and / or remains powered up sufficiently to reserve the information after an event that stops or suspends the operation of the device. Alternatively, or in addition, memory storing the blocks may be copied to such memory as part of a transition to a state where device operation is stopped or suspended, and then copied back as part of the transition back to an operating state.

[0319] Persisting blocks across events that stop or pause operation of the device enables, for example, blocks persisted at the end of a previous session to be available at the start of a current session. In some embodiments, blocks persisted from a previous session may be made available for fast use in a new session even if they are not reserved in or returned to the same memory used in the previous session. For example, upon shutdown, blocks in active memory may be transferred to local memory and / or remote memory, where they are persisted, and blocks in local memory may be transferred to remote memory, where they are persisted. After a new session begins, blocks may be selectively moved from either local or remote memory to active memory. Similarly, blocks may be selectively moved to local memory. For example, techniques as described herein may be used to determine which blocks are moved from remote memory to active memory or local memory based on the current field of view or projected field of view of the user of the device.

[0320] One or more techniques may be used to determine whether a persisted block is used in a new session. In some embodiments, for example, blocks may have metadata associated with them that may be used in determining whether the block is used. As described above, a device may track its location relative to the physical world, which may enable the device to locate relative to a coordinate system shared by multiple devices. Blocks may have associated metadata, including location information, that may be compared to the location of the device in response to starting a new session. Persisted blocks that represent a portion of the physical world in the vicinity of the device may be used in the new session.

[0321] Other metadata, such as the version of the block, may be used to determine if the block is still current. In some embodiments, a device may communicate with a cloud service that shares blocks with multiple devices and determines the latest version of a block that represents a portion of the physical world. If a device persists the latest version of a block, it may continue to use it. Alternatively, or in addition, a timestamp that reflects the time of the block's creation may be used. For example, a persisted block may be used when its timestamp is less than a threshold amount of time prior to the current time.

[0322] In some embodiments, the device may determine whether to use the persisted blocks based on a comparison between the persisted blocks and current sensor data about the physical world. For example, at start-up, the blocks persisted in various memories may be used only after the real world is recognized. For example, sensors on the device may capture 3D information about the physical world, including objects in the physical world, and represent the physical world as blocks that include the 3D reconstruction data. The device may compare the captured blocks with blocks persisted in active, local, and / or remote memory, and in response to a match, retrieve the matched blocks from the active, local, and / or remote memory for use.

[0323] Alternatively, or in addition, the persisted blocks may be compared to the physical world surrounding the device upon actuation without creating a mesh. For example, surfaces in the physical world may be detected using a sensor, such as a depth sensor, and compared to the surfaces represented by the meshes in the blocks corresponding to the locations in the physical world where the surfaces were detected. The persisted blocks may be used if the error between the sensed locations of one or more surfaces and the surfaces represented by the meshes of the persisted blocks is below a threshold.

[0324] Blocks persisted on the device that are determined to be unsuitable for use in a new session may be deleted or otherwise processed so that they are not used.

[0325] Block-based processing as described above may be based on blocks, allowing portions of the 3D representation to be processed separately and then combined with other blocks. According to some embodiments, blocks may be formatted such that when a block is changed, the changed representation maintains, for the most part or entirely, the values ​​of the block at the interfaces to other blocks. Such processing, based on changes and invariant blocks, allows a changed version of a block to be used in conjunction with versions of neighboring blocks that were not changed without creating unacceptable artifacts in the rendered scene. Figures 43A-48 illustrate such blocks.

[0326] The 3D representation of the physical world may be provided by a stereoscopic 3D reconstruction, which may create a 3D reconstruction data hierarchy of the 3D information of the physical world captured by the sensor. For example, the sensor may be a depth camera, which may capture 3D information of the physical world, e.g., a stream of depth images with a respective pose of the depth camera (i.e., camera pose). The 3D information of the physical world may be processed into a voxel grid. Each voxel may contain one or more signed distance functions (SDFs) that describe whether the voxel is inside or outside the geometry of an object in the physical world. The voxels may be grouped into "bricks". Each brick may contain, for example, 8 3 A brick may contain multiple voxels within a cubic volume, such as a voxel. Bricks may be further grouped into "tiles." Each tile may contain multiple bricks.

[0327] In some embodiments, the voxel grid may be mapped to conform to a memory structure. A tile may correspond to a memory page of a storage medium. The size of the tile may be variable, for example, depending on the size of the memory page of the storage medium in use. Thus, the 3D reconstruction data may be transmitted between storage media (e.g., active memory of the device and / or local memory and / or remote memory in the cloud) based on tiles. In some embodiments, one or more tiles may be processed to generate a block. The block may be updated, for example, when at least one voxel in one or more tiles changes.

[0328] A block is not necessarily limited to correspond to a tile. In some embodiments, a block may be generated from one or more bricks, one or more voxels, or one or more SDF samples, and the like. A block may be any suitable partition of the physical world. A block is not necessarily limited to the format of a mesh. A block may be in any suitable format of 3D reconstruction data.

[0329] 43A-D illustrate an example physical world 4300 represented by mesh blocks 4302. Each mesh block 4302 may be extracted from voxels 4304, which correspond to a given volume of the mesh block. In the illustrated example, each block represents a cubic region of voxels (e.g., 1 m) within the low-level reconstruction model. 3), each mesh block 4302 may contain a portion of the world mesh and may be processed independently. This may allow for scalability with fast local updates as a result of some blocks being changed when exploring new areas or things being moved in the environment. In the illustrated example, there are no changes to the mesh blocks except for mesh block 4306, which has a new object 4308 placed in front of an existing surface 4310. In this case, the AR system needs to update only mesh block 4306, which may save significant computational power compared to arbitrarily updating the entire mesh for the world.

[0330] 43B is a simplified schematic diagram illustrating a mesh block, according to some embodiments. In the illustrated example, the mesh block, when viewed from the inside, may have a fully connected mesh, meaning that a vertex is shared by multiple triangles.

[0331] On the other hand, individual mesh blocks may be unconnected, independent meshes. Fig. 43C is a simplified schematic diagram illustrating a crack that may exist at the edge of two adjacent mesh blocks, according to some embodiments. Fig. 44D is a simplified schematic diagram illustrating the crack in Fig. 43C being obscured by implementing an overlapping mesh skirt on the adjacent mesh blocks, according to some embodiments.

[0332] FIG. 44 is a schematic diagram illustrating a representation 4400 of a portion of the physical world in 2D segmentation according to some embodiments. The 2D representation 4400 may be obtained by connecting a set of four blocks, namely blocks A-D. The representation 4400 may be segmented, for example, into four blocks, namely blocks 4402, 4404, 4406, and 4408, illustrated by solid line 4410. In some embodiments, blocks 4402, 4404, 4406, and 4408 may each be designated as blocks A-D. An application may desire 3D reconstruction data in the format of a mesh for further processing, such as occlusion testing and generation of physical effects in a physics engine. In some embodiments, the set of blocks may be in the format of a mesh, which may be generated by a device (e.g., devices 3810, 3820), a network (e.g., the cloud, including cloud cache 3802), or a discrete application (e.g., applications 3816, 3826).

[0333] In some embodiments, the regions at the boundaries of blocks 4402, 4404, 4406, and 4408 may be skirts, for example, as illustrated by dashed line 4412. In some embodiments, blocks A-D may each include a block and a corresponding skirt. For example, block B may include block 4404 and a skirt 4414 that overlaps with boundary portions of block B's neighboring blocks A, C, and D such that cracks between the blocks may be occluded when the blocks are connected into the global mesh. Blocks A, C, and D may also include corresponding skirts. Thus, before returning the blocks containing the 3D reconstruction data to the application, the processor may occlude any cracks between the blocks.

[0334] In some embodiments, the global mesh may be a topologically connected global mesh, for example, adjacent blocks in a set of blocks may share mesh vertices within block boundaries, such as line 4410. In some embodiments, the global mesh may be visually seamless, but topologically disconnected, using any suitable technique, such as skirts and zippers.

[0335] Although a method employing a skirt is illustrated, other methods, such as zippers, may also be used to allow an altered block to be combined with an unchanged adjacent block. In the illustrated example, the portion of the physical world is represented by four 2D blocks, but it should be understood that the portion of the physical world may be represented by any suitable number of 2D and / or 3D blocks, e.g., two, three, five, six or more. Each block may correspond to a space in the physical world. In some embodiments, blocks in the 2D and / or 3D representation of the portion of the physical world may correspond to spaces of the same size (e.g., area / volume) in the physical world. In some embodiments, blocks in the 2D and / or 3D representation of the portion of the physical world may correspond to spaces of different sizes in the physical world.

[0336] FIG. 45 is a schematic diagram illustrating a 3D representation 4500 of a portion of the physical world, according to some embodiments. Similar to the 2D representation 4400, the 3D representation 4500 may be obtained by connecting eight blocks, namely blocks AH. In some embodiments, the blocks AH may be mutually exclusive, e.g., have no overlapped regions. In some embodiments, the blocks AH may have overlapping regions (e.g., skirt 4516) with neighboring blocks. In some embodiments, the blocks AH may each have versions. Each version of a block may have values ​​that represent an object in a certain region of the physical world at a certain point in time. In the illustrated example, the 3D representation 4500 includes versions of blocks AH, namely, version 4502 of block A, version 4504 of block B, version 4514 of block C, version 4512 of block D, version 4534 of block E, version 4506 of block F, version 4508 of block G, and version 4510 of block H. Version 4502 of block A may include value 4518, version 4504 of block B may include value 4522, version 4514 of block C may include value 4528, version 4512 of block D may include value 4532, version 4534 of block E may include value 4520, version 4506 may include value 4524, version 4508 may include value 4526, and version 4510 may include value 4530.

[0337] FIG. 46 is a schematic diagram illustrating a 3D representation 4600 of a portion of a physical world obtained by updating the 3D representation 4500, according to some embodiments. Compared to the 3D representation 4500, the 3D representation 4600 may have a new version 4610 of block H, including information 4630. The information 4630 may differ from the information 4530. For example, a first device may persist a version 4510 of block H in a remote memory. The version 4510 of block H may include information 4530 corresponding to a table with an empty surface. After the first device leaves the area (e.g., the field of view of the first device no longer includes block H), a second device may place a virtual and / or physical box on the surface of the table and persist a version 4610 of block H in a remote memory. The version 4610 of block H may include information 4630 corresponding to a table with virtual and / or physical boxes. When the first device returns, the first device may be able to select version 4610 of block H to view from the available versions of block H, including version 4610 of block H and version 4510 of block H.

[0338] FIG. 47 is a schematic diagram illustrating an augmented world 4700 viewable by a first device 4702 (e.g., device 3810) and a second device 4712 (e.g., device 3820). The first and second devices may include AR display systems 4704 and 4714 (e.g., AR display system 80) operating in an augmented mode. The augmented world 4700 may be obtained by connecting four blocks, namely, blocks A-D. In the illustrated example, the augmented world 4700 includes versions of blocks A-D, namely, version 4702A of block A, version 4702B of block B, version 4702C of block C, and version 4702D of block D. The first device 4702 may look in a first direction 4706 and have a first field of view (FOV) 4708. In the illustrated example, the first FOV includes version 4702B of block B and version 4702D of block D. A processor (e.g., 3812) of the second device 4704 may include computer-executable instructions for identifying blocks B and D corresponding to the first FOV and selecting version 4702B of block B and version 4702D of block D. The second device 4714 may look in a second direction 4716 and have a second FOV 4718. In the illustrated example, the second FOV includes version 4702C of block C and version 4702D of block D. A processor (e.g., 3822) of the second device 4714 may include computer-executable instructions for identifying blocks C and D corresponding to the second FOV and selecting version 4702C of block C and version 4702D of block D.

[0339] 48 is a schematic diagram illustrating an augmented world 4800 obtained by updating the augmented world 4700 with new versions of blocks, according to some embodiments. Compared to the augmented world 4700, the augmented world 4800 may include a version 4802C of block C that differs from version 4702C and a version 4802D of block D that differs from version 4702D. The first device 4702 may be looking in a third direction 4806 and have a third FOV 4808. In the illustrated example, the third FOV includes version 4802C of block C and version 4802D of block D. A processor of the first device 4702 may include computer-executable instructions for determining versions 4702C, 4802C, 4702D, and 4802D for providing an application based on, for example, a change in FOV and / or information collected using a sensor of the first device 4702. In some embodiments, the processor of the first device 4702 may include computer-executable instructions for generating version 4802C of block C and version 4802D of block D when corresponding latest versions of blocks C and D are not present in local memory (e.g., local cache 3814, 3824) or remote memory (e.g., cloud cache 3802). In some embodiments, the first device 4702 may be able to estimate a change in its FOV (e.g., from the first FOV 4708 to the third FOV 4808) and, based on the estimation, select block C and store a version of block C in memory closer to the processor (e.g., move the version of block C from the remote memory to the local cache or from the local cache to the active memory).

[0340] How to Render Occlusion with Raycasting and Live Depth

[0341] The realism with which AR and MR scenes are presented to a user may be enhanced by providing occlusion data to applications generating such scenes, where the occlusion data is derived from a combination of one or more depth data sources. The occlusion data may be formatted in any suitable manner, such as with depth data that may represent surfaces of physical objects in a scene and indicate the distance and surface from a viewpoint from which the scene should be rendered. For example, the utilization component 164 may receive the occlusion data from the perception module 160 (FIG. 6).

[0342] However, in some embodiments, one data source may be directly one or more depth cameras that sense and capture the position between the depth camera and real objects in the physical world. Data from the depth cameras may be provided directly to the usage component 164 or indirectly, such as through the perception module 160. The one or more depth cameras may provide an instantaneous view of the physical world at a frame rate that is high enough to capture changes in the physical world, but low enough not to burden the processing. In some embodiments, the frame rate may be 5 frames / sec, 10 frames / sec, 12 frames / sec, 15 frames / sec, 20 frames / sec, 24 frames / sec, 30 frames / sec, and the like. In some embodiments, the frame rate may be less than 5 frames / sec. In some embodiments, the frame rate may be greater than 30 frames / sec. Thus, in some embodiments, the frame rate may be in the range of 1-5 frames / sec, 5-10 frames / sec, 10-15 frames / sec, 15-20 frames / sec, or 20-30 frames / sec, and the like.

[0343] A second source of data may be a stereoscopic camera, which may capture a visual representation of the physical world. Depth data from the depth camera and / or image data from the visual camera may be processed to extract points that represent real objects in the physical world. Images from a visual camera, such as a stereoscopic camera, may be processed to compute a three-dimensional (3D) reconstruction of the physical world. In some embodiments, the depth data may be generated from images from the visual camera, for example, using deep learning techniques. Some or all of the 3D reconstruction may be computed prior to the occlusion data and may be stored in memory. In some embodiments, the 3D reconstruction may be maintained in computer memory by a process independent of any process that generates depth information for occlusion processing, which may access the stored 3D reconstruction as needed. In some embodiments, the 3D reconstruction may be maintained in memory, portions of which may be updated, for example, in response to an indication computed based on the depth information that there is a change in the physical world that corresponds to a portion of the 3D reconstruction. In some embodiments, the second data source may be implemented by ray casting into the 3D reconstruction of the physical world to obtain low-level 3D reconstruction data (e.g., a ray casting point cloud). Using ray casting, data from the second data source may be selected to fill any holes in the occlusion data and allow the data from the two (or more) sources to be integrated.

[0344] According to some embodiments, the depth data and / or image data and / or low level data of the 3D reconstruction may be oriented to a user of the AR or MR system. Such orientation may be achieved, for example, by using data from a sensor worn by the user. The sensor may be mounted, for example, on a head-mounted display device / unit.

[0345] In systems in which occlusion data may be generated from more than one depth data source, the system may include a filter that identifies a portion of the 3D region represented by data from each of the multiple depth data sources. The filter may apply one or more criteria to identify a portion of the region for which data from a second data source should be collected. These criteria may be an indication of the reliability of the depth data. Another criterion may be a change detected within the portion of the region since depth data was collected.

[0346] Selection between multiple depth data sources to provide data for different portions of a representation of a 3D region may reduce processing time. For example, selection may favor data from a first data source when less processing is required to derive occlusion data from data collected with a first data source than with a second data source, but may use data from the second source when data from the first source is unavailable or unacceptable. As a specific example, the first data source may be a depth camera and the second data source may be a stereoscopic camera. Data from the stereoscopic camera may be formatted as a 3D reconstruction of the physical world. In some embodiments, the 3D reconstruction may be calculated before occlusion data is requested. Alternatively, or in addition, the 3D reconstruction may be recalculated when occlusion data is requested. In some embodiments, criteria may be applied to determine whether the 3D reconstruction should be recalculated.

[0347] In some embodiments, this occlusion data is calculated by a service that provides the occlusion data through an application programming interface (API) to an application running on a computing device that will render the XR scene. The service may run on the same computing device as the application or on a remote computer. The service may include one or more of the components discussed herein, such as a filter for data from a first data source and / or an engine for selectively fetching data from a second data source based on the filtered data from the first data source. The service may also include a component for combining the filtered data from the first data source and the selected data from the second data source to generate the occlusion data.

[0348] The occlusion data may be formatted in any suitable manner to represent surfaces in the physical world. For example, the occlusion data may be formatted as a depth buffer of surfaces, storing data identifying the locations of surfaces in the physical world. This occlusion data may then be used in any suitable manner. In some embodiments, the occlusion data may be provided to one or more applications that desire to have a virtual object occluded by a real object. In some embodiments, the occlusion data may be formatted as a depth filter created by the system for an application that requests an occlusion service for the occlusion data to render a virtual object in one or more locations. The depth filter may identify locations where the application should not render image information for a virtual object because the virtual object in those locations would be occluded by a surface in the physical world. It should be understood that the "occlusion data" may be in any suitable format to provide information about surfaces in the physical world and need not be used for occlusion processing. In some embodiments, the occlusion data may be used in any application that performs processing based on a representation of surfaces in a physical world scene.

[0349] The approach as described herein provides occlusion data with lower latency and / or using fewer computing resources than traditional AR and MR systems in which applications use mesh data to perform occlusion processing. Mesh data may be obtained by processing geometric data extracted by an image sensor multiple times or with cost-intensive steps including application of marching cubes algorithms, mesh simplification, and triangle count limits. Mesh data may take hundreds of milliseconds to seconds to compute, and the delay in having an up-to-date mesh may lead to visible artifacts when the environment is dynamically changing and the application renders a scene using an outdated mesh. These artifacts become evident, for example, when virtual content appears superimposed on top of real objects when the virtual content should be rendered behind the real objects, which destroys the perception / immersion for users of such applications and provides the user with incorrect cues for 3D depth perception.

[0350] In order for an application that uses a mesh for occlusion processing to have the latest mesh, the application must either continually query for the mesh (resulting in significant continuous processing) or utilize a mechanism to determine whether there are changes and then query for a new mesh (which reduces overall processing but would still have a long latency between changes in the physical world and the time a mesh reflecting those changes arrives at the application).

[0351] By utilizing low-level data (e.g., point clouds) and live depth data directly in the 3D reconstruction data instead of meshes for occlusion, it is possible to reduce the latency between changes occurring in the environment and being reflected in the occlusion data, maintaining closer synchronization with the physical world and therefore higher perceived visual quality.

[0352] In some embodiments, a real-time depth map of the physical environment may be obtained from a depth sensor (e.g., a depth camera). Each pixel in the depth map may correspond to a discrete distance measurement captured from a 3D point in the environment. In some embodiments, these depth cameras may provide a depth map that includes a set of points at a real-time rate. However, the depth map may have holes that may result from the depth camera being unable to obtain sensor data representing an area, or obtaining incorrect or unreliable data representing an area. In some embodiments, if the depth sensor uses infrared (IR) light, the holes may result, for example, from materials or structures in the physical environment that do not reflect IR light very well or at all. In some embodiments, the holes may result, for example, from very thin structures or surfaces at an oblique incidence angle that do not reflect light toward the depth sensor. Depth sensors may also suffer from motion blur when moving at high speeds, which may also lead to missing data. Furthermore, "holes" in a depth map represent regions of the depth map that are not suitable for use in occlusion processing for any other reason. Such holes may be detected using any suitable process, such as processing the depth map to detect lack of connectivity between points or regions in the depth map. As another example, holes may be detected using a process that calculates a quality metric for regions of the depth map and treats regions with low quality metrics as holes. One such metric may be the image-to-image variability for pixels in the depth map that represent the same location in the physical world. Pixels with such variability that exceed a threshold may be classified as holes. In some embodiments, holes may be identified by pixels that meet a predetermined statistical criterion for a cluster of pixels whose quality metric is below a threshold.

[0353] In some embodiments, the depth map may first be "filtered" to identify holes. Rays from viewpoints from which the scene will be rendered into the holes may then be determined. Those rays may be "cast" into a 3D representation of the physical world, created using more than just a depth sensor, to identify data representing the area of ​​the hole. The 3D representation of the physical world may be, for example, a 3D reconstruction created from data from a stereoscopic camera. Data from the 3D reconstruction identified by such ray casting may be added to the depth map, thereby filling the hole.

[0354] The 3D reconstruction may be calculated from the image sensor data once holes are identified. Alternatively, some or all of the 3D reconstruction may be pre-calculated and stored in memory. The 3D reconstruction may be maintained by a process, e.g., in computer memory, independent of any process that generates depth information for occlusion processing, which may access the stored 3D reconstruction as needed. As a further alternative, the 3D reconstruction may be maintained in memory, but portions of it may be updated in response to indications calculated based on the depth information that there is a change in a portion of the physical world corresponding to a portion of the 3D reconstruction.

[0355] In an XR system, the light ray may have the same pose as the user's eye line of sight. In an exemplary system as described below, the depth map may also be obtained with the user's eye line of sight, since a depth sensor may be worn by the user and mounted on the user's head near the eyes. The visual camera used to form the 3D reconstruction data may also be worn by the user such that the images and data derived from those images may be related to a coordinate system that allows the light ray defined for the depth map to be cast into the 3D reconstruction calculated from the visual images. Similarly, an inertial measurement unit and / or other sensor worn by the user and / or associated with the sensor may provide data regardless of the pose of the visual camera, perform coordinate transformations, add data to the 3D representation, and relate the light ray defined for the depth map to the 3D reconstruction.

[0356] In some embodiments, the user's focus or related virtual content placement information may guide the ray casting and adapt it in the image space by casting denser rays at depth-discontinued locations to obtain high quality occlusion at object boundaries and sparse rays at the center of objects in the view to reduce processing requirements. Ray casting may additionally provide local 3D surface information such as normals and positions, which, with the depth information, may be used to enable improved time warping processing and mitigate missing visible pixels that would otherwise need to be rendered or ray traced in a typical rendering engine. Time warping is a technique in XR that modifies the rendered image before sending it to the display to compensate for head movements that are calculated to occur between rendering and display. In some embodiments, time warping may be used to synchronize data from the depth map with a representation of the 3D physical world, which may be used to generate data to fill holes in the depth map. Data from both data sources may be time warped to represent the calculated pose when displayed. In some embodiments, data from the 3D representation may be time warped to represent the calculated pose at the time the data was captured using a depth map.

[0357] In some embodiments, advanced features such as time warping may utilize 3D local surface information from ray casting. When a content frame is rendered without physical world occlusion or with eroded depth imaging, time warping may fill in all missing visible pixels that were previously occluded. Thus, the rendering engine may not need to fill in pixels, allowing for less strict decoupled rendering applications (or more independent time warping).

[0358] Processing as described above may be performed in any suitable form of hardware processor on data acquired using any suitable sensor and presented on any suitable interface. An example of a suitable system including sensors, processing, and user interfaces is presented below. In the illustrated embodiment, a "service" may be implemented as part of the XR system using computer executable instructions. Execution of those instructions may control one or more processors to access sensor data, which in turn generates and supplies depth information to applications running on the XR system. Those instructions may be executed on the same processor or device that executes the application that presents the XR scene to the user, or on a remote device accessed by the user device via a computer network.

[0359] FIG. 49 illustrates an occlusion rendering system 4900, according to some embodiments. The occlusion rendering system 4900 may include a reconstruction filter 4902. The reconstruction filter 4902 may receive depth information 4904. In some embodiments, the depth information 4904 may be a sequence of depth images captured by a depth camera. In some embodiments, the depth information 4904 may be derived from a sequence of images captured by a visual camera, for example, using structure from motion based on a single camera and / or using stereo computation based on two cameras. FIG. 50 illustrates a depth image 5000, according to some embodiments. In some embodiments, surface information may be generated from the depth information. The surface information may indicate distances to physical objects within a field of view (FOV) of a head mounted display device that includes a depth camera and / or a visual camera. The surface information may be updated in real time as the scene and FOV change.

[0360] A second source of depth information is illustrated as a 3D reconstruction 4908. The 3D reconstruction 4908 may include a representation of the 3D physical world. The representation of the 3D physical world may be created and / or maintained in a computer memory. In some embodiments, the 3D representation may be generated from images captured by a visual camera, for example, using a composition from motion based on a single camera and / or using a stereo calculation based on two cameras. In some embodiments, the 3D representation may be generated from depth images captured by a depth camera. For example, the 3D reconstruction 4908 may be created and / or updated using the depth information 4904 in combination with the pose of the depth camera relative to the world origin. The representation may be built and modified over time, for example, as a user wearing the camera looks around the physical world. In some embodiments, the depth information 4904 may also be used to generate a 3D representation of the physical world. The 3D reconstruction 4908 may be a stereo reconstruction, including 3D voxels. In some embodiments, each 3D voxel may represent a cube of space (e.g., 0.5 meters by 0.5 meters by 0.5 meters), and each 3D voxel may contain data relating to and / or describing surfaces in the real world within that cube of space.

[0361] The 3D reconstruction 4908 of the world may be stored in any suitable manner. In some embodiments, the 3D reconstruction 4908 may be stored as a "cloud" of points that represent features of objects in the physical world. In some embodiments, the 3D reconstruction 408 may be stored as a mesh, where groups of points define vertices of triangles that represent a surface. In some embodiments, the 3D reconstruction 4908 may be generated using a room layout detection system and / or other techniques, such as object detection. In some embodiments, several techniques may be used together to generate the 3D reconstruction 4908. For example, object detection may be used for known physical objects in the physical world, 3D modeling may be used for unknown physical objects in the physical world, and a room layout detection system may also be used to identify boundaries in the physical world, such as walls and floors.

[0362] The reconstruction filter 4902 may include computer-executable instructions for producing a depth map based on the depth information 4904. The depth map may include one or more pixels. Each pixel may indicate a distance to a surface point in the physical world. In some embodiments, the reconstruction filter 4902 may combine the depth information 4904 and data from the ray casting engine 4906. In some embodiments, the reconstruction filter 4902 may reduce or remove noise from the depth information 4904 and / or from the combination of the depth information 4904 and data from the ray casting engine 4906, based at least in part on the data from the ray casting engine 4902. In some embodiments, the reconstruction filter 4902 may up-amplify the depth information 4904 using deep learning techniques.

[0363] The reconstruction filter 4902 may identify regions of the depth map based on a quality metric. For example, when a pixel's quality metric exceeds a threshold, the pixel may be determined to be incorrect or noisy. Regions of the depth map that contain incorrect or noisy pixels may be referred to as holes (e.g., holes 5002).

[0364] 54A and 54B provide an alternative example of how holes can be introduced into a depth map in an embodiment where the depth map is constructed from multiple depth images. FIG. 54A is a sketch of a region imaged from a first viewpoint using a depth camera to identify voxels occupied by a surface and empty voxels. FIG. 54B is a sketch of a region imaged from multiple viewpoints using a depth camera to identify voxels occupied by a surface and empty voxels. FIG. 54B illustrates multiple voxels determined to be occupied by a surface or empty by fusing data from multiple camera images. However, voxels in region 5420 are not imaged. Region 5420 may be imaged using a depth camera at location 5422, but the camera has not moved to that location. Thus, region 5420 is an observation space where no stereoscopic information is available. The AR system may guide a user wearing it to scan the observation space.

[0365] Referring back to FIG. 49, the reconstruction filter 4902 may inform the ray casting engine 4906 about the location of the hole. The ray casting engine 4906 may generate a view of the physical world given the user's pose and may remove the hole from the depth map. The data may represent the portion of the user's current view of the physical world at the current time, or where time warping is used at the time by which the data should be time warped. The ray casting engine 4906 may generate one or more 2D images, for example, one image per eye. In some embodiments, the reconstruction filter 4902 may remove regions of the depth map that are spaced away from the location of the virtual object beyond a threshold distance, as these regions may not be relevant for occlusion testing for the virtual object.

[0366] The ray casting engine 4906, which generates a view of the physical world given the user's pose, may be implemented by any suitable technique. In some embodiments, the ray casting engine 4906 may implement a ray casting algorithm on the 3D reconstruction 4908 to extract data from it. The ray casting algorithm may take the user's pose as input. The ray casting engine 4906 may cast rays from the virtual camera to the 3D reconstruction 4908 of the physical world to obtain missing surface information (e.g., holes) from the depth map. The ray casting engine 4906 may cast dense rays to the boundaries of physical objects in the physical world to obtain high quality occlusion at the object boundaries and sparse rays to the central regions of the physical objects to reduce processing requirements. The ray casting engine 4906 may then provide the ray casting point cloud to the reconstruction filter 4902. The ray casting engine 4906 is illustrated as an example. In some embodiments, the ray casting engine 4906 may be a meshing engine. The meshing engine may implement a meshing algorithm on the 3D reconstruction 4908 to extract data from it, including, for example, triangles and triangle connectivity. The meshing algorithm may take the user's pose as input.

[0367] The reconstruction filter 4902 may combine data from the depth information 4904 and the ray casting engine 4906 and compensate for holes in the depth map from the depth information 4904 with data from the ray casting point cloud from the ray casting engine 4906. In some embodiments, the resolution of the depth map may be improved. The method may be used to generate a high resolution depth image from a sparse or low resolution depth image.

[0368] The reconstruction filter 4902 may provide the updated depth map to an occlusion service 4910. The occlusion service 4910 may calculate occlusion data based on the updated depth map and information about the location of virtual objects in the scene. The occlusion data may be a depth buffer of surfaces in the physical world. The depth buffer may store the depth of pixels. In some embodiments, the occlusion service 4910 may interface with an application 4912. In some embodiments, the occlusion service 4910 may interface with a graphics system. In these embodiments, the graphics system may expose a depth buffer to the application 4912, and the depth buffer is pre-filled with the occlusion data.

[0369] The occlusion service 4910 may provide the occlusion data to one or more applications 4912. In some embodiments, the occlusion data may correspond to a user's pose. In some embodiments, the occlusion data may be a per-pixel representation. In some embodiments, the occlusion data may be a mesh representation. The applications 4912 may be configured to execute computer-executable instructions to render virtual objects in a scene based on the occlusion data. In some embodiments, the occlusion rendering may be performed by a separate graphics system instead of the applications 4912. The separate graphics system may use time warping techniques.

[0370] In some embodiments, the reconstruction filter 4902, the ray casting engine 4906, and the occlusion service 4910 may be remote services, e.g., remote processing module 72, the 3D reconstruction 4908 may be stored in a remote memory, e.g., remote data repository 74, and the application 4912 may be on the AR display system 80.

[0371] FIG. 51 illustrates a method 5100 of occlusion rendering in an augmented reality (AR) environment, according to some embodiments. In act 5102, depth information may be captured, for example, from a depth sensor (e.g., depth sensor 51) on a head mounted display device. The depth information may indicate a distance between the head mounted display device and a physical object. In act 5104, surface information may be generated from the depth information. The surface information may indicate a distance to a physical object within the head mounted display device and / or a field of view (FOV) of a user of the head mounted display device. The surface information may be updated in real time as the scene and FOV change. In act 5106, a portion of a virtual object to render may be calculated from the surface information and information about the location of the virtual object within the scene.

[0372] FIG. 52 illustrates details of act 5104, according to some embodiments. In act 5202, depth information may be filtered to generate a depth map. The depth map may include one or more pixels. Each pixel may indicate a distance to a point of the physical object. In act 5204, low-level data of a 3D reconstruction of the physical object may be selectively obtained, for example from 3D reconstruction 4908. In act 5206, surface information may be generated based on the depth map and the selectively obtained low-level data of the 3D reconstruction of the physical object.

[0373] FIG. 53 illustrates details of act 5202, according to some embodiments. In act 5302, a quality metric for a region of the depth map may be determined. The quality metric may indicate whether the region of the depth map is incorrect or noisy. In act 5304, holes in the depth map may be identified based on the quality metric, for example, by comparison to a threshold. In act 5306, the identified holes may be removed from the depth map.

[0374] conclusion

[0375] While several aspects of several embodiments have been described above, it is to be appreciated that various alterations, modifications, and improvements will readily occur to those skilled in the art.

[0376] As an example, the embodiments are described in relation to an augmented (AR) environment, although it should be understood that some or all of the techniques described herein may be applied in MR environments, or more generally, in other XR and VR environments.

[0377] As another example, embodiments are described in connection with devices, such as wearable devices. It should be understood that some or all of the techniques described herein may be implemented via a network (such as the cloud), a discrete application, and / or any suitable combination of devices, networks, and discrete applications.

[0378] Such alterations, modifications, and improvements are intended to be part of this disclosure and are intended to be within the spirit and scope of this disclosure. Additionally, while advantages of the disclosure are set forth, it should be understood that not all embodiments of the disclosure include all described advantages. Some embodiments may not implement any feature described herein as advantageous, and in some instances. Thus, the foregoing description and drawings are by way of example only.

[0379] The foregoing embodiments of the present disclosure can be implemented in any of a number of ways. For example, the embodiments may be implemented using hardware, software, or a combination thereof. When implemented in software, the software code can be executed on any suitable processor or collection of processors, whether provided in a single computer or distributed among multiple computers. Such a processor may be implemented as an integrated circuit, with one or more processors in an integrated circuit component, including commercially available integrated circuit components known in the art, such as a CPU chip, a GPU chip, a microprocessor, a microcontroller, or a coprocessor, to name a few. In some embodiments, the processor may be implemented in a custom circuit, such as an ASIC, or in a semi-custom circuit resulting from configuring a programmable logic device. As a further alternative, the processor may be part of a larger circuit or semiconductor device, whether commercially available, semi-custom, or custom. As a specific example, some commercially available microprocessors have multiple cores, such that one or a subset of those cores may constitute a processor. However, the processor may be implemented using a circuit in any suitable format.

[0380] Further, it should be understood that a computer may be embodied in any of a number of forms, such as a rack-mounted computer, a desktop computer, a laptop computer, or a tablet computer, etc. In addition, a computer may be embodied in devices not generally considered computers but with suitable processing capabilities, including a personal digital assistant (PDA), a smart phone, or any suitable portable or fixed electronic device.

[0381] A computer may also have one or more input and output devices. These devices can be used, among other things, to present a user interface. Examples of output devices that may be used to provide a user interface include a printer or display screen for visual presentation of output, or a speaker or other sound generating device for audible presentation of output. Examples of input devices that may be used for a user interface include keyboards and pointing devices such as mice, touchpads, and digitizing tablets. As another example, a computer may receive input information through speech recognition or in other audible formats. In the illustrated embodiment, the input / output devices are illustrated as physically separate from the computing device. However, in some embodiments, the input and / or output devices may be physically integrated into the same unit as the processor or other elements of the computing device. For example, the keyboard may be implemented as a soft keyboard on a touch screen. In some embodiments, the input / output devices may be completely disconnected from the computing device and functionally integrated through a wireless connection.

[0382] Such computers may be interconnected by one or more networks in any suitable form, including as a local area network or a wide area network, such as an enterprise network or the Internet. Such networks may be based on any suitable technology and operate according to any suitable protocol, and may include wireless networks, wired networks, or fiber optic networks.

[0383] Also, the various methods and processes outlined herein may be coded as software that is executable on one or more processors employing any one of a variety of operating systems or platforms. In addition, such software may be written using any of a number of suitable programming languages ​​and / or programming or scripting tools, and may be compiled as executable machine language code or intermediate code that runs on a framework or virtual machine.

[0384] In this regard, the present disclosure may be embodied as a computer-readable storage medium (or multiple computer-readable media) (e.g., a computer memory, one or more floppy disks, compact disks (CDs), optical disks, digital video disks (DVDs), magnetic tapes, flash memories, circuitry in field programmable gate arrays or other semiconductor devices, or other tangible computer storage media) encoded with one or more programs that, when executed on one or more computers or other processors, perform methods implementing various embodiments of the present disclosure discussed above. As is evident from the foregoing examples, a computer-readable storage medium may retain information for a sufficient time to provide computer-executable instructions in a non-transitory form. Such a computer-readable storage medium or media may be transportable such that one or more programs stored thereon may be loaded onto one or more different computers or other processors to implement various aspects of the present disclosure, as described above. As used herein, the term "computer-readable storage medium" encompasses only computer-readable media that may be considered a manufacture (i.e., an article of manufacture) or machine. In some embodiments, the present disclosure may be embodied as a computer-readable medium other than a computer-readable storage medium, such as a propagated signal.

[0385] The term "program" or "software" is used herein in a general sense to refer to any type of computer code or set of computer executable instructions that can be employed to program a computer or other processor to implement various aspects of the present disclosure, as described above. In addition, according to one aspect of the present embodiment, it should be understood that one or more computer programs that, when executed, perform the methods of the present disclosure need not reside on a single computer or processor, but can be distributed in a modular manner among several different computers or processors to implement various aspects of the present disclosure.

[0386] Computer-executable instructions may be in many forms, such as program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Typically the functionality of the program modules may be combined or distributed as desired in various embodiments.

[0387] Also, the data structures may be stored in the computer-readable medium in any suitable form. For ease of illustration, the data structures may be shown to have fields that are related through locations within the data structure. Such relationships may be achieved by allocating storage for the fields with locations in the computer-readable medium that similarly convey the relationship between the fields. However, any suitable mechanism may be used to establish relationships between information in the fields of the data structures, including through the use of pointers, tags, or other mechanisms that establish relationships between data elements.

[0388] Various aspects of the disclosure may be used alone, in combination, or in various arrangements not specifically discussed in the foregoing embodiments, and therefore, its application is not limited to the details and arrangements of components set forth in the foregoing description or illustrated in the drawings. For example, aspects described in one embodiment may be combined in any manner with aspects described in other embodiments.

[0389] The present disclosure may also be embodied as a method, examples of which are provided. The acts performed as part of the method may be ordered in any suitable manner. Thus, while the illustrative embodiments are shown as sequential acts, embodiments may be constructed in which acts are performed in an order different from that shown, which may include performing some acts simultaneously.

[0390] The use of ordinal terms such as "first," "second," "third," etc. in the claims to modify claim elements does not, by itself, imply any priority, precedence, or ordering of one claim element relative to another element, or the temporal order in which acts of a method are performed, but rather the ordinal terms are used solely as markers to distinguish one claim element having a certain name from another element having the same name (due to the use of the ordinal terms) to distinguish between claim elements.

[0391] Also, the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of "including," "comprising," "having," "containing," "with," and variations thereof herein is meant to encompass the items listed thereafter and equivalents and additional items.

Claims

1. 1. A portable electronic system comprising: a sensor configured to capture three-dimensional (3D) information about objects in a physical world; and An active memory; A local memory; a transceiver configured for communication with a remote memory over a computer network; a processor, the processor being communicatively coupled to the sensor, the active memory, the local memory, and the transceiver, and configured to execute computer-executable instructions to provide a 3D representation of a portion of the physical world based at least in part on 3D information about objects in the physical world; Equipped with the 3D representation of the portion of the physical world comprises a plurality of blocks, each block having a value representing an object within an area of ​​the portion of the physical world at a point in time; The computer executable instructions include: persisting the plurality of blocks in at least one of the active memory, the local memory, and the remote memory when the portable electronic system changes to an inactive operating status at a first time; responsive to a transition to an active operating status at a second time after the first time, selecting from the persisted plurality of blocks blocks that are in a vicinity of the portable electronic system at the second time for rendering virtual content to a user of the portable electronic system; 16. A portable electronic system comprising instructions for performing

2. A portable electronic system, comprising: a sensor configured to capture three-dimensional (3D) information about objects in a physical world; and An active memory; A local memory; a transceiver configured for communication with a remote memory over a computer network; a processor, the processor being communicatively coupled to the sensor, the active memory, the local memory, and the transceiver, and configured to execute computer-executable instructions to provide a 3D representation of a portion of the physical world based at least in part on 3D information about objects in the physical world; Equipped with the 3D representation of the portion of the physical world comprises a plurality of blocks, each block having a value representing an object within an area of ​​the portion of the physical world at a point in time; The computer executable instructions include: persisting the plurality of blocks in at least one of the active memory, the local memory, and the remote memory when the portable electronic system changes to an inactive operating status; calculating one or more new blocks based at least in part on incoming 3D information about objects in the physical world; selecting from the persisted blocks surrounding the portable electronic system blocks that at least partially overlap with the one or more new blocks such that the persisted blocks can be read; 16. A portable electronic system comprising instructions for performing

3. the portion of the physical world is within a field of view of a user of the portable electronic system; 3. The portable electronic system of claim 1 or claim 2, wherein each block comprises a volumetric representation of said portion of an area of ​​said physical world at said point in time.

4. the inactive operating status is shut down or sleep; Persisting the plurality of blocks in at least one of the active memory, the local memory, and the remote memory includes: and / or when the inactive operational status is sleep, persisting the plurality of blocks in a separate memory in which the plurality of blocks currently resides such that the plurality of blocks are accessible by the portable electronic system upon waking up of the portable electronic system; When the inactive operational status is shutdown, persisting at least a portion of blocks in the local memory in the remote memory and persisting blocks in the active memory in the local memory and / or the remote memory.

3. The portable electronic system of claim 1 or 2, comprising:

5. 3. The portable electronic system of claim 2, wherein selecting from the persisted blocks those blocks that at least partially overlap with the one or more new blocks comprises comparing the one or more new blocks to the persisted blocks.

6. Comparing the one or more new blocks with the persisted block comprises: Comparing the one or more new blocks with blocks persisted in the active memory.

6. The portable electronic system of claim 5, comprising:

7. Comparing the one or more new blocks with the persisted block comprises: comparing the one or more new blocks with blocks persisted in the local memory when the active memory does not have blocks that at least partially overlap with the one or more new blocks.

7. The portable electronic system of claim 6, comprising:

8. Comparing the one or more new blocks with the persisted block comprises: comparing the one or more new blocks with blocks persisted in the remote memory when the local memory does not have blocks that at least partially overlap with the one or more new blocks; 8. The portable electronic system of claim 7, comprising:

9. The computer executable instructions include: updating said plurality of blocks with said one or more new blocks; The portable electronic system of claim 2 comprising instructions for:

10. 1. A method of operating a portable electronic system in an augmented reality system, the method comprising: Using at least one processor, Obtaining a plurality of blocks comprising three-dimensional (3D) information about a surface in the physical world; storing at least one of the plurality of blocks in an active memory; using blocks from the active memory to render virtual content to a user of the portable electronic system; persisting the block in active memory in response to a transition to an inactive operating status at a first time; responsive to a transition to an active operating status at a second time after the first time, selecting from the persisted blocks blocks that are in a surrounding area of ​​the portable electronic system at the second time for rendering virtual content to a user of the portable electronic system; The method of claim 1,

11. 1. A method of operating a portable electronic system in an augmented reality system, the method comprising: Using at least one processor, obtaining a plurality of blocks comprising a three-dimensional (3D) representation of a surface in a physical world; storing said plurality of blocks in active memory; using blocks from the active memory to render virtual content to a user of the portable electronic system; selectively persisting the block in the active memory in response to a transition to an inactive operating status at a first time; calculating one or more new blocks based at least in part on incoming 3D information about objects in the physical world; responsive to a transition to an active operating status at a second time after the first time, selecting from the persisted blocks surrounding the portable electronic system blocks that at least partially overlap with the one or more new blocks such that the persisted blocks may be read; The method of claim 1,

12. The method of claim 10 or claim 11, wherein each block comprises a volumetric representation of a portion of the physical world at a point in time.

13. The method of claim 10 or claim 11, wherein the blocks correspond to the same volume of space.

14. The method according to claim 10 or claim 11, wherein the blocks are mesh blocks.

15. paging blocks between the active memory and a local memory based on a current or projected field of view of a user of the portable electronic system; and / or and persisting the block in the local memory in response to a transition to an inactive operating status. The method of claim 10 or claim 11, comprising:

16. Using blocks from the active memory to render virtual content to the user includes: using said block for visual occlusion processing; Calculating physics-based interactions and / or environmental inferences; The method of claim 10 or claim 11, comprising:

17. A non-transitory computer readable medium having instructions stored thereon that, when executed on a processor, perform: Obtaining a plurality of blocks comprising three-dimensional (3D) information about a surface in the physical world; storing at least one of the plurality of blocks in an active memory; using the blocks from the active memory to render virtual content to a user of the portable electronic system; persisting the block in active memory in response to a transition to an inactive operating status at a first time; responsive to a transition to an active operating status at a second time after the first time, selecting from the persisted blocks blocks that are in a surrounding area of ​​the portable electronic system at the second time for rendering virtual content to a user of the portable electronic system; A non-transitory computer readable medium performing acts including:

18. A non-transitory computer readable medium having instructions stored thereon that, when executed on a processor, perform: obtaining a plurality of blocks comprising a three-dimensional (3D) representation of a surface in a physical world; storing said plurality of blocks in active memory; using the blocks from the active memory to render virtual content to a user of the portable electronic system; selectively transferring the blocks in the active memory to a non-volatile memory in response to a transition to an inactive operational status at a first time; calculating one or more new blocks based at least in part on incoming 3D information about the physical world; and responsive to a transition to an active operating status at a second time after the first time, selecting from the transferred blocks blocks that at least partially overlap with the one or more new blocks such that the transferred blocks may be read from around the portable electronic system; A non-transitory computer readable medium performing acts including:

19. 19. The non-transitory computer readable medium of claim 17 or claim 18, wherein each block comprises a volumetric representation of a portion of the physical world at a point in time.

20. 19. The non-transitory computer readable medium of claim 17 or claim 18, wherein the multiple blocks correspond to the same volume of space.

21. 19. The non-transitory computer readable medium of claim 17 or claim 18, wherein the block is a mesh block.

22. The said act is paging blocks between the active memory and a local memory based on a current or projected field of view of a user of the portable electronic system; and / or and persisting the block in the local memory in response to a transition to an inactive operating status.

19. The non-transitory computer readable medium of claim 17 or claim 18, comprising:

23. Using blocks from the active memory to render virtual content to the user includes: using said block for visual occlusion processing; Calculating physics-based interactions and / or environmental inferences; 19. The non-transitory computer readable medium of claim 17 or claim 18, comprising:

Images