Occlusion rendering method with ray casting and live depth

Abstract

To provide a more immersive user experience method and augmented reality / mixed reality system capable of achieving an increased rate of update for occlusion data by using depth sensor data that has been augmented with lower level reconstruction data.SOLUTION: In an AR system using a world reconstruction system, when a perception module operates in real-time dynamic environments, changes in the physical world are reflected quickly in the occlusion data. Occlusion rendering using live depth data augmented with lower-level 3D reconstruction data, such as a ray casting point cloud, greatly reduces the latency for visual occlusion processing. Generating occlusion data in this way may provide faster operation of an XR system using less computing resources and enable the system to be packaged in a battery-operated wearable device.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to and the benefit of U.S. Provisional Patent Application No. 62 / 610,000, filed Dec. 22, 2017, entitled "VIEWPOINT DEPENDENT BRICK SELECTION FOR FAST VOLUMETRIC RECONSTRUCTION," which is incorporated herein by reference in its entirety. This application also claims priority to and the benefit of U.S. Provisional Patent Application No. 62 / 609,990, filed Dec. 22, 2017, entitled "MULTI-STAGE BLOCK MESH SIMPLIFICATION FOR MULTIPLE TARGETS," which is incorporated herein by reference in its entirety. This application also claims priority to and the benefit of U.S. Provisional Patent Application No. 62 / 609,990, filed Dec. 22, 2017, entitled "CACHING AND UPDATING OF DENSE 3D This application claims priority to and the benefit of U.S. Provisional Patent Application No. 62 / 609,962, entitled "METHOD OF OCCLUSION RENDERING USING RAYCAST AND LIVE DEPTH," filed Dec. 22, 2017, which is incorporated herein by reference in its entirety. This application also claims priority to and the benefit of U.S. Provisional Patent Application No. 62 / 609,979, entitled "METHOD OF OCCLUSION RENDERING USING RAYCAST AND LIVE DEPTH," filed Jul. 24, 2018, which is incorporated herein by reference in its entirety. This application also claims priority to and the benefit of U.S. Provisional Patent Application No. 62 / 702,740, filed July 24, 2018, entitled "MULTI-STAGE BLOCK MESH SIMPLIFICATION," which is incorporated herein by reference in its entirety. This application also claims priority to and the benefit of U.S. Provisional Patent Application No. 62 / 702,749, filed July 24, 2018, entitled "CACHING AND UPDATING OF DENSE 3D RECONSTRUCTION DATA," which is incorporated herein by reference in its entirety. This application also claims priority to and the benefit of U.S. Provisional Patent Application No. 62 / 702,736, filed July 24, 2018, entitled "METHOD OF OCCLUSION RENDERING USING RAYCAST AND LIVE DEPTH," which is incorporated herein by reference in its entirety.

[0002] This application relates generally to cross-reality systems that use 3D world reconstruction to render scenes. [Background technology]

[0003] A computer may control a human user interface and create an X-reality (XR or cross-reality) environment in which some or all of the XR environment may be generated by the computer as it is perceived by the user. These XR environments may be virtual reality (VR), augmented reality (AR), and mixed reality (MR) environments in which some or all of the XR environment may be generated, in part, by the computer using data that describes the environment. This data may, for example, describe virtual objects that may be rendered such that a user can sense or perceive and interact with the virtual objects as part of the physical world. The user may use a user interface, such as a head-mounted display device, to interact with the virtual objects. These virtual objects may be experienced as a result of data being rendered and presented through a user-facing device. The data may be displayed for the user to see, or may control audio that is played for the user to hear, or may control a tactile (or haptic) interface, allowing the user to experience touch sensations that the user senses or perceives as they feel the virtual objects.

[0004] XR systems can be useful for many applications ranging from scientific visualization, medical training, engineering design and prototyping, remote manipulation and telepresence, and personal entertainment. AR and MR, in contrast to VR, involve one or more objects in association with real objects in the physical world. The experience of a virtual object interacting with a real object generally enhances the user's enjoyment when using an XR system and also opens up possibilities for a variety of applications that present realistic and easily understandable information about how the physical world can be altered. Summary of the Invention [Means for solving the problem]

[0005] Aspects of the present application relate to methods and apparatus for occlusion rendering. These techniques may be used together, separately, or in any suitable combination.

[0006] Some embodiments relate to a method of operating a computing system to render a virtual object in a scene comprising physical objects, the method including capturing depth information from a sensor worn by a user, the depth information indicative of a distance between the user and the physical object, generating surface information from the depth information, the surface information indicative of a distance to the physical object in a field of view of the user, the generating including updating the surface information in real time as the scene and field of view change, and calculating a portion of the virtual object to render from the surface information and information about the location of the virtual object in the scene.

[0007] In some embodiments, the depth information is a sequence of depth images at a frame rate of at least 30 frames / second.

[0008] In some embodiments, generating the surface information from the depth information includes generating the surface information from stereo depth information from a stereo camera, the stereo depth information corresponding to a region of the physical object selected based on the depth information.

[0009] In some embodiments, generating the surface information from the depth information includes generating the surface information from low-level data of a 3D reconstruction of the physical object.

[0010] In some embodiments, the low-level data of the 3D reconstruction of the physical object is in the format of a point cloud.

[0011] In some embodiments, generating the surface information from the depth information includes generating the surface information from low level data of a 3D representation of a scene comprising physical objects.

[0012] In some embodiments, a computing system includes a remote service and an application, where the surface information is generated by the remote service, and the application calculates a portion of a virtual object to render based on the surface information received from the remote service.

[0013] In some embodiments, capturing the depth information includes producing a depth map based at least in part on the depth information, the depth map comprising a plurality of pixels, each pixel indicating a distance to a point of the physical object.

[0014] In some embodiments, generating the surface information includes filtering the depth information to generate a depth map, where the depth map comprises a plurality of pixels, each pixel indicating a distance to a point of the physical object; selectively obtaining low-level data of a 3D reconstruction of the physical object; and generating the surface information based on the depth map and the selectively obtained low-level data of the 3D reconstruction of the physical object.

[0015] In some embodiments, filtering the depth information includes determining a quality metric for a region of the depth map, identifying holes in the depth map based on the quality metric, and removing the holes from the depth map.

[0016] In some embodiments, generating the surface information includes filling regions in the depth map that correspond to holes with at least a portion of low-level data of a 3D reconstruction of the physical object.

[0017] In some embodiments, the holes indicate missing or noisy pixels in the depth map.

[0018] In some embodiments, the act of removing holes includes removing regions in the depth map that are spaced away from the location of the virtual object beyond a threshold distance.

[0019] In some embodiments, selectively obtaining low-level data of the 3D reconstruction of the physical object is based on holes.

[0020] In some embodiments, the act of selectively obtaining low-level data of the 3D reconstruction of the physical object includes casting a ray from the virtual camera to the 3D reconstruction of the physical object.

[0021] In some embodiments, casting the rays includes casting dense rays to boundary regions of the 3D reconstruction of the physical object and sparse rays to a central region of the 3D reconstruction of the physical object.

[0022] In some embodiments, calculating the portion of the virtual object to render includes simulating an interaction between the virtual object and a physical object based on the surface information.

[0023] Some embodiments relate to an electronic system portable by a user. The electronic system comprises a sensor configured to capture depth information for one or more physical objects in a scene, and an application configured to execute computer-executable instructions to render a virtual object in the scene. The depth information indicates a distance between a user and the one or more physical objects. The depth information is transmitted to a remote service. The application receives from the remote service a depth buffer of surfaces of the one or more physical objects in the scene, where portions of the virtual object are occluded by the surfaces.

[0024] In some embodiments, the surface depth buffer is generated by a remote service based on the depth information and low-level data of the 3D reconstruction of the scene.

[0025] In some embodiments, the low-level data of the 3D reconstruction of the scene is generated by casting light rays from a virtual camera into a portion of the 3D reconstruction of the scene, the virtual camera being co-located with the user's eyes, the portion of the 3D reconstruction of the scene being selected based on depth information captured by a sensor.

[0026] In some embodiments, the depth information comprises a depth image having a plurality of pixels, some of which correspond to incorrect and noisy pixels of the depth image.

[0027] The foregoing summary is provided by way of illustration and is not intended to be limiting. The present invention provides, for example, the following: (Item 1) 1. A method of operating a computing system to render a virtual object in a scene comprising physical objects, the method comprising: capturing depth information from a sensor worn by a user, the depth information indicating a distance between the user and the physical object; generating surface information from the depth information, the surface information indicating distances to the physical objects in the user's field of view, the generating including updating the surface information in real time as the scene and field of view change; calculating a portion of the virtual object to render from the surface information and information about the location of the virtual object within the scene; and A method comprising: (Item 2) 2. The method of claim 1, wherein the depth information is a sequence of depth images at a frame rate of at least 30 frames per second. (Item 3) 2. The method of claim 1, wherein generating the surface information from the depth information includes generating the surface information from stereo depth information from a stereo camera, the stereo depth information corresponding to a region of the physical object selected based on the depth information. (Item 4) 2. The method of claim 1, wherein generating the surface information from the depth information comprises generating the surface information from low-level data of a 3D reconstruction of the physical object. (Item 5) 5. The method according to claim 4, wherein the low-level data of the 3D reconstruction of the physical object is in a point cloud format. (Item 6) 2. The method of claim 1, wherein generating the surface information from the depth information includes generating the surface information from low-level data of a 3D representation of the scene comprising the physical objects. (Item 7) the computing system includes a remote service and an application; the surface information is generated by the remote service; 2. The method of claim 1, wherein the application calculates the portion of the virtual object to render based on surface information received from the remote service. (Item 8) Capturing the depth information includes: producing a depth map based at least in part on the depth information, the depth map comprising a plurality of pixels, each pixel indicating a distance to a point of the physical object; The method according to item 1, comprising: (Item 9) generating the surface information filtering the depth information to generate a depth map, the depth map comprising a plurality of pixels, each pixel indicating a distance to a point of the physical object; selectively obtaining low level data of a 3D reconstruction of the physical object; generating said surface information based on said depth map and selectively obtained low level data of a 3D reconstruction of said physical object; The method according to item 1, comprising: (Item 10) Filtering the depth information includes: determining a quality metric for a region of the depth map; identifying holes in the depth map based on the quality metric; removing the hole from the depth map; 10. The method according to item 9, comprising: (Item 11) generating the surface information filling regions in the depth map corresponding to holes with at least a portion of low-level data of a 3D reconstruction of the physical object. 11. The method according to item 10, comprising: (Item 12) Item 11. The method of item 10, wherein the holes indicate missing or noisy pixels in the depth map. (Item 13) Item 11. The method of item 10, wherein the act of removing holes includes removing areas in the depth map that are spaced away from a location of the virtual object beyond a threshold distance. (Item 14) 11. The method of claim 10, wherein selectively obtaining low-level data of the 3D reconstruction of the physical object is based on the holes. (Item 15) 10. The method of claim 9, wherein the act of selectively obtaining low-level data of the 3D reconstruction of the physical object includes casting a ray from a virtual camera to the 3D reconstruction of the physical object. (Item 16) Item 16. The method of item 15, wherein casting rays includes casting dense rays into boundary regions of the 3D reconstruction of the physical object and sparse rays into a central region of the 3D reconstruction of the physical object. (Item 17) 2. The method of claim 1, wherein calculating the portion of the virtual object to render includes simulating an interaction between the virtual object and the physical object based on the surface information. (Item 18) 1. A user-portable electronic system, comprising: a sensor configured to capture depth information about one or more physical objects in a scene, the depth information being indicative of a distance between the user and the one or more physical objects, the depth information being transmitted to a remote service; 1. An application configured to execute computer-executable instructions for rendering a virtual object in the scene, the application receiving from the remote service a depth buffer of surfaces of the one or more physical objects in the scene, the portions of the virtual object being occluded by the surfaces. Communication and An electronic system comprising: (Item 19) 20. The electronic system of claim 18, wherein the surface depth buffer is generated by the remote service based on the depth information and low-level data of a 3D reconstruction of the scene. 20. The electronic system of item 19, wherein low-level data of the 3D reconstruction of the scene is generated by casting light rays from a virtual camera into a portion of the 3D reconstruction of the scene, the virtual camera being positioned at the same position as the user's eyes, and the portion of the 3D reconstruction of the scene being selected based on depth information captured by the sensor. (Item 21) 21. The electronic system of claim 20, wherein the depth information comprises a depth image having a plurality of pixels, and a portion of the 3D reconstruction of the scene corresponds to incorrect and noisy pixels of the depth image. [Brief description of the drawings]

[0028] The accompanying drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component that is illustrated in various figures is represented by a like numeral. For purposes of clarity, not every component is labeled in every figure.

[0029] [Figure 1] FIG. 1 is a sketch illustrating an example of a simplified augmented reality (AR) scene, according to some embodiments.

[0030] [Diagram 2] FIG. 2 is a sketch of an example simplified AR scene illustrating an example world reconstruction use case including visual occlusion, physics-based interaction, and environmental inference, according to some embodiments.

[0031] [Diagram 3] FIG. 3 is a schematic diagram illustrating data flow within an AR system configured to provide an experience of AR content that interacts with the physical world, according to some embodiments.

[0032] [Figure 4] FIG. 4 is a schematic diagram illustrating an example of an AR display system, according to some embodiments.

[0033] [Figure 5A] FIG. 5A is a schematic diagram illustrating a user wearing an AR display system that renders AR content as the user moves through a physical world environment, according to some embodiments.

[0034] [Figure 5B] FIG. 5B is a schematic diagram illustrating a viewing optics assembly and associated components, according to some embodiments.

[0035] [Figure 6]FIG. 6 is a schematic diagram illustrating an AR system using a world reconstruction system, according to some embodiments.

[0036] [Figure 7A] FIG. 7A is a schematic diagram illustrating a 3D space discretized into voxels, according to some embodiments.

[0037] [Figure 7B] FIG. 7B is a schematic diagram illustrating a reconstruction range for a single viewpoint, according to some embodiments.

[0038] [Figure 7C] FIG. 7C is a schematic diagram illustrating the perception range versus the reconstruction range at a single position, according to some embodiments.

[0039] [Figure 8] 8A-F are schematic diagrams illustrating the reconstructing a surface in the physical world into a voxel model by an image sensor viewing the surface from multiple points and viewpoints, according to some embodiments.

[0040] [Figure 9] FIG. 9 is a schematic diagram illustrating a scene represented by bricks containing voxels, surfaces within the scene, and a depth sensor that captures the surfaces in a depth image, according to some embodiments.

[0041] [Figure 10A] FIG. 10A is a schematic diagram illustrating a 3D space represented by eight bricks.

[0042] [Figure 10B] FIG. 10B is a schematic diagram illustrating a voxel grid within the brick of FIG. 10A.

[0043] [Figure 11] FIG. 11 is a schematic diagram illustrating a stereoscopic representation hierarchy, according to some embodiments.

[0044] [Figure 12] FIG. 12 is a flowchart illustrating a method of operating a computing system to generate a 3D reconstruction of a scene, according to some embodiments.

[0045] [Figure 13] FIG. 13 is a flowchart illustrating a method for thinning a portion of a plurality of bricks for the camera frustum of the depth sensor in FIG. 12 according to some embodiments.

[0046] [Figure 14] FIG. 14 is a flow chart illustrating a method for performing the camera frustum tolerance test in FIG. 13 according to some embodiments.

[0047] [Figure 15] FIG. 15 is a flowchart illustrating a method of thinning out a portion of the first plurality of bricks for the depth image in FIG. 12 according to some embodiments.

[0048] [Figure 16] FIG. 16 is a flow chart illustrating a method for performing the first depth image acceptance test in FIG. 15 according to some embodiments.

[0049] [Figure 17] FIG. 17 is a flow chart illustrating a method for performing the second depth image acceptance test in FIG. 15 according to some embodiments.

[0050] [Figure 18] FIG. 18 shows a table used by the method to categorize all pixels within the rectangle in FIG. 17 against a minimum brick value (bmin) and a maximum brick value (bmax) according to some embodiments.

[0051] [Figure 19]19A-F are schematic diagrams illustrating thinning bricks relative to the camera frustum, according to some embodiments.

[0052] [Figure 20] 20A-B are schematic diagrams illustrating brick thinning for a depth image that includes a surface, according to some embodiments.

[0053] [Figure 21] FIG. 21 is a schematic diagram illustrating a planar extraction system, according to some embodiments.

[0054] [Figure 22] FIG. 22 is a schematic diagram illustrating a portion of the plane extraction system of FIG. 21 along with details regarding the plane extraction, in accordance with some embodiments.

[0055] [Figure 23] FIG. 23 is a schematic diagram illustrating a scene represented by bricks containing voxels and example planar data within the scene, according to some embodiments.

[0056] [Figure 24] FIG. 24 is a schematic diagram illustrating the planar data store of FIG. 21 according to some embodiments.

[0057] [Diagram 25] FIG. 25 is a schematic diagram illustrating planar geometry extraction when a planar query is sent to the planar data store of FIG. 21 in accordance with some embodiments.

[0058] [Figure 26A] FIG. 26A is a schematic diagram illustrating steps of generating the plane covering points of FIG. 25 according to some embodiments.

[0059] [Figure 26B]FIG. 26B is a schematic diagram illustrating various example planar geometric shape representations that may be extracted from an example rasterized planar mask, according to some embodiments.

[0060] [Figure 27] FIG. 27 illustrates a mesh for a scene, according to some embodiments.

[0061] [Figure 28A] FIG. 28A shows the scene of FIG. 27 represented by an outer rectangular plane, according to some embodiments.

[0062] [Figure 28B] FIG. 28B shows the scene of FIG. 27 represented by an inner rectangular plane, according to some embodiments.

[0063] [Figure 28C] FIG. 28C shows the scene of FIG. 27 represented by a polygonal plane according to some embodiments.

[0064] [Figure 29] FIG. 29 shows the scene of FIG. 27 with the mesh denoised by flattening the mesh shown in FIG. 27 in accordance with some embodiments.

[0065] [Diagram 30] FIG. 30 is a flowchart illustrating a method of generating a model of an environment represented by a mesh, according to some embodiments.

[0066] [Diagram 31] FIG. 31 is a schematic diagram illustrating a 2D representation of a portion of the physical world by four blocks according to some embodiments.

[0067] [Diagram 32] 32A-32D are schematic diagrams illustrating mesh evolution of an example mesh block during multi-stage simplification, according to some embodiments.

[0068] [Diagram 33] 33A and 33B show representations of the same environment with and without simplification through triangle reduction, respectively.

[0069] [Diagram 34] 34A and 34B show zoomed-in representations of the same environment with and without simplification through triangle reduction, respectively.

[0070] [Diagram 35] 35A and 35B show representations of the same environment with and without planarization, respectively.

[0071] [Diagram 36] 36A and 36B show representations of the same environment with and without simplification through removal of disconnected components, respectively.

[0072] [Figure 37] FIG. 37 is a schematic diagram illustrating an electronic system that enables an interactive X-Reality environment for multiple users, according to some embodiments.

[0073] [Figure 38] FIG. 38 is a schematic diagram illustrating the interaction of components of the electronic system in FIG. 37 according to some embodiments.

[0074] [Figure 39] FIG. 39 is a flow chart illustrating a method of operating the electronic system in FIG. 37 according to some embodiments.

[0075] [Diagram 40] FIG. 40 is a flowchart illustrating a method of capturing 3D information about objects in the physical world and representing the physical world as blocks of the 3D reconstruction in FIG. 39 according to some embodiments.

[0076] [Diagram 41] FIG. 41 is a flowchart illustrating a method for selecting a version of a block that represents a subset of the blocks in FIG. 39 according to some embodiments.

[0077] [Diagram 42] FIG. 42 is a flow chart illustrating a method of operating the electronic system in FIG. 37 according to some embodiments.

[0078] [Figure 43A] FIG. 43A is a simplified schematic diagram illustrating updates being detected in a portion of a physical world represented by a mesh block, according to some embodiments.

[0079] [Figure 43B] FIG. 43B is a simplified schematic diagram illustrating a mesh block, according to some embodiments.

[0080] [Figure 43C] FIG. 43C is a simplified schematic diagram illustrating a crack at the edge of two adjacent mesh blocks, according to some embodiments.

[0081] [Fig. 43D] FIG. 43D is a simplified schematic diagram illustrating the crack in FIG. 43C being concealed by implementing an overlapping mesh skirt on adjacent mesh blocks, according to some embodiments.

[0082] [Diagram 44] FIG. 44 is a schematic diagram illustrating a 2D representation of a portion of the physical world by four blocks according to some embodiments.

[0083] [Diagram 45]FIG. 45 is a schematic diagram illustrating a 3D representation of a portion of the physical world by eight blocks, according to some embodiments.

[0084] [Figure 46] FIG. 46 is a schematic diagram illustrating a 3D representation of a portion of the physical world obtained by updating the 3D representation in FIG. 45 according to some embodiments.

[0085] [Figure 47] FIG. 47 is a schematic diagram illustrating an example of an augmented world viewable by a first and a second user wearing an AR display system, according to some embodiments.

[0086] [Figure 48] FIG. 48 is a schematic diagram illustrating an example of an augmented world obtained by updating the augmented world of FIG. 47 with a new version of a block, according to some embodiments.

[0087] [Figure 49] FIG. 49 is a schematic diagram illustrating an occlusion rendering system according to some embodiments.

[0088] [Figure 50] FIG. 50 is a schematic diagram illustrating a depth image with a hole.

[0089] [Figure 51] FIG. 51 is a flowchart illustrating a method of occlusion rendering in an augmented reality environment according to some embodiments.

[0090] [Figure 52] FIG. 52 is a flowchart illustrating details of the steps of generating surface information from depth information captured by a depth sensor worn by a user in FIG. 51 in accordance with some embodiments.

[0091] [Diagram 53] FIG. 53 is a flowchart illustrating details of the step of filtering depth information and generating the depth map in FIG. 52 according to some embodiments.

[0092] [Figure 54A] FIG. 54A is a sketch of an area being imaged with a depth camera from a first viewpoint to identify voxels that are occupied by surfaces and those that are empty.

[0093] [Figure 54B] FIG. 54B is a sketch of an area that has been imaged using depth cameras from multiple viewpoints to identify voxels that are occupied by surfaces and those that are empty, showing a "hole" for which no stereo information is available as a result of voxels within the region of the "hole" that were not imaged using the depth cameras. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0094] Described herein are methods and apparatus for creating and using three-dimensional (3D) world reconstructions in an augmented reality (AR), mixed reality (MR), or virtual reality (VR) system. To provide a user with a realistic AR / MR / VR experience, the AR / MR / VR system must understand the user's physical surroundings to properly correlate the location of virtual objects in relation to real objects. The world reconstruction may be built from images and depth information about those physical surroundings collected with sensors that are part of the AR / MR / VR system. The world reconstruction may then be used by any of a number of components of such a system. For example, the world reconstruction may be used by components that perform visual occlusion handling, calculate physics-based interactions, or perform environmental inference.

[0095] Occlusion processing identifies portions of a virtual object that should not be rendered and / or displayed to the user due to the presence of objects in the physical world that block the user's view of where the virtual object should be perceived by the user. Physics-based interactions are calculated to determine where or how a virtual object will appear to the user. For example, a virtual object may be rendered to appear as if it is resting on a physical object, moving through void, or colliding with the surface of a physical object. World reconstruction provides a model from which information about objects in the physical world can be obtained for such calculations.

[0096] Environmental inference may also use the world reconstruction in the process of generating information that can be used in calculating how to render virtual objects. For example, environmental inference may involve identifying clear surfaces by recognizing that they are window panes or glass table tops. From such identification, areas containing physical objects may be classified as not occluding virtual objects, but may be classified as interacting with the virtual object. Environmental inference may also generate information used in other ways, such as identifying stationary objects that can be tracked relative to the user's field of view and calculating the motion of the user's field of view.

[0097] However, there are significant challenges in providing such a system. Substantial processing may be required to compute the world reconstruction. Furthermore, the AR / MR / VR system must correctly figure out how to position the virtual objects relative to the user's head, body, etc. As the user's position relative to the physical environment changes, the relevant parts of the physical world may also change, which may require further processing. Furthermore, the 3D reconstruction data is often required to be updated as objects move within the physical world (e.g., a cup moves on a table). Updates to the data representing the environment the user is experiencing must be performed quickly, without using too many of the computing resources of the computer generating the AR / MR / VR environment, since it is not possible to perform other functions while performing the world reconstruction. Furthermore, processing of the reconstruction data by components that "consume" that data may exacerbate demands on computer resources.

[0098] Known AR / MR / VR systems require high computational computing power (e.g., GPU) to drive real-time world reconstruction only within a predefined reconstruction volume (e.g., a predefined voxel grid). The present inventors have recognized and understood techniques for operating AR / MR / VR systems to provide accurate 3D reconstruction data in real-time using a small amount of computational resources, such as computational power (e.g., a single ARM core), memory (e.g., less than 1 GB), and network bandwidth (e.g., less than 100 Mbps). These techniques relate to reducing the processing required to generate and maintain the world reconstruction, and providing and consuming data with low computational overhead.

[0099] These techniques may include reducing the amount of data processed when updating the world reconstruction, such as by identifying portions of sensor data that are available at any time and using them in creating or updating the world reconstruction. The sensor data may be selected, for example, based on whether it represents a portion of the physical world that is likely to be in the vicinity of a surface of an object to be represented in the world reconstruction.

[0100] In some embodiments, computational resources may be reduced by simplifying the data that represents the world reconstruction: a simpler representation may reduce resources for processing, storing, and / or managing that data and its use.

[0101] In some embodiments, computational resource usage may be reduced by representing the physical world in blocks that are stored and retrieved separately but can be combined to provide a realistic representation of the physical world. Blocks may be managed in memory to limit computational resources, and in some embodiments may allow sharing of blocks across AR / MR / VR systems operating in the same physical space, such that each AR / MR / VR system does little processing to construct the world reconstruction.

[0102] In some embodiments, the use of computational resources is to access information about the physical world. The cost of occlusion may be reduced by selecting among different representations of the physical world when accessing a virtual object. The world reconstruction may include information about the physical world, for example, captured from different sensors and / or stored in different formats. The simplest data to consume or provide may be provided to a component that uses the world reconstruction to render virtual objects. If the simpler data is not available, data obtained with different sensors that may generate a higher computational load may be accessed. As an example, the world reconstruction may include a depth map collected with a depth sensor and a more comprehensive representation of the 3D world, such as may be stored as a mesh calculated from image information. Information about the physical world, when available, may be provided to a component that performs occlusion processing based on the depth map. If holes exist in the depth map, information to fill those holes may be extracted from the mesh. In some embodiments, the depth map may be "live," representing the physical world as it is captured by the depth sensor at the time the data is accessed.

[0103] The techniques as described herein may be used with or separately from many types of devices, including wearable or portable devices with limited computing resources, that provide augmented reality scenes, and for many types of scenes.

[0104] AR System Overview

[0105] Figures 1-2 illustrate such a scene. For illustrative purposes, an AR system is used as an example of an XR system. Figures 3-8 illustrate an example AR system including one or more processors, memory, sensors, and a user interface that may operate in accordance with the techniques described herein.

[0106] With reference to Figure 1, an AR scene 4 is depicted in which a user of the AR technology sees a physical world park-like setting 6 featuring people, trees, buildings in the background, and a concrete platform 8. In addition to these items, the user of the AR technology also perceives that they "see" a robot figure 10 standing on the physical world concrete platform 8, and a flying cartoon-like avatar character 2 that appears to be an anthropomorphic bumblebee, although these elements (e.g., the avatar character 2 and the robot figure 10) do not exist within the physical world. Due to the extreme complexity of the human visual perception and nervous system, it is difficult to produce AR technology that facilitates comfortable, natural-feeling, rich presentation of virtual image elements among other virtual or physical world image elements.

[0107] Such AR scenes can be achieved using a system that includes a world reconstruction component that can build and update a representation of the physical world surfaces around the user. This representation can be used for occluding rendering, placing virtual objects into physics-based interaction states, and for virtual character path planning and navigation, or other operations where information about the physical world is used. Figure 2 depicts another example of an AR scene 200 showing an example world reconstruction use case including visual occlusion 202, physics-based interaction 204, and environment inference 206, according to some embodiments.

[0108] The example scene 200 is a living room with a wall, a bookshelf on one side of the wall, a floor lamp in the corner of the room, a floor, a sofa, and a coffee table on the floor. In addition to these physical items, a user of the AR technology also perceives virtual objects such as an image on the wall behind the sofa, a bird flying through the door, a deer peeking out from the bookshelf, and an ornament in the form of a windmill placed on the coffee table. In addition to the images on the wall, the AR technology can obtain information not only about the surface of the wall, but also about objects and surfaces in the room, such as the lamp shape. For a flying bird, the AR technique requires information about all objects and surfaces around the room to render the bird with realistic physics and avoid bouncing off the objects and surfaces or the bird in case of collision. For a deer, the AR technique requires information about surfaces such as the floor or coffee table to calculate where to place the deer. For a windmill, the system may identify it as a separate object from the table and infer that it is movable, while the corners of the shelf or the corners of the wall may be inferred to be stationary. Such specificity may be used in inferences regarding the parts of the scene to be used or updated in each of the various actions.

[0109] The scene may be presented to the user via a system that includes multiple components, including a user interface that may stimulate one or more user senses, including sight, sound, and / or touch. In addition, the system may include one or more sensors that may measure parameters of the physical portion of the scene, including the user's position and / or movement within the physical portion of the scene. Furthermore, the system may include one or more computing devices with associated computer hardware, such as memory. These components may be integrated into a single device or distributed across multiple interconnected devices. In some embodiments, some or all of these components may be integrated into a wearable device.

[0110] 3 depicts an AR system 302 configured to provide an experience of AR content that interacts with a physical world 306, according to some embodiments. The AR system 302 may include a display 308. In the illustrated embodiment, the display 308 may be worn by a user as part of a headset such that the user may wear the display over their eyes like a pair of goggles or glasses. At least a portion of the display may be transparent such that the user may observe a see-through reality 310. The see-through reality 310 may correspond to a portion of the physical world 306 within a current perspective of the AR system 302, which may correspond to the user's perspective when the user is wearing a headset that incorporates both the display and sensors of the AR system and obtains information about the physical world.

[0111] The AR content may also be presented on the display 308, overlaid on the see-through reality 310. To provide accurate interaction between the AR content and the see-through reality 310 on the display 308, the AR system 302 may include a sensor 322 configured to capture information about the physical world 306.

[0112] The sensors 322 may include one or more depth sensors that output a depth map 312. Each depth map 312 may have a number of pixels, each of which may represent a distance to a surface in the physical world 306 in a particular direction relative to the depth sensor. Raw depth data may come from the depth sensors to create a depth map. Such a depth map may be updated as fast as the depth sensors can form new images, which may be hundreds or thousands of times per second. However, the data may be noisy, incomplete, and have holes, shown as black pixels on the illustrated depth map.

[0113] The system may include other sensors, such as image sensors. The image sensors may obtain information that may be processed to represent the physical world in other ways. For example, images may be processed in the world reconstruction component 316 to create a mesh that represents connected portions of objects in the physical world. Metadata about such objects, including, for example, color and surface texture, may also be obtained using the sensors and stored as part of the world reconstruction.

[0114] The system may also obtain information about the user's head pose relative to the physical world. In some embodiments, the sensor 310 may include an inertial measurement unit, which may be used to calculate and / or determine the head pose 314. The head pose 314 for a depth map may indicate the current viewpoint of the sensor capturing the depth map, for example with six degrees of freedom (6DoF), but the head pose 314 may be used for other purposes, such as to relate image information to a particular part of the physical world or to relate the position of a display worn on the user's head to the physical world. In some embodiments, the head pose information may be derived in a manner other than from an IMU, such as from analyzing objects in an image.

[0115] The world reconstruction component 316 may receive the depth map 312 and head pose 314 and any other data from the sensors and integrate the data into a reconstruction 318, which may at least appear to be a single combined reconstruction. The reconstruction 318 may be more complete and less noisy than the sensor data. The world reconstruction component 316 may update the reconstruction 318 using spatial and temporal averaging over time of the sensor data from multiple viewpoints.

[0116] Reconstruction 318 may include a representation of the physical world in one or more data formats, including, for example, voxels, meshes, planes, etc. Different formats may represent alternative representations of the same portion of the physical world or may represent different portions of the physical world. In the illustrated example, on the left side of reconstruction 318, a portion of the physical world is presented as a global surface, and on the right side of reconstruction 318, a portion of the physical world is presented as a mesh.

[0117] Reconstruction 318 may be used for AR functions, such as producing a surface representation of the physical world for occlusion or physics-based processing. This surface representation may change as the user moves or objects in the physical world change. Aspects of reconstruction 318 may be used by component 320, for example, to produce a changing global surface representation in world coordinates, which may be used by other components.

[0118] AR content may be generated, such as by an AR application 304, based on this information. The AR application 304 may be, for example, a game program that performs one or more functions based on information about the physical world, such as visual occlusion, physics-based interactions, and environmental inference. These functions may be performed by querying data in different formats from the reconstruction 318 produced by the world reconstruction component 316. In some embodiments, the component 320 may be configured to output updates as the representation in the region of interest of the physical world changes. The region of interest may be set to approximate a portion of the physical world in the vicinity of the user of the system, such as a portion in the user's field of view, or projected (predicted / determined) to occur in the user's field of view.

[0119] The AR application 304 uses this information to generate and update AR content, the virtual portions of which may be combined with a see-through reality 310 and presented on the display 308 to create a realistic user experience.

[0120] In some embodiments, the AR experience may be provided to a user through a wearable display system. FIG. 4 illustrates an example of a wearable display system 80 (hereinafter referred to as "system 80"). System 80 is a head-mounted display. The display device 62 includes a ray device 62 (hereinafter referred to as "display device 62") and various mechanical and electronic modules and systems for supporting the functionality of the display device 62. The display device 62 may be coupled to a frame 64, which is wearable by a display system user or viewer 60 (hereinafter referred to as "user 60") and configured to position the display device 62 in front of the eyes of the user 60. According to various embodiments, the display device 62 may be a sequential display. The display device 62 may be monocular or binocular. In some embodiments, the display device 62 may be an example of the display 308 in FIG. 3.

[0121] In some embodiments, a speaker 66 is coupled to the frame 64 and positioned proximate the ear canal of the user 60. In some embodiments, another speaker, not shown, is positioned adjacent another ear canal of the user 60 to provide stereo / adjustable sound control. The display device 62 is operatively coupled, such as by wired leads or wireless connectivity 68, to a local data processing module 70, which may be mounted in a variety of configurations, such as fixedly attached to the frame 64, fixedly attached to a helmet or hat worn by the user 60, built into headphones, or otherwise removably attached to the user 60 (e.g., in a backpack configuration, in a belt-coupled configuration).

[0122] The local data processing module 70 may include a processor and digital memory, such as non-volatile memory (e.g., flash memory), both of which may be utilized to aid in processing, caching, and storing data. The data includes a) data captured (e.g., which may be operatively coupled to the frame 64 or otherwise attached to the user 60) from sensors, such as image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, and / or gyroscopes, and / or b) data obtained and / or processed using the remote processing module 72 and / or remote data repository 74, possibly for passing to the display device 62 after processing or retrieval. The local data processing module 70 may be operatively coupled to the remote processing module 72 and the remote data repository 74 by communications links 76, 78, such as via wired or wireless communication links, respectively, such that these remote modules 72, 74 are operatively coupled to each other and available as resources to the local processing and data module 70. 3 may be implemented, at least in part, within the local data processing module 70. For example, the local data processing module 70 may be configured to execute computer-executable instructions to generate, at least in part, a physical world representation based, at least in part, on at least a portion of the data.

[0123] In some embodiments, the local data processing module 70 may include one or more processors (e.g., a graphics processing unit (GPU)) configured to analyze and process the data and / or image information. In some embodiments, the local data processing module 70 may include a single processor (e.g., a single-core or multi-core ARM processor), which would limit the computational budget of the module 70 but allow for smaller devices. In some embodiments, the world reconstruction component 316 may generate the physical world representation in real time over a non-predetermined space using a computational budget of less than a single ARM core, such that the remaining computational budget of the single ARM core may be accessed for other uses, such as, for example, extracting meshes.

[0124] In some embodiments, remote data repository 74 may include a digital data storage facility, which may be connected to the Internet or other networks in a "cloud" resource configuration. The world reconstructions may be available through a networking configuration. In some embodiments, all data is stored and all calculations are performed within the local data processing module 70, allowing for fully autonomous use from remote modules. The world reconstructions may be stored in whole or in part within this repository 74, for example.

[0125] In some embodiments, the local data processing module 70 is operably coupled to a battery 82. In some embodiments, the battery 82 is a removable power source, such as a commercially available battery. In other embodiments, the battery 82 is a lithium ion battery. In some embodiments, the battery 82 includes both an internal lithium ion battery that is rechargeable by the user 60 during non-operational times of the system 80, and a removable battery, so that the user 60 may operate the system 80 for longer periods of time without having to plug in a power source and charge the lithium ion battery, or without having to shut off the system 80 and replace the battery.

[0126] 5A illustrates a user 30 wearing an AR display system that renders AR content as the user 30 moves through a physical world environment 32 (hereafter referred to as "environment 32"). The user 30 positions the AR display system at a location 34, and the AR display system records ambient information of the passable world (e.g., digital representations of real objects in the physical world that may be stored and updated with changes to the real objects in the physical world) for the location 34, such as pose relationships to mapped features or directional audio input. The location 34 is aggregated into a data input 36 and processed by at least a passable world module 38, which may be implemented by, for example, processing on the remote processing module 72 of FIG. 3. In some embodiments, the passable world module 38 may include a world reconstruction component 316.

[0127] The passable world module 38 determines where and how the AR content 40 may be placed in the physical world as determined from the data input 36. The AR content is "placed" in the physical world by presenting both a representation of the physical world and the AR content through the user interface, where the AR content is rendered as if it were interacting with objects in the physical world, and the objects in the physical world are presented as if the AR content were obscuring the user's view of those objects, when appropriate. In some embodiments, the AR content may be placed by appropriately selecting a portion of a fixed element 42 (e.g., a table) from a reconstruction (e.g., reconstruction 318) and determining the shape and position of the AR content 40. As an example, the fixed element may be a table, and the virtual content may be positioned to appear as if it were on the table. In some embodiments, the AR content may be placed in a structure in the field of view 44, which may be the current field of view or an estimated future field of view. In some embodiments, the AR content may be placed relative to a mapped mesh model 46 of the physical world.

[0128] As depicted, the fixed elements 42 serve as a substitute for any fixed elements in the physical world, which may be stored in the passable world module 38 such that the user 30 may perceive content on the fixed elements 42 without the system having to map to the fixed elements 42 each time it is visible to the user 30. The fixed elements 42 may thus be mapped mesh models from a previous modeling session or determined from a separate user, but stored on the passable world module 38 for future reference by multiple users. Thus, the passable world module 38 may allow the device of the user 30 to recognize the environment 32 from the previously mapped environment and display AR content without first mapping the environment 32, saving computational processes and cycles and avoiding latency of any rendered AR content.

[0129] A mapped mesh model 46 of the physical world may be created by the AR display system, and appropriate surfaces and metrics for interacting with and displaying the AR content 40 can be mapped and stored in the passable world module 38 for future retrieval by the user 30 or other users without the need for remapping or modeling. In some embodiments, the data inputs 36 are inputs such as geographic location, user identification, and current activity that indicate to the passable world module 38 the fixed elements 42 of one or more available fixed elements, the AR content 40 last placed on the fixed elements 42, and whether that same content should be displayed (such AR content is "persistent" content regardless of whether the user is viewing a particular passable world model).

[0130] FIG. 5B illustrates a schematic of the viewing optics assembly 48 and associated components. When directed at the user's eye 49, in some embodiments, two eye tracking cameras 50 detect metrics of the user's eye 49, such as eye shape, eyelid occlusion, pupil direction, and phosphenes on the user's eye 49. In some embodiments, a depth sensor 51, such as a time-of-flight sensor, emits relay signals into the world and determines the distance to a given object. In some embodiments, a world camera 52 records a view above the surroundings, maps the environment 32, and detects inputs that may affect the AR content. Camera 53 may also capture specific timestamps of physical world images within the user's field of view. World camera 52, camera 53, and depth sensor 51 each have separate fields of view 54, 55, and 56, and collect and record data from a physical world scene, such as the physical world environment 32 depicted in FIG. 3A.

[0131] The inertial measurement unit 57 may determine the movement and orientation of the viewing optics assembly 48. In some embodiments, each component is operably coupled to at least one other component. For example, the depth sensor 51 is operably coupled to the eye tracking camera 50 as a check of accommodation measured against the actual distance the user's eyes 49 are looking at.

[0132] Information from these sensors in the viewing optics assembly 48 may be coupled to one or more of the processors in the system. The processor may generate data that may be rendered to cause the user to perceive virtual content interacting with objects in the physical world. The rendering may be implemented in any suitable manner, including generating image data that depicts both physical and virtual objects. In other embodiments, the physical and virtual content may be depicted in a scene by modulating the opacity of a display device through which the user sees the physical world. The opacity may be controlled to create the appearance of a virtual object and also to block the user from seeing objects in the physical world that are occluded by the virtual object. Regardless of how the content is presented to the user, a model of the physical world is required so that properties of the virtual object that may be affected by the physical object, including the shape, position, motion, and visibility of the virtual object, may be correctly calculated. In some embodiments, the model may include a reconstruction of the physical world, e.g., reconstruction 318.

[0133] The model may be created from data collected from sensors on the user's wearable device, although in some embodiments the model may be created from data collected by multiple users, which may be aggregated in a computing device remote from all users (and may be "in the cloud").

[0134] The model may be created, at least in part, by a world reconstruction system, such as the world reconstruction component 316 of FIG. 3, which is depicted in further detail in FIG. 6. The world reconstruction component 316 may include a perception module 160, which may generate, update, and store representations for portions of the physical world. In some embodiments, the perception module 160 may represent a portion of the physical world within the reconstruction range of a sensor as a plurality of voxels. Each voxel may correspond to a 3D cube of a given volume in the physical world and may include surface information indicating whether a surface exists within the volume represented by the voxel. A voxel may be assigned a value indicating whether its corresponding volume has been determined to include a surface of a physical object, has been determined to be empty, or has not yet been measured with a sensor and therefore its value is unknown. It should be understood that the value indicates that voxels determined to be empty or unknown need not be explicitly stored, and that the value of a voxel may be stored in computer memory in any suitable manner, including not storing information regarding voxels determined to be empty or unknown.

[0135] 7A depicts an example of a 3D space 100 that is discretized into voxels 102. In some embodiments, the perception module 160 may set the volume of voxels to determine objects of interest, capture features of the objects of interest, and avoid redundant information. For example, the perception module 160 may be configured to identify larger objects and surfaces, such as walls, ceilings, floors, and large furniture. Thus, the volume of voxels may be relatively large in size, e.g., 4 cm 3 may be set to a cube.

[0136] The reconstruction of the physical world, including the voxels, may be referred to as a solid model. Information for creating the solid model may be created over time as the sensor moves about the physical world. Such motion may occur as a user of a wearable device, including the sensor, moves around. Figures 8A-F depict an example of reconstructing the physical world into a solid model. In the illustrated example, the physical world includes a portion of a surface 180, which is shown in Figure 8A. In Figure 8A, a sensor 182 at a first location may have a field of view 184 in which the portion of the surface 180 is visible.

[0137] The sensor 182 may be of any suitable type, such as a depth sensor. However, the depth data may be derived from an image sensor or in other ways. The perception module 160 may receive the data from the sensor 182 and then set values ​​of a number of voxels 186, as shown in FIG. 8B, to represent a portion 180 of a surface visible by the sensor 182 within a field of view 184.

[0138] In Figure 8C, the sensor 182 may move to a second location and have a field of view 188. As shown in Figure 8D, an additional group of voxels becomes visible, and the values ​​of these voxels may be set to indicate the location of the portion of the surface that has entered the field of view 188 of the sensor 182. The values ​​of these voxels may be added to a solid model for the surface.

[0139] In FIG. 8E, the sensor 182 may further move to a third location and have a field of view 190. In the illustrated embodiment, an additional portion of the surface becomes visible within the field of view 190. As shown in FIG. 8F, an additional group of voxels may become visible and the values ​​of these voxels may be set to indicate the location of the portion of the surface that entered the field of view 190 of the sensor 182. The values ​​of these voxels may be added to a solid model for the surface. As shown in FIG. 6, this information may be stored as solid information 162a as part of the persisted world. Information about the surface, such as color or texture, may also be stored. Such information may be stored, for example, as solid metadata 162b.

[0140] In addition to generating information for the persisted world representation, the perception module 160 also The R system may identify and output indications of changes in the area surrounding a user of the R system. Indications of such changes may trigger updates to volumetric data stored as part of the persisted world, or trigger other functionality, such as triggering component 304 that generates AR content and updates AR content.

[0141] In some embodiments, the perception module 160 may identify changes based on a signed distance function (SDF) model. The perception module 160 may be configured to receive sensor data, such as, for example, a depth map 160a and a head pose 160b, and then fuse the sensor data into an SDF model 160c. The depth map 160a may provide SDF information directly, or images may be processed to become SDF information. The SDF information represents distances from sensors used to capture that information. Because those sensors may be part of the wearable unit, the SDF information may represent the physical world from the line of sight of the wearable unit, and therefore the user. The head pose 160b may enable the SDF information to be related to voxels in the physical world.

[0142] Referring back to FIG. 6, in some embodiments, perception module 160 may generate, update, and store representations for portions of the physical world within a perception range. The perception range may be determined, at least in part, based on the reconstruction range of the sensor, which may be determined, at least in part, based on the limits of the observation range of the sensor. As a specific example, an active depth sensor operating using active IR pulses may operate reliably over a range of distances, creating an observation range for the sensor, which may be a few centimeters or tens of centimeters to several meters.

[0143] FIG. 7B depicts a reconstruction range for a sensor 104 having a viewpoint 106. A reconstruction of a 3D space within the viewpoint 106 may be constructed based on data captured by the sensor 104. In the illustrated example, the sensor 104 has an observation range of 40 cm to 5 m. In some embodiments, the reconstruction range of the sensor may be determined to be smaller than the observation range of the sensor because the sensor output closer to its observation limit may be noisier, incomplete, and inaccurate. For example, in the illustrated example of 40 cm to 5 m, the corresponding reconstruction range may be set to 1 to 3 m, and data collected with the sensor showing surfaces outside this range may not be used.

[0144] In some embodiments, the perception range may be larger than the reconstruction range of the sensor. If the component 164 that uses data about the physical world requests data about an area in the perception range outside the portion of the physical world in the current reconstruction range, that information may be provided from the persisted world 162. Thus, information about the physical world may be easily accessible by query. In some embodiments, an API may be provided to respond to such queries and provide information about the user's current perception range. Such techniques may reduce the time required to access an existing reconstruction and provide an improved user experience.

[0145] In some embodiments, the perception range may be a 3D space corresponding to a bounding box centered on the user location. As the user moves, the portion of the physical world within the perception range that may be queriable by component 164 may also move with the user. FIG. 7C depicts a bounding box 110 centered on location 112. It should be understood that the size of the bounding box 110 may be set to enclose the sensor's observation range with a reasonable extension since a user may not move at an unreasonable speed. In the illustrated example, the sensor worn by the user has an observation limit of 5 m. The bounding box 110 is set to a limit of 20 m. 3 It is set as a cube of .

[0146] 6, the world reconstruction component 316 may include additional modules that may interact with the perception module 160. In some embodiments, the persisted world module 162 may receive a representation for the physical world based on data obtained by the perception module 160. The persisted world module 162 may also include various formats of the representation of the physical world. For example, volumetric metadata 162b, such as voxels, may be stored along with the meshes 162c and planes 162d. In some embodiments, other information, such as depth maps, may also be saved.

[0147] In some embodiments, the perception module 160 may include modules that generate representations for the physical world in various formats, including, for example, meshes 160d, planes, and semantics 160e. These modules may generate the representations based on data within the perception range of one or more sensors at the time the representations are generated and data captured at previous times and information within the persisted world 162. In some embodiments, these components may operate on depth information captured using a depth sensor. However, the AR system may also include a vision sensor and generate such representations by analyzing monocular or binocular visual information.

[0148] In some embodiments, these modules may operate on a region of the physical world, such as a region represented by a block or tile, as described below. They may be triggered to update the block or tile or other sub-region of the physical world when perception module 160 detects a change in the physical world in that sub-region. Such a change may be detected by other criteria, such as, for example, detecting a new surface in SDF model 160c, or by a change in value of a sufficient number of voxels representing the sub-region.

[0149] The world reconstruction components 316 may include components 164 that may receive a representation of the physical world from the perception module 160. Information about the physical world may be pulled by these components, for example, according to usage requests from applications. In some embodiments, information may be pushed to the usage components, such as via indications of changes in pre-identified areas or changes in the physical world representation within a perception range. The components 164 may include, for example, game programs and other components that implement processing for visual occlusion, physics-based interactions, and environmental inference.

[0150] In response to a query from component 164, perception module 160 may transmit a representation for the physical world in one or more formats. For example, when component 164 indicates that the use is for visual occlusion or physics-based interaction, perception module 160 may transmit a representation of a surface. When component 164 indicates that the use is for environmental inference, perception module 160 may transmit meshes, planes, and semantics of the physical world.

[0151] In some embodiments, perception module 160 may include a component that formats information and provides it to component 164. An example of such a component may be ray casting component 160f. A usage component (e.g., component 164) may, for example, query information about the physical world from a particular viewpoint. Ray casting component 160f may select from one or more representations of the physical world data within a field of view from that viewpoint.

[0152] Viewpoint-dependent brick selection for fast volumetric reconstruction.

[0153] As should be understood from the foregoing description, the perception module 160 or another component of the AR system may process data and create a 3D representation of a portion of the physical world. The data to be processed may be reduced, at least in part, by thinning out portions of the 3D reconstruction volume based on the camera frustum and / or depth image, extracting and persisting planar data, capturing, persisting, and updating the 3D reconstruction data on a block-by-block basis while maintaining neighborhood consistency, providing occlusion data to an application, generating such scenes where the occlusion data is derived from a combination of one or more depth data sources, and / or performing multi-stage mesh simplification.

[0154] The world reconstruction system may integrate sensor data from multiple viewpoints of the physical world over time. The pose (e.g., position and orientation) of the sensor may be tracked as the device containing the sensor is moved. As the frame pose of the sensor and how it relates to other poses is understood, each of these multiple viewpoints of the physical world may be fused together into a single combined reconstruction. The reconstruction may be more complete and less noisy than the original sensor data by using spatial and temporal averaging (i.e., averaging data from multiple viewpoints over time).

[0155] The reconstruction may contain data of different levels of sophistication, including, for example, raw data such as live depth data, fused volumetric data such as voxels, and calculated data such as meshes.

[0156] In some embodiments, the AR and MR systems represent 3D scenes with a regular voxel grid, where each voxel may contain a signed distance field (SDF) value. The SDF value describes whether the voxel is inside or outside a surface in the scene to be reconstructed, and the distance from the voxel to the surface. Computing 3D reconstruction data representing the desired volume of the scene requires large amounts of memory and processing power. These requirements increase for scenes representing larger spaces, since the number of variables required for 3D reconstruction grows cube-wise with the number of depth images to be processed.

[0157] Described herein are efficient methods of reducing processing. According to some embodiments, a scene may be represented by one or more bricks. Each brick may include multiple voxels. The bricks processed to generate a 3D reconstruction of the scene may be selected by thinning out the set of bricks representing the scene based on a frustum derived from the field of view (FOV) of an image sensor and / or a depth image (or "depth map") of the scene created using a depth sensor.

[0158] The depth image may have one or more pixels, each representing a distance to a surface in the scene. These distances may be related to a position relative to the image sensor so that data output from the image sensor may be selectively processed. Image data may be processed for those bricks that represent a portion of the 3D scene that contains a surface that would be visible from the viewpoint (or "viewpoint") of the image sensor. Processing of some or all of the remaining bricks may be omitted. Using such an approach, the selected bricks may be those that are more likely to contain new information, which may be achieved by thinning out bricks for which the output of the image sensor is less likely to provide useful information. The data output from the image sensor is less likely to provide useful information for bricks that are either closer to or farther from the image sensor than the surface indicated by the depth map, because those bricks are either in void or behind a surface and therefore not depicted in the image from the image sensor.

[0159] In some embodiments, one or more criteria may be applied to efficiently select a set of bricks for processing. The initial set of bricks may be limited to those within the frustum of the image sensor. Many bricks outside the frustum may then be thinned out. More computationally intensive processing to update the 3D reconstruction may then be performed on the subset of bricks accepted for processing following the thinning. Thus, the 3D representation of the scene to be updated is more efficiently calculated using processing on a reduced number of voxels.

[0160] Even more reduction in processing may be achieved by thinning out bricks based on the depth image. According to some embodiments, thinning and / or acceptance of bricks may be performed by projecting a silhouette of each brick in the initial set into the depth image. Such thinning may be based on whether the brick corresponds to a portion of the scene that the depth image shows in the vicinity of the surface. Bricks that may be identified as simply being entirely in front of the surface or entirely behind it may be thinned out. In some embodiments, such a determination may be made efficiently. For example, a bounding box around the projection of the brick into the depth map may be used to determine maximum and minimum brick values ​​along a z-coordinate direction, which may be approximately perpendicular to the 2D plane of the depth image. By comparing these maximum and minimum brick values ​​against the distance represented by a pixel in the depth map, the brick may be thinned out and / or accepted for further processing. Such processing may result in the selection of bricks for initial processing that intersect the surface as reflected in the depth image and / or are in front of the surface as reflected in the depth image. In some embodiments, such processing may distinguish between bricks in front of a solid surface and bricks in front of a perforated surface (i.e., bricks representing areas for which the depth sensor was unable to reliably measure the distance to the surface).

[0161] In some embodiments, the thinning / acceptance criteria may result in classification of some or all of the bricks that are accepted for further processing, such that the processing algorithms for computational volumetric reconstruction may be tuned for the characteristics of the bricks. In some embodiments, different treatments may be selected based on whether a brick is classified as intersecting a surface, in front of a solid surface, or in front of a surface with holes.

[0162] 9 shows a cross-sectional view of a scene 400 along a plane parallel to the y- and z-coordinates. The XR system may represent the scene 400 by a grid of voxels 504. A conventional XR system may update each voxel of the grid of voxels based on every new depth image captured by a sensor 406, which may be an image sensor or a depth sensor, so that a 3D reconstruction generated from the grid of voxels may reflect changes in the scene. Updating in this manner may consume significant computational resources and may also introduce artifacts into the output of the XR system, for example due to latency caused by heavy computation.

[0163] Described herein are techniques for providing accurate 3D reconstruction data using low usage computational resources, for example, by thinning a portion of a grid of voxels 504 based, at least in part, on the camera frustum 404 of an image sensor 406 and / or a depth image captured by the image sensor.

[0164] In the illustrated example, the image sensor 406 captures a depth image (not shown) that includes the surface 402 of the scene 400. The depth image may be stored in computer memory in any convenient manner that captures the distance between a reference point in the scene 400 and the surface. In some embodiments, the depth image may be represented as values ​​in a plane parallel to the x- and y-axes, as illustrated in FIG. 9, where the reference point is the origin of the coordinate system. A location in the XY plane is , may correspond to a direction relative to a reference point, and values ​​at those pixel locations may indicate the distance from the reference point to the nearest neighbor of the surface in the direction indicated by the coordinates in the plane. Such a depth image may include a grid of pixels (not shown) in a plane parallel to the x- and y-axes. Each pixel may indicate the distance in a particular direction from the image sensor 406 to the surface 402. In some embodiments, the depth sensor may be unable to measure the distance to the surface in a particular direction. Such a condition may arise, for example, when the surface is out of range of the image sensor 406. In some embodiments, the depth sensor may be an active depth sensor that measures distance based on reflected energy, but the surface may not reflect enough energy for an accurate measurement. Thus, in some embodiments, the depth image may have "holes", that is, there are pixels for which no value is assigned.

[0165] In some embodiments, the reference point of the depth image may vary. Such a configuration allows the depth image to represent surfaces throughout the entire 3D scene and may not be limited to portions having a predetermined and limited range of angles relative to a particular reference point. In such embodiments, the depth image may indicate the distance to the surface as the image sensor 406 moves through six degrees of freedom (6DOF). In these embodiments, the depth image may include a set of pixels for each of a number of reference points. In these embodiments, the portion of the depth image may be selected based on a "camera pose," which represents the direction and / or orientation in which the image sensor 406 is facing at the time the image data is captured.

[0166] The image sensor 406 may have a field of view (FOV), which may be represented by a camera frustum 404. In some embodiments, the infinite camera frustum depicted may be reduced to a finite 3D trapezoidal prism 408 by assuming a maximum depth 410 that the image sensor 406 can provide and / or a minimum depth 412 that the image sensor 406 can provide. The 3D trapezoidal prism 408 may be a convex polyhedron bounded by six planes.

[0167] In some embodiments, one or more voxels 504 may be grouped into bricks 502. Figure 10A illustrates a portion 500 of a scene 400 that includes eight bricks 502. Figure 10B illustrates a portion 500 of a scene 400 that includes eight bricks 502. 3 Illustrated is an example brick 502 that includes voxels 504. Referring back to Figure 9, the scene 400 may include one or more bricks, 16 of which are illustrated in the diagram depicted in Figure 4. Each brick may be identifiable by a brick identification, for example,

[0000] -

[0015] .

[0168] FIG. 11 depicts a volumetric representation hierarchy implemented in some embodiments. In some embodiments, such a volumetric representation hierarchy may reduce latency for data transfer. In some embodiments, a voxel grid of the physical world may be mapped to conform to a structure of a memory architecture for a processor that computes the AR content, such as the processor on which component 304 executes. One or more voxels may be grouped into a "brick." One or more bricks may be grouped into a "tile." The size of a tile may correspond to a memory page of a storage medium local to the processor. Tiles may be moved between local and remote memory, such as via a wireless connection, based on usage or expected usage according to a memory management algorithm.

[0169] In some embodiments, uploads and / or downloads between the perception module 160 and the persisted world module 162 may be performed on multiple tiles in one operation. One or more tiles may be grouped into a "RAM tileset." The size of the RAM tileset may correspond to an area within the reconstruction of the range of the sensors worn by the user. One or more RAM tilesets may be grouped into a "global tileset." The size of the global tile set may correspond to the perception range of the world reconstruction system (e.g., the perception range for the perception module 160).

[0170] 12 is a flow chart illustrating a method 600 of operating a computing system to generate a 3D reconstruction of a scene, according to some embodiments. The method 600 may begin with representing a scene (e.g., scene 400) with one or more bricks (e.g., brick 502), each of which includes one or more voxels (e.g., voxel 504). Each brick may represent a portion of the scene. The bricks may be identifiable with respect to a persistent coordinate system such that the same brick represents the same volume in the scene even if the pose of the image sensor (e.g., image sensor 406) changes.

[0171] At act 604, the method 600 may capture a depth image (e.g., a depth image including the surface 402) from a depth sensor (e.g., depth sensor 406). The depth sensor may be, for example, an active depth sensor that transmits IR radiation and measures the time of flight for reflection. Each such measurement represents the distance from the depth sensor to the surface in a particular direction. That depth information may represent the same volume as represented by the brick.

[0172] At act 606, the method 600 may decimate a portion of one or more bricks relative to the camera frustum (e.g., the finite 3D trapezoidal prism 408 derived from the camera frustum 404) to produce a first one or more bricks that are a reduced set of bricks from the one or more bricks. Such decimation may eliminate bricks that represent portions of the scene outside the field of view of the image sensor at the time the image data being processed was acquired. That image data is therefore unlikely to contain information useful in creating or updating the bricks.

[0173] At act 608, method 600 may thin out a portion of the first one or more bricks from the depth image to produce a second one or more bricks that are a reduced set of bricks from the first one or more bricks. At act 610, method 600 may generate a 3D reconstruction of the scene based on the second one or more bricks.

[0174] 9, given the surface 402 and corresponding camera pose captured by the depth image, voxels between the image sensor 406 and the surface 402 may be empty. The further a voxel is behind the surface 402 from the image sensor 406, the less certain it may be that it represents the inside of an object or void. The degree of certainty may be represented by a weighting function, which weights the voxel update based on the distance to the surface 402. When a voxel behind the surface 402 (farther away from the image sensor 402) has a weighting function above a threshold, the voxel may receive no update or zero update (e.g., an update with zero change). Also, all voxels not within the camera frustum 404 may not be updated or examined for this depth image.

[0175] Method 600 not only improves the processing speed of stereoscopic depth image fusion, but also consumes little memory storage, which allows method 600 to run on wearable hardware. 3 A small reconstruction volume of 5m x 5m x 3m with a voxel size and 8 bytes per voxel (4 bytes for the distance value and 4 bytes for the weight value) would already require about 600MB. Method 600 can categorize bricks by their distance to the surface relative to a truncation threshold. For example, method 600 can identify empty bricks (e.g., bricks that are thinned out or bricks that are far from the surface beyond the truncation threshold) so as not to allocate memory space for empty bricks. Method 600 also applies a constant distance value of a negative truncation threshold and a weight of 1. Thus, bricks far from the surface can be identified by the truncation threshold, so that these bricks are stored. Method 600 can also identify bricks with a distance to the surface between zero and the truncation threshold, so that these bricks are stored with a constant SDF value of the positive truncation threshold, but with a variable weighting. Storing a distance or weighting value that is constant for bricks with a single value can be an entropy-based compression for a zero-entropy field.

[0176] Method 600 may allow for flagging a brick as "not containing any part of a surface" during voxel updates, which may significantly speed up processing of the brick, which may include, for example, converting an image of the portion of the scene represented by the brick into a mesh.

[0177] FIG. 13 illustrates an example method 606 of thinning a portion of one or more bricks for the camera frustum 404 of the image sensor 406, according to some embodiments. The method 606 may begin with finding a cubic axis aligned bounding box (AABB) containing the camera frustum 404. The AABB may contain one or more bricks within the scene. The method 606 may include splitting the AABB into one or more sub-AABBs (act 704) and performing a camera frustum acceptance test (act 706). If the method 606 determines in act 708 that the sub-AABB reaches the size of a brick, the method 606 may produce a first one or more bricks (act 710). If the method 606 determines in act 708 that the sub-AABB is larger than the size of a brick, the method 606 may repeat acts 704-708 until the sub-AABB reaches the size of a brick.

[0178] For example, given a 3D trapezoidal prism 408 corresponding to the camera frustum 404, an AABB may be found in constant time using the square of the side length containing the 3D trapezoidal prism 408. The AABB may be divided into eight sub-AABBs. Each of the eight sub-AABBs may be tested for intersection with the camera frustum 404. If it is determined that a sub-AABB does not intersect with the camera frustum 404, the brick corresponding to this sub-AABB may be thinned out. The thinned brick may be rejected for further processing. If it is determined that a sub-AABB does intersect with the camera frustum 404, the sub-AABB may be further divided into eight sub-AABBs of the sub-AABB. Each of the eight sub-AABBs of the sub-AABB may then be tested for intersection with the camera frustum 404. The iteration of dividing and testing continues until the sub-AABB corresponds to a single brick. To determine whether the camera frustum 404 intersects with the AABB, a two-step test may be performed. First, it may be tested whether at least one corner point of the AABB is inside each of the planes that bound the camera frustum 404. Second, it may be tested whether all corner points of the camera frustum 404 are inside the AABB, so that some cases of AABBs that do not intersect the camera frustum 404 but are incorrectly classified as partially inside (e.g., having only one corner point on the edge of the frustum) may be captured.

[0179] A desirable by-product of this two-step test is that for each brick that intersects with the camera frustum 404, it can be known whether it is fully or only partially inside the camera frustum 404. For those bricks that are fully inside the camera frustum 404, in subsequent individual voxel updates, the per-voxel test for inside the camera frustum 404 can be skipped.

[0180] 14 illustrates an example method 706 for performing a camera frustum acceptance test, according to some embodiments. The method 706 may begin by testing each of one or more sub-AABBs against each of the planes that bound the camera frustum 404 (act 802). In act 806, method 706 may determine whether the sub-AABB being tested is entirely outside of camera frustum 404. In act 806, if it is determined that the sub-AABB being tested is entirely outside of camera frustum 404, method 706 may cull all bricks contained by the sub-AABB being tested. In act 808, if it is determined that the sub-AABB being tested is not entirely outside of camera frustum 404, method 706 may determine whether the sub-AABB being tested is entirely inside camera frustum 404.

[0181] In act 810, if it is determined that the sub-AABB being tested is entirely inside the camera frustum 404, the method 706 may add all bricks contained by the sub-AABB being tested to the first one or more bricks. In act 708, if it is determined that the sub-AABB being tested is not entirely inside the camera frustum 404 (which may indicate that the sub-AABB being tested intersects with the camera frustum 404), the method 706 may determine whether the sub-AABB being tested reaches the size of a brick.

[0182] In act 814, if it is determined that the sub-AABB being tested is equal to the size of a brick, the method 706 may further determine whether all corner points of the camera frustum 404 are inside the sub-AABB being tested. If it is determined that all corner points of the camera frustum 404 are inside the sub-AABB being tested, the method 706 may thin out the bricks of the sub-AABB being tested (act 806). If it is determined that not all corner points of the camera frustum are inside the bricks of the sub-AABB being tested, the method 706 may add the bricks of the sub-AABB being tested to the first one or more bricks (act 810).

[0183] 15 illustrates an example method 608 for thinning a portion of a first one or more bricks for a depth image, according to some embodiments. The method 608 may begin by performing a first depth image acceptance test for each of the first one or more bricks (act 902). In act 904, the method 808 may determine whether the tested brick is accepted by the first depth image acceptance test. If the tested brick is determined to be accepted by the first depth image acceptance test (which may indicate that the tested brick intersects with a surface in the scene), the method 608 may apply a variable increment to the selected voxel (act 906) and add the tested brick to the second one or more bricks (act 914).

[0184] In act 908, if it is determined that the tested brick is not approved by the first depth image approval test, the method 608 may perform a second depth image approval test on the tested brick (act 908). In act 910, it is determined whether the tested brick is approved by the second depth image approval test. If it is determined that the tested brick is approved by the second depth image approval test (which may indicate that the tested brick is in front of a solid or perforated background in the scene), the method 608 may apply a constant increment to all or selected voxels (act 912) and add the tested brick to the second one or more bricks (act 914). If it is determined that the tested brick is also not approved by the second depth image approval test, the method 608 may thin out the tested brick (act 916).

[0185] 16 illustrates an example method 902 for performing a first depth image acceptance test, according to some embodiments. For each brick to be tested, the method 902 may begin by determining a minimum brick value (bmin) and a maximum brick value (bmax) along a direction parallel to the z-coordinate (act 1002). The bmin and bmax values ​​may be padded to account for an integral threshold, beyond which depth values ​​indicate a constant update to voxels within the brick. In act 1004, the method 902 projects the corners of the brick into the depth image. In act 1006, the method 902 may calculate a rectangle by building a convex hull of the 2D pixel locations of the corners of the brick. In act 1008, the method 902 may test each pixel in the rectangle against the bmin and bmax values. In act 1010, the method 902 may determine whether all pixels in the rectangle have depth values ​​between the bmin and bmax values. If it is determined that all pixels in the rectangle have depth values ​​between the bmin and bmax values, the method 902 may approve the brick (act 1012). If it is determined that all pixels in the rectangle do not have depth values ​​between the bmin and bmax values, the method 902 may perform a second depth image approval test for the brick (act 908).

[0186] FIG. 17 illustrates an example method 908 for performing a second depth image acceptance test, according to some embodiments. The method 908 may begin in act 1102 by categorizing all pixels within a rectangle for bmin and bmax values. In act 1104, the method 908 may determine whether the tested brick is in front of a solid or perforated background, for example, by using a table shown in FIG. 18. If it is determined that the tested brick is in front of a solid or perforated background, the method 908 may accept the brick (act 1106). If it is determined that the tested brick is not in front of a solid or perforated background, the method 908 may thin out the brick in act 916.

[0187] 19A-F depict an example of thinning bricks representing a scene 190 relative to a camera frustum 192. In FIG. 19A, the scene 190 is represented by a single AABB 194a, which in the illustrated example includes a 16×16 brick. In FIG. 19B, the single AABB 194a is split into four sub-AABBs 194b, each including an 8×8 brick. After performing a camera frustum acceptance test (e.g., method 706), one of the four sub-AABBs 194b fails the camera frustum acceptance test, and thus the 8×8 bricks in the failing sub-AABB 194b are thinned out and illustrated as white bricks. In FIG. 19C, each of the three sub-AABBs 194 that pass the camera frustum acceptance test are further split into four sub-AABBs 194c, each including a 4×4 brick. After performing the camera frustum acceptance test (e.g., method 706), 8 of the 16 sub-AABBs 194c fail the camera frustum acceptance test, and therefore the bricks in the failing sub-AABBs 194c are thinned out and illustrated as white bricks. Similarly, in FIG. 19D, sub-AABB 194d includes a 2×2 brick. In FIG. 19E, sub-AABB 194e includes a single brick, and therefore sub-AABB 194e and the individual bricks that passed the camera frustum test are produced as a first plurality of bricks, and in FIG. 19F, illustrated as gray bricks 196f. In the illustrated embodiment, if thinning was not performed, the world reconstruction component would compute all of the 256 bricks. With brick thinning on the camera frustum, the world reconstruction component would only need to compute the first plurality of bricks, i.e., 34 bricks, and would therefore render the result much faster.

[0188] 20A illustrates an example of further thinning 34 bricks 196f for a depth image that includes surface 220, e.g., by performing method 608. FIG. 20B illustrates the results of thinning for the depth image, showing that 12 bricks 222a of the 34 bricks 196f pass a first depth image acceptance test (e.g., method 904), 9 bricks 222b of the 34 bricks 196f pass a second depth image acceptance test (e.g., method 910), and finally, after thinning for the depth image with surface 220, 13 bricks 222c of the 34 bricks 196f have been thinned. As a result, in the illustrated example, using brick thinning for the depth image, the number of bricks calculated by the world reconstruction component is Further, the number of bricks is reduced to 21 bricks. It should be understood that the computation speed of the world reconstruction component can also be accelerated not only by the reduced number of bricks, but also by the classification of the bricks as a result of the first and second depth image acceptance tests. For example, as discussed with respect to FIG. 15, a fixed increment may be applied to the 9 bricks 222b that failed the first depth image acceptance test but passed the second depth image acceptance test. Applying a fixed increment in bulk may also improve the computation speed compared to applying a variable increment to each voxel. The geometry (e.g., planes) in the scene, such as a wall for placing a virtual screen and / or a floor for navigating a virtual robot, may be acquired in the XR system to support the application. A common representation of the geometry of the scene is a mesh, which may comprise a group of connected triangles having vertices and edges. Conventionally, geometry in a scene is obtained by generating a mesh for the scene and searching for geometry in the mesh, which takes time for processing, e.g., several seconds, and does not show the relationship between the geometry requested by different queries. For example, a first query may be for a table plane. In response to the first query, the system may find the table plane and leave the watch on the table plane. Then, a second query may be for the watch. In response to the second query, the system may find all possible table planes and check whether the watch is on each table plane until it finds the watch, since there is no indication from the response to the first query that the table plane is a table plane.

[0189] Described herein is a geometry extraction system. In some embodiments, the geometry extraction system may extract geometry while scanning a scene using a camera and / or sensor, which allows for fast efficient extraction and can adapt to dynamic environmental changes. In some embodiments, the geometry extraction system may persist the extracted geometry in local and / or remote memory. The persisted geometry may be accompanied by a unique identification, such that different queries, e.g., at different timestamps and / or from different applications, may share the persisted geometry. In some embodiments, the geometry extraction system may support different representations of the geometry according to individual queries. A plane is used as an exemplary geometry in the following description of Figures 21-29. It should be understood that instead of or in addition to a plane, the geometry extraction system may detect other geometries, including semantics such as, for example, cylinders, cubes, lines, corners, or glass surfaces or holes, and use them in subsequent processing. In some embodiments, the principles described herein with respect to geometry extraction may be applicable to object extraction and the like.

[0190] 21 illustrates a plane extraction system 1300 according to some embodiments. The plane extraction system 1300 may include a depth fusion 1304, which may receive multiple depth maps 1302. The multiple depth maps 1302 may be created by one or more users wearing depth sensors and / or downloaded from local / remote memory. The multiple depth maps 1302 may represent multiple viewpoints of the same surface. Differences may exist between the multiple depth maps, which may be resolved by the depth fusion 1304.

[0191] In some embodiments, depth fusion 1304 may generate SDF 1306 based, at least in part, on method 600. Mesh bricks 1308 may be extracted from SDF 1306, for example, by applying a marching cubes algorithm over corresponding bricks (e.g., bricks

[0000] -

[0015] in FIG. 23). Plane extraction 1310 may detect planar surfaces within mesh bricks 1308 and extract planes based, at least in part, on mesh bricks 1308. Planes Extraction 1310 may also extract surfels, brick by brick, based at least in part on the corresponding mesh brick. Surfels may include vertices in the mesh, but not edges connecting neighboring vertices, so that storing surfels consumes less memory space than the mesh. Plane data store 1312 may persist the extracted planes and surfels.

[0192] In some embodiments, an XR application may request and obtain a plane from the plane data store 1312 by a plane query 1314, which may be sent by an application programming interface (API). For example, an application may send information about its location to the plane extraction system 1300 and ask about all planes in its vicinity (e.g., within a 5 meter radius). The plane extraction system 1300 may then search its plane data store 1312 and send the selected plane to the application. The plane query 1314 may include information such as where the application needs the plane, the type of plane the application needs, and / or how the plane should look (e.g., whether horizontal or vertical or angled, which may be determined by checking the primitive normals of the plane in the plane data store).

[0193] 22 shows a portion 1400 of the plane extraction system 1300 illustrating details regarding plane extraction 1310, according to some embodiments. Plane extraction 1310 may include dividing each of the mesh bricks 1308 into sub-bricks 1402. Plane detection 1404 may be performed for each sub-brick 1402. For example, plane detection 1404 may compare primitive normals of each mesh triangle within a sub-brick, merge those mesh triangles with primitive normal difference less than a predetermined threshold into one mesh triangle, and identify mesh triangles with area greater than a predetermined area value as planes.

[0194] 23 is a schematic diagram illustrating a scene 1500, represented by bricks 0000-0015, including voxels, and example plane data, including a brick plane 1502, a global plane 1504, and surfels 1506 in the scene, according to some embodiments. FIG. 23 illustrates a brick 0011, which is divided into four sub-bricks 1508. It should be understood that a mesh brick may be divided into any suitable number of sub-bricks. The granularity of the planes detected by plane detection 1404 may be determined by the size of the sub-bricks, while the size of the bricks may be determined by the granularity of the local / remote memory that stores the volumetric 3D reconstruction data.

[0195] 22, plane detection 1404 may determine a brick plane (e.g., brick plane 1502) for each mesh brick based, at least in part, on the planes detected for each sub-brick within the mesh brick. Plane detection 1404 may also determine a global plane (e.g., global plane 1504) that extends across more than one brick.

[0196] In some embodiments, plane extraction 1310 may include plane updating 1406, which may update existing brick planes and / or global planes stored in plane data store 1312 based at least in part on planes detected by plane detection 1404. Plane updating 1406 may include adding additional brick planes, removing some of the existing brick planes, and / or replacing some of the existing brick planes with brick planes detected by plane detection 1404 and corresponding to the same brick, such that real-time changes in the scene are persisted in plane data store 1312. Plane updating 1406 may also include aggregating brick planes detected by plane detection 1404 into the existing global plane, for example, when a brick plane is detected adjacent to an existing global plane.

[0197] In some embodiments, plane extraction 1310 may further include plane merging and splitting 1408. Plane merging may merge multiple global planes into one larger global plane, for example, when a brick plane is added to connect two global planes. Plane splitting may split one global plane into multiple global planes, for example, when a brick plane in the middle of a global plane is removed.

[0198] FIG. 24 illustrates a data structure in the plane data store 1312, according to some embodiments. A global plane 1614, indexed by plane ID 1612, may be at the highest level of the data structure. While one brick plane may be persisted per brick, each global plane 1614 may contain multiple brick planes and surfels of bricks adjacent to the corresponding global plane, so that the global plane may be rendered correctly when an edge of the global plane does not qualify as a brick plane for the corresponding brick. In some embodiments, surfels of bricks adjacent to the global plane are persisted instead of surfels of all bricks in the scene, since these will be sufficient to render the global plane correctly. For example, as illustrated in FIG. 23, a global plane 1504 extends across bricks

[0008] -

[0010] and

[0006] . Brick

[0006] has a brick plane 1502 that is not part of the global plane 1504. Using the data structures in the plane data store 1312, when a plane query requests the global plane 1504, surfels for bricks

[0006] and

[0012] are checked to determine whether the global plane 1504 extends into bricks

[0006] and

[0012] . In the illustrated example, surfel 1506 indicates that the global plane 1504 extends into brick

[0006] .

[0199] 24 , the global plane 1614 may be bidirectionally associated with the corresponding brick plane 1610. The bricks may be identifiable by a brick ID 1602. The bricks may be divided into planar bricks 1604, which contain at least one plane, and non-planar bricks 1606, which do not contain a plane. Surfels for both planar and non-planar bricks may be persisted depending on whether the brick is adjacent to the global plane, not whether the brick contains a plane. It should be understood that the planes may be persisted continuously in the plane data store 1312 while the XR system observes the scene, regardless of whether a plane query 1314 is present.

[0200] 25 illustrates a plane geometry extraction 1702 that may extract planes for use by an application when the application sends a plane query 1314 to the plane data store 1312, according to some embodiments. The plane geometry extraction 1702 may be implemented as an API. The plane query 1314 may indicate a requested planar geometry representation, for example, an outer rectangular plane, an inner rectangular plane, or a polygonal plane. In accordance with the plane query 1314, the plane search 1704 may search and retrieve plane data in the plane data store 1312.

[0201] In some embodiments, rasterization from the planar cover points 1706 may generate planar cover points. An example is illustrated in Figure 26A. There are four bricks, 0000-0003, each with a brick plane 1802. The planar cover points 1806 (or "rasterized points") are generated by projecting the boundary points of the brick planes onto a global plane 1804.

[0202] 25, rasterization from the planar coverage points 1706 may also generate a rasterized planar mask from the planar coverage points. According to the planar geometry representation requested by the planar query 1314, an inner rectangle planar representation, an outer rectangle planar representation, and a polygonal planar representation may be generated. They may be extracted by extract inner rectangle 1708, extract outer rectangle 1710, and extract polygon 1712, respectively. In some embodiments, an application can receive the requested planar geometry representation within milliseconds of sending a planar query.

[0203] An exemplary rasterized planar mask 1814 is illustrated in FIG. 26B. From the rasterized planar mask, various planar geometric shape representations can be generated. In the illustrated example, a polygon 1812 is generated by connecting some of the planar coverage points of the rasterized planar mask such that none of the planar coverage points in the mask are outside the polygon. An outer rectangle 1808 is generated such that the outer rectangle 1808 is the smallest rectangle that encloses the rasterized planar mask 1814. The inner rectangle 1810 is generated by assigning a "1" to bricks with two planar coverage points and a "0" to bricks without two planar coverage points, forming a rasterization grid, determining groups of bricks that are marked as "1" and aligned within lines parallel to the edges of the bricks (e.g., bricks

[0001] ,

[0005] ,

[0009] , and

[0013] as a group, bricks

[0013] -

[0015] as a group), such that the inner rectangle is the smallest rectangle that encloses the respective group, and generating one inner rectangle for each determined group.

[0204] Figure 27 shows a mesh for a scene 1900, according to some embodiments. Figures 28A-C show the scene 1900, represented by an outer rectangular plane, an inner rectangular plane, and a polygonal plane, respectively, according to some embodiments.

[0205] FIG. 29 illustrates a less noisy 3D representation of a scene 1900, which is obtained by flattening the mesh shown in FIG. 27 based on extracted plane data, e.g., the planes illustrated in FIGS. 28A-C.

[0206] Multi-step block mesh simplification

[0207] In some embodiments, processing may be employed to reduce the complexity of the representation of the XR environment before the representation is stored or used in rendering functions such as handling occlusion or calculating the physics of interactions between objects in the XR environment. For example, meshing component 160d may simplify a mesh or part of a partial mesh before storing it as mesh 162c in persisted world 162.

[0208] Such processing may involve performing operations on the representation of the XR environment in stages. The stages may include simplification operations preceding and following region-based operations. The region-based operations, like the simplification operations, may reduce the complexity of the representation of the XR environment. By staging the operations in this manner, the total processing to produce the simplified representation of the XR environment may be reduced while maintaining the quality of the representation of the XR environment. As a result, the simplified high-quality representation may be updated frequently so that the XR environment may be updated frequently, improving the performance of the XR system, such as by presenting a more realistic environment to the user.

[0209] The XR environment may represent the physical world, and data representing the XR environment may be captured by one or more sensors. However, techniques as described herein may be applied to the XR environment regardless of the source of the data representing the environment. In some embodiments, the XR environment may be represented by a mesh, including one or more points and polygons, such as triangles, defined by a subset of points. A first simplification operation, preceding the region-based operation, may reduce the complexity of the representation of the environment. The mesh may be simplified, for example, by reducing the number of such polygons in the mesh. As a specific example, the first simplification operation may employ a triangle reduction algorithm, which may reduce the number of triangles used to represent the XR environment.

[0210] The region-based operation may be a shape detection operation, which may detect one or more shapes. A common shape detection operation is a planarization operation, where the detected shapes are planes. The detected planes may represent objects or portions of objects. Detecting planes may simplify processing for rendering the XR environment. Moving objects being rendered in the XR environment may move in an easily calculable manner, for example, when colliding with a plane. Thus, identifying a plane may simplify subsequent rendering of the moving object, compared to performing calculations based on multiple polygons representing the same portion of the environment. Other shapes, including semantics such as cylinders, cubes, lines, corners, or glass surfaces or holes, may also be detected and used in subsequent processing instead of or in addition to a plane. Such an operation may group polygons that represent the surfaces of the detected shapes.

[0211] A second simplification operation, following the region-based operation, may also further simplify the representation of the environment, such as by further reducing the number of polygons in the representation. The second simplification operation may focus on reducing the number of polygons in each region detected by the region-based operation.

[0212] Such processing may enable a meshing service that processes sensor data collected in the physical environment and provides a mesh to content generating applications. In some embodiments, the processing may provide simplified representations of virtual objects in the virtual environment.

[0213] In XR systems, such as virtual reality (VR), augmented reality (AR), and mixed reality (MR) systems, three-dimensional (3D) mesh data is often used for multiple purposes, including, for example, to occlude virtual content in a graphics / game engine based on physical objects in the environment, or to calculate effective rigid-body collisions for virtual objects in a game engine's physics engine. In some embodiments, the requirements for a mesh may be different for different uses of the mesh, and simplified meshes may be suitable for many such uses, with some simplification techniques being more suitable for some uses than others.

[0214] Thus, the processing as described herein may be implemented using any of a number of simplification techniques and / or simplification techniques that may be configured based on the intended use of the simplified mesh. The utility of a meshing service that provides simplified meshes to multiple client applications that may use the mesh in different ways may be improved using processing as described herein. Each client application may request a mesh with a different level of simplification. In some embodiments, an application that accesses the meshing service may specify a target simplification or mesh to be provided to it. Mesh simplification methods as described herein may be useful for multiple client applications, including, for example, those that perform virtual content occlusion, physics simulation, or environment geometry visualization. The mesh processing described herein may have low latency and may be flexible in that it may optimize / bias operation for different uses (e.g., flattened surfaces, variable triangle counts).

[0215] Mesh simplification methods as described herein may provide real-time performance (e.g., low latency to support on-the-fly (real-time) environmental changes), local update capabilities (e.g., refreshing portions of the mesh that have changed since the last update), and flattened surfaces (e.g., planar surfaces that are flattened to support robust physics simulations).

[0216] In some embodiments, the representation of the XR environment may be segmented into multiple blocks, some or all of which may be processed in parallel. In some embodiments, the resulting blocks may then be recombined. In some embodiments, blocks may be defined with "skirts" that overlap adjacent blocks. Skirts allow for recombination of blocks with fewer and / or less noticeable disruptions to the interfaces of the blocks being reassembled.

[0217] Thus, in some embodiments, mesh simplification methods may include mesh block segmentation, pre-simplification, mesh flattening, and post-simplification. To speed up the process, the global mesh may first be segmented into blocks of component meshes so that the mesh blocks can be handled (e.g., processed) in parallel. The mesh blocks may then be extended with skirts that span the boundaries between adjacent blocks. With skirted mesh blocks, simplification may be performed on individual mesh blocks while the global mesh may be topologically disconnected but visually seamless.

[0218] In some embodiments, the mesh simplification method may be suitable for use by applications that use simplified meshes to represent object interactions in an XR environment, such as by performing a simplification process plane recognition. To simplify the mesh, a three-step simplification process may be implemented. The mesh may first be moderately pre-simplified using a relatively high target triangle count. Planar areas may then be detected by a region growing algorithm. The mesh may be flattened by projecting corresponding triangles onto the detected plane. In some embodiments, the mesh may be regularized by adjusting the plane (or primitive) normals to be approximately perpendicular and parallel to the detected plane. A post-simplification process may then be launched over the flattened mesh. The post-simplification process may focus more on the detected planar areas, for example, simplifying the mesh of each detected planar area to reach a desired level of complexity (e.g., a complexity metric) as may be indicated by a target value of one or more metrics.

[0219] FIG. 30 illustrates a method 3000 of generating a model of an environment represented by a mesh, according to some embodiments. In some embodiments, the method 3000 may be implemented on a meshing service on an XR platform. The method 3000 may begin, at act 3002, with an input mesh representing the environment. In some embodiments, the input mesh may have a high resolution, which may be indicated by the number of triangles. The input mesh may be generated by a reconstruction system (e.g., a stereoscopic 3D reconstruction system), and the input mesh may include 3D reconstruction data.

[0220] In some embodiments, the reconstruction system may generate a stereoscopic 3D representation of the environment, which may create a data hierarchy of 3D information of the environment captured by one or more sensors. For example, the sensor may be a depth camera, which may capture a stream of 3D information of the environment, e.g., depth images, using a distinct pose (i.e., camera pose) of the depth camera. The 3D information of the environment may be processed into a voxel grid. Each voxel may contain one or more signed distance functions (SDFs) that describe whether the voxel is inside or outside the geometry of an object in the environment. The voxels may be grouped into "bricks." Each brick may contain, for example, 8 3 A brick may contain multiple voxels within a cubic volume, such as a voxel. Bricks may be further grouped into "tiles." Each tile may contain multiple bricks.

[0221] The size of the tiles may be selected to facilitate memory operations within a computing device, for example by accommodating the active memory of the device that processes such data. The tiles may be selected based on the amount of information about the environment that is maintained in the active memory. The system may, for example, move tiles between active memory, typically local to the device, and other memories that have more latency, such as non-volatile memory or remote memory in the cloud. One or more whole or partial tiles may contain information that represents a "block" in a mesh or other representation of the environment.

[0222] In some embodiments, the volumetric 3D reconstruction system may generate the input mesh 3002 as a topologically connected global mesh. In some embodiments, the volumetric 3D reconstruction system may generate the input mesh 3002 as a visually seamless, but topologically disconnected, global mesh. For example, a topologically disconnected global mesh may be composed of multiple mesh blocks, each generated from a block.

[0223] The reconstruction system may be configured to capture substantial detail of the environment, which allows the system to distinguish adjacent portions of the representation that have relatively small differences in their characteristics. Adjacent regions of different nature may be identified as different surfaces, resulting in the system identifying multiple surfaces in the environment. However, such a system may capture detail that is unnecessary for many applications, but is processed nonetheless. For example, the reconstruction system may unnecessarily present irregularities with many triangles on a wall, when two triangles that make a rectangle would be a sufficient representation of the wall for a client application that requests the mesh from the meshing service. In some embodiments, when an application requests a mesh from the meshing service, it may specify a target simplification level of the requested mesh. The target simplification level may be expressed in a degree of compression, a number of triangles per unit area, or any other suitable manner.

[0224] From the input mesh, the method 3000 may efficiently generate a model of the environment that is sufficient for the client application. At act 3004, the input mesh may be segmented into one or more first mesh blocks, each of which may correspond to a block in the data hierarchy of the stereoscopic 3D representation of the environment.

[0225] Each first mesh block may represent a portion of the environment and may have a first value of a complexity metric (e.g., mesh resolution). In some embodiments, the complexity metric of a mesh block indicates the number of triangles in the mesh block. In some embodiments, processing may be performed on mesh blocks in a sequential and / or parallel manner. However, simplification processing as described herein may be applied to the entire mesh or any suitable portion (e.g., one or more mesh blocks).

[0226] Act 3006 represents a sub-process that is performed on each of the multiple mesh blocks. The sub-process may be performed on multiple mesh blocks independently, such that processing may be easily performed in parallel for some or all of the mesh blocks. The sub-process may be performed on all mesh blocks or a subset of the mesh blocks selected for further processing. The subset of mesh blocks may be selected based, at least in part, on the field of view of the device on which the application requesting the simplified mesh is executed.

[0227] In act 3006, some of the first mesh blocks may be selected, for example, based on the location of the first mesh block or the object depicted in the first mesh block. For each selected first mesh block, a multi-stage simplification may be performed. In some embodiments, the multi-stage simplification on the selected first mesh blocks may be performed in parallel, such that the simplification on the selected first mesh blocks may be performed in parallel. The simplification of may be completed at approximately the same time, which may depend on the complexity metric of each mesh block of the selected first mesh block.

[0228] The multi-stage simplification may include a pre-simplification operation, a region-based operation (e.g., a flattening operation), and a post-simplification operation. In some embodiments, the multi-stage simplification may be performed based on input values ​​from a client application. The input values ​​may indicate a mesh complexity (e.g., mesh resolution) required by the client application. The input values ​​from the client application may be the same or different for each selected first mesh block.

[0229] In act 3012, a pre-simplification operation may be performed on the selected first mesh block to generate a second mesh block. The pre-simplification operation may reduce the complexity of the block. For mesh blocks, the pre-simplification may reduce the number of polygons in the mesh block. In some embodiments, the amount of pre-simplification in act 3012 may be configurable. A target value may be provided as an input to the process, for example, in act 3012, for example, by a client application. The target value may be a single value or multiple values ​​of one or more prescribed or predefined metrics. The one or more metrics may include, for example, an absolute triangle count, a percentage of the initial triangle count, and / or a quadratic error metric, which may measure the mean squared distance between the simplified mesh and the original mesh (e.g., input mesh 3002).

[0230] The target values ​​may be provided in any suitable manner. An instance of method 3000 may, for example, be pre-configured with the target values. In some embodiments, the target values ​​may be supplied through an API by an application that requests a mesh from a meshing service that implements method 3000. For example, the target values ​​for act 3012 may be the final targets requested by a rendering function (e.g., a requesting application). In some embodiments, the target values ​​provided as input may be adjusted or overwritten to ensure that enough data remains in the mesh for subsequent processing. For example, processing in act 3014 may require a minimum number of triangles, and the target value provided by the application may be replaced by the minimum value if the target value falls below the minimum number of triangles.

[0231] In such embodiments, the pre-simplified mesh may have values ​​of one or more metrics such that the pre-simplified mesh may be processed faster than the segmented input mesh of the original block during region-based operations while still containing all or most of the region of the segmented input mesh of the original block.

[0232] If the values ​​of one or more metrics are not controlled, the simplified mesh may be too coarse, non-uniformly distributed, and / or miss many regions of the segmented input mesh of the original block that are required in subsequent region-based operations.

[0233] The second mesh block produced in act 3012 may have a second value of the complexity metric, which may be less than the first value of the complexity metric. In some embodiments, the pre-simplification operation of act 3012 may be performed using a triangle reduction algorithm.

[0234] In act 3014, a shape detection operation may be performed on the second mesh block to generate a third mesh block. A planarization operation is illustrated as an example. The third mesh block may have a third value of the complexity metric. In some embodiments, the third value of the complexity metric may be the same as the second value of the complexity metric. In some embodiments, the third value of the complexity metric may be less than the second value of the complexity metric. The planarization operation may include detecting planar areas in the second mesh block, e.g., using a region growing algorithm, projecting meshes of the detected planar areas onto the corresponding planes, adjusting plane normals of the detected planar areas to be approximately perpendicular to the corresponding planes, and simplifying the projected mesh on each of the corresponding planes, e.g., based on a target triangle count. In some embodiments, the plane normals of the detected planar areas may be adjusted prior to projecting the meshes of the detected planar areas onto the corresponding planes.

[0235] In act 3016, a post simplification operation may be performed on the third mesh block to generate a fourth mesh block. In some embodiments, the processing in act 3014 may desirably be performed on a mesh with a resolution greater than that required in the simplified mesh that will be output from method 3000. In some embodiments, the processing in act 3016 may simplify the entire mesh block to reach a desired level of complexity (e.g., a complexity metric) as may be indicated by target values ​​of one or more metrics, which may be the same or different targets as those provided to act 3012. In some embodiments, the post simplification operation in act 3016 may focus on reducing the number of polygons in each plane detected by the planarization operation in act 3014.

[0236] The fourth mesh block may have a fourth value of the complexity metric, which may be less than the third value of the complexity metric. In some embodiments, the percentage reduction between the third value of the complexity metric and the fourth value of the complexity metric may be greater than the percentage reduction between the first value of the complexity metric and the second value of the complexity metric. In some embodiments, the percentage reduction between the third value of the complexity metric and the fourth value of the complexity metric may be at least two times greater than the percentage reduction between the first value of the complexity metric and the second value of the complexity metric. In some embodiments, the post-simplification operation of act 3016 may be performed using a triangle reduction algorithm. In some embodiments, the post-simplification operation of act 3016 may be performed using the same simplification algorithm as the pre-simplification operation of act 3012.

[0237] In act 3008, the simplified selected blocks may be combined with other selected mesh blocks similarly processed as in act 3006, and / or may be combined with unselected blocks into a new mesh of the environment. In act 3010, the new mesh of the environment may be provided to the client application. In some embodiments, the new mesh of the environment may be referred to as a simplified mesh.

[0238] In some embodiments, act 3008 may be skipped. The simplified mesh blocks may be sent directly to the client application, where the mesh blocks may be visually seamless, but topologically disconnected.

[0239] FIG. 31 illustrates an example of segmenting a mesh representation 3100 of an environment into mesh blocks, according to some embodiments. The mesh representation 3100 may be segmented into four mesh blocks, namely mesh blocks AD. In some embodiments, the mesh blocks may correspond to regions with the same volume in the physical world that belongs to the environment. In some embodiments, the mesh blocks may correspond to regions with different volumes in the physical world that belongs to the environment. For example, when the physical world is an office room, the office room may be divided into regions, each of which may be one cubic foot. One block contains the 3D representation of one area of ​​the office room. That's fine.

[0240] Although the mesh representation 3100 of the environment is illustrated in two dimensions (2D), it should be understood that the environment may be three-dimensional, correspondingly represented by a 3D mesh representation. Although the mesh representation 3100 of the environment is illustrated as a combination of four mesh blocks, it should be understood that the environment may be represented by any suitable number of mesh blocks, for example, two, three, five, six, or more.

[0241] Representation 3100 may be segmented, for example, into four portions, namely portions 3102, 3104, 3106, and 3108, illustrated by solid lines 3110. In some embodiments, portions 3102, 3104, 3106, and 3108 may each be designated as a mesh block AD.

[0242] When a mesh block is updated, it may continue to have interface contact with adjacent blocks that were not updated. As a result, discontinuities at the boundaries between mesh blocks may occur. If the area represented by adjacent blocks has a discontinuity, the fused mesh may be interpreted in subsequent processing as indicating that a crack exists between the adjacent blocks. In some embodiments, such a crack in the representation of the physical world space may be interpreted as a space with infinite depth. Thus, the space may not actually be a feature, but an artifact of the representation of the physical world. Any application that uses such a fused mesh to generate a representation of an object in the physical world may not correctly generate output. For example, an application that renders a virtual character on a surface in the physical world may render the character to appear as if it were falling through the crack, which would not create the desired appearance of the object.

[0243] To reduce the appearance of such cracks, in some embodiments, portions of adjacent blocks may represent the same region of the physical world. For example, the interface region between adjacent blocks may be represented by a portion of each of the adjacent blocks, which may allow for easy independent updating and / or rendering that takes into account the level of detail (LOD) (e.g., reducing the complexity of the 3D reconstruction of a portion of the physical world as that portion moves out of the user's field of view). Even if one block is updated and its adjacent block is not updated, the fused mesh may be represented by combining data representing the interface region from both of the blocks. As a specific example, when fusing an updated block with an adjacent block, the physics engine may determine the overlap region of the adjacent blocks, for example, based on which of the adjacent blocks should be observable within that overlap region. A block-based data structure may be employed along with a skirt, zipper, or any other suitable method for representing the interface region between adjacent blocks, such that when a block is updated, it will remain in interface contact with an adjacent block that was not updated. Such an approach may have the appearance of "hiding" the crack between adjacent blocks. Thus, blocks may be updated independently of their adjacent blocks.

[0244] In the example illustrated in FIG. 31 , the areas at the boundaries of portions 3102, 3104, 3106, and 3108 may be designated as skirts, illustrated by dashed lines 3112. In some embodiments, mesh block A-D may each include one of portions 3102, 3104, 3106, and 3108 and a corresponding skirt. For example, mesh block B may include portion 3104 and skirt 3114 that overlaps with boundary portions of mesh block B adjacent mesh blocks A, C, and D such that cracks between the mesh blocks may be concealed when the blocks are connected into a single mesh. Mesh blocks A, C, and D may also include corresponding skirts. Therefore, the processor can mask out any cracks between mesh blocks before returning a single connected 3D mesh representation to the application.

[0245] In some embodiments, the block mesh, including the skirt, may be sent directly to the application without being combined so as to be topologically connected to the global mesh. The application may have a global mesh, made up of the block meshes, that is visually seamless, but may be topologically disconnected.

[0246] 32A-32D illustrate the mesh evolution of an example mesh block 3201 during multi-stage simplification, according to some embodiments. Mesh block 3201 may include vertices 3206, edges 3208, and faces 3210. Each face may have a normal, which may be represented by multiple coordinates, for example, shown in FIG. 32A as x, y, z.

[0247] A pre-simplification operation may be performed on mesh block 3201 to generate mesh block 3202. An edge collapse transformation may be used. In the illustrated example, mesh block 3202 reduces the number of faces of mesh block 3201 from 10 to 8. Each of the resulting faces of mesh block 3202 may have a corresponding set of normals (e.g., x1, y1, z1; x2, y2, z2; ... x8, y8, z8).

[0248] A planarization operation may be performed on the mesh block 3202 to generate a mesh block 3203. The planarization operation may include, for example, detecting planar areas within the mesh 3202 based on the plane (or primitive) normals of the faces. Values ​​of the plane normals x1, y1, z1 of the first face 3212 and the plane normals x2, y2, z2 of the second face 3214 may be compared. A comparison result of the plane normals of the first and second faces may indicate an angle between the plane normals (e.g., the angle between x1 and x2). When the comparison result is within a threshold, it may be determined that the first and second planes are coplanar. In the illustrated example, planes 3212, 3214, 3216, and 3218 may be determined to be on a first planar area corresponding to plane 3228, and planes 3220, 3222, 3224, and 3226 may be determined to be on a second, coplanar area corresponding to plane 3230.

[0249] The planarization operation may also include projecting the triangles formed by the edges of planes 3212, 3214, 3216, and 3218 onto plane 3228 as illustrated by dashed line 3232, and projecting the triangles formed by the edges of planes 3220, 3222, 3224, and 3226 onto plane 3230 as illustrated by dashed line 3234. The planarization operation may also include adjusting the plane normals for planes 3212, 3214, 3216, and 3218 to be identical to the plane normal (x_a, y_a, z_a) for plane 3228, and adjusting the plane normals for planes 3220, 3222, 3224, and 3226 to be identical to the plane normal (x_b, y_b, z_b) for plane 3230.

[0250] A post-simplification operation may be performed on mesh block 3203 to produce mesh block 3204. In the illustrated example, mesh block 3204 reduces the number of faces of mesh block 3203 from eight to four.

[0251] 33A and 33B-36A and 36B illustrate the effect of simplification, showing side-by-side the same portion of the physical world with and without simplification applied. These figures provide a graphical illustration of how simplification can provide usable information for operating an AR system while providing less data that needs to be processed.

[0252] 33A and 33B show the same environment with and without simplification through triangle reduction, respectively. Such processing is an example of processing that may be implemented in pre-simplification and post-simplification blocks 3012 and 3016 in FIG.

[0253] 34A and 34B show zoomed-in representations of the same environment with and without simplification through triangle reduction, respectively.

[0254] 35A and 35B show representations of the same environment with and without flattening, respectively. Such processing is an example of processing that may be performed in flattening block 3014 in FIG.

[0255] 36A and 36B show representations of the same environment with and without simplification through removal of disconnected components, respectively. Such processing is an example of an alternative embodiment of the region-based operation that may be implemented in block 3014 in FIG.

[0256] Caching and updating dense 3D reconstruction data

[0257] In some embodiments, 3D reconstruction data may be captured, persisted, and updated on a block-by-block basis, which may enable local updates while maintaining neighborhood consistency. The block-based 3D reconstruction data representation may be used in conjunction with a multi-layer cache mechanism to efficiently retrieve, pre-fetch, and store 3D data for AR and MR applications, including both single-device and multi-device applications. For example, stereo information 162a and / or mesh 162c (FIG. 6) may be stored on a block-by-block basis. The usage component 164 may use this block-based representation to receive information about the physical world. Similarly, the perception component 160 may store and retrieve such information on a block-by-block basis.

[0258] These techniques extend the capabilities of portable devices with limited computing resources to present AR and MR content with high realism. Such techniques may be used, for example, to efficiently update and manage real-time or offline reconstruction and scanning outputs in mobile devices with limited resources and a connection (continuous or non-continuous) to the Internet. The techniques make up-to-date, accurate, and comprehensive 3D reconstruction data available to a variety of mobile AR and MR applications, either in single-device applications or multi-device applications that share and update the same 3D reconstruction data. These 3D reconstruction data may be in any suitable format, including meshes, point clouds, voxels, and the like.

[0259] Some AR and MR systems attempt to simplify the presentation of MR and AR scenes by limiting the amount of 3D reconstruction data being processed at any given time. The sensor used to capture 3D information may have a maximum reconstruction range that may define a bounding volume around the sensor's field of view. To reduce the amount of 3D reconstruction data, some reconstruction systems keep only the area near the sensor's field of view in active working memory and store other data in secondary storage. For example, the area near the sensor's field of view is stored in CPU memory, while other data is persisted in a local cache (e.g., disk) or remote storage (e.g., in the cloud) via a network.

[0260] The computational cost of generating the information stored in the CPU memory, even if limited, can still be relatively high. Some AR and MR systems continually recalculate the global representation of the environment previously seen by the reconstruction system in order to select information for storage in the CPU memory, which can be prohibitively expensive for interactive applications. Other AR and MR systems similarly use some approaches to compute only local updates to the connected representation, which can be expensive, especially for simplified meshes, since it requires decomposing the existing mesh, computing another mesh with the same boundary, and reconnecting the mesh parts.

[0261] In some embodiments, the 3D reconstruction data may be segmented into blocks. The 3D reconstruction data may be transmitted between storage media based on blocks. For example, blocks may be paged from active memory and persisted to a local or remote cache. The system may implement a paging algorithm in which an active memory associated with a wearable device (e.g., a head mounted display device) stores blocks representing a portion of the 3D reconstruction of the physical world within a field of view of a user of the wearable device. The wearable device may capture data about a portion of the physical world in conjunction with a field of view of a user of the wearable device. As the physical world changes within the user's field of view, the blocks representing that region of the physical world may be in active memory, from which they can be easily updated. As the user's field of view changes, blocks representing regions of the physical world that move out of the user's field of view may be moved to a cache such that blocks representing regions of the physical world that have come into the user's field of view may be loaded into the active memory.

[0262] In some embodiments, a coordinate frame may be created for the portion of the physical world to be 3D reconstructed, and each block in the 3D representation of the portion of the physical world may correspond to a different region of the physical world identifiable using the coordinate frame.

[0263] In some embodiments, when a block is updated, the updated block may continue to interface with adjacent blocks that may not be updated. If the areas represented by adjacent blocks do not overlap, cracks may exist in the fused meshes of adjacent blocks. In some embodiments, such cracks in the representation of the physical world space may be interpreted as spaces with infinite depth. Thus, the spaces may not actually be features, but rather artifacts of the representation of the physical world. Any application that uses such fused meshes to generate a representation of an object in the physical world may not generate the output correctly. For example, an application that renders a virtual character on a surface in the physical world may render the character to appear as if it were falling through a crack, which would not create the desired appearance of the object. Thus, in some embodiments, some of the adjacent blocks may represent the same region of the physical world, e.g., the interface between adjacent blocks, which may allow for easy independent updating and / or rendering that takes into account level of detail (LOD) (e.g., reducing the complexity of the 3D reconstruction of a portion of the physical world as that portion moves out of the user's field of view). For example, when a block is updated, its adjacent blocks may not be updated. When blending an updated block with an adjacent block, the physics engine may determine the overlapping area of ​​the adjacent blocks, for example, based on which of the adjacent blocks should be observable in that overlapping area. A block-based data structure may be employed with a skirt, zipper, or any other suitable method for representing the interface between adjacent blocks, such that when a block is updated, it will remain in interface contact with an adjacent block that was not updated. Such an approach may have the appearance of "unmasking" the cracks between adjacent blocks. Thus, the changed block may be updated independently of the adjacent blocks.

[0264] In some embodiments, these techniques may be used within an AR and / or MR “platform” that receives and processes data from sensors worn by one or more users. This sensor data creates and updates 3D reconstruction data, which is then used to A 3D reconstruction service may be used to represent a portion of the physical world encountered by a user. A reconstruction service may continually reconstruct a 3D representation of the physical world as sensors capture and update data. One or more techniques may be used to determine blocks affected by changes in the physical world, and those blocks may be updated. This 3D reconstruction data may then be provided to an application that uses the 3D reconstruction data to render the scene and depict virtual reality objects positioned in or interacting with objects in the physical world. This data may be provided to the application through an application programming interface (API). The API may be a push or pull interface that either pushes data to the application when relevant parts change, or in response to requests from the application for up-to-date information.

[0265] In an embodiment of a pull delivery interface, when an application requests 3D reconstruction data of the physical world, the reconstruction service may determine the appropriate version of each block that needs to be provided to the application, allowing the reconstruction service to start from the most recent block. The reconstruction service may, for example, retrieve blocks that were previously persisted. A single device system may allow a single device to contribute to 3D reconstruction data for the physical world. In a single device system, if a requested area of ​​the physical world is outside or extends outside of an active area (e.g., an area within the device's current field of view), the persisted block may be used directly as the most recent block because the persisted block would not have been updated since it was persisted when that area moved outside the device's field of view. On the other hand, a multi-device system may allow multiple devices to contribute to 3D reconstruction data for the physical world, such as by using cloud persistence or peer-to-peer local cache sharing of blocks. Each device may update the areas within its active area that may be persisted. A multi-device system may create a coordinate frame such that blocks generated by different devices are identifiable using the coordinate frame. Thus, blocks requested by an application generating data for a first device may be based on updates from other devices if those updates were made after any version made by the first device. Blocks constructed using data from the first device and other devices may be merged by using coordinate frames.

[0266] The selected block may be used to provide 3D reconstruction data for the physical world in any suitable format, but a mesh is used as an example of a suitable representation herein. The mesh may be created by processing the image data and identifying points of interest in the environment, such as edges of an object. These points may be connected to form a mesh. Points in the mesh associated with the same object or part thereof, conventionally a group of three points, define a surface of the object or part thereof. Information stored in conjunction with the group of points describes the surface in the environment. This information may then be used in various ways to render and / or display information for the environment. The selected block may be used to provide 3D reconstruction data in any suitable manner. In some embodiments, a newest block may be provided. In some embodiments, the newest block may be used to determine whether an update of the block is required.

[0267] For example, in some embodiments, in a multi-device system, when a block required by an application is identified, the reconfiguration service checks the blocks persisted by other devices, determines whether there are any significant updates (e.g., via a threshold or timestamp of the magnitude of the geometric change), restarts meshing on the blocks that have changed, and then persists these updated mesh blocks. This may be allowed.

[0268] In some embodiments, when the set of blocks required by an application is identified, if the application requires a connected mesh, the set of blocks may be processed into a global mesh that is topologically connected using any suitable technique, such as skirts and zippers, or that is visually seamless but may be topologically disconnected.

[0269] In some embodiments, an application (e.g., a graphics / game engine) may update its internal blocks (e.g., blocks stored in active memory and / or local cache) when a block change occurs. The reconstruction service may know what blocks the application has and therefore can calculate other (e.g., nearby) blocks that need to be updated in the engine to maintain the correct overlap with the skirt / zipper as an in-view block update.

[0270] In some embodiments, an AR and / or MR platform may be implemented, for example, on a mobile device, to support the execution of AR and / or MR applications. An application running on or generating data for presentation through a user interface may request 3D reconstruction data representing the physical world. The 3D reconstruction data may be sourced from an active memory on the device, which may be updated with 3D reconstruction data representing the physical world within the field of view of the device as the user changes its field of view. The 3D reconstruction data in the active memory may represent an active area for the mobile device. In some embodiments, the 3D reconstruction data outside the active area of ​​the device may be stored in other memory, such as in a local cache on the device or coupled to the device with a low latency connection. In some embodiments, the 3D reconstruction data outside the active area of ​​the device may also be stored in a remote cache, such as in the cloud, accessible to the device via a longer latency connection. As the user changes its field of view, the platform may access / load the 3D reconstruction data from the cache and add it to the active memory, representing areas that have moved into the user's field of view. The platform may move other data representing areas that have moved out of the user's field of view to the cache.

[0271] Predictions of device movements, which may result in areas moving into the device's field of view and other areas moving out of the device's field of view, may be used to initiate transfers of 3D reconstruction data between the active memory and the cache. Predictions of movement may be used, for example, to select 3D reconstruction data for transfer into and / or out of the active memory. In some embodiments, predicted movements may be used to transfer 3D reconstruction data into and / or out of the local cache by retrieving or transferring the 3D reconstruction data from a remote cache. Exchange of 3D reconstruction data between the local cache and the remote cache based on predicted movements of the user may ensure that the 3D reconstruction data is available with low latency for moving into the active memory.

[0272] In embodiments where regions of the physical world are represented using blocks, initiating transfer of blocks may involve pre-requesting blocks representing areas predicted to be within the user's field of view. For example, if the platform determines, based on sensor or other data, that a user is walking in a particular direction at a particular speed, it may identify areas that are likely to be within the user's field of view and transfer blocks representing these areas to a local cache on the mobile device. If the mobile device is a wearable device, such as a set of glasses, predicting movement may involve predicting the location, orientation, and / or rotation of the user's head. The method may involve receiving sensor data indicative of rotation.

[0273] FIG. 37 illustrates a system 3700 that enables an interactive X-reality environment for multiple users, according to some embodiments. The system 3700 may include a computing network 3705 consisting of one or more computer servers 3710 connected through one or more high-bandwidth interfaces 3715. The servers in the computing network need not be co-located. The one or more servers 3710 may each include one or more processors for executing program instructions. The servers also include memory for storing program instructions and data used and / or generated by processes performed by the servers under the direction of the program instructions. The system 3700 may include one or more devices 3720, including, for example, an AR display system 80 (e.g., the viewing optics assembly 48 in FIG. 3B).

[0274] The computing network 3705 communicates data between the servers 3710 and between the servers and the devices 3720 via one or more data network connections 3730. Examples of such data networks include, but are not limited to, all types of public and private data networks, both mobile and wired, including, for example, the many interconnections of such networks commonly referred to as the Internet. No particular medium, topology, or protocol is intended to be implied by the illustration.

[0275] In some embodiments, the devices may be configured to communicate directly with either the computing network 3705 or the server 3710. In some embodiments, the devices 3720 may communicate locally with the remote server 3710, and optionally with other devices and AR display systems, through a local gateway 3740 to process data and / or communicate data between the network 3705 and one or more devices 3720.

[0276] As shown, the gateway 3740 is implemented as a separate hardware component, including a processor for executing software instructions and memory for storing software instructions and data. The gateway has its own wired and / or wireless connection with a data network to communicate with a server 3710 comprising the computing network 3705. In some embodiments, the gateway 3740 can be integrated with a device 3720 worn or carried by a user. For example, the gateway 3740 may be implemented as a downloadable software application that is installed and runs on a processor included within the device 3720. The gateway 3740, in one embodiment, provides one or more users with access to the computing network 3705 via the data network 3730. In some embodiments, the gateway 3740 may include communication links 76 and 78.

[0277] The servers 3710 each include, for example, working memory and storage devices for storing data and software programs, a microprocessor for executing program instructions, a graphics processor, and other specialized processors for rendering and generating graphic, image, video, audio, and multimedia files. The computing network 3705 may also include devices for storing data that is accessed, used, or created by the servers 3710. In some embodiments, the computing network 3705 may include a remote processing module 72 and a remote data repository 74.

[0278] Software programs running on the server, and optionally on the device 3720 and gateway 3740, are used to generate a digital world (also referred to herein as a virtual world) with which the user interacts with the device 3720. The digital world is represented by data and processes that describe and / or define virtual non-existent entities, environments, and conditions that may be presented to the user through the device 3720 for the user to experience and interact with. For example, some types of objects, entities, or items that would appear to be physically present when instantiated within a scene viewed or experienced by the user may include descriptions of their appearance, their behavior, the way the user is allowed to interact with them, and other characteristics. Data used to create the environment of the virtual world (including virtual objects) may include, for example, atmospheric data, terrain data, weather data, temperature data, location data, and other data used to define and / or describe the virtual environment. In addition, data defining various conditions that govern the operation of the virtual world may include, for example, physical laws, time, spatial relationships, and other data that may be used to define and / or create various conditions that govern the operation of the virtual world (including virtual objects).

[0279] Entities, objects, conditions, properties, behaviors, or other features of the digital world will generally be referred to herein as objects (e.g., digital objects, virtual objects, rendered physical objects, etc.) unless otherwise indicated by context. Objects may be any type of animate or inanimate object, including, but not limited to, buildings, plants, vehicles, people, animals, living things, machines, data, videos, text, photos, and other users. Objects may also be defined in the digital world to store information about items, behaviors, or conditions that actually exist in the physical world. Data that describes or defines an entity, object, or item, or that stores its current state, is generally referred to herein as object data. This data is processed by the server 3710, or, depending on the implementation, by the gateway 3740 or the device 3720, to instantiate instances of the object and render the object in an appropriate manner for a user to experience through the device 3720.

[0280] Programmers who develop and / or curate a digital world create or define objects and the conditions under which they are instantiated. However, a digital world may allow others to create or modify objects. Once an object is instantiated, the state of the object may be allowed to be altered, controlled, or manipulated by one or more users experiencing the digital world.

[0281] For example, in one embodiment, the development, production, and management of the digital world is generally provided by one or more system administration programmers. In some embodiments, this may include the development, design, and / or execution of storylines, themes, and events within the digital world, and the distribution of storylines through various forms of events and media, such as, for example, film, digital, networked, mobile, augmented reality, and live entertainment. System administration programmers may also handle the technical management, coordination, and curation of the digital world and its associated user community, as well as other tasks typically performed by network administrators.

[0282] A user interacts with one or more digital worlds using some type of local computing device, generally designated as device 3720. Examples of such devices include, but are not limited to, smartphones, tablet devices, heads-up displays (HUDs), gaming consoles, or any other devices that communicate data and interact with the digital world. and any other device capable of providing a user with a user interface or display, and combinations of such devices. In some embodiments, device 3720 may include or communicate with local peripherals or input / output components, such as, for example, a keyboard, a mouse, a joystick, a game controller, a haptic interface device, a motion capture controller, audio equipment, voice equipment, a projector system, a 3D display, and holographic 3D contact lenses.

[0283] FIG. 38 is a schematic diagram illustrating an electronic system 3800, according to some embodiments. In some embodiments, the system 3800 may be part of the system 3700 of FIG. 37. The system 3800 may include a first device 3810 (e.g., a first portable device of a first user) and a second device 3820 (e.g., a second portable device of a second user). The devices 3810 and 3820 may be, for example, the device 3720 and the AR display system 80 of FIG. 37. The devices 3810 and 3820 may communicate with a cloud cache 3802 through networks 3804a and 3804b, respectively. In some embodiments, the cloud cache 3802 may be implemented in memory of one or more servers 3710 of FIG. 37. The networks 3804a and 3804b may be examples of the data network 3730 and / or the local gateway 3740 of FIG. 37.

[0284] The devices 3810 and 3820 may be individual AR systems (e.g., device 3720). In some embodiments, the devices 3810 and 3820 may include an AR display system worn by its respective user. In some embodiments, one of the devices 3810 and 3820 may be an AR display system worn by the user, and the other may be a smartphone held by the user. Although two devices 3810 and 3820 are illustrated in the example, it should be understood that the system 3800 may include one or more devices, and one or more devices may operate the same type of AR system or different types of AR systems.

[0285] The devices 3810 and 3820 may be portable computing devices. The first device 3810 may include, for example, a processor 3812, a local cache 3814, and one or more AR applications 3816. The processor 3812 may include a computing portion 3812a configured to execute computer-executable instructions and provide a 3D representation (e.g., 3D reconstruction data) of a portion of the physical world based, at least in part, on data collected using one or more sensors (e.g., the depth sensor 51, the world camera 52, and / or the inertial measurement unit 57 of FIG. 3B).

[0286] The computing portion 3812a may represent the physical world as one or more blocks. Each block may represent an object in a different region of the physical world. Each region may have a corresponding volume. In some embodiments, the blocks may represent regions with the same volume. In some embodiments, the blocks may represent regions with different volumes. For example, when the physical world is an office room, the office room may be divided into cubes, each of which may be one cubic foot. One block may contain a 3D representation (e.g., 3D reconstruction data) of one cubic foot of the office room. In some embodiments, the office room may be divided into regions with various volumes, and each volume may contain a similar amount of 3D information (e.g., 3D reconstruction data) such that the data size of the 3D representation of each region may be similar. The representation determines whether a virtual object is occluded by a physical object. The blocks may be formatted to facilitate further processing, such as occlusion processing to render a virtual object visible, or physics processing to determine how a virtual object should move or deform when interacting with a physical object in the physical world. The blocks may be formatted as mesh blocks, where features of objects in the physical world, such as corners, become points in the mesh block or are used as points to create mesh blocks. The connections between points in the mesh may represent groups of points on the same surface of the physical object.

[0287] Each block may have one or more versions, each version containing data (e.g., volumetric 3D reconstruction data such as voxels and / or meshes that may represent surfaces within the region represented by the individual blocks) that represent its corresponding region based on data from a point in time. Computing portion 3812a may create new versions of blocks as additional data becomes available, such as data indicating that objects in the physical world have changed or additional data from which a more accurate representation of the physical world may be created. The additional data may come from sensors on the device (e.g., devices 3810 and / or 3820). In some embodiments, the additional data may come from remote sensors, e.g., obtained via a network connection.

[0288] The processor 3812 may also include an active memory 3812b that may be configured to store blocks within the field of view of the device. In some embodiments, the active memory 3812b may store blocks outside the field of view of the device. In some embodiments, the active memory 3812b may store blocks adjacent to blocks within the field of view of the device. In some embodiments, the active memory 3812b may store blocks that are predicted to be within the field of view of the device. In some embodiments, the processor 3812 maintains a block in the active memory 3812b if it is within the field of view of the device at that time. The field of view may be determined by imaging an area of ​​one or more of the sensors. In some embodiments, the field of view may be determined by the amount of the physical world that is presented to a user of the device or that an average user may perceive without the use of the AR system. The field of view may thus depend on the location of the user within the physical world and the orientation of the wearable components of the device.

[0289] The processor 3812 may treat a block as inactive if it is out of view of the device 3810 as the user moves. Inactive blocks may be paged from active memory to a cache. The cache may be a local cache or a remote cache. In the embodiment of Figure 38, the block is first paged through the local gateway 3818b to the local cache 3814. In some embodiments, the local cache 3814 may be the only available cache.

[0290] In some embodiments, there may be a remote cache accessible via a network. In the illustrated embodiment, a cloud cache (e.g., remote cache) 3802, accessed through a network 3804a, is an example of a remote cache. The processor 3812 may manage when blocks are moved between the local cache 3814 and the cloud cache 3802. For example, when the local cache 3814 is full, the processor 3812 may page blocks to the cloud cache 3802 over the network 3804a. Because blocks in the local cache 3814 are accessible for rendering a scene with less latency than blocks in the cloud cache, the processor 3812 may attempt to keep in the local cache 3814 the blocks that are most likely to be active. An algorithm, as shown, may be used to select blocks to be paged out of the local cache 3814. Such an algorithm may be based on time of access. In some embodiments, the algorithm may be based on predictions of device movement that will change the device's field of view.

[0291] An application (e.g., a computer game) that renders a scene may obtain information that represents parts of the physical world that affect the scene to be rendered. The application 3816 may obtain active blocks from the active memory 3812b through the local gateway 3818a. In some embodiments, the local gateway 3818a may be implemented as an application programming interface (API) such that the processor 3812 implements a "service" for the application 3816. In embodiments where the data of the physical world is represented as a mesh, the service may be a "meshed service." The API may be implemented as a push or pull interface, or may have attributes of both. In a pull interface, for example, the application 3816 may indicate parts of the physical world for which it requests data, and the service may provide the data for those parts. In a push system, for example, the service may provide data about parts of the physical world as such data changes or becomes available.

[0292] The portions of the physical world for which data is provided may be limited to those portions that the application 3816 has indicated are relevant, such as data within the field of view of the device or data representing portions of the physical world within a threshold distance of the field of view of the device. In a pull / push delivery system, an application 3816 may request data about a portion of the physical world, and the service may provide data about the requested portion, plus any adjacent portions where the data has changed. In order to limit the information to those that have changed, the service may keep track of versions of the blocks that have been provided to each of the applications 3816, in addition to maintaining blocks that describe the physical world. The operations for determining portions of the representation of the physical world to be updated and where the updates have occurred may be partitioned between the applications 3816 and the service in any suitable manner. Similarly, where updated data is incorporated into the representation of the physical world may also be partitioned in any suitable manner.

[0293] In some embodiments, the reconstruction service may continually reconstruct a 3D representation of the physical world as sensors capture and update data. This data may then be provided to applications 3816, which use the 3D reconstruction data to render the scene and depict both the physical world and virtual reality objects that are positioned within or interact with objects in the physical world. This data may be provided to applications 3816 through an API, which may be implemented as a push interface that pushes data to applications 3816 as relevant parts change, or as a pull interface that responds to requests from applications 3816 for up-to-date information, or both.

[0294] For example, application 3816 may operate on a mesh representation of a portion of the physical world that constitutes a 45 degree viewing angle over a distance of 10 meters relative to an origin defined by the device's current location and the direction the device is facing within it. As this area changes, or as data becomes available indicating physical changes within this area, a mesh may be calculated to represent this area. The mesh may be calculated within application 3816 based on data provided by the service, or may be calculated within the service and provided to application 3816. In either case, the service may use the mesh to determine the location of the area. The memory may store information in the physical world that simplifies computation of the mesh. As described herein, blocks implemented with zippers, skirts, or other techniques to facilitate "hiding" of cracks between adjacent blocks may be used to allow processing of only those portions of the representation of the physical world that have changed. The changed portions of the representation of the physical world may then replace the corresponding portions in the previous representation of the physical world.

[0295] Efficient access to representations of portions of the physical world used to generate meshes to be used by applications 3816 to render the scene to the user reduces computer resources, allows XR systems to be easily implemented on portable or other devices with limited computing resources, and may result in a more realistic user experience because the XR scene better matches the physical world. Thus, instead of or in addition to using blocks with skirts, zippers, or other techniques to facilitate concealment of cracks between blocks, as described anywhere herein, algorithms used to page blocks in and out of active memory and / or local caches may be selected to reduce the access time to blocks needed to compute a mesh at any given time.

[0296] 38, the gateway 3818a is a pull delivery interface. When an AR application 3816 requests information about an area of ​​the physical world, but the block representing the area is not in active memory 3812b, the processor 3812 may search for a block persisted in local cache 3814. If the processor 3812 cannot find the block in both active memory 3812b and local cache 3814, the processor 3812 may search for a block persisted in cloud cache 3802. Because access to active memory 3812b has lower latency than access to data in local cache 3814 (which has lower latency than access to data in cloud cache 3802), the overall speed in generating the mesh may be increased by the service implementing a paging algorithm that loads blocks into active memory before they are requested, or moves them from cloud cache 3802 to local cache 3814 before they are requested.

[0297] Similar to the first device 3810, the second device 3820 may include a processor 3822 having a computing portion 3822a and an active memory 3822b, a local cache 3824, and one or more AR applications 3826. The AR applications 3826 may communicate with the processor 3822 through a local gateway 3828a. The local cache 3824 may communicate with the processor 3822 through a local gateway 3828b.

[0298] Thus, the cloud cache 3802 may persist blocks transmitted from both devices 3810 and 3820. The first device 3810 may access blocks in the cloud cache 3802 that were captured and transmitted from the second device 3820. Similarly, the second device 3820 may access blocks in the cloud cache 3802 that were captured and transmitted from the first device 3810.

[0299] Devices 3801 and 3802 are provided as examples of portable AR devices, any suitable device may be used and implemented as well, for example a smartphone.

[0300] FIG. 39 illustrates a block diagram of a system (e.g., system 3700) for operating in accordance with some embodiments. 39 is a flow chart illustrating a method 3900 of rendering a 3D image of a physical world, including objects within the physical world. In act 3902, the device may capture 3D information about the physical world, including objects within the physical world, and represent the physical world as blocks that include 3D reconstruction data. In some embodiments, the 3D reconstruction data may be captured by a single system and used to render information about only that system. In some embodiments, the 3D reconstruction data may be captured by multiple systems and used to render information about any one of the multiple systems or any other system. In these embodiments, the 3D reconstruction data from the multiple systems may be combined and made accessible to the multiple systems or any other system.

[0301] For example, several users, each wearing an AR system, may set their devices to an augmented mode while exploring a warehouse. A sensor on each device may capture 3D information (e.g., 3D reconstruction data, including depth maps, images, and the like) of the warehouse within the sensor's field of view, including objects (e.g., tables, windows, doors, floors, ceilings, walls) in the warehouse. Each device may segment the warehouse into regions with corresponding volumes and represent individual regions as blocks. The blocks may have versions. Each version of a block may have a value that represents an object in a region of the physical world at a point in time.

[0302] When an application requests information about the physical world, a version of the block to represent that portion of the physical world may be selected and used to generate that information. Such selection processing may be performed by or distributed across any suitable processor, although according to some embodiments the processing may occur locally at the device on which the application requesting the data is running.

[0303] Thus, in act 3904, a processor (e.g., processor 3812 or 3822) may respond to a request from an application (e.g., AR application 3816 or 3826) for 3D reconstruction data. In some embodiments, regardless of whether an application requests 3D reconstruction data, the device may continue to capture 3D information, including 3D reconstruction data about the physical world, and represent the physical world as blocks of 3D reconstruction data. That 3D reconstruction data may be used to create new versions of the blocks.

[0304] If the application requests 3D reconstruction data, processing may proceed to act 3906, where the processor may identify a subset of blocks corresponding to the portion of the physical world required to deliver 3D reconstruction data in accordance with the request. Identification of the blocks may be based, for example, on data collected by sensors (e.g., depth sensor 51, world camera 52, inertial measurement unit 57, global positioning system, and / or the like). The multi-device system may create a common coordinate frame such that blocks generated by different devices that are associated with corresponding portions of the physical world may be created using a common coordinate frame regardless of which device provided the 3D reconstruction data for reconstructing the portion of the physical world represented by that block. As one example of how a common coordinate frame may be created, data from devices within the same general vicinity may be routed to the same server or one or more servers for processing. There, data from each device may first be represented in a device-specific coordinate frame. Once enough data from each of the devices has been collected to identify features within the common portion of the physical world, those features may be correlated to provide a transformation from one device-specific coordinate frame to the other. One of these device-specific coordinate frames is the common coordinate frame and the other coordinate frames. A common coordinate frame may be specified as a transformation between the first and second devices, which may be used to transform data from device-specific coordinate frames to a coordinate frame designated as the common coordinate frame. Regardless of the specific mechanism by which the common coordinate frame is created, once created, 3D reconstruction data requested by an application generating data for the first device may be based on updates from the other device if those updates were made after any version made by the first device. Blocks from both the first and other devices may be merged, for example, by using a common coordinate frame.

[0305] The specific processing in act 3906 may depend on the nature of the request. In some embodiments, if the application requesting the 3D reconstruction data maintains its own information about the blocks and requests specific blocks, the request for the 3D reconstruction data in act 3904 may include a reference to the specific subset of blocks, and identifying the subset of blocks in act 3906 may include determining the subset of blocks that corresponds to the specific subset of blocks. In some embodiments, the request for the 3D reconstruction data in act 3904 may include a reference to a field of view of the device on which the application is running, and identifying the subset of blocks in act 3906 may include determining the subset of blocks that corresponds to the referenced field of view of the device.

[0306] Regardless of the manner in which the blocks are identified / determined, in act 3908, the processor may select a version of the block of the subset of blocks. This selection may be based on one or more criteria. The criteria may be based, for example, on the most recent version of the block from an available source. In the illustrated embodiment, the version of the block may be stored in active memory, local cache, or remote cache. Act 3908 may include, for example, selecting the version in active memory if available, or if not, selecting the version in local cache if available, or selecting the version from remote cache if available. If no version of the block is available, the selection may involve generating the block, for example, from data (e.g., 3D reconstruction data) collected using sensors (e.g., depth sensor 51, world camera 52, and / or inertial measurement unit 57), etc. An algorithm for block selection may be used within the system to manage the background processes, block versions stored within each of the possible locations, etc. An example management process is described below in connection with FIG. 41.

[0307] In act 3910, the processor may provide information to the application based on the version of the selected block. Processing at act 3910 may simply involve providing the block to the application, which may be appropriate when the application uses the block directly. If the application receives a mesh, processing at act 3910 may involve generating a mesh from the block and / or a subset of the block and providing the mesh or any suitable portion of the mesh to the application.

[0308] 40 is a flowchart illustrating details of capturing 3D information about objects in the physical world and representing the physical world as blocks of 3D reconstruction data, according to some embodiments. In some embodiments, FIG. 40 is a flowchart illustrating details of act 3902 of FIG. 39. In act 4002, one or more sensors (e.g., depth sensor 51, world camera 52, inertial measurement unit 57, and / or the like) of a system (e.g., system 3700) capture 3D information about objects in the physical world and represent the physical world as blocks of 3D reconstruction data. Captures 3D information about objects in a scene.

[0309] In act 4004, a processor of the system (e.g., processor 3812 or 3822) may create a version of the block that includes 3D reconstruction data of the physical world based on 3D information captured by one or more sensors. In some embodiments, each block may be formatted as one or more portions of a mesh. In some embodiments, other representations of the physical world may be used.

[0310] A block may have versions such that each time information about an area of ​​the physical world is captured by any device, a new version of the block may be stored. Each version of a block may have 3D reconstruction data that includes values ​​that represent objects in a certain area of ​​the physical world at a certain point in time. In some embodiments, such processing may be performed locally on the device, resulting in a new version of the block being stored in active memory. In some embodiments, in a multi-device system, a similar process may be performed in a server (e.g., server 3710 of FIG. 37) that may manage versions of blocks such that the most recent version of each block available in its remote cache is provided when requested by any device.

[0311] Because blocks represent a physical world, much of which may be unchanged, new versions of blocks are not necessarily created when new 3D reconstruction data representing corresponding regions of the physical world become available. Rather, managing block versions may involve processing the 3D reconstruction data representing the physical world to determine whether there has been sufficient change since the last version of the block representing those regions of the physical world to justify a change. In some embodiments, sufficient change may be indicated by the magnitude of a block metric exceeding a threshold since the last version was stored.

[0312] In some embodiments, other criteria may be applied to determine which version of a block is served as the current version when the block is requested, such as the version having the lowest value of a metric indicating the completeness or correctness of the data in the block. Similar processes may be performed on each device, resulting in versions of the block being stored in a local cache on the device.

[0313] One or more techniques may be used to manage the versions of blocks available to the service on each device. For example, if an acceptable version of a block already calculated exists, rather than creating a new version of the block from the sensor data, the processor may access the previously stored block. Such access may be implemented by efficiently managing the storage of the versions of the block. In act 4006, the processor of the device may page a version of the block of the 3D reconstruction data of the physical world from an active memory (e.g., active memory 3812b or 3822b). Paging may include the processor accessing the sensor data to continually update the blocks in the active memory / local cache / cloud cache, for example, as a function of the field of view of the device. As the field of view of the device changes, blocks corresponding to the new field of view may be transferred (e.g., paged) from the local cache and / or cloud cache into the active memory, and blocks corresponding to areas just outside the new field of view (e.g., blocks adjacent to blocks in the new field of view) may be transferred (e.g., paged) from the active memory and / or cloud cache to the local cache. For example, in act 4008, the version of the block that is paged out by the processor is the local The cached version of the block may be persisted in local memory (e.g., local cache 3814 or 3824) and / or remote memory (e.g., cloud cache 3802). In some embodiments, as each new version of the block is created, for example on a device, that version may be sent to the remote memory so that other users may access it.

[0314] FIG. 41 is a flowchart illustrating an exemplary process for performing a selection of a version of a block representing a subset of blocks, according to some embodiments. In some embodiments, FIG. 41 is a flowchart illustrating details of act 3908 of FIG. 39. To select a version of each block in the subset of blocks, in act 4102, a processor (e.g., processor 3812 or 3822) may query whether a latest version is stored in the active memory (e.g., active memory 3812b or 3822b). In some embodiments, whether a version is latest may be determined by comparing values ​​tied to the version (e.g., magnitude of geometric change, timestamp, and the like) with data collected using a sensor (e.g., depth sensor 51, world camera 52, and / or inertial measurement unit 57). In some embodiments, a comparison may be made between the current sensor data and the version of the block stored in the active memory. Based on the degree of difference, which may represent, for example, a change in the physical world or the quality of the version in the active memory, the version in the active memory may be considered latest.

[0315] If the latest version is stored in active memory, processing continues to act 4104 where the latest version is selected. If the latest version is not stored in active memory, processing continues to act 4106 where the processor may query whether the latest version is stored in local memory (e.g., local cache 3814 or 3824). The query may be performed using the criteria described above in connection with act 4102 or any other suitable criteria. If the latest version is stored in local memory, then in act 4108 the latest version is selected.

[0316] If the latest version is not stored in the local memory, then in act 4110 the processor may query whether the latest version is stored in a remote memory (e.g., cloud cache 3802). The query may also be performed using the criteria described above in connection with act 4102 or any other suitable criteria. If the latest version is stored in the remote memory, then in act 4112 the latest version is selected.

[0317] If the latest version is not stored in the remote memory, processing may proceed to act 4114, where the device's processor may generate a new version of the block based on 3D information (e.g., 3D reconstruction data) captured by the sensor. In some embodiments, in act 4116, the processor may identify neighboring blocks of the block with the new version and update the identified neighboring blocks according to the new version of the block.

[0318] 42 is a flow chart illustrating a method 4200 of operating the system according to some embodiments, in which paging may be managed based on a projection of the device's field of view based on device motion, rather than pulling blocks into active memory and / or local cache as the device requests them.

[0319] Similar to act 3902, in act 4202, sensors on the device may capture 3D information about the physical world, including objects in the physical world, and represent the physical world as blocks containing 3D reconstruction data.

[0320] In act 4204, a processor (e.g., processor 3812 or 3822) may calculate, based at least in part on the output of the sensors, an area of ​​the physical world to which the portable pointable component (e.g., depth sensor 51, world camera 52, and / or inertial measurement unit 57) will be pointed at a future time. In some embodiments, the processor may calculate based on motion data from the inertial sensors or analysis of captured images. In a simple calculation, for example, to obtain fast results, the processor may calculate based on the translation and rotation of the user's head. When applying a more comprehensive algorithm, the processor may calculate based on objects in the scene. For example, the algorithm may take into account that a user walking towards a wall or table is unlikely to walk through the wall or table.

[0321] In act 4206, the processor may select a block based on the calculated region. In act 4208, the processor may update the active memory (e.g., active memory 3812b or 3822b) with the selected block. In some embodiments, the processor may select the block based on the flowchart of Figure 41. In act 4210, the processor may select a block from the active memory and provide it to the application (e.g., application 3816 or 3826), for example through an API, based on changes to each of the blocks since a version of the block was last provided to the application.

[0322] In some embodiments, in act 4206, the processor may request the selected block from a remote memory (e.g., cloud cache 3802) and update information stored in the local cache (e.g., 3814 or 3824) so ​​that the local cache stores the selected block. Act 4206 may be similar to act 3908 illustrated in FIG.

[0323] Block-based processing as described above may be based on blocks, allowing portions of the 3D representation to be processed separately and then combined with other blocks. According to some embodiments, blocks may be formatted such that when a block is changed, the changed representation maintains, for the most part or entirely, the values ​​of the block at the interfaces to other blocks. Such processing, based on changes and invariant blocks, allows a changed version of a block to be used in conjunction with versions of neighboring blocks that were not changed without creating unacceptable artifacts in the rendered scene. Figures 43A-48 illustrate such blocks.

[0324] The 3D representation of the physical world may be provided by a stereoscopic 3D reconstruction, which may create a 3D reconstruction data hierarchy of the 3D information of the physical world captured by the sensor. For example, the sensor may be a depth camera, which may capture 3D information of the physical world, e.g., a stream of depth images with a respective pose of the depth camera (i.e., camera pose). The 3D information of the physical world may be processed into a voxel grid. Each voxel may contain one or more signed distance functions (SDFs) that describe whether the voxel is inside or outside the geometry of an object in the physical world. The voxels may be grouped into "bricks". Each brick may contain, for example, 8 3 A brick may contain multiple voxels within a cubic volume, such as a voxel. Bricks may be further grouped into "tiles." Each tile may contain multiple bricks.

[0325] In some embodiments, the voxel grid may be mapped to conform to a memory structure. A tile may correspond to a memory page of a storage medium. The size of the tile may be variable, for example, depending on the size of the memory page of the storage medium in use. Thus, the 3D reconstruction data may be transmitted between storage media (e.g., active memory of the device and / or local memory and / or remote memory in the cloud) based on tiles. In some embodiments, one or more tiles may be processed to generate a block. The block may be updated, for example, when at least one voxel in one or more tiles changes.

[0326] A block is not necessarily limited to correspond to a tile. In some embodiments, a block may be generated from one or more bricks, one or more voxels, or one or more SDF samples, and the like. A block may be any suitable partition of the physical world. A block is not necessarily limited to the format of a mesh. A block may be in any suitable format of 3D reconstruction data.

[0327] 43A-D illustrate an example physical world 4300 represented by mesh blocks 4302. Each mesh block 4302 may be extracted from voxels 4304, which correspond to a given volume of the mesh block. In the illustrated example, each block represents a cubic region of voxels (e.g., 1 m) in the low-level reconstruction model. 3), each mesh block 4302 may contain a portion of the world mesh and may be processed independently. This may allow for scalability with fast local updates as a result of some blocks being changed when searching for new areas or things moved in the environment. In the illustrated example, there are no changes to the mesh blocks except for mesh block 4306, which has a new object 4308 placed in front of an existing surface 4310. In this case, the AR system needs to update only mesh block 4306, which may save significant computational power compared to arbitrarily updating the entire mesh for the world.

[0328] 43B is a simplified schematic diagram illustrating a mesh block, according to some embodiments. In the illustrated example, the mesh block, when viewed from the inside, may have a fully connected mesh, meaning that a vertex is shared by multiple triangles.

[0329] On the other hand, individual mesh blocks may be unconnected, independent meshes. Fig. 43C is a simplified schematic diagram illustrating a crack that may exist at the edge of two adjacent mesh blocks, according to some embodiments. Fig. 44D is a simplified schematic diagram illustrating the crack in Fig. 43C being obscured by implementing an overlapping mesh skirt on the adjacent mesh blocks, according to some embodiments.

[0330] FIG. 44 is a schematic diagram illustrating a representation 4400 of a portion of the physical world in 2D segmentation, according to some embodiments. The 2D representation 4400 may be obtained by connecting a set of four blocks, namely blocks A-D. The representation 4400 may be segmented, for example, into four blocks, namely blocks 4402, 4404, 4406, and 4408, illustrated by solid line 4410. In some embodiments, blocks 4402, 4404, 4406, and 4408 may each be designated as blocks A-D. An application may desire 3D reconstruction data in the format of a mesh for further processing, such as occlusion testing and generation of physical effects in a physics engine. In some embodiments, the set of blocks may be in the format of a mesh, which may be shared across devices (e.g., devices 3810, 3820), networks (e.g., including cloud cache 3802), and the like. , cloud), or by a discrete application (e.g., applications 3816, 3826).

[0331] In some embodiments, the regions at the boundaries of blocks 4402, 4404, 4406, and 4408 may be skirts, for example, as illustrated by dashed line 4412. In some embodiments, blocks A-D may each include a block and a corresponding skirt. For example, block B may include block 4404 and a skirt 4414 that overlaps with boundary portions of block B's neighboring blocks A, C, and D such that cracks between the blocks may be occluded when the blocks are connected into the global mesh. Blocks A, C, and D may also include corresponding skirts. Thus, before returning the blocks containing the 3D reconstruction data to the application, the processor may occlude any cracks between the blocks.

[0332] In some embodiments, the global mesh may be a topologically connected global mesh, for example, adjacent blocks in a set of blocks may share mesh vertices within block boundaries, such as line 4410. In some embodiments, the global mesh may be visually seamless, but topologically disconnected, using any suitable technique, such as skirts and zippers.

[0333] Although a method employing a skirt is illustrated, other methods, such as zippers, may also be used to allow an altered block to be combined with an unchanged adjacent block. In the illustrated example, the portion of the physical world is represented by four 2D blocks, but it should be understood that the portion of the physical world may be represented by any suitable number of 2D and / or 3D blocks, e.g., two, three, five, six or more. Each block may correspond to a space in the physical world. In some embodiments, blocks in the 2D and / or 3D representation of the portion of the physical world may correspond to spaces of the same size (e.g., area / volume) in the physical world. In some embodiments, blocks in the 2D and / or 3D representation of the portion of the physical world may correspond to spaces of different sizes in the physical world.

[0334] FIG. 45 is a schematic diagram illustrating a 3D representation 4500 of a portion of the physical world, according to some embodiments. Similar to the 2D representation 4400, the 3D representation 4500 may be obtained by connecting eight blocks, namely blocks AH. In some embodiments, the blocks AH may be mutually exclusive, e.g., have no overlapped regions. In some embodiments, the blocks AH may have overlapping regions (e.g., skirt 4516) with neighboring blocks. In some embodiments, the blocks AH may each have versions. Each version of a block may have values ​​that represent an object in a certain region of the physical world at a certain point in time. In the illustrated example, the 3D representation 4500 includes versions of blocks AH, namely, version 4502 of block A, version 4504 of block B, version 4514 of block C, version 4512 of block D, version 4534 of block E, version 4506 of block F, version 4508 of block G, and version 4510 of block H. Version 4502 of block A may include value 4518, version 4504 of block B may include value 4522, version 4514 of block C may include value 4528, version 4512 of block D may include value 4532, version 4534 of block E may include value 4520, version 4506 may include value 4524, version 4508 may include value 4526, and version 4510 may include value 4530.

[0335] FIG. 46 illustrates a 3D representation 4500 obtained by updating the 3D representation 4500, according to some embodiments. FIG. 4 is a schematic diagram illustrating a 3D representation 4600 of a portion of a physical world to be displayed. Compared to 3D representation 4500, 3D representation 4600 may have a new version 4610 of block H, including information 4630. Information 4630 may differ from information 4530. For example, a first device may persist a version 4510 of block H in a remote memory. Version 4510 of block H may include information 4530 corresponding to a table with an empty surface. After the first device leaves the area (e.g., the field of view of the first device no longer includes block H), a second device may place a virtual and / or physical box on the surface of the table and persist a version 4610 of block H in a remote memory. Version 4610 of block H may include information 4630 corresponding to a table with virtual and / or physical boxes. When the first device returns, the first device may be able to select version 4610 of block H to view from the available versions of block H, including version 4610 of block H and version 4510 of block H.

[0336] FIG. 47 is a schematic diagram illustrating an augmented world 4700 viewable by a first device 4702 (e.g., device 3810) and a second device 4712 (e.g., device 3820). The first and second devices may include AR display systems 4704 and 4714 (e.g., AR display system 80) operating in an augmented mode. The augmented world 4700 may be obtained by connecting four blocks, namely, blocks A-D. In the illustrated example, the augmented world 4700 includes versions of blocks A-D, namely, version 4702A of block A, version 4702B of block B, version 4702C of block C, and version 4702D of block D. The first device 4702 may look in a first direction 4706 and have a first field of view (FOV) 4708. In the illustrated example, the first FOV includes version 4702B of block B and version 4702D of block D. A processor (e.g., 3812) of the second device 4704 may include computer-executable instructions for identifying blocks B and D corresponding to the first FOV and selecting version 4702B of block B and version 4702D of block D. The second device 4714 may look in a second direction 4716 and have a second FOV 4718. In the illustrated example, the second FOV includes version 4702C of block C and version 4702D of block D. A processor (e.g., 3822) of the second device 4714 may include computer-executable instructions for identifying blocks C and D corresponding to the second FOV and selecting version 4702C of block C and version 4702D of block D.

[0337] 48 is a schematic diagram illustrating an augmented world 4800 obtained by updating the augmented world 4700 with new versions of blocks, according to some embodiments. Compared to the augmented world 4700, the augmented world 4800 may include a version 4802C of block C that differs from version 4702C and a version 4802D of block D that differs from version 4702D. The first device 4702 may be looking in a third direction 4806 and have a third FOV 4808. In the illustrated example, the third FOV includes version 4802C of block C and version 4802D of block D. A processor of the first device 4702 may include computer-executable instructions for determining versions 4702C, 4802C, 4702D, and 4802D for providing an application based on, for example, a change in FOV and / or information collected using a sensor of the first device 4702. In some embodiments, the processor of the first device 4702 may include computer-executable instructions for generating a version 4802C of block C and a version 4802D of block D when the corresponding latest versions of blocks C and D do not exist in a local memory (e.g., local cache 3814, 3824) or a remote memory (e.g., cloud cache 3802). In some embodiments, the first device 4702 may be able to estimate a change in its FOV (e.g., from the first FOV 4708 to the third FOV 4808) and, based on the estimate, select block C and store a version of block C in memory closer to the processor (e.g., moving a version of block C from a remote memory to a local cache or from the local cache to active memory).

[0338] How to Render Occlusion with Raycasting and Live Depth

[0339] The realism with which AR and MR scenes are presented to a user may be enhanced by providing occlusion data to applications generating such scenes, where the occlusion data is derived from a combination of one or more depth data sources. The occlusion data may be formatted in any suitable manner, such as with depth data that may represent surfaces of physical objects in a scene and indicate the distance and surface from a viewpoint from which the scene should be rendered. For example, the utilization component 164 may receive the occlusion data from the perception module 160 (FIG. 6).

[0340] However, in some embodiments, one data source may be directly one or more depth cameras that sense and capture the position between the depth camera and real objects in the physical world. Data from the depth cameras may be provided directly to the usage component 164 or indirectly, such as through the perception module 160. The one or more depth cameras may provide an instantaneous view of the physical world at a frame rate that is high enough to capture changes in the physical world, but low enough not to burden the processing. In some embodiments, the frame rate may be 5 frames / sec, 10 frames / sec, 12 frames / sec, 15 frames / sec, 20 frames / sec, 24 frames / sec, 30 frames / sec, and the like. In some embodiments, the frame rate may be less than 5 frames / sec. In some embodiments, the frame rate may be greater than 30 frames / sec. Thus, in some embodiments, the frame rate may be in the range of 1-5 frames / sec, 5-10 frames / sec, 10-15 frames / sec, 15-20 frames / sec, or 20-30 frames / sec, and the like.

[0341] A second source of data may be a stereoscopic camera, which may capture a visual representation of the physical world. Depth data from the depth camera and / or image data from the visual camera may be processed to extract points that represent real objects in the physical world. Images from a visual camera, such as a stereoscopic camera, may be processed to compute a three-dimensional (3D) reconstruction of the physical world. In some embodiments, the depth data may be generated from images from the visual camera, for example, using deep learning techniques. Some or all of the 3D reconstruction may be computed prior to the occlusion data and may be stored in memory. In some embodiments, the 3D reconstruction may be maintained in computer memory by a process independent of any process that generates depth information for occlusion processing, which may access the stored 3D reconstruction as needed. In some embodiments, the 3D reconstruction may be maintained in memory, portions of which may be updated, for example, in response to an indication computed based on the depth information that there is a change in the physical world that corresponds to a portion of the 3D reconstruction. In some embodiments, the second data source may be implemented by ray casting into the 3D reconstruction of the physical world to obtain low-level 3D reconstruction data (e.g., a ray casting point cloud). Using ray casting, data from the second data source may be selected to fill any holes in the occlusion data and allow the data from the two (or more) sources to be integrated.

[0342] According to some embodiments, the depth data and / or image data and / or the 3D The low-level data of the D reconstruction may be oriented to a user of the AR or MR system. Such orientation may be achieved, for example, by using data from a sensor worn by the user. The sensor may be mounted, for example, on a head-mounted display device / unit.

[0343] In systems in which occlusion data may be generated from more than one depth data source, the system may include a filter that identifies a portion of the 3D region represented by data from each of the multiple depth data sources. The filter may apply one or more criteria to identify a portion of the region for which data from a second data source should be collected. These criteria may be an indication of the reliability of the depth data. Another criterion may be a change detected within the portion of the region since depth data was collected.

[0344] Selection between multiple depth data sources to provide data for different portions of a representation of a 3D region may reduce processing time. For example, selection may favor data from a first data source when less processing is required to derive occlusion data from data collected with a first data source than with a second data source, but may use data from the second source when data from the first source is unavailable or unacceptable. As a specific example, the first data source may be a depth camera and the second data source may be a stereoscopic camera. Data from the stereoscopic camera may be formatted as a 3D reconstruction of the physical world. In some embodiments, the 3D reconstruction may be calculated before occlusion data is requested. Alternatively, or in addition, the 3D reconstruction may be recalculated when occlusion data is requested. In some embodiments, criteria may be applied to determine whether the 3D reconstruction should be recalculated.

[0345] In some embodiments, this occlusion data is calculated by a service that provides the occlusion data through an application programming interface (API) to an application running on a computing device that will render the XR scene. The service may run on the same computing device as the application or on a remote computer. The service may include one or more of the components discussed herein, such as a filter for data from a first data source and / or an engine for selectively fetching data from a second data source based on the filtered data from the first data source. The service may also include a component for combining the filtered data from the first data source and the selected data from the second data source to generate the occlusion data.

[0346] The occlusion data may be formatted in any suitable manner to represent surfaces in the physical world. For example, the occlusion data may be formatted as a depth buffer of surfaces, storing data identifying the locations of surfaces in the physical world. This occlusion data may then be used in any suitable manner. In some embodiments, the occlusion data may be provided to one or more applications that desire to have a virtual object occluded by a real object. In some embodiments, the occlusion data may be formatted as a depth filter created by the system for an application that requests an occlusion service for occlusion data to render a virtual object in one or more locations. The depth filter may identify locations where the application should not render image information for the virtual object because the virtual object in those locations would be occluded by a surface in the physical world. "Occlusion data" may be formatted in any suitable format to provide information about surfaces in the physical world. It should be understood that the occlusion data may be in the matte and need not be used for occlusion processing. In some embodiments, the occlusion data may be used in any application that performs processing based on a representation of surfaces in a physical world scene.

[0347] The approach as described herein provides occlusion data with lower latency and / or using fewer computing resources than traditional AR and MR systems in which applications use mesh data to perform occlusion processing. Mesh data may be obtained by processing geometric data extracted by an image sensor multiple times or with cost-intensive steps including application of marching cubes algorithms, mesh simplification, and triangle count limits. Mesh data may take hundreds of milliseconds to seconds to compute, and the delay in having an up-to-date mesh may lead to visible artifacts when the environment is dynamically changing and the application renders a scene using an outdated mesh. These artifacts become evident, for example, when virtual content appears superimposed on top of real objects behind which the virtual content is supposed to be rendered, which destroys the perception / immersion for users of such applications and provides the user with incorrect cues for 3D depth perception.

[0348] In order for an application that uses a mesh for occlusion processing to have the latest mesh, the application must either continually query for the mesh (resulting in significant continuous processing) or utilize a mechanism to determine whether there are changes and then query for a new mesh (which reduces overall processing but would still have a long latency between changes in the physical world and the time a mesh reflecting those changes arrives at the application).

[0349] By utilizing low-level data (e.g., point clouds) and live depth data directly in the 3D reconstruction data instead of meshes for occlusion, it is possible to reduce the latency between changes occurring in the environment and being reflected in the occlusion data, maintaining closer synchronization with the physical world and therefore higher perceived visual quality.

[0350] In some embodiments, a real-time depth map of the physical environment may be obtained from a depth sensor (e.g., a depth camera). Each pixel in the depth map may correspond to a discrete distance measurement captured from a 3D point in the environment. In some embodiments, these depth cameras may provide a depth map that includes a set of points at a real-time rate. However, the depth map may have holes that may result from the depth camera being unable to obtain sensor data representing an area, or obtaining incorrect or unreliable data representing an area. In some embodiments, if the depth sensor uses infrared (IR) light, the holes may result, for example, from materials or structures in the physical environment that do not reflect IR light very well or at all. In some embodiments, the holes may result, for example, from very thin structures or surfaces at an oblique incidence angle that do not reflect light toward the depth sensor. Depth sensors may also suffer from motion blur when moving at high speeds, which may also lead to missing data. Additionally, "holes" in a depth map represent regions of the depth map that are not suitable for use in occlusion processing for any other reason. Such holes may be detected using any suitable process, such as processing the depth map to detect lack of connectivity between points or regions in the depth map. As another example, holes may be detected using a process that calculates a quality metric for regions of the depth map and treats regions with low quality metrics as holes. One such metric may be the image-to-image variability for pixels in the depth map that represent the same location in the physical world. Pixels with such variability that exceed a threshold may be classified as holes. In some embodiments, holes may be identified by pixels that meet a predetermined statistical criterion for a cluster of pixels whose quality metric is below a threshold.

[0351] In some embodiments, the depth map may first be "filtered" to identify holes. Rays from viewpoints from which the scene will be rendered into the holes may then be determined. Those rays may be "cast" into a 3D representation of the physical world, created using more than just a depth sensor, to identify data representing the area of ​​the hole. The 3D representation of the physical world may be, for example, a 3D reconstruction created from data from a stereoscopic camera. Data from the 3D reconstruction identified by such ray casting may be added to the depth map, thereby filling the hole.

[0352] The 3D reconstruction may be calculated from the image sensor data once holes are identified. Alternatively, some or all of the 3D reconstruction may be pre-calculated and stored in memory. The 3D reconstruction may be maintained by a process, e.g., in computer memory, independent of any process that generates depth information for occlusion processing, which may access the stored 3D reconstruction as needed. As a further alternative, the 3D reconstruction may be maintained in memory, but portions of it may be updated in response to indications calculated based on the depth information that there is a change in the physical world corresponding to a portion of the 3D reconstruction.

[0353] In an XR system, the light ray may have the same pose as the user's eye line of sight. In an exemplary system as described below, the depth map may also be obtained using the user's eye line of sight, since a depth sensor may be worn by the user and mounted on the user's head near the eyes. The visual camera used to form the 3D reconstruction data may also be worn by the user such that the images and data derived from those images may be related to a coordinate system that allows the light ray defined for the depth map to be cast into the 3D reconstruction calculated from the visual images. Similarly, an inertial measurement unit and / or other sensor worn by the user and / or associated with the sensor may provide data regardless of the pose of the visual camera, perform coordinate transformations, add data to the 3D representation, and relate the light ray defined for the depth map to the 3D reconstruction.

[0354] In some embodiments, the user's focus or related virtual content placement information may guide the ray casting and adapt it in the image space by casting denser rays at depth-discontinued locations to obtain high quality occlusion at object boundaries and sparse rays at the center of objects in the view to reduce processing requirements. Ray casting may additionally provide local 3D surface information such as normals and positions, which, with the depth information, may be used to enable improved time warping processing and mitigate missing visible pixels that would otherwise need to be rendered or ray traced in a typical rendering engine. Time warping is a technique in XR that modifies the rendered image before sending it to the display to compensate for head movements that are calculated to occur between rendering and display. In some embodiments, time warping may be used to synchronize data from the depth map with a 3D representation of the physical world, which may be used to generate data to fill holes in the depth map. Data from both data sources may be time warped to represent the calculated pose when displayed. In some embodiments, data from the 3D representation may be time warped to represent the calculated pose at the time the data was captured using a depth map.

[0355] In some embodiments, advanced features such as time warping may utilize 3D local surface information from ray casting. When a content frame is rendered without physical world occlusion or with eroded depth imaging, time warping may fill in all missing visible pixels that were previously occluded. Thus, the rendering engine may not need to fill in pixels, allowing for less strict decoupled rendering applications (or more independent time warping).

[0356] Processing as described above may be performed in any suitable form of hardware processor on data acquired using any suitable sensor and presented on any suitable interface. An example of a suitable system including sensors, processing, and user interfaces is presented below. In the illustrated embodiment, a "service" may be implemented as part of the XR system using computer executable instructions. Execution of those instructions may control one or more processors to access sensor data, which in turn generates and supplies depth information to applications running on the XR system. Those instructions may be executed on the same processor or device that executes the application that presents the XR scene to the user, or on a remote device accessed by the user device via a computer network.

[0357] FIG. 49 illustrates an occlusion rendering system 4900 according to some embodiments. The occlusion rendering system 4900 may include a reconstruction filter 4902. The reconstruction filter 4902 may receive depth information 4904. In some embodiments, the depth information 4904 may be a sequence of depth images captured by a depth camera. In some embodiments, the depth information 4904 may be derived from a sequence of images captured by a visual camera, for example, using structure from motion based on a single camera and / or using stereo computation based on two cameras. FIG. 50 illustrates a depth image 5000 according to some embodiments. In some embodiments, surface information may be generated from the depth information. The surface information may indicate distances to physical objects within a field of view (FOV) of a head mounted display device that includes a depth camera and / or a visual camera. The surface information may be updated in real time as the scene and FOV change.

[0358] A second source of depth information is illustrated as a 3D reconstruction 4908. The 3D reconstruction 4908 may include a 3D representation of the physical world. The 3D representation of the physical world may be created and / or maintained in a computer memory. In some embodiments, the 3D representation may be generated from images captured by a visual camera, for example, using a composition from motion based on a single camera and / or using a stereo calculation based on two cameras. In some embodiments, the 3D representation may be generated from depth images captured by a depth camera. For example, the 3D reconstruction 4908 may be created and / or updated using the depth information 4904 in combination with the pose of the depth camera relative to the world origin. The representation may be built and modified over time, for example, as a user wearing the camera looks around the physical world. In some embodiments, the depth information 4904 may also be used to generate a 3D representation of the physical world. The 3D reconstruction 4908 may be a stereo reconstruction, including 3D voxels. In some embodiments, each 3D voxel may represent a cube of space (e.g., 0.5 meters by 0.5 meters by 0.5 meters), and each 3D voxel may contain data relating to and / or describing surfaces in the real world within that cube of space.

[0359] The 3D reconstruction of the world 4908 may be stored in any suitable manner. In some embodiments, the 3D reconstruction 4908 is stored as a set of points that represent features of objects in the physical world. In some embodiments, the 3D reconstruction 4908 may be stored as a "cloud." In some embodiments, the 3D reconstruction 408 may be stored as a mesh, where groups of points define vertices of triangles that represent a surface. In some embodiments, the 3D reconstruction 4908 may be generated using a room layout detection system and / or other techniques, such as object detection. In some embodiments, several techniques may be used together to generate the 3D reconstruction 4908. For example, object detection may be used for known physical objects in the physical world, 3D modeling may be used for unknown physical objects in the physical world, and a room layout detection system may also be used to identify boundaries in the physical world, such as walls and floors.

[0360] The reconstruction filter 4902 may include computer-executable instructions for producing a depth map based on the depth information 4904. The depth map may include one or more pixels. Each pixel may indicate a distance to a surface point in the physical world. In some embodiments, the reconstruction filter 4902 may combine the depth information 4904 and data from the ray casting engine 4906. In some embodiments, the reconstruction filter 4902 may reduce or remove noise from the depth information 4904 and / or from the combination of the depth information 4904 and data from the ray casting engine 4906, based at least in part on the data from the ray casting engine 4902. In some embodiments, the reconstruction filter 4902 may up-amplify the depth information 4904 using deep learning techniques.

[0361] The reconstruction filter 4902 may identify regions of the depth map based on a quality metric. For example, when a pixel's quality metric exceeds a threshold, the pixel may be determined to be incorrect or noisy. Regions of the depth map that contain incorrect or noisy pixels may be referred to as holes (e.g., holes 5002).

[0362] 54A and 54B provide an alternative example of how holes can be introduced into a depth map in an embodiment where the depth map is constructed from multiple depth images. FIG. 54A is a sketch of a region imaged from a first viewpoint using a depth camera to identify voxels occupied by a surface and empty voxels. FIG. 54B is a sketch of a region imaged from multiple viewpoints using a depth camera to identify voxels occupied by a surface and empty voxels. FIG. 54B illustrates multiple voxels determined to be occupied by a surface or empty by fusing data from multiple camera images. However, voxels in region 5420 are not imaged. Region 5420 may be imaged using a depth camera at location 5422, but the camera has not moved to that location. Thus, region 5420 is an observation space where no stereoscopic information is available. The AR system may guide a user wearing it to scan the observation space.

[0363] Referring back to FIG. 49, the reconstruction filter 4902 may inform the ray casting engine 4906 about the location of the hole. The ray casting engine 4906 may generate a view of the physical world given the user's pose and may remove the hole from the depth map. The data may represent the portion of the user's current view of the physical world at the current time, or where time warping is used at the time by which the data should be time warped. The ray casting engine 4906 may generate one or more 2D images, for example, one image per eye. In some embodiments, the reconstruction filter 4902 may remove regions of the depth map that are spaced away from the location of the virtual object beyond a threshold distance, as these regions may not be relevant for occlusion testing for the virtual object.

[0364] A ray-casting engine that generates a view of the physical world given the user's pose. The gin 4906 may be implemented by any suitable technique. In some embodiments, the ray casting engine 4906 may implement a ray casting algorithm on the 3D reconstruction 4908 to extract data therefrom. The ray casting algorithm may take the user's pose as input. The ray casting engine 4906 may cast rays from the virtual camera to the 3D reconstruction 4908 of the physical world to obtain missing surface information (e.g., holes) from the depth map. The ray casting engine 4906 may cast dense rays to the boundaries of physical objects in the physical world to obtain high quality occlusion at the object boundaries and sparse rays to the central regions of the physical objects to reduce processing requirements. The ray casting engine 4906 may then provide the ray casting point cloud to the reconstruction filter 4902. The ray casting engine 4906 is illustrated as an example. In some embodiments, the ray casting engine 4906 may be a meshing engine. The meshing engine may implement a meshing algorithm on the 3D reconstruction 4908 to extract data from it, including, for example, triangles and triangle connectivity. The meshing algorithm may take the user's pose as input.

[0365] The reconstruction filter 4902 may combine data from the depth information 4904 and the ray casting engine 4906 and compensate for holes in the depth map from the depth information 4904 with data from the ray casting point cloud from the ray casting engine 4906. In some embodiments, the resolution of the depth map may be improved. The method may be used to generate a high resolution depth image from a sparse or low resolution depth image.

[0366] The reconstruction filter 4902 may provide the updated depth map to an occlusion service 4910. The occlusion service 4910 may calculate occlusion data based on the updated depth map and information about the location of virtual objects in the scene. The occlusion data may be a depth buffer of surfaces in the physical world. The depth buffer may store the depth of pixels. In some embodiments, the occlusion service 4910 may interface with an application 4912. In some embodiments, the occlusion service 4910 may interface with a graphics system. In these embodiments, the graphics system may expose a depth buffer to the application 4912, and the depth buffer is pre-filled with the occlusion data.

[0367] The occlusion service 4910 may provide the occlusion data to one or more applications 4912. In some embodiments, the occlusion data may correspond to a user's pose. In some embodiments, the occlusion data may be a per-pixel representation. In some embodiments, the occlusion data may be a mesh representation. The applications 4912 may be configured to execute computer-executable instructions to render virtual objects in a scene based on the occlusion data. In some embodiments, the occlusion rendering may be performed by a separate graphics system instead of the applications 4912. The separate graphics system may use time warping techniques.

[0368] In some embodiments, the reconstruction filter 4902, the ray casting engine 4906, and the occlusion service 4910 may be remote services, e.g., the remote processing module 72, the 3D reconstruction 4908 may be stored in a remote memory, e.g., the remote data repository 74, and the application 4912 may be on the AR display system 80.

[0369] FIG. 51 illustrates an occlusion rendering in an augmented reality (AR) environment, according to some embodiments. 5 illustrates a method 5100 of depth mapping. In act 5102, depth information may be captured, for example, from a depth sensor (e.g., depth sensor 51) on a head mounted display device. The depth information may indicate a distance between the head mounted display device and a physical object. In act 5104, surface information may be generated from the depth information. The surface information may indicate a distance to a physical object within the head mounted display device and / or a field of view (FOV) of a user of the head mounted display device. The surface information may be updated in real time as the scene and FOV change. In act 5106, a portion of a virtual object to render may be calculated from the surface information and information about the location of the virtual object within the scene.

[0370] FIG. 52 illustrates details of act 5104, according to some embodiments. In act 5202, depth information may be filtered to generate a depth map. The depth map may include one or more pixels. Each pixel may indicate a distance to a point of the physical object. In act 5204, low-level data of a 3D reconstruction of the physical object may be selectively obtained, for example from 3D reconstruction 4908. In act 5206, surface information may be generated based on the depth map and the selectively obtained low-level data of the 3D reconstruction of the physical object.

[0371] FIG. 53 illustrates details of act 5202, according to some embodiments. In act 5302, a quality metric for a region of the depth map may be determined. The quality metric may indicate whether the region of the depth map is incorrect or noisy. In act 5304, holes in the depth map may be identified based on the quality metric, for example, by comparison to a threshold. In act 5306, the identified holes may be removed from the depth map.

[0372] conclusion

[0373] While several aspects of several embodiments have been described above, it will be appreciated that various alterations, modifications, and improvements will readily occur to those skilled in the art.

[0374] As an example, the embodiments are described in relation to an augmented (AR) environment, it being understood that some or all of the techniques described herein may be applied in MR environments, or more generally, in other XR and VR environments.

[0375] As another example, embodiments are described in connection with devices, such as wearable devices. It should be understood that some or all of the techniques described herein may be implemented via a network (such as the cloud), a discrete application, and / or any suitable combination of devices, networks, and discrete applications.

[0376] Such alterations, modifications, and improvements are intended to be part of this disclosure and are intended to be within the spirit and scope of this disclosure. Additionally, advantages of the disclosure are set forth, but it should be understood that not all embodiments of the disclosure include all described advantages. Some embodiments may not implement any feature described herein as advantageous, and in some instances. Thus, the foregoing description and drawings are by way of example only.

[0377] The foregoing embodiments of the present disclosure can be implemented in any of numerous ways. For example, the embodiments may be implemented using hardware, software, or a combination thereof. When implemented in software, the software code can be executed on any suitable processor or collection of processors, whether provided in a single computer or distributed among multiple computers. Such a processor may be implemented as an integrated circuit, with one or more processors in an integrated circuit component, including commercially available integrated circuit components known in the art, such as a CPU chip, a GPU chip, a microprocessor, a microcontroller, or a coprocessor, to name a few. In some embodiments, the processor may be implemented in a custom circuit, such as an ASIC, or in a semi-custom circuit resulting from configuring a programmable logic device. As a further alternative, the processor may be part of a larger circuit or semiconductor device, whether commercially available, semi-custom, or custom. As a specific example, some commercially available microprocessors have multiple cores, such that one or a subset of those cores may constitute a processor. However, the processor may be implemented using circuitry in any suitable format.

[0378] Further, it should be understood that a computer may be embodied in any of a number of forms, such as a rack-mounted computer, a desktop computer, a laptop computer, or a tablet computer, etc. In addition, a computer may be embodied in devices not generally considered computers but with suitable processing capabilities, including a personal digital assistant (PDA), a smart phone, or any suitable portable or fixed electronic device.

[0379] A computer may also have one or more input and output devices. These devices can be used, among other things, to present a user interface. Examples of output devices that may be used to provide a user interface include a printer or display screen for visual presentation of output, or a speaker or other sound generating device for audible presentation of output. Examples of input devices that may be used for a user interface include keyboards and pointing devices such as mice, touchpads, and digitizing tablets. As another example, a computer may receive input information through speech recognition or in other audible formats. In the illustrated embodiment, the input / output devices are illustrated as physically separate from the computing device. However, in some embodiments, the input and / or output devices may be physically integrated into the same unit as the processor or other elements of the computing device. For example, the keyboard may be implemented as a soft keyboard on a touch screen. In some embodiments, the input / output devices may be completely disconnected from the computing device and functionally integrated through a wireless connection.

[0380] Such computers may be interconnected by one or more networks in any suitable form, including as a local area network or a wide area network, such as an enterprise network or the Internet. Such networks may be based on any suitable technology and operate according to any suitable protocol, and may include wireless networks, wired networks, or fiber optic networks.

[0381] Also, the various methods and processes outlined herein may be coded as software that is executable on one or more processors employing any one of a variety of operating systems or platforms. In addition, such software may be written using any of a number of suitable programming languages ​​and / or programming or scripting tools, and may be compiled as executable machine language code or intermediate code that runs on a framework or virtual machine.

[0382] In this aspect, the present disclosure relates to a method and apparatus for implementing the present invention on one or more computers or other processors. At times, the computer-readable storage medium may be embodied as a computer-readable storage medium (or a plurality of computer-readable media) (e.g., a computer memory, one or more floppy disks, compact disks (CDs), optical disks, digital video disks (DVDs), magnetic tapes, flash memories, circuitry in field programmable gate arrays or other semiconductor devices, or other tangible computer storage media) encoded with one or more programs that perform methods implementing the various embodiments discussed above. As is evident from the foregoing examples, a computer-readable storage medium may retain information for a sufficient time to provide computer-executable instructions in a non-transitory form. Such a computer-readable storage medium or media may be transportable, as described above, such that one or more programs stored thereon may be loaded onto one or more different computers or other processors to implement various aspects of the present disclosure. As used herein, the term "computer-readable storage medium" encompasses only computer-readable media that may be considered a manufacture (i.e., an article of manufacture) or machine. In some embodiments, the present disclosure may be embodied as a computer-readable medium other than a computer-readable storage medium, such as a propagating signal.

[0383] The term "program" or "software" is used herein in a general sense to refer to any type of computer code or set of computer executable instructions that can be employed to program a computer or other processor to implement various aspects of the present disclosure, as described above. In addition, according to one aspect of the present embodiment, it should be understood that one or more computer programs that, when executed, perform the methods of the present disclosure need not reside on a single computer or processor, but can be distributed in a modular manner among several different computers or processors to implement various aspects of the present disclosure.

[0384] Computer-executable instructions may be in many forms, such as program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Typically the functionality of the program modules may be combined or distributed as desired in various embodiments.

[0385] Also, the data structures may be stored in the computer-readable medium in any suitable form. For ease of illustration, the data structures may be shown to have fields that are related through locations within the data structure. Such relationships may be achieved by allocating storage for the fields with locations in the computer-readable medium that similarly convey the relationship between the fields. However, any suitable mechanism may be used to establish relationships between information in the fields of the data structures, including through the use of pointers, tags, or other mechanisms that establish relationships between data elements.

[0386] Various aspects of the disclosure may be used alone, in combination, or in various arrangements not specifically discussed in the foregoing embodiments, and therefore, its application is not limited to the details and arrangements of components set forth in the foregoing description or illustrated in the drawings. For example, aspects described in one embodiment may be combined in any manner with aspects described in other embodiments.

[0387] The present disclosure may also be embodied as a method, examples of which are provided. The acts performed as part of the method may be ordered in any suitable manner. Thus, while the illustrative embodiments are shown as sequential acts, embodiments may be constructed in which acts are performed in an order different from that shown, which may include performing some acts simultaneously.

[0388] The use of ordinal terms such as "first," "second," "third," etc. in the claims to modify claim elements does not, by itself, imply any priority, precedence, or ordering of one claim element relative to another element, or the temporal order in which acts of a method are performed, but rather the ordinal terms are used solely as markers to distinguish one claim element having a certain name from another element having the same name (due to the use of the ordinal terms) to distinguish between claim elements.

[0389] Also, the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of "including," "comprising," "having," "containing," "with," and variations thereof herein is meant to encompass the items listed thereafter and equivalents and additional items.

Claims

1. A method of operating a computing system to generate a three-dimensional (3D) reconstruction of a surface in a scene, the method comprising: accessing information describing a plurality of bricks for the scene, each brick comprising a plurality of voxels; Accessing a depth image comprising a plurality of pixels; performing a first acceptance test for each of the plurality of bricks, the first acceptance test including accepting the brick for further processing based at least in part on determining that a pixel of the plurality of pixels intersects with the brick; generating the 3D reconstruction based on the approved bricks; A method comprising:

2. The method of claim 1, further comprising performing a second recognition test including recognizing bricks having a position in front of a background indicated by pixels in the depth image, each pixel of the depth image indicating a distance to an area adjacent to the surface in the scene.

3. The method described in claim 2, wherein the second acceptance test includes accepting bricks having a position in front of a solid or perforated background indicated by a pixel in the depth image.

4. The method of claim 3, wherein generating the 3D reconstruction includes selecting a treatment for an approved pixel based on whether the pixel was approved in the first acceptance test or the second acceptance test.

5. The method described in claim 2, wherein the first and second acceptance tests are performed sequentially such that the second acceptance test is performed on bricks that are neither approved nor rejected in the first acceptance test.

6. The method of claim 5, wherein determining that a pixel of the plurality of pixels intersects with the brick comprises: calculating, in coordinates of the depth image, minimum and maximum distances to a bounding box around a projection of the brick into the depth image; for at least one pixel, accepting the brick when the distance value of the pixel is between the minimum dimension and the maximum dimension; The method of claim 1 , comprising:

7. Each pixel of the depth image indicates a distance to an area adjacent to the surface in the scene; 2. The method of claim 1, further comprising: performing a second acceptance test on at least a portion of the bricks not accepted in the first acceptance test, the second acceptance test comprising, for each such brick, accepting the brick if its maximum dimension is less than a distance value for any pixel in the depth image.

8. The method of claim 7, further comprising, for each brick approved in the second approval test, selecting a treatment for the approved pixel based on whether a minimum dimension is greater than the distance value for any pixel in the depth image.

9. The method of claim 1, further comprising rejecting the brick for further processing when the brick is determined to be behind a surface represented in the depth image.

10. The method of claim 1, wherein the further processing includes updating the plurality of voxels of the approved brick based on image data of the scene.

11. The method of claim 1, wherein determining that a pixel intersects with the brick includes projecting a brick silhouette of the brick into the depth image, the brick silhouette being a minimum bounding box for the brick projected onto the depth image.

12. The method of claim 11, further comprising: determining whether each of the portions of the plurality of pixels intersects with the brick; calculating minimum and maximum padded values ​​for said bricks; for each of said portion of said plurality of pixels, comparing the distance indicated by said pixel with said minimum and maximum padded values ​​of said brick; determining that the pixel intersects the brick when the distance indicated by the pixel is between the maximum padded value and the minimum padded value; The method of claim 1 , comprising:

13. The depth image corresponds to a camera frustum, the camera frustum being in the shape of a 3D trapezoidal prism; the method including culling a portion of the plurality of bricks relative to the camera frustum to produce a reduced set of bricks prior to performing the first acceptance test; The method of claim 1 , wherein the first acceptance test is performed on a reduced set of the bricks.

14. Thinning out a portion of the plurality of bricks with respect to the camera frustum comprises: Finding a cube axis aligned bounding box (AABB) to contain said trapezoidal prism; conducting a third acceptance test, the third acceptance test comprising: Dividing the AABB into a plurality of sub-AABBs; determining that each of the plurality of sub-AABBs intersects with the trapezoidal prism; rejecting a brick corresponding to a sub-AABB as being outside the camera frustum when the sub-AABB is determined not to intersect with the camera frustum; and repeating said third acceptance test until the sub-AABB corresponds to only one brick; The method of claim 13, comprising:

15. Determining that each of the plurality of sub-AABBs intersects with the trapezoidal prism includes: determining that at least one corner point of the sub-AABB is inside each of the planes bounding the camera frustum; determining that all corner points of the camera frustum are inside the sub-AABB; 15. The method of claim 14, comprising:

16. A computing device configured to generate a three-dimensional (3D) reconstruction of a portion of a scene, the computing device comprising: a central processing unit (CPU) coupled to access the depth image and the visual image; at least one non-transitory computer-readable medium comprising computer-executable instructions configured for execution on the CPU; Equipped with The computer-executable instructions, when executed by the CPU, selecting a brick representing a portion of the scene based at least in part on an intersection between the brick and the depth image; calculating a 3D representation based on a portion of the visual image that corresponds to the selected brick; A computing device performing a method including:

17. The computing device of claim 16, further comprising a depth sensor, the CPU being coupled to receive the depth image from the depth sensor.

18. The computing device of claim 16, comprising a support member comprising a portion of the head-mounted device.

19. The computing device of claim 18, wherein the head-mounted device further comprises a display.

20. An image sensor having a field of view defining a frustum within the scene, the visual image being from the image sensor; The computing device of claim 16 , wherein the portion of the scene is defined by the frustum.

Images