Caching and updating of dense 3D reconstruction data

Abstract

A method for efficient updating and managing the output of real-time or offline 3D reconstruction and scanning in mobile devices with limited resources and connected to the Internet is provided. The method is capable of providing up-to-date, accurate, and comprehensive 3D reconstruction data to various mobile XR applications in single-user applications or multi-user applications that share and update the same 3D reconstruction data. The method includes a block-based 3D data representation that simultaneously allows local updates and maintains neighbor consistency, and a multi-tiered cache mechanism for efficient retrieval, prefetching, and storage of 3D data for XR applications. Between sessions of an XR device, blocks can be kept on the device or in remote storage in one or more cache tiers. The device can selectively use blocks from one or more cache tiers when starting a new session.

Description

TECHNICAL FIELD

[0001] The present application generally relates to extended reality systems that render scenes using 3D world reconstruction. BACKGROUND

[0002] Computers can control human user interfaces to create X-reality (XR or cross-reality) environments in which some or all of the XR environment is generated by a computer and perceived by a user. These XR environments can be virtual reality (VR), augmented reality (AR), and mixed reality (MR) environments in which some or all of the XR environment can be generated by a computer in part using data that describes the environment. This data can describe, for example, virtual objects that can be rendered in a way that a user senses or perceives as part of the physical world and can interact with the virtual objects. Because the data is rendered and presented through a user interface device (e.g., a head-mounted display device), the user can experience these virtual objects. The data can be displayed for the user to see, or can control audio that is played for the user to hear, or can control a tactile (or haptic) interface that enables the user to experience a sensation of touch that the user senses or perceives as feeling the virtual object.

[0003] XR systems can be useful for many applications, spanning fields such as scientific visualization, medical training, engineering design and prototyping, tele-manipulation and tele-presence, and personal entertainment. In contrast to VR, AR and MR include one or more virtual objects that are related to real objects of the physical world. The experience of virtual objects interacting with real objects greatly enhances the enjoyment of users using XR systems and opens the door to a variety of applications that present real and easily understood information about how to change the physical world. SUMMARY

[0004] Some aspects of the present application relate to methods and apparatus for caching and updating 3D reconstruction data. The inventors have recognized and appreciated techniques for caching and updating dense 3D reconstruction data in real time on devices with limited computing resources, such as mobile devices. These techniques can be used together, separately, or in any suitable combination.

[0005] Some embodiments relate to a portable electronic system. The portable electronic system includes a sensor configured to capture three-dimensional (3D) information about objects in a physical world; an active memory; a local memory; a transceiver configured to communicate with a remote memory over a computer network; and a processor communicatively coupled to the sensor, the active memory, the local memory, and the transceiver. The processor is configured to execute computer-executable instructions to provide a 3D representation of a portion of the physical world based at least in part on the 3D information about the objects in the physical world. The 3D representation of the portion of the physical world includes a plurality of tiles, each tile having a value representing an object in a partial area of the physical world at a point in time. The computer-executable instructions include instructions to maintain the plurality of tiles in at least one of the active memory, the local memory, and the remote memory when the portable electronic system changes to an inactive operating state.

[0006] In some embodiments, the inactive operating state is a shutdown or hibernation.

[0007] In some embodiments, maintaining the plurality of tiles in at least one of the active memory, the local memory, and the remote memory includes maintaining the plurality of tiles in respective memories in which the plurality of tiles currently reside such that the plurality of tiles are accessible by the portable electronic system when the portable electronic system wakes up when the inactive operating state is hibernation.

[0008] In some embodiments, maintaining the plurality of tiles in at least one of the active memory, the local memory, and the remote memory includes maintaining at least a portion of the tiles in the local memory to the remote memory and maintaining tiles in the active memory to the local memory and / or remote memory when the inactive operating state is a shutdown.

[0009] In some embodiments, the computer-executable instructions include instructions to compute one or more new tiles based at least in part on incoming 3D information about objects in the physical world and select tiles from the maintained tiles that at least partially overlap with the one or more new tiles such that the maintained tiles around the portable electronic system can be retrieved.

[0010] In some embodiments, selecting tiles from the maintained tiles that at least partially overlap with the one or more new tiles includes comparing the one or more new tiles to the maintained tiles.

[0011] In some embodiments, comparing the one or more new tiles to the maintained tiles includes comparing the one or more new tiles to tiles maintained in the active memory.

[0012] In some embodiments, comparing the one or more new tiles to the maintained tiles includes comparing the one or more new tiles to tiles maintained in the local memory when the active memory does not have tiles that at least partially overlap the one or more new tiles.

[0013] In some embodiments, comparing the one or more new tiles to the maintained tiles includes comparing the one or more new tiles to tiles maintained in the remote memory when the local memory does not have tiles that at least partially overlap the one or more new tiles.

[0014] In some embodiments, the computer-executable instructions include instructions to update the plurality of tiles with the one or more new tiles.

[0015] Some embodiments relate to a method of operating a portable electronic system in an augmented reality system. The method includes, with at least one processor: obtaining a plurality of tiles comprising three-dimensional (3D) information about surfaces in a physical world; storing at least one of the plurality of tiles in an active memory; using tiles from the active memory for rendering virtual content to a user of the portable electronic system; maintaining the tiles in the active memory when transitioning to an inactive operating state at a first time; and determining whether to use the maintained tiles for rendering virtual content to the user of the portable electronic system when transitioning to an active operating state at a second time after the first time.

[0016] In some embodiments, the tiles are grid tiles.

[0017] In some embodiments, the method includes paging tiles between the active memory and the local memory based on a current field of view or a projected field of view of the user of the portable electronic system.

[0018] In some embodiments, the method includes maintaining the tiles in the local memory when transitioning to an inactive operating state.

[0019] In some embodiments, using tiles from the active memory for rendering virtual content to the user includes using the tiles for visual occlusion processing, and computing physics-based interactions and / or environmental reasoning.

[0020] Some embodiments relate to a non-transitory computer-readable medium storing instructions that, when executed on a processor, perform actions comprising: obtaining a plurality of tiles comprising three-dimensional (3D) information about surfaces in a physical world; storing at least one of the plurality of tiles in an active memory; using tiles from the active memory for rendering virtual content to a user of the portable electronic system; upon transitioning to an inactive operating state at a first time, retaining the tiles in active memory; and upon transitioning to an active operating state at a second time after the first time, determining whether to use the retained tiles for rendering virtual content to a user of the portable electronic system.

[0021] The above summary provides a simplified introduction, and is not intended to be limiting. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings are not intended to be drawn to scale. In the drawings, each identical, or nearly identical, component that is illustrated in various figures is represented with a like numeral. For purposes of clarity, not every component is called out in every drawing. In the drawings:

[0023] FIG. 1 is a schematic diagram illustrating an example of a simplified augmented reality (AR) scene in accordance with some embodiments.

[0024] FIG. 2 is a sketch of an example simplified AR scene in accordance with some embodiments, showing an example world reconstruction use case including visual occlusion, physics-based interaction, and environmental reasoning.

[0025] FIG. 3 is a schematic diagram illustrating data flow in an AR system configured to provide an experience of AR content interacting with a physical world in accordance with some embodiments.

[0026] FIG. 4 is a schematic diagram illustrating an example of an AR display system in accordance with some embodiments.

[0027] FIG. 5A is a schematic diagram illustrating an AR display system rendering AR content as a user moves through a physical world environment in accordance with some embodiments.

[0028] FIG. 5B is a schematic diagram illustrating a viewing optics assembly and accompanying components in accordance with some embodiments.

[0029] FIG. 6 is a schematic diagram illustrating an AR system using a world reconstruction system in accordance with some embodiments.

[0030] FIG. 7Ais a diagram illustrating a discretized 3D space according to some embodiments.

[0031] FIG. 7B is a diagram illustrating a reconstruction range relative to a single viewpoint according to some embodiments.

[0032] FIG. 7C is a diagram illustrating a perceived range relative to a reconstruction range at a single location according to some embodiments.

[0033] FIG. 8A to FIG. 8F is a diagram illustrating reconstructing a surface in a physical world as a voxel model by image sensors viewing the surface from multiple locations and viewpoints according to some embodiments.

[0034] FIG. 9 is a diagram illustrating a scene represented by bricks comprising voxels, a surface in the scene, and a depth sensor capturing the surface in a depth image according to some embodiments.

[0035] FIG. 10A is a diagram illustrating a 3D space represented by eight bricks.

[0036] FIG. 10B is a diagram illustrating a voxel grid in a brick of FIG. 10A .

[0037] FIG. 11 is a diagram illustrating a volumetric representation hierarchy according to some embodiments.

[0038] FIG. 12 is a flowchart illustrating a method of operating a computing system to generate a 3D reconstruction of a scene according to some embodiments.

[0039] FIG. 13 is a flowchart illustrating a method of selecting a portion of a plurality of bricks for a depth sensor in FIG. 12 according to some embodiments.

[0040] FIG. 14 is a flowchart illustrating a method of performing a camera frustum acceptance test in FIG. 13 according to some embodiments.

[0041] FIG. 15 is a flowchart illustrating a method of rejecting a portion of a first plurality of bricks for a depth image in FIG. 12 according to some embodiments.

[0042] FIG. 16 is a flowchart illustrating a method of performing a first depth image acceptance test in FIG. 15 according to some embodiments.

[0043] FIG. 17 is a flowchart illustrating a method of performing a second depth image acceptance test in FIG. 15 is a flowchart illustrating a method of performing a second depth image acceptance test in

[0044] FIG. 18 illustrates a method of classifying all pixels in a rectangle relative to a minimum brick value (bmin) and a maximum brick value (bmax) in FIG. 17

[0045] FIG. 19A to FIG. 19F is a schematic diagram illustrating a camera frustum culling brick according to some embodiments.

[0046] FIG. 20A to FIG. 20B is a schematic diagram illustrating a surface culling brick according to some embodiments.

[0047] FIG. 21 is a schematic diagram illustrating a plane extraction system according to some embodiments.

[0048] FIG. 22 is a schematic diagram illustrating portions of a plane extraction system of FIG. 21 according to some embodiments with details regarding plane extraction.

[0049] FIG. 23 is a schematic diagram illustrating a scene represented by bricks comprising voxels and exemplary plane data in the scene according to some embodiments.

[0050] FIG. 24 is a schematic diagram illustrating a plane data store of FIG. 21 according to some embodiments.

[0051] FIG. 25 is a schematic diagram illustrating plane geometry extraction when a plane query is sent to a plane data store of FIG. 21 according to some embodiments.

[0052] FIG. 26A is a schematic diagram illustrating generation of plane cover points of FIG. 25 according to some embodiments.

[0053] FIG. 26B is a schematic diagram illustrating various exemplary plane geometry representations that can be extracted from an exemplary rasterized plane mask according to some embodiments.

[0054] FIG. 27 illustrates a mesh for a scene according to some embodiments.

[0055] FIG. 28A illustrates an external rectangular plane representation according to some embodiments.​FIG. 27 the scene.

[0056] FIG. 28B shows a scene represented by a polygonal plane according to some embodiments. FIG. 27 the scene.

[0057] FIG. 28C shows a scene represented by a polygonal plane according to some embodiments. FIG. 27 the scene.

[0058] FIG. 29 shows a scene with denoised meshes resulting from planarization of the mesh shown in FIG. 27 according to some embodiments. FIG. 27 the scene.

[0059] FIG. 30 is a flowchart showing a method of generating a model of an environment represented by a mesh according to some embodiments.

[0060] FIG. 31 is a schematic diagram showing a 2D representation of a portion of a physical world of four tiles according to some embodiments.

[0061] FIG. 32A-32D is a schematic diagram showing mesh evolution of an exemplary mesh tile during multi-level simplification according to some embodiments.

[0062] FIG. 33A and 33B show representations of the same environment without simplification and with simplification by triangle reduction, respectively.

[0063] FIG. 34A and 34B show close-up representations of the same environment without simplification and with simplification by triangle reduction, respectively.

[0064] FIG. 35A and 35B show representations of the same environment without planarization and with planarization, respectively.

[0065] FIG. 36A and 36B show representations of the same environment without simplification and with simplification by removing disconnected components, respectively.

[0066] FIG. 37 is a schematic diagram showing an electronic system to enable interactive X-reality environments for multiple users according to some embodiments.

[0067] FIG. 38 is a schematic diagram showing interaction of components of an electronic system in FIG. 37 according to some embodiments.

[0068] FIG. 39 is a flowchart illustrating a method of capturing 3D information about objects in a physical world and representing the physical world as a 3D reconstruction of blocks in FIG. 37 according to some embodiments.

[0069] FIG. 40 is a flowchart illustrating a method of capturing 3D information about objects in a physical world and representing the physical world as a 3D reconstruction of blocks in FIG. 39 according to some embodiments.

[0070] FIG. 41 is a flowchart illustrating a method of selecting a version of a block representing a subset of blocks in FIG. 39 according to some embodiments.

[0071] FIG. 42 is a flowchart illustrating a method of operating an electronic system in FIG. 37 according to some embodiments.

[0072] FIG. 43A is a simplified schematic diagram illustrating an update detected in a portion of a physical world represented by grid blocks according to some embodiments.

[0073] FIG. 43B is a simplified schematic diagram illustrating a grid block according to some embodiments.

[0074] FIG. 43C is a simplified schematic diagram illustrating a crack at an edge of two adjacent grid blocks according to some embodiments.

[0075] FIG. 43D is a simplified schematic diagram illustrating masking a crack in FIG. 43C by implementing a grid skirt overlapping an adjacent grid block according to some embodiments.

[0076] FIG. 44 is a schematic diagram illustrating a 2D representation of a portion of a physical world by four blocks according to some embodiments.

[0077] FIG. 45 is a schematic diagram illustrating a 3D representation of a portion of a physical world by eight blocks according to some embodiments.

[0078] FIG. 46 is a schematic diagram illustrating a 3D representation of a portion of a physical world obtained by updating a 3D representation in FIG. 45 according to some embodiments.

[0079] FIG. 47 is a schematic diagram illustrating an example of an augmented world viewable by first and second users wearing AR display systems according to some embodiments.

[0080] FIG. 48 is a diagram illustrating an example of an augmented world obtained by updating the augmented world of FIG. 47 with a new version of a block according to some embodiments.

[0081] FIG. 49 is a diagram illustrating an occlusion rendering system according to some embodiments.

[0082] FIG. 50 is a diagram illustrating a depth image with holes.

[0083] FIG. 51 is a flowchart illustrating a method of occlusion rendering in an augmented reality environment according to some embodiments.

[0084] FIG. 52 is a flowchart illustrating details of generating surface information from depth information captured by a depth sensor worn by a user in FIG. 51 according to some embodiments.

[0085] FIG. 53 is a flowchart illustrating details of filtering depth information to generate a depth map in FIG. 52 according to some embodiments.

[0086] FIG. 54A is a sketch of imaging with a depth camera from a first point of view to identify regions of voxels occupied by surfaces and regions of empty voxels.

[0087] FIG. 54B is a sketch of imaging with a depth camera from multiple points of view to identify regions of voxels occupied by surfaces and empty voxels and to indicate "holes" for which there is no available volume information because voxels in the region of "holes" were not imaged with the depth camera. DETAILED DESCRIPTION

[0088] Methods and apparatuses for creating and using a three-dimensional (3D) world reconstruction in an augmented reality (AR), mixed reality (MR), or virtual reality (VR) system are described herein. In order to provide a realistic AR / MR / VR experience to a user, an AR / MR / VR system must understand the user's physical environment in order to correctly relate the position of virtual objects to real objects. A world reconstruction can be constructed from images and depth information about those physical environments, which are collected by sensors that are part of the AR / MR / VR system. This world reconstruction can then be used by any of a number of components of such a system. For example, the world reconstruction can be used by components that perform visual occlusion processing, compute physics-based interactions, or perform environment reasoning.

[0089] Occlusion processing identifies portions of virtual objects that should not be rendered and / or displayed to a user because the presence of some object in the physical world prevents the user from viewing the location of the virtual object that would be perceived by the user. Physics-based interactions are computed to determine how virtual objects are displayed to the user. For example, a virtual object can be rendered to appear to be resting on a physical object, to move through empty space, or to collide with the surface of a physical object. The world reconstruction provides a model from which information about objects in the physical world can be obtained to make such computations.

[0090] Environmental reasoning can also use the world reconstruction in generating information that can be used to compute how to render virtual objects. For example, environmental reasoning can involve identifying a transparent surface by recognizing whether the transparent surface is a window pane or a glass tabletop. From such identification, an area containing a physical object can be classified as not occluding a virtual object, but can be classified as interacting with a virtual object. Environmental reasoning can also generate information that is used in other ways, such as identifying a stationary object that can be tracked relative to the field of view of the user to compute motion of the field of view of the user.

[0091] However, there are significant challenges in providing such systems. Computing a world reconstruction can require a large amount of processing. Moreover, an AR / MR / VR system must correctly know how to position virtual objects relative to the user’s head, body, etc. As the user’s position relative to the physical environment changes, the relevant portion of the physical world can also change, which can require further processing. Also, the 3D reconstruction data often needs to be updated as objects move in the physical world (e.g., a cup moves on a table). Updates to data representing the environment that the user is experiencing must be performed quickly without requiring a large amount of computing resources of the computer generating the AR / MR / VR environment because other functions cannot be performed while the world reconstruction is being performed. Moreover, processing the reconstruction data by components that “consume” the data exacerbates the demand for computer resources.

[0092] Known AR / MR / VR systems require high computing power (e.g., GPUs) to run real-time world reconstruction only within a predefined reconstruction volume (e.g., a predefined voxel grid). The inventors have recognized and appreciated techniques for operating an AR / MR / VR system to provide accurate 3D reconstruction data in real-time with little use of computing resources (e.g., computing power (e.g., a single ARM core), memory (e.g., less than 1 GB), and network bandwidth (e.g., less than 100 Mbps)). These techniques involve reducing the processing required to generate and maintain a world reconstruction, as well as providing and consuming data with low computational overhead.

[0093] These techniques can include, for example, reducing the amount of data processed in updating a world reconstruction by identifying portions of sensor data that are available at the time to create or update the world reconstruction. For example, the sensor data can be selected based on whether it represents a portion of the physical world that is likely to be proximate to a surface of an object represented in the world reconstruction.

[0094] In some embodiments, computing resources can be reduced by simplifying the data representing the world reconstruction. A simpler representation can reduce the resources used for processing, storing, and / or managing the data and its use.

[0095] In some embodiments, the use of computing resources can be reduced by representing the physical world in blocks that can be stored and retrieved individually, but combined in a way that provides a realistic representation of the physical world. These blocks can be managed in memory to limit computing resources, and in some embodiments, can enable sharing of blocks between AR / MR / VR systems operating in the same physical space, so that each AR / MR / VR system does less processing to build the world reconstruction. Alternatively or additionally, blocks can be persisted across sessions of a portable user system in an AR / MR / VR system. At the start of a new session of a portable user system, blocks persisted from a previous session can be selectively used, reducing the time for the user system to fully represent the physical world, while at the same time enabling the system to render virtual content to the user in a realistic way that reflects the interaction of the virtual content with the physical world.

[0096] In some embodiments, the use of computing resources can be reduced by selecting from different representations of the physical world when accessing information about the physical world. A world reconstruction, for example, can include information about the physical world captured from different sensors and / or stored in different formats. The simplest data to consume or provide can be provided to components that use the world reconstruction to render virtual objects. If the simpler data is not available, data acquired using other sensors can be accessed, which can generate a higher computational load. As an example, a world reconstruction can include a depth map collected with a depth sensor and a more cumbersome representation of the 3D world, such as a mesh that can be stored as computed from image information. Information about the physical world can be provided to components that perform occlusion processing based on the depth map, if available. In the case of holes in the depth map, information to fill in these holes can be extracted from the mesh. In some embodiments, the depth map can be“live,” representing the physical world as captured by a depth sensor at the time the data is accessed.

[0097] The techniques as described herein can be used with or separate from a variety of types of devices and for a variety of types of scenarios, including wearable or portable devices with limited computing resources that provide an augmented reality scenario.

[0098] AR System Overview

[0099] FIG. 1-2 Such scenarios are shown. For illustrative purposes, AR systems are used as examples of XR systems. FIG. 3 -8 shows an example AR system, including one or more processors, memory, sensors, and user interfaces that can operate in accordance with the techniques described herein.

[0100] Referring to FIG. 1 depicts an AR scenario 4 in which a user of AR technology sees a physical world park-like setting 6 featuring people, trees, buildings in the background, and a concrete platform 8. In addition to these items, the user of AR technology also sees what appears to be a robotic statue 10 standing on the physical world concrete platform 8 and a personified cartoon-like avatar character 2 flying by that looks like a yellowjacket, even though these elements (e.g., avatar character 2 and robotic statue 10) do not exist in the physical world. Due to the extreme complexity of the human visual perception and nervous system, it is challenging to produce AR technology that facilitates a comfortable, natural-feeling, rich presentation of virtual image elements in other virtual or physical world image elements.

[0101] Such AR scenarios can be implemented by systems that include a world reconstruction component that can build and update a representation of the physical world surfaces around the user. This representation can be used for occlusion rendering in physics-based interactions, placement of virtual objects, and for virtual character path planning and navigation, or for other operations that use information about the physical world. FIG. 2 depicts another example of an AR scenario 200 showing example world reconstruction use cases, including visual occlusion 202, physics-based interaction 204, and environment reasoning 206, in accordance with some embodiments.

[0102] The example scenario 200 is a living room with a wall, a bookshelf on one side of the wall, a floor lamp in a corner of the room, a floor, a sofa, and a coffee table on the floor. In addition to these physical items, a user of AR technology perceives virtual objects such as an image on the wall behind the sofa, a bird flying through the door, a deer peeking from the bookshelf, and a decoration in the form of a windmill placed on the coffee table. For the image on the wall, the AR technology needs information not only about the surface of the wall, but also about the objects and surfaces within the room that are occluding the image (e.g., the shape of the lamp) to correctly render the virtual object. For the flying bird, the AR technology needs information about all the objects and surfaces around the room to render the bird in realistic physics, to avoid or bounce off of objects and surfaces as the bird collides with them. For the deer, the AR technology needs information about the surface (e.g., the floor or the coffee table) to calculate where to place the deer. For the windmill, the system can identify that it is an object separate from the table and can infer that it is movable, while the corner of the bookshelf or the corner of the wall can be inferred to be stationary. Such distinctions can be used to infer which parts of the scene are used or updated in each of various operations.

[0103] A scene can be presented to a user by a system that includes a plurality of components, including a user interface that can stimulate one or more user senses (including visual, sound, and / or touch). In addition, the system can include one or more sensors that can measure parameters of physical portions of the scene, including a user's position and / or motion within a physical portion of the scene. Further, the system can include one or more computing devices with associated computer hardware (e.g., memory). These components can be integrated into a single device or distributed among multiple interconnected devices. In some embodiments, some or all of these components can be integrated into a wearable device.

[0104] FIG. 3 An AR system 302 configured to provide an experience of AR content interacting with a physical world 306 is depicted in accordance with some embodiments. The AR system 302 can include a display 308. In the illustrated embodiment, the display 308 can be worn by a user as part of a head-mounted device so that the user can wear the display over their eyes like a pair of goggles or glasses. At least a portion of the display can be transparent so that the user can observe see-through reality 310. The see-through reality 310 can correspond to a portion of the physical world 306 within a current viewpoint of the AR system 302, which can correspond to the viewpoint of the user if the user is wearing a head-mounted device that incorporates a display and sensors of the AR system to acquire information about the physical world.

[0105] AR content can also be rendered on display 308, overlaid on the see-through reality 310. To provide accurate interaction between the AR content and the see-through reality 310 on display 308, AR system 302 can include sensors 322 configured to capture information about the physical world 306.

[0106] The sensors 322 can include one or more depth sensors that output depth maps 312. Each depth map 312 can have a plurality of pixels, each of which can represent a distance to a surface in the physical world 306 in a particular direction relative to the depth sensor. Raw depth data can come from the depth sensors to create the depth maps. Such depth maps can be updated as fast as the depth sensors can form new images, up to hundreds of thousands of times per second. However, this data can be noisy and incomplete, and have holes shown as black pixels on the illustrated depth map.

[0107] The system can include other sensors, such as image sensors. Image sensors can acquire information that can be otherwise processed to represent the physical world. For example, images can be processed in a world reconstruction component 316 to create a mesh that represents connected portions of objects in the physical world. Metadata about such objects, including for example color and surface texture, can similarly be acquired by the sensors and stored as part of the world reconstruction.

[0108] The system can also acquire information about the head pose of the user relative to the physical world. In some embodiments, the sensors can include an inertial measurement unit that can be used to compute and / or determine a head pose 314. The head pose 314 for a depth map can indicate the current viewpoint at which the sensor captured the depth map, for example in six degrees of freedom (6DoF), but the head pose 314 can be used for other purposes, such as relating image information to a particular portion of the physical world or relating the position of a display worn on the user’s head to the physical world. In some embodiments, head pose information can be derived in other ways than an IMU, for example by analyzing objects in images.

[0109] The world reconstruction component 316 can receive the depth maps 312 and head poses 314 from the sensors, as well as any other data, and integrate that data into a reconstruction 318, which can at least appear to be a single combined reconstruction. The reconstruction 318 can be more complete and less noisy than the sensor data. The world reconstruction component 316 can update the reconstruction 318 using spatial and temporal averaging of sensor data from multiple viewpoints over time.

[0110] The reconstruction 318 can include representations of the physical world in one or more data formats including, for example, voxels, meshes, planes, etc. Different formats can represent alternative representations of the same portion of the physical world, or can represent different portions of the physical world. In the example shown, on the left side of the reconstruction 318, a portion of the physical world is presented as a global surface; on the right side of the reconstruction 318, a portion of the physical world is presented as a mesh.

[0111] The reconstruction 318 can be used for AR functions, such as generating a surface representation of the physical world for occlusion processing or physics-based processing. This surface representation can change as the user moves or objects in the physical world change. Aspects of the reconstruction 318 can be used, for example, by a component 320 that produces a global surface representation that changes in world coordinates, which can be used by other components.

[0112] AR content can be generated based on this information, for example, by the AR application 304. The AR application 304 can be, for example, a game program that performs one or more functions based on information about the physical world (e.g., visual occlusions, physics-based interactions, and environmental reasoning). It can perform these functions by querying data in different formats from the reconstruction 318 produced by the world reconstruction component 316. In some embodiments, the component 320 can be configured to output updates when the representation in a region of interest of the physical world changes. This region of interest can be set, for example, to approximate a portion of the physical world near a user of the system, such as a portion within the user's field of view, or projected (predicted / determined) to enter the user's field of view.

[0113] The AR application 304 can use this information to generate and update AR content. Virtual portions of the AR content can be presented on the display 308 in conjunction with the see-through reality 310, creating a realistic user experience.

[0114] In some embodiments, an AR experience can be provided to a user through a wearable display system. FIG. 4 An example of a wearable display system 80 (hereafter "system 80") is shown. The system 80 includes a head-mounted display device 62 (hereafter "display device 62") and various mechanical and electronic modules and systems to support the functionality of the display device 62. The display device 62 can be coupled to a frame 64 that can be worn by a display system user or viewer 60 (hereafter "user 60") and configured to position the display device 62 in front of the eyes of the user 60. According to various embodiments, the display device 62 can be a sequential display. The display device 62 can be monocular or binocular. In some embodiments, the display device 62 can be a head-mounted display (HMD). FIG. 3 An example of a display 308 in a wearable display system 80.

[0115] In some embodiments, a speaker 66 is coupled to the frame 64 and positioned proximate an ear canal of the user 60. In some embodiments, another speaker, not shown, is positioned proximate the other ear canal of the user 60 to provide stereo / shapeable sound control. The display device 62 is operatively coupled, e.g., by a wired lead or wireless connectivity 68, to a local data processing module 70, which can be mounted in a variety of configurations, i.e., fixedly attached to the frame 64, fixedly attached to a helmet or hat donned by the user 60, embedded in the ear piece, or otherwise detachably attached to the user 60 (e.g., in a backpack-style configuration, in a belt-coupling style configuration).

[0116] The local data processing module 70 can include a processor, as well as digital memory, such as nonvolatile memory (e.g., flash memory, etc.) both of which can be used to assist in the processing, caching, storage, and / or retrieval of data. Data includes a) data captured from sensors (which can be, e.g., operatively coupled to the frame 64) or otherwise attached to the user 60 (such as image capture devices (e.g., cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, radio devices, and / or gyros), and / or b) data acquired using remote processing modules 72 and / or remote data repositories 74, possibly for passage to the display device 62 after such processing or retrieval. The local data processing module 70 can be operatively coupled by communication links 76, 78 (e.g., via wired or wireless communication links) to the remote processing modules 72 and remote data repositories 74, respectively, such that these remote modules 72, 74 are operatively coupled to each other and available as resources to the local processing and data module 70. In some embodiments, FIG. 3 The world reconstruction component 316 in the system 300 can be implemented at least partially in the local data processing module 70. For example, the local data processing module 70 can be configured to execute computer-executable instructions to generate a physical world representation based at least in part on at least a portion of the data.

[0117] In some embodiments, the local data processing module 70 can include one or more processors (e.g., graphics processing units (GPUs)) configured to analyze and process data and / or image information. In some embodiments, the local data processing module 70 can include a single processor (e.g., a single core or multi-core ARM processor), which would limit the computational budget of the module 70 but enable a smaller device. In some embodiments, the world reconstruction component 316 can use less than the computational budget of a single ARM core to generate a physical world representation in real-time on a non-predefined space, such that the remaining computational budget of the single ARM core can be accessed for other purposes, such as extracting meshes.

[0118] In some embodiments, the remote data repository 74 can comprise a digital data storage facility, which can be used through the Internet or other network configuration in a "cloud" resource configuration. In some embodiments, all data is stored and all computations are performed in the local data processing module 70, allowing fully autonomous use from a remote module. World reconstruction, for example, can be stored in this repository 74 in whole or in part.

[0119] In some embodiments, the local data processing module 70 is operatively coupled to a battery 82. In some embodiments, the battery 82 is a removable power source, such as over a counter battery. In other embodiments, the battery 82 is a lithium ion battery. In some embodiments, the battery 82 comprises both an internal lithium ion battery that can be charged by the user 60 during non-operation time of the system 80 and a removable battery, so that the user 60 can operate the system 80 for longer periods of time without having to connect to a power source to charge the lithium ion battery or having to shut down the system 80 to replace the battery.

[0120] FIG. 5A A user 30 is shown wearing an AR display system that renders AR content as the user 30 moves in a physical world environment 32 (hereinafter "environment 32"). The user 30 places the AR display system at a location 34, and the AR display system records passable world information (e.g., digital representations of real objects in the physical world, which can be stored and updated as real objects in the physical world change) relative to the environment of location 34, such as poses related to mapping features or directional audio inputs. The location 34 is aggregated to a data input 36, and is processed by at least a passable world module 38, which can be implemented through processing on a remote processing module 72 of FIG. 3 In some embodiments, the passable world module 38 can comprise a world reconstruction component 316.

[0121] The tethered world module 38 determines where and how AR content 40 can be placed in the physical world as determined from the data input 36. By presenting a representation of the physical world and AR content via a user interface, the AR content is "placed" in the physical world, where the AR content is rendered as if it is interacting with objects in the physical world, and objects in the physical world are rendered as if the AR content is occluding the user's view of those objects when appropriate. In some embodiments, AR content 40 can be placed by determining its shape and position from appropriately selecting portions of a fixed element 42 (e.g., a table) from a reconstruction (e.g., reconstruction 318). As an example, the fixed element can be a table, and virtual content can be placed so that it appears to be on that table. In some embodiments, AR content can be placed within structures in the field of view 44, which can be the current field of view or an estimated future field of view. In some embodiments, AR content can be placed with respect to a mapped mesh model 46 of the physical world.

[0122] As depicted, the fixed element 42 serves as a proxy for any fixed element within the physical world, which can be stored in the tethered world module 38 so that the user 30 can perceive content on the fixed element 42 without the system having to map to the fixed element 42 every time the user 30 sees it. The fixed element 42 can thus be a mapped mesh model from a previous modeling session, or determined from a separate user, but both stored on the tethered world module 38 for future reference by multiple users. Thus, the tethered world module 38 can recognize the environment 32 from a previously mapped environment and display AR content without the user's 30 device having to first map the environment 32, saving computational processes and cycles and avoiding any delay in rendered AR content.

[0123] The mapped mesh model 46 of the physical world can be created by the AR display system, and appropriate surfaces and metrics for interacting with and displaying AR content 40 can be mapped and stored in the tethered world module 38 for future retrieval by the user 30 or other users without having to re-map or model. In some embodiments, the data input 36 is input such as a geographic location, user identification, and current activity to indicate to the tethered world module 38 which of one or more fixed elements 42 is available, which AR content 40 was last placed on the fixed element 42, and whether to display that same content (which is "persistent" content regardless of whether the user views a particular tethered world model or not).

[0124] FIG. 5BA schematic view of the viewing optics assembly 48 and accompanying components is shown. In some embodiments, facing the user's eyes 49, two eye tracking cameras 50 detect metrics of the user's eyes 49, such as eye shape, eyelid occlusion, pupil direction, and glints on the user's eyes 49. In some embodiments, a depth sensor 51 (e.g., a time-of-flight sensor) emits a relay signal to the world to determine distance to a given object. In some embodiments, a world camera 52 records a greater-than-peripheral view to map the environment 32 and detect inputs that can affect AR content. A camera 53 can further capture a specific timestamp of the physical world image within the user's field of view. Each of the world camera 52, camera 53, and depth sensor 51 has a respective field of view 54, 55, and 56 to collect data from and record the physical world scene of the physical world environment 32 shown, such as FIG. 5A

[0125] An inertial measurement unit 57 can determine motion and orientation of the viewing optics assembly 48. In some embodiments, each component is operatively coupled to at least one other component. For example, the depth sensor 51 can be operatively coupled to the eye tracking cameras 50 as a confirmation of measured accommodation relative to the actual distance the user's eyes 49 are looking.

[0126] Information from these sensors in the viewing optics assembly 48 can be coupled to one or more processors in the system. The processors can generate data that can be rendered to cause the user to perceive virtual content that interacts with objects in the physical world. This rendering can be accomplished in any suitable manner, including generating image data that depicts both physical and virtual objects. In other embodiments, physical and virtual content can be depicted in one scene by modulating the opacity of a display device through which the user views the physical world. The opacity can be controlled to create the appearance of virtual objects and also to block the user from seeing objects in the physical world that are occluded by the virtual objects. Regardless of how the content is presented to the user, a model of the physical world is needed so that properties of virtual objects that can be affected by physical objects, including the shape, position, motion, and visibility of virtual objects, can be correctly calculated. In some embodiments, the model can include a reconstruction of the physical world, such as the reconstruction 318.

[0127] This model can be created from data collected from sensors on the user's wearable device. However, in some embodiments, the model can be created from data collected by multiple users, which can be aggregated in a computing device that is remote from all of the users (and can be "in the cloud").

[0128] The model can be created at least in part by a world reconstruction system (e.g., FIG. 6 depicted in greater detail in​FIG. 3 The world reconstruction component 316 is used to create the model. The world reconstruction component 316 may include a sensing module 160 that generates, updates, and stores a representation of a portion of the physical world. In some embodiments, the sensing module 160 may represent a portion of the physical world within the reconstruction range of a sensor as a plurality of voxels. Each voxel may correspond to a 3D cube of a predetermined volume in the physical world and includes surface information indicating the presence of a surface within the volume represented by the voxel. Values ​​may be assigned to voxels indicating whether their corresponding volume has been determined to contain a surface of a physical object, is determined to be empty, or has not yet been measured by a sensor and therefore its value is unknown. It should be understood that it is not necessary to explicitly store the values ​​of voxels indicating that they are determined to be empty or unknown, as the values ​​of voxels can be stored in computer memory in any suitable manner, including not storing information about voxels determined to be empty or unknown.

[0129] FIG. 7A An example of a 3D space 100 discrete as voxels 102 is depicted. In some embodiments, the perception module 160 can identify objects of interest and set the volume of the voxels to capture features of the objects of interest and avoid redundant information. For example, the perception module 160 can be configured to identify large objects and surfaces, such as walls, ceilings, floors, and large furniture. Therefore, the volume of the voxels can be set to a relatively large size, such as 4 cm. 3 A cube.

[0130] The reconstruction of the physical world, including voxels, can be called a volumetric model. As sensors move through the physical world, information is created over time to build the volumetric model. This occurs when a user of a wearable device that includes sensors moves around. FIG. 8A -F depicts an example of reconstructing the physical world as a volumetric model. In the example shown, the physical world includes... FIG. 8A A portion 180 of the surface shown. FIG. 8A In the first position, the sensor 182 may have a field of view 184, within which a portion 180 of the surface is visible.

[0131] Sensor 182 can be of any suitable type, such as a depth sensor. However, depth data can be obtained from an image sensor or otherwise. Sensing module 160 can receive data from sensor 182 and then... FIG. 8B The values ​​of multiple voxels 186 are set to represent portions 180 of the surface visible in the field of view 184 by the sensor 182.

[0132] exist FIG. 8C In this configuration, sensor 182 can be moved to a second position and has a field of view 188. For example...FIG. 8D As shown, another set of voxels becomes visible, and the values of these voxels can be set to indicate the location where the surface has entered the portion of the field of view 188 of the sensor 182. The values of these voxels can be added to the volume model for the surface.

[0133] In FIG. 8E , the sensor 182 can further move to a third position and have a field of view 190. In the illustrated example, additional portions of the surface become visible in the field of view 190. As FIG. 8F shown, another set of voxels can become visible, and the values of these voxels can be set to indicate the location where the surface has entered the portion of the field of view 190 of the sensor 182. The values of these voxels can be added to the volume model for the surface. As FIG. 6 shown, this information can be stored as part of the persistent world as volume information 162a. Information about the surface, such as color or texture, can also be stored. Such information can be stored as, for example, volume metadata 162b.

[0134] In addition to generating information for the persistent world representation, the perception module 160 can also identify and output indications of changes in the area around the user of the AR system. Such indications of changes can trigger updates to volume data stored as part of the persistent world, or trigger other functions, such as triggering the trigger component 304 that generates AR content to update that AR content.

[0135] In some embodiments, the perception module 160 can identify changes based on a signed distance function (SDF) model. The perception module 160 can be configured to receive sensor data such as depth maps 160a and head pose 160b, and then fuse the sensor data into an SDF model 160c. Depth maps 160a can provide SDF information directly, and images can be processed to obtain SDF information. The SDF information represents distances from the sensors used to capture that information. Since those sensors can be part of a wearable unit, the SDF information can represent the physical world from the perspective of the wearable unit, and thus from the perspective of the user. The head pose 160b can enable the SDF information to be related to voxels in the physical world.

[0136] Referring back to FIG. 6In some embodiments, the perception module 160 can generate, update, and store a representation of the portion of the physical world within the perception range. The perception range can be determined based at least in part on the reconstruction range of the sensor, which can be determined based at least in part on the limits of the observation range of the sensor. As a particular example, an active depth sensor operating using active IR pulses can reliably operate over a range of distances, creating an observation range for the sensor that can be from a few centimeters or tens of centimeters to a few meters.

[0137] FIG. 7B A reconstruction range is depicted relative to the sensor 104 having a viewpoint 106. A reconstruction of the 3D space within the viewpoint 106 can be constructed based on data captured by the sensor 104. In the example shown, the observation range of the sensor 104 is from 40 cm to 5 m. In some embodiments, the reconstruction range of a sensor can be determined to be less than the observation range of the sensor, as sensor output near its observation limits can be more noisy, incomplete, and inaccurate. For example, in the example shown of 40 cm to 5 m, the corresponding reconstruction range can be set to be from 1 to 3 m, and data collected by the sensor indicating surfaces outside of this range can not be employed.

[0138] In some embodiments, the perception range can be greater than the reconstruction range of the sensor. If a component 164 that uses data about the physical world requires data about regions within the perception range that are outside of the portion of the physical world within the current reconstruction range, this information can be provided from the persistent world 162. Accordingly, information about the physical world is readily accessible by querying. In some embodiments, an API can be provided to respond to such queries, providing information about the current perception range of the user. Such techniques can reduce the time required to access existing reconstructions and provide an improved user experience.

[0139] In some embodiments, the perception range can be a 3D space corresponding to a bounding box centered around the user's location. As the user moves, the portion of the physical world within the perception range that is queryable by the component 164 can move with the user. FIG. 7C A bounding box 110 centered at location 112 is depicted. It will be appreciated that the size of the bounding box 110 can be set to encompass the observation range of the sensor with a reasonable extension, as the user cannot move at an unreasonable speed. In the example shown, the observation limit of the sensor worn by the user is 5 m. The bounding box 110 is set to be a 20 m 3 cube.

[0140] Returning FIG. 6In some embodiments, the persistent world module 162 can receive a representation of the physical world based on data acquired by the perception module 160. The persistent world module 162 can also include representations of the physical world in various formats. For example, volumetric metadata such as voxels 162b can be stored as well as meshes 162c and planes 162d. In some embodiments, other information such as depth maps can be saved.

[0141] In some embodiments, the perception module 160 can include modules that generate representations for the physical world in various formats including, for example, meshes 160d, planes, and semantics 160e. These modules can generate representations based on data within the perception range of one or more sensors at the time the representation is generated as well as data captured at previous times and information in the persistent world 162. In some embodiments, these components can operate on depth information captured with a depth sensor. However, an AR system can include visual sensors and can generate such representations by analyzing monocular or binocular visual information.

[0142] In some embodiments, these modules can operate on regions of the physical world such as regions represented by tiles or patches as described below. When the perception module 160 detects a change in the physical world in other sub-regions, those modules can be triggered to update the tile or patch of the physical world or the sub-region. For example, such a change can be detected by detecting a new surface in the SDF model 160c or other criteria such as changing the value of a sufficient number of voxels representing the sub-region.

[0143] The world reconstruction component 316 can include components 164 that can receive representations of the physical world from the perception module 160. Information about the physical world can be pulled by these components according to, for example, usage requests from applications. In some embodiments, information can be pushed to usage components, for example, via an indication of a change in a pre-identified region or a change in the representation of the physical world within the perception range. The components 164 can include, for example, game programs and other components that perform processing for visual occlusion, physics-based interactions, and environmental reasoning.

[0144] In response to queries from the components 164, the perception module 160 can send representations of the physical world in one or more formats. For example, when the component 164 indicates that the usage is for visual occlusion or physics-based interactions, the perception module 160 can send a representation of surfaces. When the component 164 indicates that the usage is for environmental reasoning, the perception module 160 can send meshes, planes, and semantics of the physical world.

[0145] In some embodiments, the perception module 160 can include components that format information to provide components 164. An example of such a component can be a raycast component 160f. Using components (e.g., components 164) can query information about the physical world from a particular viewpoint, for example. The raycast component 160f can select from one or more representations of the physical world data within the field of view from that viewpoint.

[0146] Viewpoint-dependent brick selection for fast volume reconstruction

[0147] From the foregoing description, it should be appreciated that the perception module 160 or another component of the AR system can process data to create a 3D representation of a portion of the physical world. The data to be processed can be reduced by culling portions of the 3D reconstruction volume based at least in part on camera frustums and / or depth images, extracting and maintaining planar data, capturing, maintaining, and updating 3D reconstruction data in tiles that allow for local updates while maintaining neighbor consistency, providing occlusion data to applications that generate such scenes (where the occlusion data is derived from a combination of one or more depth data sources), and / or performing multi-level mesh simplification.

[0148] The world reconstruction system can integrate sensor data over time from multiple viewpoints of the physical world. As the device including the sensors moves, the pose (e.g., position and orientation) of the sensors can be tracked. Because the frame pose of the sensors is known and how it relates to other poses, each of these multiple viewpoints of the physical world can be fused into a single, combined reconstruction. By using spatial and temporal averaging (i.e., averaging data over time from multiple viewpoints), the reconstruction can be more complete and less noisy than the original sensor data.

[0149] The reconstruction can contain data of different levels of complexity, including, for example, raw data (e.g., real-time depth data), fused volumetric data (e.g., voxels), and computed data (e.g., meshes).

[0150] In some embodiments, AR and MR systems represent a 3D scene with a regular grid of voxels, where each voxel can contain a signed distance field (SDF) value. The SDF value describes whether the voxel is inside or outside a surface in the scene to be reconstructed and the distance from the voxel to that surface. Computing 3D reconstruction data for the volume needed to represent the scene requires a large amount of memory and processing power. Because the number of variables needed for the 3D reconstruction grows cubically with the number of depth images processed, these requirements also increase for scenes that represent larger spaces.

[0151] Described herein are efficient ways to reduce processing. According to some embodiments, a scene can be represented by one or more bricks. Each brick can include a plurality of voxels. A set of bricks representing the scene can be culled by selecting bricks to be processed to generate a 3D reconstruction of the scene based on a frustum derived from a field of view (FOV) of an image sensor and / or a depth image (or "depth map") of the scene created using a depth sensor.

[0152] A depth image can have one or more pixels, each pixel representing a distance to a surface in the scene. These distances can be relative to a position of the image sensor, such that data output from the image sensor can be selectively processed. Image data can be processed for those bricks representing portions of the 3D scene that contain surfaces visible from the perspective (or "point of view") of the image sensor. Processing of some or all of the remaining bricks can be omitted. In this way, the selected bricks can be bricks that can contain new information that can be obtained by culling bricks for which the output of the image sensor is unlikely to provide useful information about. Data output from the image sensor is unlikely to provide useful information about bricks that are closer or further away from the image sensor than the surfaces indicated by the depth map, as these are empty space or behind the surface and thus not depicted in the image from the image sensor.

[0153] In some embodiments, one or more criteria can be applied to efficiently select a set of bricks for processing. An initial set of bricks can be limited to those bricks within a frustum of the image sensor. A large number of bricks outside the frustum can then be culled. Further computationally intensive processing to update the 3D reconstruction can then be performed on a subset of bricks accepted for processing after culling. Thus, using processing of a reduced number of voxels, the 3D representation of the scene to be updated can be computed more efficiently.

[0154] By culling bricks based on depth images, greater reductions in processing can be achieved. According to some embodiments, culling and / or acceptance of bricks can be performed by projecting an outline of each brick in an initial set into a depth image. Such culling can be based on whether the brick corresponds to a portion of the scene indicated by the depth image to be near a surface. Bricks that can be simply identified as being entirely in front of a surface or entirely behind a surface can be culled. In some embodiments, such determinations can be made efficiently. For example, a bounding box around the projection of a brick into a depth image can be used to determine a maximum brick value and a minimum brick value in a z-coordinate direction, which can be substantially perpendicular to the 2D plane of the depth image. By comparing these maximum and minimum brick values to distances represented by pixels in the depth image, bricks can be culled and / or accepted for further processing. Such processing can result in selection of bricks for initial processing that intersect a surface reflected in the depth image and / or are in front of a surface reflected in the depth image. In some embodiments, such processing can distinguish between bricks that are in front of a solid surface and bricks that are in front of a porous surface (i.e., bricks representing areas for which the depth sensor cannot reliably measure a distance to a surface).

[0155] In some embodiments, the culling / acceptance criteria can result in classification of some or all bricks accepted for further processing such that processing algorithms for computing volume reconstructions can be tailored to characteristics of the bricks. In some embodiments, different processing can be selected based on whether a brick is classified as intersecting a surface, being in front of a solid surface, or being in front of a porous surface.

[0156] FIG. 9 A cross-sectional view of a scene 400 along a plane parallel to the y- and z- coordinates is shown. An XR system can represent the scene 400 by a voxel grid 504. Conventional XR systems can update each voxel of the voxel grid based on each new depth image captured by a sensor 406, which can be an image sensor or a depth sensor, such that a 3D reconstruction generated from the voxel grid can reflect changes in the scene. Updating in this way can consume a large amount of computational resources and also cause artifacts at the output of the XR system due to, for example, heavy computation causing time delays.

[0157] Described herein are techniques to provide accurate 3D reconstruction data with low computational resource utilization, for example, by culling portions of the voxel grid 504 based at least in part on the camera frustum 404 of the image sensor 406 and / or depth images captured by the image sensor.

[0158] In the illustrated example, image sensor 406 captures a depth image (not shown) that includes surface 402 of scene 400. The depth image can be stored in computer memory in any convenient way that captures distances between some reference point and surfaces in scene 400. In some embodiments, the depth image can be represented as values in a plane parallel to the x- and y-axes, as shown in FIG. 9 where the reference point is the origin of the coordinate system. Positions in the X-Y plane can correspond to directions relative to the reference point, and the values at those pixel positions can indicate distances from the reference point to the nearest surface in the direction indicated by the coordinate in the plane. Such a depth image can include a grid (not shown) of pixels in a plane parallel to the x- and y-axes. Each pixel can indicate a distance from image sensor 406 to surface 402 in a particular direction. In some embodiments, a depth sensor can be unable to measure a distance to a surface in a particular direction. This can occur, for example, if the surface is beyond the range of image sensor 406. In some embodiments, a depth sensor can be an active depth sensor that measures distances based on reflected energy, but a surface can not reflect enough energy for an accurate measurement. Thus, in some embodiments, a depth image can have "holes" at which pixels are not assigned a value.

[0159] In some embodiments, the reference point of a depth image can vary. Such a configuration can allow a depth image to represent surfaces in an entire 3D scene, without being limited to a portion having a predetermined and limited angular range relative to a particular reference point. In such embodiments, a depth image can indicate distances to surfaces as image sensor 406 moves through six degrees of freedom (6DOF). In these embodiments, a depth image can include a set of pixels for each of a plurality of reference points. In these embodiments, a portion of a depth image can be selected based on a "camera pose," which represents a direction and / or orientation in which image sensor 406 is pointed when image data is captured.

[0160] Image sensor 406 can have a field of view (FOV), which can be represented by camera frustum 404. In some embodiments, the depicted infinite camera frustum can be reduced to a finite 3D trapezoidal prism 408 by assuming a maximum depth 410 that image sensor 406 can provide and / or a minimum depth 412 that image sensor 406 can provide. 3D trapezoidal prism 408 can be a convex polyhedron bounded at six planes.

[0161] In some embodiments, one or more voxels 504 can be grouped into a brick 502. FIG. 10A A portion 500 of scene 400 is shown, which includes eight bricks 502. FIG. 10B A portion 500 of scene 400 is shown, which includes eight bricks 502. 3Example brick 502 of voxel 504. (Reference) FIG. 9 Scene 400 may include one or more bricks, in FIG. 4 The view shown illustrates sixteen of them. Each brick-connecting type brick is identified by, for example,

[0000] -

[0015] .

[0162] FIG. 11 A volumetric representation hierarchy that can be implemented in some embodiments is depicted. In some embodiments, such a volumetric representation hierarchy can reduce latency in data transmission. In some embodiments, a voxel grid of the physical world can be mapped to conform to the structure of the storage architecture of a processor used to compute AR content, such as a processor executing component 304 thereon. One or more voxels can be grouped into “bricks.” One or more bricks can be grouped into “tiles.” The size of a tile can correspond to a storage page on the processor’s local storage medium. Tiles can be moved between local and remote storage, for example, via a wireless connection, based on usage or anticipated usage, according to a storage management algorithm.

[0163] In some embodiments, uploads and / or downloads between the sensing module 160 and the persistent world module 162 can be performed on multiple tiles in a single operation. One or more tiles can be grouped into a “RAM tile set.” The size of the RAM tile set can correspond to the area within the reconstruction range of a sensor worn by the user. One or more RAM tile sets can be grouped into a “global tile set.” The size of the global tile set can correspond to the sensing range of the world reconstruction system (e.g., the sensing range of the sensing module 160).

[0164] FIG. 12 This is a flowchart illustrating a method 600 for generating a 3D reconstruction of a scene using an operational computing system according to some embodiments. Method 600 may begin by representing a scene (e.g., scene 400) with one or more tiles (e.g., tile 502), each tile comprising one or more voxels (e.g., voxel 504). Each tile may represent a portion of the scene. The tiles may be identifiable relative to a persistent coordinate system such that even if the pose of an image sensor (e.g., image sensor 406) changes, the same tile represents the same volume in the scene.

[0165] In action 604, method 600 can capture a depth image (e.g., a depth image including surface 402) from a depth sensor (e.g., depth sensor 406). The depth sensor can be an active depth sensor that transmits, for example, IR radiation for reflection and measures time of flight. Each such measurement represents the distance from the depth sensor to the surface in a specific direction. This depth information can represent the same volume as the volume represented by the brick.

[0166] At action 606, the method 600 can cull a portion of the one or more bricks for the camera frustum (e.g., the finite 3D trapezoidal prism 408 derived from the camera frustum 404) to produce a first one or more bricks, which is a reduced set of bricks from the one or more bricks. Such culling can eliminate bricks representing portions of the scene that were outside the field of view of the image sensor at the time the image data being processed was acquired. The image data is therefore less likely to contain information useful to create or update a brick.

[0167] At action 608, the method 600 can cull a portion of the first one or more bricks for the depth image to produce a second one or more bricks, which is a reduced set of bricks from the first one or more bricks. At action 610, the method 600 can generate a 3D reconstruction of the scene based on the second one or more bricks.

[0168] Returning to FIG. 9 , given a surface 402 captured by a depth image and a corresponding camera pose, voxels between the image sensor 406 and the surface 402 can be empty. The further a voxel is behind the surface 402 from the image sensor 406, the less likely it is to determine that the voxel represents an interior or empty space of an object. The degree of certainty can be represented by a weight function that weights voxel updates based on distance to the surface 402. When the weight function of a voxel located behind the surface 402 (farther from the image sensor 406) is above a threshold, the voxel can not be updated or have a zero update (e.g., an update of zero change). Also, all voxels that do not fall into the camera frustum 404 can not be updated or investigated for the depth image.

[0169] The method 600 can not only improve the processing speed of volumetric depth image fusion, but also consume less memory storage, which makes the method 600 possible to run on wearable hardware. For example, having 1 cm 3A small reconstruction volume of 5m*5m*3m with a voxel size and 8 bytes per voxel (4 bytes for distance value and 4 bytes for weight value) would already require approximately 600MB. The method 600 can classify the bricks according to their distance to the surface relative to a truncated threshold. For example, the method 600 can identify empty bricks (e.g., bricks that are culled, or bricks that are farther than the truncated threshold from the surface) so that non-empty bricks are allocated memory space. The method 600 can also identify bricks that are farther than the truncated threshold from the surface so that these bricks are stored with a constant distance value of negative truncated threshold and weight 1. The method 600 can also identify bricks that have a distance to the surface between zero and the truncated threshold so that these bricks are stored with a constant SDF value of positive truncated threshold but with a varying weight. Storing distance or weight values that are constant for bricks with a single value can be an entropy-based compression for zero-entropy fields.

[0170] The method 600 can allow bricks to be marked as "not containing any part of the surface" during voxel updates, which can significantly speed up processing of the bricks. The processing can include, for example, converting an image of a portion of the scene represented by the bricks to a mesh.

[0171] FIG. 13 An exemplary method 606 of culling a portion of one or more bricks for a camera frustum 404 of an image sensor 406 is shown in accordance with some embodiments. The method 606 can begin by finding an axis-aligned bounding box (AABB) for containing the camera frustum 404. The AABB can contain one or more bricks in the scene. The method 606 can include dividing (act 704) the AABB into one or more sub-AABBs, and performing (act 706) a camera frustum acceptance test. If the method 606 determines at act 708 that a sub-AABB reaches the size of a brick, the method 606 can generate (act 710) a first one or more bricks. If the method 606 determines at act 708 that a sub-AABB is larger than the size of a brick, the method 606 can repeat acts 704-708 until a sub-AABB reaches the size of a brick.

[0172] For example, given a 3D trapezoidal prism 408 corresponding to a camera frustum 404, an AABB whose side length is a power of two and which contains the 3D trapezoidal prism 408 can be found in constant time. The AABB can be divided into eight sub-AABBs. Each of the eight sub-AABBs can be tested for intersection with the camera frustum 404. When a sub-AABB is determined not to intersect the camera frustum 404, the brick corresponding to that sub-AABB can be culled. The culled brick can be rejected from further processing. When a sub-AABB is determined to intersect the camera frustum 404, the sub-AABB can be further divided into eight sub-AABBs of the sub-AABB. Each of the eight sub-AABBs of the sub-AABB can then be tested for intersection with the camera frustum 404. The iteration of dividing and testing continues until the sub-AABBs correspond to individual bricks. To determine whether the camera frustum 404 intersects an AABB, a two-step test can be performed. First, at least one corner point of the AABB can be tested to see if it lies within each plane that bounds the camera frustum 404. Second, each corner point of the camera frustum 404 can be tested to see if it lies inside the AABB so that some cases of an AABB that does not intersect the camera frustum 404 but is incorrectly classified as partially inside (e.g., only one corner point on the edge of the frustum) can be captured.

[0173] An ideal byproduct of this two-step test is that for each brick that intersects the camera frustum 404, it can be known whether it is entirely inside the camera frustum 404 or only partially inside the camera frustum 404. For bricks that are entirely inside the camera frustum 404, later when the individual voxels are updated, a test of whether each voxel is inside the camera frustum 404 can be skipped.

[0174] FIG. 14 An exemplary method 706 of performing camera frustum acceptance testing is shown in accordance with some embodiments. The method 706 can begin by testing (act 802) each of one or more sub-AABBs with respect to each plane that bounds a camera frustum 404. At act 804, the method 706 can determine whether the tested sub-AABB is entirely outside the camera frustum 404. At act 806, if the tested sub-AABB is determined to be entirely outside the camera frustum 404, the method 706 can cull all bricks contained by the tested sub-AABB. At act 808, if the tested sub-AABB is determined not to be entirely outside the camera frustum 404, the method 706 can determine whether the tested sub-AABB is entirely inside the camera frustum 404.

[0175] At action 810, if it is determined that the tested sub-AABB is entirely inside the camera frustum 404, the method 706 can add all the bricks encompassed by the tested sub-AABB to the first one or more bricks. At action 708, if it is determined that the tested sub-AABB is not entirely inside the camera frustum 404, which can indicate that the tested sub-AABB intersects the camera frustum 404, the method 706 can determine whether the tested sub-AABB reaches the size of a brick.

[0176] At action 814, if it is determined that the tested sub-AABB is equal to the size of a brick, the method 706 can further determine whether each corner point of the camera frustum 404 is inside the tested sub-AABB. If it is determined that each corner point of the camera frustum 404 is inside the brick of the tested sub-AABB, the method 706 can cull (action 806) the brick of the tested sub-AABB. If it is determined that not every corner point of the camera frustum is inside the brick of the tested sub-AABB, the method 706 can add (action 810) the brick of the tested sub-AABB to the first one or more bricks.

[0177] FIG. 15 An exemplary method 608 of culling a portion of the first one or more bricks for a depth image is shown in accordance with some embodiments. The method 608 can begin by performing (action 902) a first depth image acceptance test on each of the first one or more bricks. At action 904, the method 808 can determine whether the tested brick is accepted by the first depth image acceptance test. If it is determined that the tested brick is accepted by the first depth image acceptance test, which can indicate that the tested brick intersects a surface in the scene, the method 608 can apply (action 906) a varying delta to the selected voxels and add (action 914) the tested brick to the second one or more bricks.

[0178] At action 908, if it is determined that the tested brick is not accepted by the first depth image acceptance test, the method 608 can perform (action 908) a second depth image acceptance test on the tested brick. At action 910, the method 608 can determine whether the tested brick is accepted by the second depth image acceptance test. If it is determined that the tested brick is accepted by the second depth image acceptance test, which can indicate that the tested brick is in front of a solid or a hole background in the scene, the method 608 can apply (action 912) a constant delta to all voxels or selected voxels and then add (action 914) the tested brick to the second one or more bricks. If it is determined that the tested brick is not accepted by the second depth image acceptance test, the method 608 can cull (action 916) the tested brick.

[0179] FIG. 16An exemplary method 902 of performing a first depth image acceptance test is shown in accordance with some embodiments. For each tile to be tested, the method 902 can begin by determining (act 1002) a minimum tile value (bmin) and a maximum tile value (bmax) along a direction parallel to the z-coordinate. The bmin value and bmax value can be populated to account for an integration threshold, beyond which depth values indicate a constant update to the voxels in the tile. At act 1004, the method 902 can compute the 2D pixel positions of the corners of the tile being tested by projecting the corners of the tile into the depth image. At act 1006, the method 902 can compute a rectangle by establishing a convex hull of the 2D pixel positions of the corners of the tile. At act 1008, the method 902 can test each pixel in the rectangle against the bmin value and bmax value. At act 1010, the method 902 can determine whether all pixels in the rectangle have depth values between the bmin value and bmax value. If it is determined that all pixels in the rectangle have depth values between the bmin value and bmax value, the method 902 can accept (act 1012) the tile. If it is determined that not all pixels in the rectangle have depth values between the bmin value and bmax value, the method 902 can perform (act 908) a second depth image acceptance test for the tile.

[0180] FIG. 17 An exemplary method 908 of performing a second depth image acceptance test is shown in accordance with some embodiments. The method 908 can begin at act 1102 by classifying all pixels in the rectangle against the bmin value and bmax value. At act 1104, the method 908 can determine whether the tile being tested is in front of a solid or a hole background, for example, by using the table shown in FIG. 18

[0181] FIG. 19A -F depicts an example of a tile representing a scene being culled for a camera frustum 192. In FIG. 19A FIG. 19B ​​In the middle, a single AABB 194a is divided into four sub-AABBs 194b, each of which includes 8x8 tiles. After performing camera frustum acceptance testing (e.g., method 706), one of the four sub-AABBs 194b fails the camera frustum acceptance testing, so the 8x8 tiles in the failed sub-AABB 194b are culled and illustrated as white tiles. In FIG. 19C In the middle, each of the three sub-AABBs 194 that passed the camera frustum acceptance testing is further divided into four sub-AABBs 194c, each of which includes 4x4 tiles. After performing camera frustum acceptance testing (e.g., method 706), eight of the sixteen sub-AABBs 194c fail the camera frustum acceptance testing, so the tiles in the failed sub-AABBs 194c are culled and illustrated as white tiles. Similarly, in FIG. 19D In the middle, the sub-AABB 194d includes 2x2 tiles. In FIG. 19E In the middle, the sub-AABB 194e includes a single tile, and so the sub-AABB 194e that passed the camera frustum test and the corresponding tile are produced as the first plurality of tiles and illustrated as a gray tile 196f in FIG. 19F In the middle, the sub-AABB 194e includes a single tile, and so the sub-AABB 194e that passed the camera frustum test and the corresponding tile are produced as the first plurality of tiles and illustrated as a gray tile 196f in

[0182] FIG. 20A An example of further culling 34 tiles 196f for a depth image including surface 220 by performing, for example, method 608 is depicted. FIG. 20B The culling result for the depth image is depicted, showing that 12 tiles 222a of the 34 tiles 196f passed a first depth image acceptance test (e.g., method 904), 9 tiles 222b of the 34 tiles 196f passed a second depth image acceptance test (e.g., method 910), and finally, 13 tiles 222c of the 34 tiles 196f were culled after culling for a depth image including surface 220. As a result, in the illustrated example, the number of tiles computed by the world reconstruction component was further reduced to 21 tiles by tile culling for the depth image. It should also be understood that, as a result of the first and second depth image acceptance tests, the computation speed of the world reconstruction component can be accelerated not only by reducing the number of tiles, but also by the classification of tiles. For example, as discussed with respect to FIG. 15As discussed, a constant delta can be applied to the 9 bricks 222b that did not pass the first depth image acceptance test but passed the second depth image acceptance test. Batch applying a constant delta can further improve the speed of computation compared to applying a variable delta to each voxel. The geometry (e.g., planes) in a scene can be obtained in an XR system to support applications, such as placing a wall for a virtual screen and / or a floor for navigating a virtual robot. A common representation of the geometry of a scene is a mesh, which can include a set of connected triangles with vertices and edges. Conventionally, the geometry in a scene is obtained by generating a mesh for the scene and searching the mesh for the geometry, which takes some time (e.g., seconds) to process and does not indicate relationships between the geometry requested by different queries. For example, a first query can be for a table plane. In response to the first query, the system can find the table plane and leave a watch on the table plane. A second query can then be for the watch. In response to the second query, the system can find all possible table planes and check whether the watch is on each table plane until the watch is found, as there is no indication from the response to the first query whether the table plane is the table plane.

[0183] A geometry extraction system is described herein. In some embodiments, the geometry extraction system can extract geometry as a scene is scanned with cameras and / or sensors, which allows for fast and efficient extraction that can adapt to dynamic environmental changes. In some embodiments, the geometry extraction system can maintain extracted geometry in local and / or remote storage. The maintained geometry can have a unique identification such that, for example, different queries at different timestamps and / or from different applications can share the maintained geometry. In some embodiments, the geometry extraction system can support different representations of geometry depending on individual queries. In the description below, FIG. 21-29 In the description below, planes are used as an example geometry. It should be understood that the geometry extraction system can detect other geometries, instead of or in addition to planes, for use in subsequent processing, including, for example, cylinders, cubes, lines, corners, or semantics such as glass surfaces or holes. In some embodiments, the principles described herein with respect to geometry extraction can be applicable to object extraction, etc.

[0184] FIG. 21 A plane extraction system 1300 according to some embodiments is shown. The plane extraction system 1300 can include a depth fusion 1304, which can receive a plurality of depth maps 1302. The plurality of depth maps 1302 can be created by one or more users wearing depth sensors and / or downloaded from local / remote storage. The plurality of depth maps 1302 can represent multiple views of the same surface. There can be discrepancies between the plurality of depth maps, which can be reconciled by the depth fusion 1304.

[0185] In some embodiments, the deep fusion 1304 can generate SDFs 1306 based at least in part on the method 600. Mesh tiles 1308 can be extracted from the SDFs 1306 by, for example, applying a marching cube algorithm on the corresponding tiles (e.g., tiles

[0000] through

[0015] in FIG. 23 The plane extraction 1310 can detect planes in the mesh tiles 1308 and extract the planes based at least in part on the mesh tiles 1308. The plane extraction 1310 can also extract a facelet for each tile based at least in part on the corresponding mesh tile. A facelet mesh can include vertices in the mesh but not edges connecting adjacent vertices, such that storing facelets consumes less storage space than the mesh. A plane data store 1312 can hold the extracted planes and facelets.

[0186] In some embodiments, an XR application can request and obtain planes from the plane data store 1312 through a plane query 1314, which can be sent by an application program interface (API). For example, an application can send information about its location to the plane extraction system 1300 and request all planes within a certain radius (e.g., within a five-meter radius) around it. The plane extraction system 1300 can then search its plane data store 1312 and send the selected planes to the application. The plane query 1314 can include information such as where the application needs the planes, what kind of planes the application needs, and / or how the planes should look (e.g., horizontal, vertical, or angled, which can be determined by checking the original normals of the planes in the plane data store).

[0187] FIG. 22 A portion 1400 of the plane extraction system 1300 is shown, which shows details about the plane extraction 1310, according to some embodiments. The plane extraction 1310 can include dividing each mesh tile 1308 into sub-tiles 1402. Plane detection 1404 can be performed on each sub-tile 1402. For example, the plane detection 1404 can: compare the original normals of each mesh triangle in the sub-tile; merge those mesh triangles with original normal differences less than a predetermined threshold value into one mesh triangle; and identify mesh triangles with areas greater than a predetermined area value as planes.

[0188] FIG. 23 is a diagram showing a scene 1500 represented by tiles

[0000] through

[0015] including voxels and exemplary plane data including tile planes 1502, global planes 1504, and facelets 1506 in the scene, according to some embodiments. FIG. 23A tile

[0011] is shown that is divided into four sub-tiles 1508. It should be appreciated that a grid tile can be divided into any suitable number of sub-tiles. The granularity of the planes detected by plane detection 1404 can be determined by the size of the sub-tiles, which can be determined by the granularity of the local / remote memory storing the volume 3D reconstruction data.

[0189] Referring back to FIG. 22 , plane detection 1404 can determine tile planes (e.g., tile plane 1502) for each grid tile based at least in part on the detected planes for each sub-tile in the grid tile. Plane detection 1404 can also determine global planes (e.g., global plane 1504) that extend more than one tile.

[0190] In some embodiments, plane extraction 1310 can include plane update 1406, which can update existing tile planes and / or global planes stored in plane data store 1312 based at least in part on the planes detected by plane detection 1404. Plane update 1406 can include adding additional tile planes, removing some existing tile planes, and / or replacing some existing tile planes with tile planes detected by plane detection 1404 and corresponding to the same tile, such that real-time changes in the scene are maintained in plane data store 1312. Plane update 1406 can also include aggregating tile planes detected by plane detection 1404 to existing global planes, e.g., when tile planes are detected adjacent to an existing global plane.

[0191] In some embodiments, plane extraction 1310 can further include plane merging and splitting 1408. Plane merging can merge multiple global planes into one large global plane, e.g., when a tile plane is added and connects two global planes. Plane splitting can split one global plane into multiple global planes, e.g., when a tile plane in the middle of a global plane is removed.

[0192] FIG. 24 A data structure in plane data store 1312 according to some embodiments is shown. Global planes 1614, indexed by plane ID 1612, can be at the highest level of the data structure. Each global plane 1614 can include multiple tile planes and facelets of tiles adjacent to the corresponding global plane, such that one tile plane can be maintained for each tile, while the global plane can be accurately rendered when the edges of the global plane do not conform to the requirements of the tile planes of the corresponding tiles. In some embodiments, only the facelets of tiles adjacent to the global plane, instead of the facelets of all tiles in the scene, are maintained, as this is sufficient to accurately render the global plane. For example, as shown in FIG. 16, global plane 1614 can include tile planes 1616 and facelets 1618 of tiles adjacent to global plane 1614. Global plane 1614 can be rendered by rendering tile planes 1616 and facelets 1618 of tiles adjacent to global plane 1614. FIG. 23The global plane 1504 extends across tiles

[0008] -

[0010] and

[0006] as shown. Tile

[0006] has a tile plane 1502 that is not part of the global plane 1504. With the data structure in the plane data store 1312, when a plane query requests the global plane 1504, the facets of tiles

[0006] and

[0012] are examined to determine if the global plane 1504 extends into tiles

[0006] and

[0012] . In the example shown, facet 1506 indicates that the global plane 1504 extends into tile

[0006] .

[0193] Referring back to FIG. 24 The global plane 1614 can be bi-directionally associated with the corresponding tile plane 1610. Tiles can be identified by tile ID 1602. Tiles can be divided into plane tiles 1604 that include at least one plane and non-plane tiles 1606 that do not include a plane. Facets can be maintained for both plane and non-plane tiles, depending on whether the tile is adjacent to a global plane and not depending on whether the tile includes a plane. It should be understood that planes can be continuously maintained in the plane data store 1312 while the XR system is viewing the scene, whether or not there are plane queries 1314.

[0194] FIG. 25 Plane geometry extraction 1702 is shown that can extract a plane for use by an application when the application sends a plane query 1314 to the plane data store 1312, according to some embodiments. The plane geometry extraction 1702 can be implemented as an API. The plane query 1314 can indicate a requested plane geometry representation, such as an exterior rectangular plane, an interior rectangular plane, or a polygonal plane. From the plane query 1314, a plane search 1704 can search and obtain plane data in the plane data store 1312.

[0195] In some embodiments, rasterization from the plane coverage points 1706 can generate plane coverage points. An example is shown in FIG. 26A An example is shown in FIG. 18. There are four tiles

[0000] -

[0003] each having a tile plane 1802. By projecting the boundary points of the tile planes onto a global plane 1804, plane coverage points 1806 (or “rasterization points”) are generated.

[0196] Referring back to FIG. 25The rasterization from planar overlay point 1706 can also generate a rasterized planar mask from that planar overlay point. Based on the planar geometry representation requested by planar query 1314, the inner rectangle planar representation, outer rectangle planar representation, and polygon planar representation can be extracted respectively via inner rectangle extraction 1708, outer rectangle extraction 1710, and polygon extraction 1712. In some embodiments, the application can receive the requested planar geometry representation within milliseconds of sending the planar query.

[0197] FIG. 26B An exemplary rasterized planar mask 1814 is shown. Various planar geometric representations can be generated from the rasterized planar mask. In the example shown, a polygon 1812 is generated by connecting some planar overlay points of the rasterized planar mask such that none of the planar overlay points in the mask are outside the polygon. An outer rectangle 1808 is generated such that the outer rectangle 1808 is the smallest rectangle surrounding the rasterized planar mask 1814. The inner rectangle 1810 is generated by the following operations: assigning “1” to bricks with two planar coverage points and assigning “0” to bricks without two planar coverage points to form a raster grid, identifying groups of bricks marked with “1” and aligned on lines parallel to the edges of the bricks (e.g., bricks

[0001] ,

[0005] ,

[0009] and

[00013] as one group, and bricks

[0013] -

[0015] as another group), and generating an inner rectangle for each identified group such that the inner rectangle is the smallest rectangle surrounding the corresponding group.

[0198] FIG. 27 A mesh for scene 1900 is shown according to some embodiments. FIG. 28A -C illustrates a scenario 1900 represented by an outer rectangular plane, an inner rectangular plane, and a polygonal plane, respectively, according to some embodiments.

[0199] FIG. 29 The diagram shows a less noisy 3D representation of scene 1900, which is based on extracted planar data (e.g. FIG. 28A The plane shown in -C) will FIG. 27 The mesh shown is obtained by planarizing it.

[0200] Multi-level block grid simplification

[0201] In some embodiments, processing can be employed to reduce the complexity of the representation of the XR environment before it is stored or used for rendering functions such as occlusion handling or computational physics interaction between objects in the XR environment. For example, a mesh component can simplify the mesh or a portion of the mesh before storing the mesh or a portion of the mesh as mesh 162c in persistent world 162.

[0202] Such processing can require performing operations hierarchically on the representation of the XR environment. These levels can include simplification operations before and after region-based operations. Like the simplification operations, the region-based operations can reduce the complexity of the representation of the XR environment. By hierarchically staging the operations in this way, the total processing to produce a simplified representation of the XR environment can be reduced while maintaining the quality of the representation of the XR environment. Thus, a simplified high-quality representation can be frequently updated so that the XR environment can be frequently updated, thereby improving the performance of the XR system, such as by presenting a more realistic environment to the user.

[0203] The XR environment can represent the physical world, and the data representing the XR environment can be captured by one or more sensors. However, the techniques described herein can be applied to an XR environment regardless of the source of the data representing the environment. In some embodiments, the XR environment can be represented by a mesh comprising one or more points and polygons (e.g., triangles) defined by subsets of the points. A first simplification operation before the region-based operation can reduce the complexity of the representation of the environment. For example, the mesh can be simplified by reducing the number of such polygons in the mesh. As a particular example, the first simplification operation can employ a triangle reduction algorithm, which can reduce the number of triangles used to represent the XR environment.

[0204] The region-based operation can be a shape detection operation that can detect one or more shapes. A common shape detection operation is a planarization operation, in which the shape detected is a plane. The detected plane can represent an object or a portion of an object. The detection of a plane can simplify the process of rendering the XR environment. For example, a moving object rendered in the XR environment can move in an easily calculated manner when colliding with a plane, for example. Thus, identifying a plane can simplify the subsequent rendering of the moving object compared to performing calculations based on multiple polygons representing the same portion of the environment. As an alternative or in addition to a plane, other shapes can be detected and used in subsequent processing, including cylinders, cubes, straight lines, corners, or semantics such as glass surfaces or holes. Such operations can group polygons representing a surface of the detected shape.

[0205] A second simplification operation after the region-based operation can further simplify the representation of the environment, such as by further reducing the number of polygons in the representation. The second simplification operation can focus on reducing the number of polygons within each region detected by the region-based operation.

[0206] Such processing can enable a mesh partitioning service that processes sensor data collected in a physical environment and provides a mesh for applications generating content. In some embodiments, the processing can provide a simplified representation of virtual objects in a virtual environment.

[0207] In XR systems (e.g., virtual reality (VR), augmented reality (AR), and mixed reality (MR) systems), three-dimensional (3D) mesh data is often used for a variety of purposes, including, for example, occluding virtual content in a graphics / game engine based on physical objects in the environment, or computing rigid body collision effects for virtual objects in a physics engine of the game engine. In some embodiments, requirements for the mesh can differ for different uses of the mesh, and simplified meshes can be suitable for many such uses, with some simplification techniques being more suitable for certain uses than others.

[0208] Thus, the processes described herein can be implemented with a variety of simplification techniques and / or any of the simplification techniques that can be configured based on the intended use of the simplified mesh. The processes described herein can be used to improve the utility of a mesh partitioning service that provides simplified meshes to a plurality of client applications that can use the mesh in different ways. Each client application can need a mesh with a different level of simplification. In some embodiments, an application that accesses the mesh partitioning service can specify a target simplification or a mesh to be provided to it. The mesh simplification methods described herein can be used for a plurality of client applications, including, for example, those that perform virtual content occlusion, physics simulation, or environmental geometry visualization. The mesh processes described herein can have low latency and can be flexible in that it can optimize / bias operations for different uses (e.g., flat surfaces, varying triangle counts).

[0209] The mesh simplification methods as described herein can provide real-time performance (e.g., low latency to support fly (real-time) environmental changes), local update capability (e.g., renew the mesh for the parts that have changed since the last update), and flattened surfaces (e.g., flattened flat surfaces to support robust physics simulation).

[0210] In some embodiments, a representation of an XR environment can be partitioned into a plurality of tiles, some or all of which can be processed in parallel. In some embodiments, the resulting tiles can then be reassembled. In some embodiments, the tiles can be defined with “flanges” that overlap with adjacent tiles. The flanges enable reassembling the tiles with less and / or less noticeable discontinuities at the interface of the reassembled tiles.

[0211] Thus, in some embodiments, the mesh simplification method can include mesh tile partitioning, pre-simplification, mesh flattening, and post-simplification. To speed up the process, a global mesh can first be partitioned into tiles of component meshes, so that the mesh tiles can be operated on (e.g., processed) in parallel. Then, the mesh tiles can be extended with flanges on the boundaries between adjacent tiles. With the flanged mesh tiles, simplification can be performed on individual mesh tiles, while the global mesh can be visually seamless although topologically disconnected.

[0212] In some embodiments, the mesh simplification method can be adapted for use by applications that use the simplified mesh to represent objects in an XR environment for interaction, e.g., by making the simplification process plane-aware. To simplify the mesh, a three-step simplification process can be implemented. The mesh can first be moderately pre-simplified using a relatively high target triangle count. Then, planar regions can be detected by a region growing algorithm. The mesh can be planarized by projecting the corresponding triangles to the detected planar regions. In some embodiments, the mesh can be normalized by adjusting the plane (or original) normal to be substantially perpendicular and parallel to the detected planar regions. Thereafter, a post-simplification process can be run on the planarized mesh. The post-simplification process can pay more attention to the detected planar regions, e.g., simplifying the mesh for each detected planar region to a desired level of complexity (e.g., a metric complexity), e.g., as indicated by a target value of one or more metrics.

[0213] FIG. 30 A method 3000 of generating a model of an environment represented by a mesh is shown in accordance with some embodiments. In some embodiments, the method 3000 can be performed on a mesh partitioning service on an XR platform. The method 3000 can start at act 3002 from an input mesh representing an environment. In some embodiments, the input mesh can have a high resolution, which can be indicated by a number of triangles. The input mesh can be generated by a reconstruction system (e.g., a volumetric 3D reconstruction system), and the input mesh can include 3D reconstruction data.

[0214] In some embodiments, the reconstruction system can generate a volumetric 3D representation of the environment, which can create a data hierarchy of 3D information of the environment captured by one or more sensors. For example, the sensors can be depth cameras, which can capture 3D information of the environment, e.g., a stream of depth images with corresponding poses (i.e., camera poses) of the depth cameras. The 3D information of the environment can be processed into a voxel grid. Each voxel can contain one or more signed distance functions (SDFs) describing whether the voxel is inside or outside the geometry of an object in the environment. The voxels can be grouped into “bricks.” Each brick can include a plurality of voxels, e.g., a cubic volume of 8 3 voxels. The bricks can be further grouped into “tiles.” Each tile can include a plurality of bricks.

[0215] The size of the tiles can be selected to facilitate memory operations in a computing device. For example, the size can be selected based on an amount of information about the environment that is maintained in an active memory of a device that processes such data. For example, the system can pass tiles between an active memory that is typically local to the device and other memory (e.g., non-volatile memory or remote memory in the cloud) that has a greater latency. One or more complete or partial tiles can contain information representing a "tile" in the mesh or other representation of the environment.

[0216] In some embodiments, the volumetric 3D reconstruction system can generate the input mesh 3002 as a topologically connected global mesh. In some embodiments, the volumetric 3D reconstruction system can generate the input mesh 3002 as a global mesh that is topologically disconnected but visually seamless. For example, a topologically disconnected global mesh can be composed of multiple mesh tiles, each mesh tile generated by a tile generation.

[0217] A reconstruction system can be configured to capture substantial details of an environment, which enables the system to distinguish between adjacent portions of a representation that have relatively small characteristic differences. Adjacent regions having different properties can be identified as different surfaces, resulting in the system identifying a large number of surfaces in the environment. However, such a system can capture unnecessary but still processed details for many applications. For example, for a client application that requests a mesh from a mesh partitioning service, a reconstruction system can unnecessarily display bumps on a wall in many triangles when two triangles that make a rectangle would be a sufficient representation of the wall. In some embodiments, when a mesh is requested from a mesh partitioning service, the application can specify a target level of simplification for the requested mesh. The target level of simplification can be expressed as a degree of compression, a number of triangles per unit area, or in any other suitable manner.

[0218] The method 3000 can efficiently generate a model of an environment sufficient for a client application from an input mesh. At act 3004, the input mesh can be partitioned into one or more first mesh tiles, each first mesh tile can correspond to a tile in a data hierarchy of a volumetric 3D representation of the environment.

[0219] Each first mesh tile can represent a portion of the environment, and a complexity measure (e.g., mesh resolution) can be a first value. In some embodiments, the complexity measure of a mesh tile indicates a number of triangles in the mesh tile. In some embodiments, processing can be performed on the mesh tiles sequentially and / or in parallel. However, the simplification processing as described herein can be applied to the entire mesh or any suitable portion (e.g., one or more mesh tiles).

[0220] Action 3006 represents a sub-process performed on each of the plurality of grid tiles. This sub-process can be performed independently on the plurality of grid tiles, such that it can be easily performed in parallel on some or all of the grid tiles. This sub-process can be performed on all of the grid tiles or a subset of the grid tiles selected for further processing. The subset of grid tiles can be selected based at least in part on a field of view of a device on which the application requesting the simplified grid is executing.

[0221] In action 3006, some of the first grid tiles can be selected based on, for example, objects described in the first grid tile or a location of the first grid tile. For each selected first grid tile, a multi-level simplification can be performed. In some embodiments, the multi-level simplification on the selected first grid tiles can be performed in parallel, as a result, the simplification on the selected first grid tiles can be completed at approximately the same point in time, although this can depend on a complexity measure of each grid tile of the selected first grid tiles.

[0222] The multi-level simplification can include a pre-simplification operation, a region-based operation (e.g., a planarization operation), and a post-simplification operation. In some embodiments, the multi-level simplification can be performed based on an input value from the client application. This input value can indicate a grid complexity (e.g., a grid resolution) required by the client application. The input value from the client application can be the same or different for each selected first grid tile.

[0223] At action 3012, a pre-simplification operation can be performed on the selected first grid tile to generate a second grid tile. This pre-simplification operation can reduce a complexity of the tile. For a grid tile, the pre-simplification can reduce a number of polygons in the grid tile. In some embodiments, an amount of pre-simplification at action 3012 can be configurable. For example, a target value can be provided by, for example, the client application as an input to the process at action 3012. The target value can be a single value or multiple values of one or more specified or predetermined measures. The one or more measures can include, for example, an absolute triangle count, a percentage of an initial triangle count, and / or a quadratic error measure, which can measure an average squared distance between a simplified grid and an original grid (e.g., the input grid 3002).

[0224] The target value can be provided in any suitable manner. For example, an instance of the method 3000 can be preconfigured with a target value. In some embodiments, the target value can be provided by an application requesting a mesh partition from the mesh partitioning service performing the method 3000 via an API. For example, the target value of the action 3012 can be a final target requested by a rendering function (e.g., a requesting application). In some embodiments, the target value provided as input can be adjusted or overridden to ensure that sufficient data is maintained in the mesh for subsequent processing. For example, the processing in the action 3014 can require a minimum number of triangles, and if the target value is lower than the minimum number of triangles, the target value provided by the application can be replaced by the minimum value.

[0225] In such embodiments, the pre-simplified mesh can have values for one or more metrics such that the pre-simplified mesh can be processed faster than the original tile-partitioned input mesh during region-based operations while still containing all or most of the regions of the original tile-partitioned input mesh.

[0226] Because the values for the one or more metrics are not controlled, the simplified mesh can be too coarse, unevenly distributed, and / or lose many of the regions of the original tile-partitioned input mesh that are needed for the following region-based operations.

[0227] The complexity metric of the second mesh tile produced in the action 3012 can be a second value, which can be less than the first value of the metric complexity. In some embodiments, the pre-simplification operation of the action 3012 can be performed using a triangle reduction algorithm.

[0228] In the action 3014, a shape detection operation can be performed on the second mesh tile to generate a third mesh tile. Take the planarization operation as an example. The complexity metric of the third mesh tile can be a third value. In some embodiments, the third value of the metric complexity can be the same as the second value of the metric complexity. In some embodiments, the third value of the metric complexity can be less than the second value of the metric complexity. The planarization operation can include detecting planar regions in the second mesh tile, projecting the mesh of the detected planar regions to a corresponding plane, adjusting the plane normal of the detected planar regions to be substantially perpendicular to the corresponding plane, and simplifying the projected mesh on each corresponding plane based on, for example, a target triangle count, for example using a region growing algorithm. In some embodiments, the plane normal of the detected planar regions can be adjusted before the mesh of the detected planar regions is projected to the corresponding plane.

[0229] At action 3016, a post-simplification operation can be performed on the third mesh chunk to generate a fourth mesh chunk. In some embodiments, the processing at action 3014 can desirably be performed on the mesh at a higher resolution than is required in the simplified mesh that is to be output from method 3000. In some embodiments, the processing at action 3016 can simplify the entire mesh chunk to achieve a desired level of complexity (e.g., a metric complexity), for example, can be indicated by a target value of one or more metrics, which can be the same or different than the target provided to action 3012. In some embodiments, the post-simplification operation at action 3016 can focus on reducing the number of polygons within each plane detected by the planarization operation at action 3014.

[0230] The metric complexity of the fourth mesh chunk can be a fourth value, which can be less than the third value of the metric complexity. In some embodiments, the percentage reduction between the third value of the metric complexity and the fourth value of the metric complexity can be greater than the percentage reduction between the first value of the metric complexity and the second value of the metric complexity. In some embodiments, the percentage reduction between the third value of the metric complexity and the fourth value of the metric complexity can be at least one time greater than the percentage reduction between the first value of the metric complexity and the second value of the metric complexity. In some embodiments, the post-simplification operation at action 3016 can be performed using a triangle reduction algorithm. In some embodiments, the post-simplification operation at action 3016 can be performed using the same simplification algorithm as the pre-simplification operation at action 3012.

[0231] At action 3008, the simplified selected chunk can be combined with other selected mesh chunks that were similarly processed in action 3006, and / or can be combined with unselected chunks into a new mesh for the environment. At action 3010, the new mesh for the environment can be provided to the client application. In some embodiments, the new mesh for the environment can be referred to as a simplified mesh.

[0232] In some embodiments, action 3008 can be skipped. The simplified mesh chunks can be sent directly to the client application, where the mesh chunks can be visually seamless, despite being topologically disjoint.

[0233] FIG. 31An example of segmenting a grid representation 3100 of an environment into grid tiles is shown, according to some embodiments. The grid representation 3100 can be segmented into four grid tiles: grid tiles A-D. In some embodiments, the grid tiles can correspond to regions that belong to the physical world of the environment that have the same volume. In some embodiments, the grid tiles can correspond to regions that belong to the physical world of the environment that have different volumes. For example, when the physical world is an office, the office can be divided into a plurality of regions, each of which can be one cubic foot. One tile can include a 3D representation of one region of the office.

[0234] Although the grid representation 3100 of the environment is shown in two dimensions (2D), it should be understood that the environment can be three-dimensional, correspondingly represented by a 3D grid representation. Although the grid representation 3100 of the environment is illustrated as a combination of four grid tiles, it should be understood that the environment can be represented by any suitable number of grid tiles (e.g., two, three, five, six, or more).

[0235] The representation 3100 can be segmented into four parts: parts 3102, 3104, 3106, and 3108, shown by solid lines 3110, for example. In some embodiments, the parts 3102, 3104, 3106, and 3108 can be designated as grid tiles A-D, respectively.

[0236] When a grid tile is updated, it can continue to interface with neighboring tiles that are not updated. As a result, discontinuities can arise at the boundaries between grid tiles. If the region represented by a neighboring tile has a discontinuity, the fused grid can be interpreted in subsequent processing as indicating that there is a crack between the neighboring tiles. In some embodiments, such cracks in the representation of the physical world space can be interpreted as spaces with infinite depth. As such, the spaces can be artifacts of the representation of the physical world, rather than actual features. Any application that uses such a fused grid to generate a representation of an object in the physical world can fail to generate the output correctly. For example, an application that renders a virtual character on a surface in the physical world can render the character as if it fell into the crack, which does not create the desired appearance of the object.

[0237] To reduce the occurrence of such cracks, in some embodiments, a portion of an adjacent tile can represent the same area of the physical world. For example, the abutment area between adjacent tiles can be represented by a portion of each adjacent tile, which can enable easy independent updating and / or rendering considering level of detail (LOD) (e.g., reducing the complexity of a 3D reconstruction of a portion of the physical world as that portion moves out of the user’s field of view). Even if one tile is updated while its adjacent tile is not, the fused mesh can represent the abutment area from both tiles by combining the data representing the abutment area from both tiles. As a particular example, when fusing an updated tile with an adjacent tile, the physics engine can determine the overlapping area of the adjacent tile based on, for example, which of the adjacent tiles is observable in its overlapping area. The tile-based data structure can employ a side, a zipper, or any other suitable method to represent the abutment area between adjacent tiles, such that when a tile is updated, it will continue to abut the adjacent tile that is not updated. The appearance of this approach can be to “paper over” the cracks between adjacent tiles. Thus, a tile can be updated independently of its adjacent tiles.

[0238] In FIG. 31 In the illustrated example, the area at the boundary of portions 3102, 3104, 3106, and 3108 can be designated as a side, as shown by dashed line 3112. In some embodiments, each mesh tile A-D can include one of portions 3102, 3104, 3106, and 3108 and a corresponding side. For example, mesh tile B can include portion 3104 and side 3114, which overlaps with the boundary portions of adjacent mesh tiles A, C, and D of mesh tile B, such that cracks between the mesh tiles can be papered over when the tiles are connected into a single mesh. Mesh tiles A, C, and D can also include corresponding sides. Thus, the processor can paper over any cracks between the mesh tiles before returning a single connected 3D mesh representation to the application.

[0239] In some embodiments, the tile mesh including the sides can be sent directly to the application without combining it into a topologically connected global mesh. The application can have a global mesh composed of tile meshes that is visually seamless, although topologically disconnected.

[0240] FIG. 32A- FIG. 32D A mesh evolution of an example mesh tile 3201 during multi-level simplification is shown, in accordance with some embodiments. Mesh tile 3201 can include vertices 3206, edges 3208, and faces 3210. Each face can have a normal, which can be represented by a plurality of coordinates (e.g., x, y, z, shown in FIG. 32A

[0241] ​A pre-simplification operation can be performed on mesh patch 3201 to generate mesh patch 3202. An edge collapse transformation can be used. In the example shown, mesh patch 3202 reduces the number of faces of mesh patch 3201 from ten to eight. The resulting faces of mesh patch 3202 can each have a corresponding set of normals (e.g., xl, yl, zl; x2, y2, z2;...; x8, y8, z8).

[0242] A planarization operation can be performed on mesh patch 3202 to generate mesh patch 3203. The planarization operation can include detecting planar regions in the mesh based on, for example, the planar (or original) normals of the faces. The values of the planar normal xl, yl, zl of first face 3212 and the planar normal x2, y2, z2 of second face 3214 can be compared. The result of the comparison of the planar normals of the first and second faces can indicate an angle between the planar normals (e.g., the angle between xl and x2). When the result of the comparison is within a threshold, the first and second planes can be determined to be on the same planar region. In the example shown, planes 3212, 3214, 3216, and 3218 can be determined to be on a first planar region corresponding to plane 3228; and planes 3220, 3222, 3224, and 3226 can be determined to be on a second same planar region corresponding to plane 3230.

[0243] The planarization operation can also include projecting the triangle formed by the edges of planes 3212, 3214, 3216, and 3218 to plane 3228 as shown by dashed lines 3232; and projecting the triangle formed by the edges of planes 3220, 3222, 3224, and 3226 to plane 3230 as shown by dashed lines 3234. The planarization operation also includes adjusting the planar normals of planes 3212, 3214, 3216, and 3218 to be the same as the planar normal of plane 3228 (xa, ya, za); and adjusting the planar normals of planes 3220, 3222, 3224, and 3226 to be the same as the planar normal of plane 3230 (xb, yb, zb).

[0244] A post-simplification operation can be performed on mesh patch 3203 to generate mesh patch 3204. In the example shown, mesh patch 3204 reduces the number of faces of mesh patch 3203 from eight to four.

[0245] FIG. 33A and FIG. 33B to FIG. 36A and FIG. 36B The effect of the simplification is shown, with the same portion of the physical world shown side-by-side with and without the simplification applied. These figures provide a graphical illustration that the simplification can provide usable information to operate the AR system while providing less data that needs to be processed.

[0246] FIG. 33A and FIG. 33B show representations of the same environment without simplification and with simplification by triangle reduction, respectively. Such processing is an example of processing that can be performed at pre-simplification block 3012 and post-simplification block 3016 in FIG. 30

[0247] FIG. 34A and 34B show close-up representations of the same environment without simplification by triangle reduction and with simplification, respectively.

[0248] FIG. 35A and 35B show representations of the same environment without planarization and with planarization, respectively. Such processing is an example of processing that can be performed at planarization block 3014 in FIG. 30

[0249] FIG. 36A and 36B show representations of the same environment without simplification and with simplification by removal of disconnected components, respectively. Such processing is an example of an alternative embodiment of region-based operations that can be performed at block 3014 in FIG. 30

[0250] Caching and updating of dense 3D reconstruction data

[0251] In some embodiments, 3D reconstruction data can be captured, maintained, and updated in the form of tiles, which can allow for local updates while maintaining neighborhood consistency. Tile-based 3D reconstruction data representations can be used in conjunction with a multi-tiered caching mechanism that efficiently retrieves, prefetches, and stores 3D data for AR and MR applications, including single-device and multi-device applications. For example, volumetric information 162a and / or mesh 162c FIG. 6 ) can be stored in tiles. Using components 164 can use this tile-based representation to receive information about the physical world. Similarly, perception components can store and retrieve such information in tiles.

[0252] ​​​These techniques extend the functionality of portable devices with limited computing resources to present AR and MR content in a highly realistic manner. Such techniques can be used, for example, to efficiently update and manage the output of real-time or offline reconstruction and scanning in mobile devices that are resource-limited and connected to the Internet (continuously or not). These techniques can provide up-to-date, accurate, and comprehensive 3D reconstruction data for various mobile AR and MR applications in single-device applications or multi-device applications that share and update the same 3D reconstruction data. These 3D reconstruction data can be in any suitable format, including meshes, point clouds, voxels, etc.

[0253] Some AR and MR systems have attempted to simplify the presentation of MR and AR scenes by limiting the amount of 3D reconstruction data processed at a time. Sensors for capturing 3D information can have a maximum reconstruction range that can limit a bounding volume around the field of view of the sensor. To reduce the amount of 3D reconstruction data, some reconstruction systems only keep the area near the field of view of the sensor in active working memory and store other data in secondary storage. For example, the area near the field of view of the sensor is stored in CPU memory, while other data is kept in local cache (e.g., disk) or to remote storage (e.g., in the cloud) over a network.

[0254] The computational cost of generating the information stored in CPU memory, although limited, can still be relatively high. Some AR and MR systems constantly recompute a global representation of the environment seen by the reconstruction system in order to select the information to store in CPU memory, which can be very expensive for interactive applications. Other AR and MR systems can be equally expensive using certain methods to only compute local updates to the connected representation, especially for simplified meshes, as it requires decomposing the existing mesh, computing another mesh with the same boundary, and then reconnecting the mesh parts.

[0255] In some embodiments, 3D reconstruction data can be segmented into tiles. The 3D reconstruction data can be sent between storage media based on the tiles. For example, tiles can be paged out of active memory and held in local or remote caches. The system can implement a paging algorithm in which an active memory associated with a wearable device (e.g., a head-mounted display device) stores tiles representing a portion of a 3D reconstruction of the physical world in the field of view of a user of the wearable device. The wearable device can capture data about the portion of the physical world commensurate with the field of view of the user of the wearable device. As the physical world changes in the field of view of the user, tiles representing that region of the physical world can be in active memory, which can be readily updated from active memory. As the field of view of the user changes, tiles representing regions of the physical world that move out of the field of view of the user can be moved to a cache, so that tiles representing regions of the physical world that move into the field of view of the user can be loaded into active memory.

[0256] In some embodiments, a coordinate system can be created for a portion of the physical world to be 3D reconstructed. Each tile in the 3D representation of the portion of the physical world can correspond to a different region of the physical world that can be identified using the coordinate system.

[0257] In some embodiments, when a tile is updated, the updated tile can continue to interface with neighboring tiles that can not have been updated. If the areas represented by the neighboring tiles do not overlap, there can be a crack in the merged mesh of the neighboring tiles. In some embodiments, such cracks in the representation of the physical world space can be interpreted as a space with infinite depth. As such, the space can be an artifact of the representation of the physical world, rather than an actual feature. Any application using such merged meshes to generate a representation of an object in the physical world can not generate output correctly. For example, an application rendering a virtual character on a surface in the physical world can present the character as if it fell into the crack, which does not create the desired appearance of the object. Thus, in some embodiments, a portion of a neighboring tile can represent the same area of the physical world, e.g., the interface between neighboring tiles can represent the same area of the physical world, which can enable easy independent updating and / or rendering that takes into account level of detail (LOD) (e.g., reducing the complexity of a 3D reconstruction of a portion of the physical world because that portion moves out of the field of view of a user). For example, when a tile is updated, its neighboring tiles can not be updated. When merging the updated tile with the neighboring tiles, the physics engine can determine the overlapping area of the neighboring tiles based on, e.g., which of the neighboring tiles is observable in its overlapping area. The data structure of the tiles can employ a side, a zipper, or any other suitable method to represent the interface area between neighboring tiles, such that when a tile is updated, it will continue to interface with neighboring tiles that are not updated. The appearance of this method can be to "cover up" on the crack between neighboring tiles. Thus, a tile that has changed can be updated independently of neighboring tiles.

[0258] In some embodiments, these techniques can be used in an AR and / or MR "platform" that receives and processes data from one or more sensors worn by a user. The sensor data can be used to create and update 3D reconstruction data representing a portion of the physical world encountered by the user. While the sensors are capturing and updating data, a reconstruction service can continuously reconstruct a 3D representation of the physical world. One or more techniques can be used to determine tiles that are affected by changes to the physical world, and those tiles can be updated. The 3D reconstruction data can then be provided to an application that uses the 3D reconstruction data to render a scene to depict a virtual reality object placed in or interacting with an object in the physical world. The data can be provided to the application through an application programming interface (API). The API can be a push or pull interface, pushing data to the application when relevant portions change or in response to a request from the application for up-to-date information.

[0259] In the example of a pull interface, when an application requests 3D reconstruction data for a physical world, the reconstruction service can determine the appropriate version of each tile that needs to be provided to the application, thereby enabling the reconstruction service to start from the most recent tile. The reconstruction service may, for example, search for previously held tiles. A single device system can enable a single device to contribute 3D reconstruction data about a physical world. In a single device system, if the requested region of the physical world is in the active region (e.g., the region in the current field of view of the device) or extends beyond the active region, the held tiles can be used directly as the most recent tiles, as these held tiles will not be updated as they are held when the region moves out of the field of view of the device. On the other hand, a multi-device system can enable multiple devices to provide 3D reconstruction data about a physical world, for example, using cloud persistence or peer-to-peer local cache sharing of tiles. Each device can update the regions within its active region that can be held. The multi-device system can create a coordinate system such that tiles generated by different devices can be identified using the coordinate system. Thus, if those updates are made after any version by the first device, the tiles requested by the application for the data generated for the first device can be based on updates from other devices. Tiles constructed using data from the first device and other devices can be merged by using the coordinate system.

[0260] The selected tiles can be used to provide 3D reconstruction data about the physical world in any suitable format, although meshes are used as an example of a suitable representation here. A mesh can be created by processing image data to identify points of interest in the environment (e.g., edges of objects). These points can be connected, forming the mesh. Groups of points (typically three points) in the mesh that are associated with the same object or portion thereof define a surface of the object or portion thereof. Information stored with the group of points describes the surface in the environment. This information can then be used in various ways to render and / or display information about the environment. The selected tiles can be used to provide 3D reconstruction data in any suitable manner. In some embodiments, the most recent tile can be provided. In some embodiments, the most recent tile can be used to determine whether tiles need to be updated.

[0261] For example, in some embodiments, in a multi-device system, when the tiles requested by an application have been identified, the reconstruction service can check tiles held by other devices to determine whether there are any significant updates (e.g., by a geometry change threshold or a timestamp), re-run mesh partitioning on the tiles that have changed, and then hold these updated mesh tiles.

[0262] In some embodiments, when a set of tiles requested by an application has been identified, if the application requested a connected mesh, the set of tiles can be processed as a global mesh, which can be topologically connected, or visually seamless although topologically disconnected using any suitable technique (e.g., side-borders and zippers).

[0263] In some embodiments, when a tile change occurs, an application (e.g., a graphics / game engine) can update its internal tiles (e.g., tiles stored in active memory and / or local cache). The reconstruction service can know which tiles the application has, and thus can compute which other (e.g., neighboring) tiles need to be updated in the engine to maintain the correct overlap with the side-borders / zippers as the tiles in the field of view update.

[0264] In some embodiments, an AR and / or MR platform can be implemented to support, for example, executing AR and / or MR applications on a mobile device. An application executing on a user interface or generating data for presentation can request 3D reconstruction data representing a physical world. The 3D reconstruction data can be provided from active memory on the device, which can be updated with 3D reconstruction data representing the physical world in the field of view of the device as the user changes her field of view. The 3D reconstruction data in the active memory can represent the active area of the mobile device. In some embodiments, 3D reconstruction data outside the active area of the device can be stored in other memory, such as a local cache on the device, or coupled to the device through a low-latency connection. In some embodiments, 3D reconstruction data outside the active area of the device can also be stored in a remote cache, such as in the cloud, which can be accessed by the device through a higher-latency connection. As the user changes her field of view, the platform can access / load 3D reconstruction data from the cache to add to the active memory to represent areas moving into the user's field of view. The platform can move other data representing areas moving out of the user's field of view to the cache.

[0265] Prediction of movement of the device, which can cause areas to move into the field of view of the device and other areas to move out of the field of view of the device, can be used to initiate transfer of 3D reconstruction data between the active memory and the cache. Prediction of movement can be used, for example, to select 3D reconstruction data to transfer into and / or out of the active memory. In some embodiments, by retrieving 3D reconstruction data from or transferring 3D reconstruction data to a remote cache, predicted movement can be used to transfer 3D reconstruction data to and / or from a local cache. Exchanging 3D reconstruction data between the local cache and the remote cache based on predicted movement of the user can ensure that 3D reconstruction data is available with low latency to move into the active memory.

[0266] In embodiments in which the physical world is represented using tiles, initiating the transmission of tiles can require pre-requesting tiles representing areas predicted to enter the user's field of view. For example, if the platform determines based on sensor data or other data that the user is walking in a particular direction at a particular speed, it can identify areas likely to enter the user's field of view and transmit tiles representing those areas to a local cache on the mobile device. If the mobile device is a wearable device, such as a pair of glasses, predicting motion can require receiving sensor data indicating the position, direction, and / or rotation of the user's head.

[0267] FIG. 37 A system 3700 for enabling interactive XR environments for multiple users is shown in accordance with some embodiments. The system 3700 can include a computing network 3705 that includes one or more computer servers 3710 connected by one or more high-bandwidth interfaces 3715. The servers in the computing network need not be co-located. The one or more servers 3710 can each include one or more processors for executing program instructions. The servers also include memory for storing program instructions and data used and / or generated by the processes executed by the servers under the direction of the program instructions. The system 3700 can include one or more devices 3720 that include, for example, AR display systems 80 (e.g., viewing optics assembly 48 in FIG. 1) and / or other devices that can be used to interact with the XR environment. FIG. 5B

[0268] The computing network 3705 communicates data between the servers 3710 and between the servers and the devices 3720 over one or more data networks connections 3730. Examples of such data networks include, but are not limited to, any and all types of public and private data networks, mobile and wired, including, for example, the interconnection of many such networks, commonly referred to as the Internet. This figure is not intended to imply any specific media, topology, or protocol.

[0269] In some embodiments, the devices can be configured to communicate directly with the computing network 3705 or any of the servers 3710. In some embodiments, the devices 3720 can communicate with the remote servers 3710 and, optionally, communicate locally with other devices and AR display systems through a local gateway 3740 for processing and / or communicating data between the network 3705 and one or more devices 3720.

[0270] ​As shown, the gateway 3740 is implemented as a separate hardware component that includes a processor for executing software instructions and a memory for storing software instructions and data. The gateway has its own wired and / or wireless connection to the data network for communicating with the servers 3710 of the computing network 3705. In some embodiments, the gateway 3740 can be integrated with the device 3720 worn or carried by the user. For example, the gateway 3740 can be implemented as a downloadable software application installed and running on a processor included in the device 3720. In one embodiment, the gateway 3740 provides one or more users with access to the computing network 3705 through the data network 3730. In some embodiments, the gateway 3740 can include the communication links 76 and 78.

[0271] The servers 3710 each include, for example, a working memory and storage device for storing data and software programs, a microprocessor for executing program instructions, a graphics processor, and other special processors for rendering and generating graphics, images, video, audio, and multimedia files. The computing network 3705 can also include devices for storing data accessed, used, or created by the servers 3710. In some embodiments, the computing network 3705 can include the remote processing module 72 and the remote data repository 74.

[0272] The software programs running on the servers, and optionally the device 3720 and the gateway 3740, are used to generate a digital world (also referred to herein as a virtual world) through which a user interacts with the device 3720. The digital world is represented by data and processes that describe and / or define virtual, non-existent entities, environments, and conditions that can be presented to the user through the device 3720 for the user to experience and interact with. For example, a certain type of object, entity, or item that appears to be physically present when instantiated in a scene being viewed or experienced by the user can include a description of its appearance, its behavior, how the user is allowed to interact with it, and other characteristics. The data used to create the environment of the virtual world, including virtual objects, can include, for example, atmospheric data, terrain data, weather data, temperature data, location data, and other data used to define and / or describe the virtual environment. Additionally, data defining various conditions governing the operation of the virtual world can include, for example, physical laws, time, spatial relationships, and other data that can be used to define and / or create various conditions governing the operation of the virtual world, including virtual objects.

[0273] Unless otherwise indicated by context, references herein will generically refer to entities, objects, conditions, characteristics, behaviors, or other features of the digital world as objects (e.g., digital objects, virtual objects, rendered physical objects, etc.). Objects can be any type of animate or inanimate object, including but not limited to buildings, plants, vehicles, people, animals, living beings, machines, data, videos, text, pictures, and other users. Objects can also be defined in the digital world to store information about items, behaviors, or conditions that actually exist in the physical world. Data describing or defining entities, objects, or items or storing their current state are generally referred to herein as object data. This data is processed by the servers 3710, or by the gateways 3740 or devices 3720 as the case can be, to instantiate instances of objects and render the objects in an appropriate manner for users to experience through the devices 3720.

[0274] Programmers developing and / or curating the digital world can create or define objects and conditions under which the objects are instantiated. However, the digital world can allow others to create or modify objects. Once an object is instantiated, one or more users experiencing the digital world can be allowed to change, control, or manipulate the state of the object.

[0275] For example, in one embodiment, the development, production, and management of the digital world is generally provided by one or more system administrators. In some embodiments, this can include the development, design, and / or execution of storylines, themes, and events in the digital world, as well as the distribution of stories through various forms of events and media (e.g., movies, digital, web, mobile, augmented reality, and live entertainment). System administrators can also handle technical management, moderation, and curation of the digital world and the user community associated therewith, as well as other tasks typically performed by network administrators.

[0276] Users interact with one or more digital worlds using some type of local computing device, generally referred to as a device 3720. Examples of such devices include, but are not limited to, a smartphone, a tablet device, a heads-up device display (HUD), a game console, or any other device that is capable of communicating data to a user and providing an interface or display, as well as combinations of these devices. In some embodiments, the device 3720 can include or communicate with local peripheral devices or input / output components such as keyboards, mice, joysticks, game controllers, haptic interface devices, motion capture controllers, audio devices, voice devices, projector systems, 3D displays, and holographic 3D contact lenses.

[0277] FIG. 38 is a schematic diagram illustrating an electronic system 3800 in accordance with some embodiments. In some embodiments, the system 3800 can be a FIG. 37part of the system 3700. The system 3800 can include a first device 3810 (e.g., a first portable device of a first user) and a second device 3820 (e.g., a second portable device of a second user). The devices 3810 and 3820, for example, can be FIG. 37 the device 3720 and the AR display system 80. The devices 3810 and 3820 can communicate with the cloud cache 3802 through networks 3804a and 3804b, respectively. In some embodiments, the cloud cache 3802 can be implemented in the memory of one or more servers 3710 of the system 3700. The networks 3804a and 3804b can be examples of the data network 3730 and / or the local gateway 3740. FIG. 37 FIG. 5B

[0278] The devices 3810 and 3820 can be separate AR systems (e.g., the device 3720). In some embodiments, the devices 3810 and 3820 can include AR display systems worn by their respective users. In some embodiments, one of the devices 3810 and 3820 can be an AR display system worn by a user; the other can be a smartphone held by the user. Although two devices 3810 and 3820 are shown in this example, it should be understood that the system 3800 can include one or more devices, and that the one or more devices can run the same type of AR system or different types of AR systems.

[0279] The devices 3810 and 3820 can be portable computing devices. The first device 3810, for example, can include a processor 3812, a local cache 3814, and one or more AR applications 3816. The processor 3812 can include a computing portion 3812a configured to execute computer-executable instructions based at least in part on data collected by one or more sensors (e.g., a depth sensor 51, a world camera 52, and / or an inertial measurement unit 57) of the device 3810 to provide a 3D representation (e.g., 3D reconstruction data) of a portion of the physical world. FIG. 38

[0280] ​​​Computing portion 3812a can represent the physical world as one or more tiles. Each tile can represent objects in a different region of the physical world. Each region can have a corresponding volume. In some embodiments, the tiles can represent regions having the same volume. In some embodiments, the tiles can represent regions having different volumes. For example, when the physical world is an office, the office can be divided into cubes, each of which can be a cubic foot. One tile can include a 3D representation (e.g., 3D reconstruction data) of one cubic foot of the office. In some embodiments, the office can be divided into regions having various volumes, and each volume can include a similar amount of 3D information (e.g., 3D reconstruction data) so that the data size of the 3D representation of each region can be similar. The representation can be formatted to facilitate further processing, such as occlusion processing, to determine whether a virtual object is occluded by a physical object or physical process, to determine how a virtual object should move or deform when interacting with a physical object in the physical world. The tiles can be formatted, for example, as mesh tiles, in which features (e.g., corners) of objects in the physical world become points in the mesh tiles, or are used as points to create the mesh tiles. Connections between points in the mesh can indicate groups of points on the same surface of a physical object.

[0281] Each tile can have one or more versions, each containing data representing its corresponding region based on data at a point in time (e.g., volumetric 3D reconstruction data such as voxels, and / or a mesh that can represent surfaces in the region represented by the corresponding tile). Computing portion 3812a can create new versions of tiles when additional data becomes available, such as data indicating that objects in the physical world have changed or additional data from which a more accurate representation of the physical world can be created. The additional data can come from sensors on devices (e.g., devices 3810 and / or 3820). In some embodiments, the additional data can come from remote sensors and can be obtained, for example, through a network connection.

[0282] The processor 3812 can also include an active memory 3812b, which can be configured to store tiles in the field of view of the device. In some embodiments, the active memory 3812b can store tiles outside the field of view of the device. In some embodiments, the active memory 3812b can store tiles adjacent to tiles in the field of view of the device. In some embodiments, the active memory 3812b can store tiles predicted to be in the field of view of the device. In some embodiments, the processor 3812 maintains tiles in the active memory 3812b if the tiles are within the field of view of the device at the time. The field of view can be determined by the imaging area of one or more sensors. In some embodiments, the field of view can be determined by the amount of the physical world presented to a user of the device or perceivable by an average user when not using an AR system. Thus, the field of view can depend on the user's location in the physical world and the orientation of the wearable components of the device.

[0283] If a tile becomes outside the field of view of the device 3810 when the user moves, the processor 3812 can consider the tile inactive. Inactive tiles can be paged out of the active memory to a cache. The cache can be a local cache or a remote cache. In FIG. 38 In embodiments, tiles are first paged out to a local cache 3814 through a local gateway 3818b. In some embodiments, the local cache 3814 can be the only cache available.

[0284] In some embodiments, there can be a remote cache accessible over a network. In the illustrated embodiment, a cloud cache (e.g., remote cache) 3802 accessed over a network 3804a is an example of a remote cache. The processor 3812 can manage when to move tiles between the local cache 3814 and the cloud cache 3802. For example, when the local cache 3814 is full, the processor 3812 can page tiles out to the cloud cache 3802 over the network 3804a. Because tiles in the local cache 3814 are accessible for rendering a scene with lower latency than tiles in the cloud cache, the processor 3812 can use an algorithm that aims to keep tiles that are most likely to become active in the local cache 3814 to select tiles to page out of the local cache 3814. Such an algorithm can be based on time of access. In some embodiments, the algorithm can be based on a prediction of motion of the device that will change the field of view of the device.

[0285] An application that renders a scene (e.g., a computer game) can obtain information representing portions of the physical world that affect the scene to be rendered. The application 3816 can obtain active tiles from the active memory 3812b through a local gateway 3818a. In some embodiments, the local gateway 3818a can be implemented as an application programming interface (API) such that the processor 3812 implements a "service" for the application 3816. In embodiments in which the data representing the physical world is represented as a mesh, the service can be a "mesh partitioning service." The API can be implemented as a push or pull interface, or can have attributes of both. In a pull interface, for example, the application 3816 can indicate portions of the physical world for which it needs data, and the service can provide data for those portions. In a push system, for example, the service can provide data about portions of the physical world as such data changes or becomes available.

[0286] The portions of the physical world for which data is provided can be limited to relevant portions indicated by the application 3816, e.g., data within the field of view of a device or data representing portions of the physical world within a threshold distance of the field of view of the device. In a pull / push system, the application 3816 can request data for a portion of the physical world, and the service can provide data about the requested portion plus any adjacent portions in which data has changed. To limit the information to changed information, in addition to maintaining tiles describing the physical world, the service can track which versions of the tiles are provided to each application 3816. The operations of determining which portions of the representation of the physical world are to be updated and where the updates occur can be partitioned between the application 3816 and the service in any suitable manner. Likewise, the locations at which the updated data is incorporated into the representation of the physical world can be partitioned in any suitable manner.

[0287] In some embodiments, the reconstruction service can continuously reconstruct a 3D representation of the physical world as sensors are capturing and updating data. The data can then be provided to applications 3816 that use the 3D reconstruction data to render scenes depicting the physical world and virtual reality objects located in or interacting with objects in the physical world. The data can be provided to the applications 3816 through an API, which can be implemented as a push interface that pushes data to the applications 3816 as relevant portions change, or a pull interface that responds to requests from the applications 3816 for the most recent information, or both.

[0288] For example, application 3816 can operate on a grid representation of a portion of the physical world that forms a 45-degree view relative to an origin defined by the device's current location and the direction the device is facing. A grid can be computed to represent the region when changes in that region or data indicating physical changes within that region become available. This grid can be computed in application 3816 based on data provided by a service, or it can be computed in the service and provided to application 3816. In both cases, the service can store information in the physical world, thereby simplifying grid computation. As described herein, blocks with zippers, side edges, or other techniques can be used to "mask" cracks between adjacent blocks, allowing only the changed portions of the physical world representation to be processed. The changed portions of the physical world representation can then replace the corresponding portions in the previous representation of the physical world.

[0289] Efficient access to representations of portions of the physical world used to generate the mesh that application 3816 will use to render the scene to the user reduces computing resources, making XR systems easier to implement on portable devices or other devices with limited computing resources, and producing a more realistic user experience because the XR scene better matches the physical world. Accordingly, instead of using blocks with side edges, zippers, or other techniques to facilitate masking on cracks between blocks, as described elsewhere in this document, algorithms for page-in and page-out of blocks into and out of active memory and / or local cache can be chosen to reduce the time spent accessing the blocks required to compute the mesh at any given time.

[0290] exist FIG. 39 In an exemplary embodiment, gateway 3818a is a pull interface. When AR application 3816 requests information about a region in the physical world, but the block representing the region is not in active memory 3812b, processor 3812 can search for the block held in local cache 3814. If processor 3812 cannot find the block in either active memory 3812b or local cache 3814, processor 3812 can search for the block held in cloud cache 3802. Since access to active memory 3812b has lower latency than access to data in local cache 3814, and access to data in local cache 3814 has lower latency than access to data in cloud cache 3802, the overall speed of mesh generation can be improved by implementing a paging algorithm service that loads blocks into active memory before they are requested, or moves blocks from cloud cache 3802 to local cache 3814 before they are requested.

[0291] Similar to the first device 3810, the second device 3820 can include a processor 3822 having a compute portion 3822a and an active memory 3822b, a local cache 3824, and one or more AR applications 3826. The AR applications 3826 can communicate with the processor 3822 through a local gateway 3828a. The local cache 3824 can communicate with the processor 3822 through a local gateway 3828b.

[0292] Accordingly, the cloud cache 3802 can maintain blocks transmitted from both devices 3810 and 3820. The first device 3810 can access blocks captured and transmitted from the second device 3820 in the cloud cache 3802; likewise, the second device 3820 can access blocks captured and transmitted from the first device 3810 in the cloud cache 3802.

[0293] Devices 3801 and 3802 are provided as examples of portable AR devices. Any suitable device, such as a smartphone, can be similarly used and implemented.

[0294] FIG. 41 is a flowchart illustrating a method 3900 of operating a system (e.g., system 3700) in accordance with some embodiments. At act 3902, a device can capture 3D information about a physical world including objects in the physical world and represent the physical world as blocks including 3D reconstruction data. In some embodiments, the 3D reconstruction data can be captured by a single system and used only to render information on that system. In some embodiments, the 3D reconstruction data can be captured by multiple systems and used to render information on any of the multiple systems or any other system. In these embodiments, the 3D reconstruction data from the multiple systems can be combined and accessible to the multiple systems or any other system.

[0295] For example, several users each wearing an AR system can set their devices to augmented mode while exploring a warehouse. The sensors of each device can be capturing 3D information (e.g., 3D reconstruction data including depth maps, images, etc.) of the warehouse in the file of view of the sensors, including objects in the warehouse (e.g., tables, windows, doors, floors, ceilings, walls). Each device can segment the warehouse into regions having corresponding volumes and represent the individual regions as blocks. The blocks can have versions. Each version of a block can have values representing the objects in the region of the physical world at a certain point in time.

[0296] When an application requires information about the physical world, a version of the tile representing the portion of the physical world can be selected and used to generate the information. Although this selection process can be performed by any suitable processor or distributed across any suitable processors, according to some embodiments, the process can be performed locally on a device on which the application requesting the data is executing.

[0297] Accordingly, at act 3904, a processor (e.g., processor 3812 or 3822) can respond to a request for 3D reconstruction data from an application (e.g., AR application 3816 or 3826). In some embodiments, regardless of whether an application requests 3D reconstruction data, a device can continue to capture 3D information including 3D reconstruction data about the physical world and represent the physical world as tiles of 3D reconstruction data. The 3D reconstruction data can be used to create new versions of the tiles.

[0298] If the application requests 3D reconstruction data, the process can proceed to act 3906, in which the processor can identify a subset of tiles corresponding to a portion of the physical world needed to deliver the 3D reconstruction data according to the request. The identification of the tiles can be based on, for example, data collected by sensors (e.g., depth sensor 51, world camera 52, inertial measurement unit 57, global positioning system, etc.). A multi-device system can create a common coordinate system, so that the different tiles generated by different devices associated with a corresponding portion of the physical world can be created using the common coordinate system without having to consider which device provided the 3D reconstruction data to reconstruct the portion of the physical world represented by the tile. As one example of how a common coordinate system can be created, data from devices in approximately the same vicinity can be routed to the same server or servers for processing. There, the data from each device can initially be represented in a device-specific coordinate system. Once enough data has been collected from each device to identify features in a common portion of the physical world, the features can be related, providing a translation from one device-specific coordinate system to the others. One of the device-specific coordinate systems can be designated as the common coordinate system, while translations between the other coordinate systems can be made, and this coordinate system can be used to translate data from a device-specific coordinate system to the coordinate system designated as the common coordinate system. Other examples of creating tracking maps and localizing according to them are provided in U.S. Patent Application No. 62 / 928,833, which is hereby incorporated by reference.

[0299] Regardless of the specific mechanism by which the common coordinate system is created, once the common coordinate system is created, 3D reconstruction data generated by an application requesting data for the first device can be based on updates from other devices if those updates were made after any version made by the first device. Blocks from the first device and other devices can be merged by using, for example, the common coordinate system.

[0300] The particular processing in act 3906 can depend on the nature of the request. In some embodiments, if the application requesting the 3D reconstruction data maintains its own information about blocks and requests a particular block, the request for 3D reconstruction data at act 3904 can include a reference to a particular subset of blocks, and identifying the subset of blocks at act 3906 can include determining the subset of blocks corresponding to the particular subset of blocks. In some embodiments, the request for 3D reconstruction data at act 3904 can include a reference to a field of view of a device on which the application is executing, and identifying the subset of blocks at act 3906 can include determining the subset of blocks corresponding to the reference field of view of the device.

[0301] Regardless of the manner in which the blocks are identified / determined, at act 3908 the processor can select versions of the blocks of the subset of blocks. The selection can be based on one or more criteria. The criteria can be based on, for example, the most recent version of the block from available sources. In the illustrated embodiment, the versions of the blocks can be stored in active memory, local cache, or remote cache. Act 3908 can include, for example, selecting the version in active memory if available, or selecting the version in local cache if available, or selecting the version from remote cache if available. If no block versions are available, the selection can require generating the block, for example, from data (e.g., 3D reconstruction data) collected with sensors (e.g., depth sensor 51, world camera 52, and / or inertial measurement unit 57). Such algorithms for block selection can be used in a system that manages, for example, background processes, block versions stored in each of the possible locations. An example management process is described below in connection with FIG. 39B. FIG. 40 An example management process is described.

[0302] At act 3910, the processor can provide information based on the selected block versions to the application. The processing at act 3910 can require simply providing the blocks to the application, which can be appropriate when the application uses the blocks directly. In the case of the application receiving meshes, the processing at act 3910 can require generating the meshes from the blocks and / or the subset of blocks, and providing the meshes or any appropriate portions of the meshes to the application.

[0303] FIG. 40is a flowchart showing details of an action 3902 of FIG. 39 is a flowchart showing details of an action 3902 of FIG. 37 is a flowchart showing details of an action 3902 of

[0304] At action 4004, a processor (e.g., processor 3812 or 3822) of the system can create a version of the tiles of 3D reconstruction data of the physical world based on the 3D information captured by the one or more sensors. In some embodiments, each tile can be formatted as one or more portions of a mesh. In some embodiments, other representations of the physical world can be used.

[0305] The tiles can have versions such that whenever information about a region of the physical world is captured by any device, a new version of the tile can be stored. Each version of a tile can have 3D reconstruction data that includes values representing objects in the region of the physical world at a certain point in time. In some embodiments, such processing can be performed locally on a device, resulting in a new version of a tile being stored in active memory. In some embodiments, in a multi-device system, similar processing can be performed in a server (e.g., server 3710 of FIG. 41 ), which can manage versions of the tiles such that the most recent version of each tile available in its remote cache is provided whenever requested by any device.

[0306] Because the tiles represent the physical world, most of which will remain unchanged, a new version of a tile does not necessarily have to be created when new 3D reconstruction data representing the corresponding region of the physical world is available. Rather, managing versions of the tiles can entail processing 3D reconstruction data representing the physical world to determine whether there has been sufficient change since the last version of the tiles representing those regions of the physical world. In some embodiments, sufficient change can be indicated by the size of a tile metric becoming above a threshold since the last version has been stored.

[0307] In some embodiments, other criteria can be applied when a tile is requested to determine which version of the tile is provided as the current version, such as the version having the smallest value of a metric indicating the completeness or accuracy of the data in the tile. Similar processing can be performed on each device, resulting in versions of the tiles stored in local caches on the devices.

[0308] One or more techniques can be used to manage versions of blocks available to services on each device. For example, if there is an acceptable version of a block that has already been computed, rather than creating a new version of the block from sensor data, the processor can access the previously stored block. Such access can be performed efficiently by managing storage of versions of blocks. At action 4006, the processor of the device can page a version of a block of 3D reconstruction data of the physical world from the active memory (e.g., active memory 3812b or 3822b). Paging can include the processor accessing sensor data to continuously update the blocks in the active memory / local cache / cloud cache, e.g., according to the field of view of the device. When the field of view of the device changes, blocks corresponding to the new field of view can be transferred (e.g., paged) from the local cache and / or cloud cache to the active memory, and blocks corresponding to areas just outside the new field of view (e.g., blocks adjacent to the blocks in the new field of view) can be transferred (e.g., paged) from the active memory and / or cloud cache to the local cache. For example, at action 4008, the version of the block paged by the processor can be maintained in local memory (e.g., local cache 3814 or 3824) and / or remote memory (e.g., cloud cache 3802). In some embodiments, when each new version of a block is created, e.g., on a device, the version can be sent to remote memory so that it can be accessed by other users.

[0309] FIG. 41 is a flow diagram illustrating an example process for performing selecting a version of a block that represents a subset of blocks, in accordance with some embodiments. In some embodiments, FIG. 39 is a flow diagram illustrating FIG. 42 action 3908 of FIG. 39. To select a version of each block in the subset of blocks, at action 4102, the processor (e.g., processor 3812 or 3822) can query whether the latest version is stored in the active memory (e.g., active memory 3812b or 3822b). In some embodiments, whether a version is the latest can be determined by comparing a value (e.g., a geometry change size, a timestamp, etc.) attached to the version with data collected with sensors (e.g., depth sensor 51, world camera 52, and / or inertial measurement unit 57). In some embodiments, a comparison can be made between current sensor data and a version of a block stored in the active memory. Based on the degree of difference, which can represent changes in the physical world or quality of the version in the active memory, for example, the version in the active memory can be considered the latest.

[0310] If the latest version is stored in the active memory, processing proceeds to act 4104, in which the latest version is selected. If the latest version is not stored in the active memory, processing proceeds to act 4106, in which the processor can query whether the latest version is stored in a local memory (e.g., local cache 3814 or 3824). The query can be performed using the criteria described above in connection with act 4102 or any other suitable criteria. If the latest version is stored in the local memory, the latest version is selected at act 4108.

[0311] If the latest version is not stored in the local memory, the processor can query whether the latest version is stored in a remote memory (e.g., cloud cache 3802) at act 4110. The query can also be performed using the criteria described above in connection with act 4102 or any other suitable criteria. If the latest version is stored in the remote memory, the latest version is selected at act 4112.

[0312] If the latest version is not stored in the remote memory, processing can proceed to act 4114, in which the processor of the device can generate a new version of the tile based on 3D information (e.g., 3D reconstruction data) captured by the sensors. In some embodiments, the processor can identify neighboring tiles of the tile with the new version at act 4116 and update the identified neighboring tiles according to the new version of the tile.

[0313] FIG. 41 FIG. 42 is a flowchart illustrating a method 4200 of operating a system in accordance with some embodiments. In method 4200, rather than pulling tiles into the active memory and / or local cache when the device needs those tiles, paging can be managed based on the projection of the field of view of the device based on the motion of the device.

[0314] Similar to act 3902, at act 4202, sensors on the device can capture 3D information about a physical world including objects in the physical world and represent the physical world as tiles including 3D reconstruction data.

[0315] At action 4204, a processor (e.g., processor 3812 or 3822) can compute a region of the physical world based at least in part on the output of the sensors in which the portable pointable component (e.g., depth sensor 51, world camera 52, and / or inertial measurement unit 57) will be pointed at some time in the future. In some embodiments, the processor can make the computation based on motion data from inertial sensors or analysis of captured images. In a simple computation, e.g., to obtain a quick result, the processor can make the computation based on the translation and rotation of the user's head. In applying a more comprehensive algorithm, the processor can make the computation based on objects in the scene. For example, an algorithm can consider a user walking toward a wall or a table to be less likely to walk through the wall or table.

[0316] At action 4206, the processor can select tiles based on the computed region. At action 4208, the processor can update the active memory (e.g., active memory 3812b or 3822b) with the selected tiles. In some embodiments, the processor can select tiles based on the computed region and the state of the active memory. For example, the processor can select tiles that are not already in the active memory. FIG. 39 At action 4210, the processor can select tiles from the active memory to provide to an application (e.g., application 3816 or 3826), e.g., through an API, based on changes in each tile since the tile's version was last provided to the application.

[0317] In some embodiments, at action 4206, the processor can request the selected tiles from a remote memory (e.g., cloud cache 3802) and update information stored in a local cache (e.g., 3814 or 3824) so that the local cache stores the selected tiles. Action 4206 can be similar to action 3908 described in FIG. 43A- FIG. 48

[0318] ​In some embodiments, the tiles can be maintained across shutdown / startup periods, sleep / wake periods, and various other scenarios. When a portable user device used in an XR system transitions to an inactive operating state, the tiles can be maintained by ensuring that the tiles are stored in memory that will retain the tiles until a new session of the device is turned on by the user. When the new session begins and the device transitions to an active operating state, the maintained information can be selectively used. For example, tiles maintained in various memory (e.g., active memory, local memory, remote memory) during a shutdown / sleep time (e.g., a time including device shutdown / power-off / sleep of active memory / local memory) can be retained in the respective memory and thus available during a startup / wake time (e.g., a time including device startup / power-on / wake of active memory / local memory). This approach can be used, for example, where the memory storing the tiles is non-volatile memory and / or is sufficiently powered to retain the information after the event that stops or suspends device operation. Alternatively or additionally, the memory storing the tiles can be copied to such memory as part of transitioning to a state where device operation is stopped or suspended, and then copied back as part of transitioning back to an operating state.

[0319] Maintaining tiles across events that stop or suspend device operation makes tiles maintained at the end of a previous session available at the beginning of a current session. In some embodiments, the maintained tiles from a previous session can be used quickly in a new session, even without being retained in or returned to the same memory they were used in the previous session. For example, at shutdown, tiles in active memory can be transferred to local memory and / or remote memory where they are maintained, and tiles in local memory can be transferred to remote memory where they are maintained. After the new session begins, tiles can be selectively moved from local or remote memory to active memory. Likewise, tiles can be selectively moved to local memory. For example, techniques described herein can be used to determine which tiles to move from remote memory to active memory or local memory based on a current or projected field of view of a device user.

[0320] One or more techniques can be used to determine whether to use the maintained tiles in a new session. In some embodiments, for example, tiles can have metadata associated with them that can be used to determine whether to use the tiles. As described above, a device can track its location relative to the physical world, which can enable the device to be positioned relative to a coordinate system shared by multiple devices. Tiles can have associated metadata including location information that can be compared to the device location when a new session begins. Maintained tiles representing portions of the physical world in the vicinity of the device can be used in the new session.

[0321] Other metadata, such as a block version, can be used to determine whether a block is still up to date. In some embodiments, a device can communicate with a cloud service that shares blocks with multiple devices to determine the most up to date version of a block representing a portion of the physical world. If the device has saved the most up to date block, it can continue to use it. Alternatively or additionally, a timestamp reflecting the time of creation of a block can be used. For example, a saved block can be used when its timestamp is less than a threshold amount of time before the current time.

[0322] In some embodiments, a device can determine whether to use a saved block based on a comparison between the saved block and current sensor data about the physical world. For example, at startup, a saved block in various memory can only be used after the real world is identified. For example, sensors on a device can capture 3D information about the physical world, including objects in the physical world, and represent the physical world as blocks including 3D reconstruction data. The device can compare the captured blocks to saved blocks in active memory, local memory, and / or remote memory, and in response to a match, retrieve the matching block from active memory, local memory, and / or remote memory for use.

[0323] Alternatively or additionally, a saved block can be compared to the physical world around a device at startup without creating a mesh. For example, surfaces in the physical world can be detected with sensors such as depth sensors and compared to surfaces indicated by meshes in blocks corresponding to locations in the physical world where the surfaces were detected. If there is an error between the sensed locations of one or more surfaces and surfaces indicated by meshes of saved blocks, the saved blocks can be used.

[0324] Blocks saved on a device that are determined to be unsuitable for use in a new session can be deleted or otherwise handled so that they are not used.

[0325] Block-based processing as described above can be based on blocks that allow portions of 3D representations to be processed separately and then combined with other blocks. According to some embodiments, blocks can be formatted so that when a block changes, the representation of the change largely or completely preserves the value of the block at the interface with other blocks. Such processing enables changed versions of blocks to be used with versions of adjacent blocks that have not changed without creating unacceptable artifacts in scenes rendered based on the changed and unchanged blocks. FIG. 43A Such blocks are illustrated.

[0326] A 3D representation of the physical world can be provided by volumetric 3D reconstruction, which can create a hierarchy of 3D reconstruction data from 3D information of the physical world captured by sensors. For example, the sensors can be depth cameras, which can capture 3D information of the physical world, e.g., a stream of depth images with various poses of the depth cameras (i.e., camera poses). The 3D information of the physical world can be processed into a voxel grid. Each voxel can contain one or more signed distance functions (SDFs) that describe whether the voxel is inside or outside the geometry of an object in the physical world. The voxels can be grouped into“bricks.” Each brick can include, e.g., a cubic volume of voxels, e.g., 8 3 voxels. The bricks can be further grouped into“tiles.” Each tile can include multiple bricks.

[0327] In some embodiments, the voxel grid can be mapped to conform to a memory structure. The tiles can correspond to the memory pages of the storage medium. The size of the tiles can be variable, e.g., depending on the size of the memory pages of the storage medium used. Thus, the 3D reconstruction data can be sent between storage media (e.g., the device’s main and / or local memory, and / or remote memory in the cloud) on a tile basis. In some embodiments, one or more tiles can be processed to generate a chunk. The chunk can be updated, e.g., when at least one voxel in one or more tiles changes.

[0328] The chunks can not necessarily be limited to corresponding to the tiles. In some embodiments, the chunks can be generated from one or more bricks, one or more voxels, or one or more SDF samples, etc. The chunks can be any suitable partitioning of the physical world. The chunks do not necessarily have to be limited to a grid format. The chunks can be in any suitable format of the 3D reconstruction data.

[0329] FIG. 43B -D shows an example physical world 4300 represented by grid chunks 4302. Each grid chunk 4302 can be extracted from voxels 4304 corresponding to a predetermined volume of the grid chunk. In the example shown, each chunk can be the output of a cubic region (e.g., 1 m 3 ) of voxels in a low-level reconstruction model. Each grid chunk 4302 can contain a portion of the world grid and can be processed independently. Since certain chunks change when moving things around in a new area or environment, scalability can be achieved by fast local updates. In the example shown, none of the grid chunks changed except for grid chunk 4306, which has a new object 4308 placed in front of an existing surface 4310. In this case, the AR system only needs to update grid chunk 4306, which can save a lot of computational power compared to updating the entire grid of the world.

[0330] FIG. 43C FIG. 4A is a simplified schematic diagram illustrating a mesh tile according to some embodiments. In the example shown, the mesh tile can have a fully connected mesh inside, meaning that vertices are shared by multiple triangles.

[0331] On the other hand, individual mesh tiles can be disconnected independent meshes. FIG. 44 FIG. 4B is a simplified schematic diagram illustrating a crack that can exist at the edge of two adjacent mesh tiles according to some embodiments. FIG. 43C D is a simplified schematic diagram illustrating masking of cracks in FIG. 4B by implementing mesh side edges that overlap with adjacent mesh tiles according to some embodiments. FIG. 44

[0332] FIG. 45 FIG. 4E is a schematic diagram illustrating a representation 4400 of a portion of a physical world partitioned in 2D according to some embodiments. The 2D representation 4400 can be obtained by connecting a set of four tiles (tiles A-D). The representation 4400 can be partitioned into four tiles: tile 4402, 4404, 4406, and 4408, shown by solid lines 4410, for example. In some embodiments, the tiles 4402, 4404, 4406, and 4408 can be designated as tiles A-D, respectively. An application can need 3D reconstruction data in mesh format for further processing, such as occlusion testing, and for generating physics in a physics engine. In some embodiments, the set of tiles can be in the format of a mesh, which can be generated by a device (e.g., devices 3810, 3820), a network (e.g., a cloud including cloud cache 3802), or a discrete application (e.g., applications 3816, 3826).

[0333] In some embodiments, the areas at the boundaries of the tiles 4402, 4404, 4406, and 4408 can be side edges, shown by dashed lines 4412, for example. In some embodiments, each of the tiles A-D can include a tile and a corresponding side edge. For example, tile B can include tile 4404 and side edges 4414 that overlap with the boundary portions of the adjacent tiles A, C, and D of tile B, so that cracks on the boundaries between the tiles can be masked when the tiles are connected into a global mesh. Tiles A, C, and D can also include corresponding side edges. Thus, any cracks between the tiles can be masked by a processor before the tiles including 3D reconstruction data are returned to an application.

[0334] In some embodiments, the global mesh can be a topologically connected global mesh. For example, adjacent tiles in the set of tiles can share mesh vertices at tile boundaries such as lines 4410. In some embodiments, the global mesh can be visually seamless despite being topologically disconnected using any suitable technique, such as side edges and zippers. ​

[0335] Although a method employing side edges is shown, other methods can be used to enable the changed block to join with the unchanged adjacent block, such as a zipper. Although in the example shown, a portion of the physical world is represented by four 2D blocks, it should be appreciated that a portion of the physical world can be represented by any suitable number (e.g., two, three, five, six, or more) of 2D and / or 3D blocks. Each block can correspond to a space in the physical world. In some embodiments, the blocks in the 2D and / or 3D representation of the portion of the physical world can correspond to spaces of the same size (e.g., area / volume) in the physical world. In some embodiments, the blocks in the 2D and / or 3D representation of the portion of the physical world can correspond to spaces of different sizes in the physical world.

[0336] FIG. 46 is a diagram illustrating a 3D representation 4500 of a portion of a physical world, according to some embodiments. Similar to the 2D representation 4400, the 3D representation 4500 can be obtained by connecting eight blocks (blocks A-H). In some embodiments, the blocks A-H can be exclusive of each other, e.g., have no overlapping areas. In some embodiments, the blocks A-H can have areas that overlap with adjacent blocks (e.g., side edges 4516). In some embodiments, each of the blocks A-H can have a version. Each version of a block can have values representing objects in the block's area at a certain point in time. In the example shown, the 3D representation 4500 includes versions of blocks A-H: version 4502 of block A, version 4504 of block B, version 4514 of block C, version 4512 of block D, version 4534 of block E, version 4506 of block F, version 4508 of block G, and version 4510 of block H. Version 4502 of block A can include values 4518; version 4504 of block B can include values 4522; version 4514 of block C can include values 4528; version 4512 of block D can include values 4532; version 4534 of block E can include values 4520; version 4506 can include values 4524; version 4508 can include values 4526; and version 4510 can include values 4530.

[0337] FIG. 47is a diagram illustrating a 3D representation 4600 of a portion of the physical world obtained by updating the 3D representation 4500, according to some embodiments. In comparison to the 3D representation 4500, the 3D representation 4600 can have a new version 4610 of tile H that includes information 4630. The information 4630 can be different from the information 4530. For example, the first device can have maintained the version 4510 of tile H in the remote memory. The version 4510 of tile H can include the information 4530 corresponding to the table having an empty surface. After the first device leaves the area (e.g., the field of view of the first device no longer includes tile H), a second device can place a virtual and / or physical box on the surface of the table and then maintain the version 4610 of tile H in the remote memory. The version 4610 of tile H can include the information 4630 corresponding to the table having a virtual and / or physical box. If the first device returns, the first device can be able to select the version 4610 of tile H to view from the available versions of tile H, including the versions 4610 and 4510 of tile H.

[0338] FIG. 48 is a diagram illustrating an augmented world 4700 that a first device 4702 (e.g., device 3810) and a second device 4712 (e.g., device 3820) can view. The first and second devices can include AR display systems 4704 and 4714 (e.g., AR display systems 80) operating in an augmented mode. The augmented world 4700 can be obtained by connecting four tiles (tiles A-D). In the illustrated example, the augmented world 4700 includes versions of tiles A-D: a version 4702A of tile A, a version 4702B of tile B, a version 4702C of tile C, and a version 4702D of tile D. The first device 4702 can be looking in a first direction 4706 and have a first field of view (FOV) 4708. In the illustrated example, the first FOV includes the version 4702B of tile B and the version 4702D of tile D. A processor (e.g., 3812) of the second device 4704 can include computer-executable instructions to identify tiles B and D corresponding to the first FOV and select the version 4702B of tile B and the version 4702D of tile D. The second device 4714 can be looking in a second direction 4716 and have a second FOV 4718. In the illustrated example, the second FOV includes the version 4702C of tile C and the version 4702D of tile D. A processor (e.g., 3822) of the second device 4714 can include computer-executable instructions to identify tiles C and D corresponding to the second FOV and select the version 4702C of tile C and the version 4702D of tile D.

[0339] Method for occlusion rendering using ray casting and real-time depthis a schematic diagram illustrating an augmented world 4800 obtained by updating the augmented world 4700 with new versions of the tiles, according to some embodiments. Compared to the augmented world 4700, the augmented world 4800 can include a version 4802C of tile C that is different from the version 4702C, and a version 4802D of tile D that is different from the version 4702D. The first device 4702 can look in a third direction 4806 and have a third FOV 4808. In the example shown, the third FOV includes the version 4802C of tile C and the version 4802D of tile D. The processor of the first device 4702 can include computer-executable instructions to determine which of the versions 4702C, 4802C, 4702D, and 4802D to provide to the application based on, for example, changes in the FOV and / or information collected with the sensors of the first device 4702. In some embodiments, the processor of the first device 4702 can include computer-executable instructions to generate the version 4802C of tile C and the version 4802D of tile D when the corresponding latest versions of tiles C and D are not in local memory (e.g., local cache 3814, 3824) or remote memory (e.g., cloud cache 3802). In some embodiments, the first device 4702 can be able to estimate changes in its FOV (e.g., from the first FOV 4708 to the third FOV 4808), select tile C based on the estimate, and store the version of tile C to memory closer to the processor (e.g., move the version of tile C from remote memory to local cache, or from local cache to active memory).

[0340] FIG. 6

[0341] The realism of presenting AR and MR scenes to a user can be enhanced by providing the application that generates the AR and MR scenes with occlusion data, where the occlusion data is derived from a combination of one or more depth data sources. The occlusion data can represent surfaces of physical objects in the scene, and can be formatted in any suitable manner, such as by depth data indicating a distance from a viewpoint at which the scene is to be rendered and a surface. For example, using component 164, the occlusion data can be received from perception module 160. FIG. 49

[0342] ​However, in some embodiments, one data source can be one or more depth cameras that directly sense and capture positions between the depth cameras and real objects in the physical world. Data from the depth cameras can be provided directly to the usage component 164, or can be provided indirectly, e.g., through the perception module 160. The one or more depth cameras can provide an instantaneous view of the physical world at a frame rate that is high enough to capture changes in the physical world but low enough to not add processing burden. In some embodiments, the frame rate can be 5 frames per second, 10 frames per second, 12 frames per second, 15 frames per second, 20 frames per second, 24 frames per second, 30 frames per second, etc. In some embodiments, the frame rate can be less than 5 frames per second. In some embodiments, the frame rate can be greater than 30 frames per second. Thus, in some embodiments, the frame rate can be in the range of 1-5 frames per second, 5-10 frames per second, 10-15 frames per second, 15-20 frames per second, or 20-30 frames per second, etc.

[0343] The second data source can be a stereo vision camera that can capture a visual representation of the physical world. Depth data from the depth cameras and / or image data from the vision cameras can be processed to extract points representing real objects in the physical world. Images from a vision camera, such as a stereo camera, can be processed to compute a three-dimensional (3D) reconstruction of the physical world. In some embodiments, depth data can be generated from images from the vision cameras, e.g., using deep learning techniques. Some or all of the 3D reconstruction can be computed prior to occlusion data and stored in memory. In some embodiments, the 3D reconstruction can be maintained in computer memory by a process that is independent of any process that generates depth information for occlusion processing, which can access the stored 3D reconstruction as needed. In some embodiments, the 3D reconstruction can be maintained in memory, and portions thereof can be updated in response to indications, e.g., based on depth information, that there is a change in the physical world corresponding to a portion of the 3D reconstruction. In some embodiments, the second data source can be implemented through ray casting into the 3D reconstruction of the physical world to obtain low-level 3D reconstruction data (e.g., a ray cast point cloud). Through ray casting, data from the second data source can be selected to fill in any holes in the occlusion data, enabling integration of data from both (or more) sources.

[0344] According to some embodiments, the depth data and / or the image data and / or the low-level data of the 3D reconstruction can be oriented with respect to a user of the AR or MR system. For example, such orientation can be achieved through the use of data from sensors worn by the user. The sensors can be worn on, e.g., a head-mounted display device / unit.

[0345] In systems in which occlusion data can be generated from multiple depth data sources, the system can include a filter that identifies which portions of a 3D region are represented by data from each of the multiple depth data sources. The filter can apply one or more criteria to identify portions of the region for which data from a second data source will be collected. These criteria can be an indication of the reliability of the depth data. Another criterion can be a detected change in the portion of the region as a result of which the depth data is collected.

[0346] Selecting between multiple depth data sources to provide data for different portions of a representation of a 3D region can reduce processing time. For example, when the processing required to derive occlusion data from data collected with a first data source is less than the processing required with a second data source, the selection can favor data from the first data source, but use data from the second source when data from the first data source is unavailable or unacceptable. As a particular example, the first data source can be a depth camera, and the second data source can be a stereo vision camera. Data from the stereo camera can be formatted as a 3D reconstruction of the physical world. In some embodiments, the 3D reconstruction can be computed in advance of the need for occlusion data. Alternatively or additionally, the 3D reconstruction can be recomputed when occlusion data is needed. In some embodiments, criteria can be applied to determine whether the 3D reconstruction should be recomputed.

[0347] In some embodiments, the occlusion data is computed by a service that provides the occlusion data to an application executing on a computing device that is rendering an XR scene through an application programming interface (API). The service can execute on the same computing device as the application, or can execute on a remote computer. The service can include one or more of the components discussed herein, such as a filter for data from a first data source, and / or an engine for selectively obtaining data from a second data source based on filtered data from the first data source. The service can also include components that combine the filtered data from the first data source with the selected data from the second data source to generate the occlusion data.

[0348] Occlusion data can be formatted in any suitable way to represent surfaces in the physical world. For example, occlusion data can be formatted as a depth buffer for a surface, thereby storing data identifying the location of the surface in the physical world. The occlusion data can then be used in any suitable manner. In some embodiments, occlusion data can be provided to one or more applications that want to occlude virtual objects with real objects. In some embodiments, occlusion data can be formatted as a depth filter created by the system for applications that request occlusion data from an occlusion service to render virtual objects at one or more locations. The depth filter can identify locations for which the application should not render image information for virtual objects, because the virtual objects at these locations will be occluded by surfaces in the physical world. It should be understood that “occlusion data” can have a suitable format to provide information about surfaces in the physical world and does not need to be used for occlusion processing. In some embodiments, occlusion data can be used in any application that performs processing based on a representation of surfaces in a scene in the physical world.

[0349] Compared to conventional AR and MR systems that use mesh data to perform occlusion processing, the method described in this paper provides occlusion data with less latency and / or lower computational resources. Mesh data can be obtained from geometric data extracted by processing image sensors using multiple time- or cost-intensive steps, including a traveling cubes algorithm, mesh simplification, and the application of triangle counting constraints. Computing mesh data can take hundreds of milliseconds to seconds, and the latency of having the latest mesh can lead to visible artifacts when the environment changes dynamically and the application renders the scene using an outdated mesh. These artifacts manifest, for example, as virtual content appearing superimposed on top of real objects when virtual content is rendered as if it were rendered behind them, disrupting the user's sense of immersion and providing incorrect cues regarding 3D depth perception.

[0350] For applications that use meshes for occlusion processing to have the latest mesh, the application must either continuously query the mesh (resulting in a large amount of continuous processing) or use a mechanism to determine if there are changes and then query the new mesh (this reduces overall processing, but still has high latency between changes in the physical world and when the mesh reflecting those changes arrives at the application).

[0351] By using low-level data from 3D reconstruction data (such as point clouds) and real-time depth data instead of meshes for occlusion, the latency between changes occurring in the environment and being reflected in the occlusion data can be reduced, thus maintaining closer synchronization with the physical world and thereby achieving higher perceived visual quality.

[0352] In some embodiments, a real-time depth map of the physical environment can be obtained from a depth sensor (e.g., a depth camera). Each pixel in the depth map can correspond to a discrete distance measurement captured from a 3D point in the environment. In some embodiments, these depth cameras can provide a depth map comprising a set of points at a real-time rate. However, the depth map can have holes, which can result from the depth camera being unable to acquire sensor data representing an area or acquiring incorrect or unreliable data representing an area. In some embodiments, if the depth sensor uses infrared (IR) light, these holes can be generated, for example, due to materials or structures in the physical environment not reflecting the IR light well or not reflecting it at all. In some embodiments, these holes can be produced, for example, by very thin structures or surfaces at grazing angles of incidence that do not reflect light toward the depth sensor. The depth sensor can also experience motion blur when moving quickly, which can also result in lost data. In addition, a "hole" in the depth map represents an area of the depth map that is not suitable for use in occlusion processing due to any other reason. Such holes can be detected using any appropriate processing, such as processing the depth map to detect insufficient connectivity between points or areas in the depth map. As another example, holes can be detected using processing that computes a quality metric for areas of the depth map and processing areas with low quality metrics as holes. One such metric can be the inter-image variation of pixels in the depth map that represent the same location in the physical world. Pixels with such variation exceeding a threshold can be classified as holes. In some embodiments, holes can be identified by pixels that satisfy a predetermined statistical criterion for clusters of pixels in which the quality metric is below a threshold.

[0353] In some embodiments, the depth map can first be "filtered" to identify holes. Then, rays from the perspective of the scene from which the holes will be rendered can be determined. These rays can be "cast" into a 3D representation of the physical world created using multiple sensors rather than a single depth sensor to identify data representing areas of the holes. The 3D representation of the physical world can be, for example, a 3D reconstruction created from data from stereo vision cameras. The data from the 3D reconstruction identified by this ray casting can be added to the depth map, filling in the holes.

[0354] When holes are identified, the 3D reconstruction can be computed from image sensor data. Alternatively, some or all of the 3D reconstruction can be pre-computed and stored in memory. For example, the 3D reconstruction can be maintained in computer memory by a process that is independent of any process that generates the depth information used for occlusion processing, and the stored 3D reconstruction can be accessed as needed by the process. As a further alternative, the 3D reconstruction can be maintained in memory, but portions of it can be updated in response to indications computed based on the depth information that there is a change in the physical world corresponding to the portion of the 3D reconstruction.

[0355] In an XR system, the rays can have the same pose as the user's eye gaze. In the example system described below, the depth map can similarly be acquired with the same eye gaze as the user, as the depth sensor can be worn by the user and can be mounted on the user's head close to the eyes. The user can similarly wear the visual cameras used to form the 3D reconstruction data, so that the images and data derived from these images can be related to a coordinate system that enables rays defined with respect to the depth map to be projected into the 3D reconstruction computed from the visual images. Inertial measurement units and / or other sensors similarly worn by the user and / or associated with the sensors can provide data to perform coordinate transformations to add data to the 3D representation independent of the pose of the visual cameras, and to relate rays defined with respect to the depth map to the 3D reconstruction.

[0356] In some embodiments, the user's focus or related virtual content placement information can guide the ray casting to be adaptive in image space by casting denser rays at depth discontinuities to obtain high quality occlusion at object boundaries and sparse rays in the center of objects to reduce processing requirements. The ray casting can additionally provide local 3D surface information (e.g., normals and positions) that can be used to improve temporal warping processing with depth information and mitigate loss of visible pixels that would otherwise need to be rendered or ray traced in a typical rendering engine. Temporal warping is a technique in XR that modifies rendered images before sending them to a display to correct for computed head movement that would occur between rendering and display. In some embodiments, temporal warping can be used to synchronize data from the depth map with a 3D representation of the physical world that can be used to generate data to fill holes in the depth map. Data from both data sources can be temporally warped to represent the pose computed at the time of display. In some embodiments, data from the 3D representation can be temporally warped to represent the pose computed at the time the data was captured with the depth map.

[0357] In some embodiments, advanced features such as temporal warping can utilize the 3D local surface information from the ray casting. Temporal warping can fill all lost visible pixels that were previously occluded when rendering a frame of content without physical world occlusion or with eroded depth imaging. As a result, the rendering engine can not be required to fill the pixels, enabling a more loosely decoupled rendering application (or more independent temporal warping).

[0358] The processing described above can be performed on data acquired by many suitable sensors and presented on many suitable interfaces in many suitable forms of hardware processor. An example of a suitable system including sensors, processing, and user interfaces is presented below. In the illustrated embodiment, a "service" can be implemented as part of an XR system having computer-executable instructions. Execution of these instructions can control one or more processors to access sensor data, then generate and provide depth information to an application executing on the XR system. These instructions can be executed on the same processor or same device as the application that executes to present an XR scene to a user, or can be executed on a remote device accessed by the user device over a computer network.

[0359] FIG. 50 An occlusion rendering system 4900 is shown in accordance with some embodiments. The occlusion rendering system 4900 can include a reconstruction filter 4902. The reconstruction filter 4902 can receive depth information 4904. In some embodiments, the depth information 4904 can be a sequence of depth images captured by a depth camera. In some embodiments, the depth information 4904 can be derived from a sequence of images captured by a vision camera, e.g., using structure from motion based on a single camera and / or using stereo computation based on two cameras. FIG. 54A A depth image 5000 is shown in accordance with some embodiments. In some embodiments, surface information can be generated from the depth information. The surface information can be indicative of distances to physical objects in a field of view (FOV) of a head-mounted display device that includes a depth camera and / or a vision camera. The surface information can be updated in real-time as the scene and FOV change.

[0360] A second source of depth information is shown as a 3D reconstruction 4908. The 3D reconstruction 4908 can include a 3D representation of the physical world. The 3D representation of the physical world can be created and / or maintained in computer memory. In some embodiments, the 3D representation can be generated from images captured by a vision camera, e.g., using structure from motion based on a single camera and / or using stereo computation based on two cameras. In some embodiments, the 3D representation can be generated from depth images captured by a depth camera. For example, the 3D reconstruction 4908 can use the depth information 4904 in conjunction with a pose of the depth camera relative to a world origin to create and / or update. The representation can be built and modified as time passes, e.g., as a user wearing the camera looks around in the physical world. In some embodiments, the depth information 4904 can also be used to generate the 3D representation of the physical world. The 3D reconstruction 4908 can be a volumetric reconstruction including 3D voxels. In some embodiments, each 3D voxel can represent a cubic space (e.g., 0.5 meters by 0.5 meters by 0.5 meters), and each 3D voxel can include data related to and / or describing a surface in the real world in the cubic space.

[0361] The 3D reconstruction 4908 of the world can be stored in any suitable manner. In some embodiments, the 3D reconstruction 4908 can be stored as a“cloud” of points representing features of objects in the physical world. In some embodiments, the 3D reconstruction 408 can be stored as a mesh, where groups of points define vertices of triangles representing surfaces. In some embodiments, the 3D reconstruction 4908 can be generated using other techniques such as a room layout detection system and / or object detection. In some embodiments, multiple techniques can be used together to generate the 3D reconstruction 4908. For example, object detection can be used for known physical objects in the physical world, 3D modeling can be used for unknown physical objects in the physical world, and a room layout detection system can also be used to identify boundaries in the physical world such as walls and floors.

[0362] The reconstruction filter 4902 can include computer-executable instructions to generate a depth map based on the depth information 4904. The depth map can include one or more pixels. Each pixel can indicate a distance to a point of a surface in the physical world. In some embodiments, the reconstruction filter 4902 can synthesize the depth information 4904 and data from the ray casting engine 4906. In some embodiments, the reconstruction filter 4902 can reduce or remove noise from the depth information 4904 based at least in part on data from the ray casting engine 4906 and / or data from a synthesis of the depth information 4904 and data from the ray casting engine 4906. In some embodiments, the reconstruction filter 4902 can upsample the depth information 4904 using a deep learning technique.

[0363] The reconstruction filter 4902 can identify regions of the depth map based on a quality metric. For example, a pixel can be determined to be incorrect or noisy when the quality metric of the pixel is above a threshold. A region of the depth map containing incorrect or noisy pixels can be referred to as a hole (e.g., hole 5002).

[0364] FIG. 54A And 54B Alternative examples of how holes are produced in a depth map in embodiments where the depth map is composed of multiple depth images are provided. FIG. 54B is a diagram of imaging with a depth camera from a first perspective to identify regions of voxels occupied by surfaces and voxels that are empty. FIG. 54B is a diagram of imaging with a depth camera from multiple perspectives to identify regions of voxels occupied by surfaces and voxels that are empty. FIG. 49A number of voxels are shown that were determined to be occupied by a surface or empty by fusing data from multiple camera images. However, the voxels in region 5420 have not yet been imaged. Region 5420 can have been imaged at location 5422 using a depth camera, but the camera has not yet moved to location. Thus, region 5420 is a space of observation for which no volumetric information is available. The AR system can direct the user wearing it to scan the space of observation.

[0365] Returning to FIG. 51 The reconstruction filter 4902 can inform the ray casting engine 4906 about the location of the holes. The ray casting engine 4906 can generate a view of the physical world given the user pose and can remove the holes from the depth map. The data can represent a portion of the user’s current view of the physical world at the current time or at a time that the data will be time warped using time warping. The ray casting engine 4906 can generate one or more 2D images, e.g., one image per eye. In some embodiments, the reconstruction filter 4902 can remove regions of the depth map that are spaced apart from the location of the virtual object by more than a threshold distance, as these regions can be irrelevant to the occlusion test for the virtual object.

[0366] The ray casting engine 4906 can be implemented by any appropriate technique that generates a view of the physical world given the user’s pose. In some embodiments, the ray casting engine 4906 can implement a ray casting algorithm on the 3D reconstruction 4908 to extract data therefrom. The ray casting algorithm can take the user’s pose as input. The ray casting engine 4906 can cast rays from a virtual camera to the 3D reconstruction 4908 of the physical world to obtain surface information missing in the depth map (e.g., holes). The ray casting engine 4906 can cast dense rays at the boundaries of physical objects in the physical world to obtain high quality occlusion at the object boundaries and sparse rays in the central regions of the physical objects to reduce processing requirements. The ray casting engine 4906 can then provide the ray cast point cloud to the reconstruction filter 4902. The ray casting engine 4906 is shown as an example. In some embodiments, the ray casting engine 4906 can be a meshing engine. The meshing engine can implement a meshing algorithm on the 3D reconstruction 4908 to extract data therefrom, e.g., including triangles and connectivity of these triangles. The meshing algorithm can take the user’s pose as input.

[0367] The reconstruction filter 4902 can synthesize the depth information 4904 and data from the raycast engine 4906 to compensate for holes in the depth map from the depth information 4904 with data from the raycast engine 4906's raycast point cloud. In some embodiments, the resolution of the depth map can be improved. This approach can be used to generate high resolution depth images from sparse or low resolution depth images.

[0368] The reconstruction filter 4902 can provide the updated depth map to an occlusion service 4910. The occlusion service 4910 can compute occlusion data based on the updated depth map and information about the location of virtual objects in the scene. The occlusion data can be a depth buffer for surfaces in the physical world. A depth buffer can store the depth of a pixel. In some embodiments, the occlusion service 4910 can be an interface to the application 4912. In some embodiments, the occlusion service 4910 can interface with a graphics system. In these embodiments, the graphics system can expose the depth buffer to the application 4912, where the depth buffer is pre-populated with occlusion data.

[0369] The occlusion service 4910 can provide the occlusion data to one or more applications 4912. In some embodiments, the occlusion data can correspond to the pose of the user. In some embodiments, the occlusion data can be a per-pixel representation. In some embodiments, the occlusion data can be a mesh representation. The applications 4912 can be configured to execute computer-executable instructions based on the occlusion data to render virtual objects in the scene. In some embodiments, the occlusion rendering can be performed by a separate graphics system rather than the applications 4912. The separate graphics system can use time warp techniques.

[0370] In some embodiments, the reconstruction filter 4902, the raycast engine 4906, and the occlusion service 4910 can be remote services, such as the remote processing module 72; or the 3D reconstruction 4908 can be stored in a remote memory, such as the remote data repository 74; and the applications 4912 can be on the AR display system 80.

[0371] FIG. 52A method 5100 of occlusion rendering in an augmented reality (AR) environment is shown in accordance with some embodiments. In act 5102, depth information can be captured from, for example, a depth sensor (e.g., depth sensor 51) on a head-mounted display device. The depth information can be indicative of distances between the head-mounted display device and physical objects. In act 5104, surface information can be generated from the depth information. The surface information can be indicative of distances to physical objects in a field of view (FOV) of the head-mounted display device and / or a user of the head-mounted display device. The surface information can be updated in real-time as the scene and FOV change. In act 5106, portions of a virtual object to be rendered can be computed from the surface information and information about a location of the virtual object in the scene.

[0372] FIG. 53 Details of act 5104 are shown in accordance with some embodiments. In act 5202, the depth information can be filtered to generate a depth map. The depth map can include one or more pixels. Each pixel can be indicative of a distance to a point of a physical object. In act 5204, low-level data of a 3D reconstruction of the physical object can be selectively obtained, for example, from 3D reconstruction 4908. In act 5206, the surface information can be generated based on the depth map and the selectively obtained low-level data of the 3D reconstruction of the physical object.

[0373] CONCLUSION Details of act 5202 are shown in accordance with some embodiments. In act 5302, a quality metric of a region of the depth map can be determined. The quality metric can be indicative of whether the region of the depth map is incorrect or noisy. In act 5304, a hole in the depth map can be identified based on the quality metric, for example, by comparison to a threshold. In act 5306, the identified hole can be removed from the depth map.

[0374] ​

[0375] Thus, having described several aspects of some embodiments, it is to be appreciated that various alterations, modifications, and improvements will readily occur to those skilled in the art.

[0376] As one example, embodiments are described in connection with an augmented (AR) environment. It should be appreciated that some or all of the techniques described herein can be applied in an MR environment or more generally in other XR environments and VR environments.

[0377] As another example, embodiments are described in connection with a device such as a wearable device. It should be appreciated that some or all of the techniques described herein can be implemented via a network (e.g., cloud), discrete applications, and / or any suitable combination of devices, networks, and discrete applications.

[0378] Such alterations, modifications, and improvements as are or can be made to the present disclosure are to be considered within the scope of the present disclosure. Further, while the present disclosure has been presented with reference to the embodiment(s) illustrated, the content and form of the present disclosure are to be understood as not being limited to the embodiment(s) described. Rather, the spirit and scope of the present disclosure are to be understood and encompass any and all embodiments within the spirit and broadest scope of the appended claims. Additionally, although the features and elements of the present disclosure are described in particular combinations, each feature or element can be used alone without the other features and elements or in any other combination with the other features and elements. The scope of the present disclosure is to be determined with reference to the appended claims.

[0379] The above-described embodiments of the present disclosure can be implemented in any of numerous ways. For example, the embodiments can be implemented using hardware, software or a combination thereof. When implemented in software, the software code can be executed on any suitable processor or collection of processors, whether provided in a single computer or distributed among multiple computers. Such processors can be implemented as integrated circuits, with one or more processors in an integrated circuit component, including commercially available integrated circuit components such as CPUs, GPUs, microcontrollers, microprocessors, or co-processors. In some embodiments, a processor can be implemented in custom circuitry, in an ASIC, or by a combination of custom circuitry and ASIC. As another alternative, the processor can be a portion of a larger circuit or semiconductor device, whether provided in a single package or distributed among multiple packages. As a specific example, some commercially available microprocessors have multiple cores such that one or a subset of those cores can constitute a processor. However, a processor can be implemented using circuitry in any suitable format.

[0380] Further, it should be appreciated that a computer can be embodied in any of a number of forms, such as a rack-mounted computer, a desktop computer, a laptop computer, or a tablet computer. Additionally, a computer can be embedded in a device not generally regarded as a computer but with suitable processing capabilities, including a Personal Digital Assistant (PDA), a smart phone or any other suitable portable or fixed electronic device.

[0381] Also, a computer can have one or more input and output devices. These devices can be used, among other things, to present a user interface. Examples of output devices that can be used to provide a user interface include printers or display screens for visual presentation of output and speakers or other sound generating devices for audible presentation of output. Examples of input devices that can be used for a user interface include keyboards, and pointing devices, such as mice, touch pads, and digitizing tablets. As another example, a computer can receive input information through speech recognition or in other audible formats. In the illustrated embodiment, the input / output devices are shown as physically separate from the computing device. However, in some embodiments, the input and / or output devices can be physically integrated into the same unit as the processor or other elements of the computing device. For example, a keyboard can be implemented as a

[0382] Such computers can be interconnected by one or more networks in any suitable form, including as a local area network or a wide area network, such as an enterprise network or the Internet. Such networks can be based on any suitable technology and can operate according to any suitable protocol and can include wireless networks, wired networks or fiber optic networks.

[0383] Also, the various methods or processes outlined herein can be encoded as software that is executable on one or more processors that employ any one of a variety of operating systems or platforms. Additionally, such software can be written using any of a number of suitable programming languages and / or programming or scripting tools, and also can be compiled as executable machine language code or intermediate code that is executed on a framework or virtual machine.

[0384] In this regard, the present disclosure can be embodied as a computer readable storage medium (or multiple computer readable media) (e.g., a computer memory, one or more floppy discs, compact discs (CD), optical discs, digital video disks (DVD), magnetic tapes, flash memories, circuit configurations in Field Programmable Gate Arrays or other semiconductor devices, or other tangible computer storage medium) encoded with one or more programs that, when executed on one or more computers or other processors, perform methods that implement the various embodiments of the present disclosure discussed above. As is apparent from the foregoing examples, computer readable storage media can retain information for a sufficient amount of time to provide computer-executable instructions. Such computer readable storage media or media can be removable from the computer or other processor reading the instructions, so that the one or more programs stored thereon can be loaded into the one or more different computers or other processors to implement various aspects of the present disclosure as described above. As used herein, the term "computer- readable storage media" encompasses only computer-readable media that are considered tangible, in that the computer-readable media is not a signal per se. In some embodiments, the present disclosure can be embodied as computer readable media other than a computer readable storage media, such as a propagating signal.

[0385] The terms "program" or "software" are used herein in a generic sense to refer to any type of computer code or set of computer-executable instructions that can be employed to program a computer or other processor to implement various aspects of the present disclosure as discussed above. Additionally, it should be appreciated that according to one aspect of this embodiment, one or more computer programs that when executed perform methods of the present disclosure need not reside on a single computer or processor, but can be distributed in a modular fashion amongst a number of different computers or processors to implement various aspects of the present disclosure.

[0386] Computer-executable instructions can be in many forms, such as program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Typically the functionality of the program modules can be combined or distributed as desired in various embodiments.

[0387] Also, data structures can be stored in computer readable media in any suitable form. For simplicity of illustration, data structures can be shown to have fields that are related through location in the data structure. Such relationships can likewise be achieved by assigning storage for the fields with locations in a computer readable medium that are related through the computer program instruction and executed to implement an interface for the data in a structure. However, any suitable mechanisms can be used to establish relationships among the information stored in the data structures, including through the use of pointers, tags or other mechanisms that establish relationships among data elements.

[0388] The various aspects of the present disclosure can be used alone, in combination, or in various arrangements not specifically discussed in the foregoing examples, and therefore are not limited to the details and arrangements set forth in the foregoing description or illustrated in the drawings. For example, aspects described in one embodiment can be combined with aspects described in other embodiments in any manner.

[0389] Furthermore, the present disclosure can be embodied as a method, examples of which have been provided. The acts performed as part of the method can be ordered in any suitable way. Accordingly, embodiments can be constructed in which acts are performed in an order different than illustrated, which can include performing some acts simultaneously even though shown as being performed sequentially in illustrative embodiments.

[0390] The use of ordinal terms such as "first," "second," "third," etc. in the claims to modify a claim element does not imply a sequence or order to the claim elements but is used to distinguish one claim element from another.

[0391] In addition, the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of "including," "comprising," "having," "containing," "involving," and variations thereof herein, is meant to encompass the items listed thereafter and equivalents thereof as well as additional items.

Claims

1. A portable electronic system comprising: a sensor configured to capture three-dimensional (3D) information about objects in a physical world; an active memory; a local memory; a transceiver configured to communicate with a remote memory over a computer network; and a processor communicatively coupled to the sensor, the active memory, the local memory, and the transceiver, and configured to execute computer-executable instructions to provide a 3D representation of a portion of the physical world based at least in part on the 3D information about the objects in the physical world, wherein: the 3D representation of the portion of the physical world comprises a plurality of tiles, each tile having a value representing objects in a region of the portion of the physical world at a certain point in time; and the computer-executable instructions comprise instructions to: hold the plurality of tiles in at least one of the active memory, the local memory, and the remote memory when the portable electronic system changes to an inactive operating state at a first time; select tiles from the held tiles that are located around the portable electronic system at a second time after the first time when transitioning to an active state; and render virtual content to a user of the portable electronic system based at least in part on the selected tiles. the inactive operating state is a shutdown or hibernation.

2. The portable electronic system of claim 1, wherein, each tile comprises a volumetric representation of the region of the portion of the physical world at the certain point in time.

3. The portable electronic system of claim 1 or claim 2, wherein, holding the plurality of tiles in at least one of the active memory, the local memory, and the remote memory comprises:

4. The portable electronic system of claim 1 or claim 2, wherein, when the inactive operating state is hibernation, holding the plurality of tiles in respective memories in which the plurality of tiles currently exist such that the plurality of tiles are accessible by the portable electronic system when the portable electronic system wakes up. holding the plurality of tiles in at least one of the active memory, the local memory, and the remote memory comprises:

5. The portable electronic system of claim 1 or claim 2, wherein, when the inactive operating state is a shutdown, holding at least a portion of the tiles in the local memory to the remote memory and holding tiles in the active memory to the local memory and / or remote memory. the computer-executable instructions comprise instructions to:

6. The portable electronic system of claim 1 or claim 2, wherein, compute one or more new tiles based at least in part on incoming 3D information about objects in the physical world, and select tiles from the held tiles that at least partially overlap with the one or more new tiles to enable retrieval of the held tiles around the portable electronic system. selecting tiles from the held tiles that at least partially overlap with the one or more new tiles comprises comparing the one or more new tiles to the held tiles.

7. The portable electronic system of claim 6, wherein, comparing the one or more new tiles to the held tiles comprises:

8. The portable electronic system of claim 7, wherein, comparing the one or more new tiles to tiles held in the active memory. comparing the one or more new tiles to the held tiles comprises:

9. The portable electronic system of claim 8, wherein, comparing the one or more new tiles to tiles held in the local memory. when the active memory does not have a block that at least partially overlaps the one or more new blocks, comparing the one or more new blocks to blocks maintained in the local memory.

10. The portable electronic system of claim 9, wherein, Comparing the one or more new blocks to the maintained blocks includes: when the local memory does not have a block that at least partially overlaps the one or more new blocks, comparing the one or more new blocks to blocks maintained in the remote memory.

11. The portable electronic system of claim 6, wherein, The computer-executable instructions include instructions for performing operations comprising: updating the plurality of blocks with the one or more new blocks.

12. A method of operating a portable electronic system in an augmented reality system, the method comprising: with at least one processor: obtaining a plurality of blocks comprising three-dimensional (3D) information about surfaces in a physical world; storing at least one of the plurality of blocks in an active memory; using blocks from the active memory for rendering virtual content to a user of the portable electronic system; upon transitioning to an inactive operating state at a first time, maintaining the blocks in the active memory; and upon transitioning to an active operating state at a second time after the first time, selecting from the maintained blocks blocks that are located around the portable electronic system at the second time; and rendering virtual content to the user of the portable electronic system based at least in part on the selected blocks.

13. The method of claim 12, wherein, Each block comprises a volumetric representation of a portion of the physical world at a point in time.

14. The method of claim 12 or claim 13, wherein, The plurality of blocks correspond to a same volumetric space.

15. The method of claim 12 or claim 13, wherein, The blocks are grid blocks.

16. The method of claim 12 or claim 13, comprising: paging blocks between the active memory and a local memory based on a current field of view or a projected field of view of a user of the portable electronic system.

17. The method of claim 12 or claim 13, further comprising: upon transitioning to an inactive operating state, maintaining the blocks in the local memory.

18. The method of claim 12 or claim 13, wherein, using blocks from the active memory for rendering virtual content to the user includes: using the blocks for visual occlusion processing, and computing physics-based interactions and / or environmental reasoning.

19. A non-transitory computer-readable medium having instructions stored thereon that, when executed on a processor, perform an action comprising: obtaining a plurality of blocks comprising three-dimensional (3D) information about surfaces in a physical world; storing at least one of the plurality of blocks in an active memory; using blocks from the active memory for rendering virtual content to a user of a portable electronic system; upon transitioning to an inactive operating state at a first time, maintaining the blocks in the active memory; and upon transitioning to an active operating state at a second time after the first time, selecting from the maintained blocks blocks that are located around the portable electronic system at the second time; and rendering virtual content to the user of the portable electronic system based at least in part on the selected blocks.

20. The non-transitory computer-readable medium of claim 19, wherein, Each block comprises a volumetric representation of a portion of the physical world at a point in time.

21. The non-transitory computer-readable medium of claim 19 or claim 20, wherein, The plurality of tiles correspond to a certain volume of space.

22. The non-transitory computer-readable medium of claim 19 or claim 20, wherein, The tiles are grid tiles.

23. The non-transitory computer-readable medium of claim 19 or claim 20, wherein, The actions include: paging tiles between the active memory and a local memory based on a current field of view or a projected field of view of a user of the portable electronic system.

24. The non-transitory computer-readable medium of claim 19 or claim 20, wherein, The actions include:

25. The non-transitory computer-readable medium of claim 19 or claim 20, wherein, maintaining the tiles in the local memory upon transitioning to an inactive operating state. using tiles from the active memory for rendering virtual content to the user includes: using the tiles for visual occlusion processing, and computing physics-based interactions and / or environmental reasoning.

Images