Cross-reality system

Abstract

An augmented reality viewing system is described. A local coordinate framework of the local content will be transformed into a world coordinate framework. The head coordinate frame is further transformed, and the camera coordinate frame including all pupil positions of the eye is further transformed. One or more users may interact with the viewing system in a separate session. If the canonical map is available, the earlier map is downloaded onto the user's viewing device. The viewing device then generates another map and locates the subsequent map to the canonical map.

Description

[0001] This application is a divisional application of the Chinese patent application with application number 201980066842.8. The application date of the original application is August 12, 2019, the priority date is August 13, 2018, the date of entering the Chinese national phase is April 9, 2021, and the name of the invention is “Cross-reality System”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to and the benefit of U.S. Provisional Patent Application Serial No. 62 / 718,357, filed on August 13, 2018, and entitled “VIEWING DEVICE OR VIEWING DEVICES HAVING ONE ORMORE COORDINATE FRAME TRANSFORMERS,” which is hereby incorporated by reference in its entirety. This application also claims priority to and the benefit of U.S. Provisional Patent Application Serial No. 62 / 742,237, filed on October 5, 2018, and entitled “COORDINATE FRAME PROCESSING AUGMENTED REALITY,” which is hereby incorporated by reference in its entirety. This patent application also claims priority to and the benefit of U.S. Provisional Patent Application Serial No. 62 / 812,935, filed on March 1, 2019, and entitled “MERGING APLURALITY OF INDIVIDUALLY MAPPED ENVIRONMENTS,” which is hereby incorporated by reference in its entirety. U.S. Provisional Patent Application Serial No. 62 / 815,955, filed on March 8, 2019, and entitled “VIEWING DEVICE OR VIEWING DEVICES HAVING ONE OR MORE COORDINATE FRAME TRANSFORMERS,” which is hereby incorporated by reference in its entirety. This patent application also claims priority and the benefit of U.S. Provisional Patent Application Serial No. 62 / 868,786, filed on June 28, 2019, and entitled “RANKING AND MERGING A PLURALITY OF ENVIRONMENT MAPS”, which is incorporated herein by reference in its entirety. This patent application also claims priority and the benefit of U.S. Provisional Patent Application Serial No. 62 / 870,954, filed on July 5, 2019, and entitled “RANKING AND MERGING A PLURALITY OF ENVIRONMENT MAPS”, which is incorporated herein by reference in its entirety. This patent application also claims priority and the benefit of U.S. Provisional Patent Application Serial No. 62 / 884,109, filed on August 8, 2019, and entitled “A VIEWING SYSTEM”, which is incorporated herein by reference in its entirety. Technical Field

[0004] The present application relates generally to cross-reality systems. Background Art

[0005] A computer can control a human user interface to create an X Reality (XR or cross-reality) environment in which some or all of the XR environments perceived by the user are generated by the computer. These XR environments can be virtual reality (VR), augmented reality (AR), and mixed reality (MR) environments, some or all of which can be generated by a computer in part using data describing the environment. For example, the data can describe a virtual object that can be rendered in a way that the user feels or perceives it as part of the physical world, and can interact with the virtual object. Since the data is rendered and presented by a user interface device (such as, for example, a head-mounted display device), the user can experience these virtual objects. The data can be displayed to the user, or the audio played to the user can be controlled, or a tactile (or tactile) interface can be controlled, so that the user can experience the touch sensation that the user feels or perceives as feeling the virtual object.

[0006] XR systems can be used for many applications across scientific visualization, medical training, engineering design and prototyping, telemanipulation and telepresence, and personal entertainment. Compared to VR, AR and MR include one or more virtual objects related to real objects in the physical world. The experience of virtual objects interacting with real objects significantly enhances the user's enjoyment of using XR systems, and also opens the door to a variety of applications that present realistic and easy-to-understand information about how to change the physical world.

[0007] An XR system can represent the physical world surrounding a user of the system as a "mesh." The mesh can be represented by a plurality of interconnected triangles. Each triangle has edges that connect points on the surface of an object in the physical world, so that each triangle represents a portion of that surface. Information about the surface portion, such as color, texture, or other characteristics, can be stored in association with the triangle. In operation, the XR system can process image information to detect points and surfaces in order to create or update a mesh. Summary of the invention

[0008] Aspects of the present application relate to methods and apparatus for providing an X-reality (cross-reality or XR) scene. The techniques described herein may be used together, separately, or in any suitable combination.

[0009] Some embodiments relate to an XR system including a first XR device, the first XR device including a first processor, a first computer-readable medium connected to the first processor, a first origin coordinate frame stored on the first computer-readable medium, a first destination coordinate frame stored on the computer-readable medium, a first data channel for receiving data representing local content, a first coordinate frame transformer executable by the first processor to transform the positioning of the local content from the first origin coordinate frame to a first destination coordinate frame, and a first display system adapted to display the local content to a first user after transforming the positioning of the local content from the first origin coordinate frame to the first destination coordinate frame.

[0010] Some embodiments relate to a viewing method, which includes: storing a first origin coordinate frame; storing a first destination coordinate frame; receiving data representing local content; transforming the positioning of the local content from the first origin coordinate frame to the first destination coordinate frame; and after transforming the positioning of the local content from the first origin coordinate frame to the first destination coordinate frame, displaying the local content to a first user.

[0011] Some embodiments relate to an XR system comprising: a map storage routine that stores a first map, the first map being a canonical map having multiple anchor points, each anchor point of the first map having a set of coordinates; a real object detection device that is positioned to detect the location of a real object; an anchor point recognition system that is connected to the real object detection device to detect anchor points of a second map based on the location of the real object, each anchor point of the second map having a set of coordinates; and a positioning module that is connected to the canonical map and the second map and is executable to position the second map to the canonical map by matching a first anchor point of the second map with a first anchor point of the canonical map and matching a second anchor point of the second map with a second anchor point of the canonical map.

[0012] Some embodiments relate to a viewing method, which includes: storing a first map, which is a canonical map having multiple anchor points, each anchor point of the canonical map has a set of coordinates; detecting the location of a real object; based on the location of the real object, detecting anchor points of a second map, each anchor point of the second map has a set of coordinates; and positioning the second map to the canonical map by matching the first anchor point of the second map with the first anchor point of the first map, and matching the second anchor point of the second map with the second anchor point of the canonical map.

[0013] Some embodiments relate to an XR system, comprising: a server, which may have: a processor; a computer-readable medium connected to the processor; a plurality of canonical maps on the computer-readable medium; a corresponding canonical map identifier on the computer-readable medium associated with each corresponding canonical map, the canonical map identifiers being different from each other to uniquely identify the canonical map; a location detector, which is located on the computer-readable medium and is executable by the processor to receive and store the location identifier from the XR device; a first filter, which is located on the computer-readable medium and is executable by the processor to compare the location identifier with the canonical map identifier to determine one or more canonical maps that form a first filtered selection; and a map transmitter, which is located on the computer-readable medium and is executable by the processor to send one or more canonical maps to the XR device based on the first filtered selection.

[0014] Some embodiments relate to a viewing method comprising: storing multiple canonical maps on a computer-readable medium, each canonical map having a corresponding canonical map identifier associated with the corresponding canonical map, the canonical map identifiers being different from each other to uniquely identify the canonical map; receiving and storing location identifiers from an XR device using a processor connected to the computer-readable medium; comparing the location identifiers with the canonical map identifiers using the processor to determine one or more canonical maps that form a first filter selection; and sending the multiple canonical maps to the XR device based on the first filter selection using the processor.

[0015] Some embodiments relate to an XR system comprising: a processor; a computer-readable medium connected to the processor; a multi-layer perception unit on the computer-readable medium and executable by the processor, which receives multiple features of an image and converts each feature into a corresponding digital string; and a maximum pooling unit on the computer-readable medium and executable by the processor, which combines the maximum value of each digital string into a global feature string representing the image.

[0016] Some embodiments relate to a viewing method, which includes: receiving, with a processor, multiple features of an image; converting, with the processor, each feature into a corresponding digital string; and combining, with the processor, a maximum value of each digital string into a global feature string representing the image.

[0017] Some embodiments relate to a method of operating a computing system for identifying one or more environmental maps stored in a database for merging with a tracking map calculated based on sensor data collected by a device worn by a user, wherein the device receives a signal of an access point of a computer network while calculating the tracking map, the method comprising: determining at least one area attribute of the tracking map based on communication characteristics with the access point; determining a geographic location of the tracking map based on the at least one area attribute; identifying a set of environmental maps stored in the database corresponding to the determined geographic location; filtering the set of environmental maps based on similarity of one or more identifiers of network access points associated with the tracking map and an environmental map in a set of environmental maps; filtering the set of environmental maps based on similarity of metrics representing the content of the tracking map and an environmental map in a set of environmental maps; and filtering the set of environmental maps based on a degree of match between a portion of the tracking map and a portion of an environmental map in a set of environmental maps.

[0018] Some embodiments relate to a cloud computing environment for an augmented reality system, the augmented reality system being configured to communicate with a plurality of user devices including sensors, the plurality of user devices including: a user database storing area identifications indicating areas in which the plurality of user devices are used, the area identifications including parameters of wireless networks detected by the user devices in use; a map database storing a plurality of environment maps constructed from data provided by the plurality of user devices and associated metadata, the associated metadata including area identifications derived from the area identifications of the plurality of user devices providing data from which the maps are constructed, the area identifications including parameters of wireless networks detected by the user devices providing data from which the maps are constructed; a non-transitory computer storage medium storing computer executable instructions, the instructions being stored in a computer program product; When executed by at least one processor in a cloud computing environment: receiving a message including parameters of a wireless network detected by the user device from a plurality of user devices, calculating an area identifier of the user device, and updating a user database based on the received parameters and / or the calculated area identifier; and receiving a request for an environmental map from a plurality of user devices, determining an area identifier associated with the user device requesting the environmental map, identifying a set of environmental maps from a map database based at least in part on the area identifier, filtering the set of environmental maps, and sending the filtered set of environmental maps to the user device, wherein filtering the set of environmental maps is based on similarity between parameters of the wireless network detected by the user device from which the request for the environmental map originates and parameters of the wireless network in the map database for environmental maps in the set of environmental maps.

[0019] Some embodiments relate to an XR system, which includes: a real object detection device that detects multiple surfaces of a real-world object; an anchor recognition system that is connected to the real object detection device to generate a map based on the real-world object; a persistent coordinate frame (PCF) generation system that generates a first PCF based on the map and associates the first PCF with the map; corresponding first storage media and second storage media on a first XR device and a second XR device; and at least a first processor and a second processor of the first XR device and the second XR device, which respectively store the first PCF in the first storage medium of the first XR device and the second storage medium of the second XR device.

[0020] Some embodiments relate to a viewing method, which includes: using at least one detection device to detect multiple surfaces of a real-world object; using at least one processor to generate a map based on the real-world object; using at least one processor to generate a first PCF based on the map; using at least one processor to associate the first PCF with the map; and using at least the first processor and the second processor of a first XR device and a second XR device to store the first PCF in a first storage medium of the first XR device and a second storage medium of the second XR device, respectively.

[0021] Some embodiments relate to an XR system, which includes: a first XR device, which may include: a first XR device processor; a first XR device storage device, which is connected to the first XR device processor; a set of instructions on the first XR device processor, which includes: a download system, which is executable by the first XR device processor to download a persistent gesture from a server; a PCF fetcher, which is executable by the first XR device processor to obtain a PCF from the first storage device of the first XR device based on the persistent gesture; and a coordinate frame calculator, which is executable by the first XR device processor to calculate a coordinate frame based on the PCF fetched from the first storage device.

[0022] Some embodiments relate to a viewing method, which includes: using a first XR device processor of a first XR device to download a persistent posture from a server; using the first XR device processor to obtain a PCF from a first storage device of the first XR device based on the persistent posture; and using the first XR device processor to calculate a coordinate frame based on the PCF obtained from the first storage device.

[0023] Some embodiments relate to a viewing method, the viewing method comprising: a server, which may include: a server processor; a server storage device connected to the server processor; a map storage routine, which can be executed by the server processor of the server to store a first PCF in association with a map on the server storage device of the server; and a map sender, which uses the server processor and can be executed by the server processor to send the map and the first PCF to a first XR device. The present invention also provides a viewing method, the viewing method comprising: using the server processor of the server to store the first PCF in association with the map on the server storage device of the server; and using the server processor of the server to send the map and the first PCF to the first XR device.

[0024] Some embodiments relate to a viewing method, which includes: tracking head posture by a processor of an XR device by capturing an environmental surface using a capture device on a head-mounted frame fixed to a user's head and determining an orientation of the head-mounted frame relative to the surface; determining by the processor whether the head posture is lost due to an inability to determine the orientation of the head-mounted frame relative to the surface; and if the head posture is lost, entering a posture recovery mode by the processor to establish the head posture by determining the orientation of the head-mounted frame relative to the surface. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component shown in various figures is represented by a like numeral. For clarity, not every component may be labeled in every figure. In the drawings:

[0026] Figure 1 is a schematic diagram illustrating a cross-reality (XR) system according to some embodiments;

[0027] Figure 2 According to some embodiments Figure 1 A block diagram of a first XR device of an XR system;

[0028] Figure 3 is a schematic diagram illustrating transforming an origin coordinate frame into a destination coordinate frame, for example, to correctly render local XR content, according to some embodiments;

[0029] Figure 4 is a top plan view illustrating a pupil-based coordinate framework according to some embodiments;

[0030] Figure 5 is a top plan view showing a camera coordinate frame including all pupil locations according to some embodiments;

[0031] Figure 6 is according to some embodiments including an additional second XR device Figure 1 Block diagram of the XR system;

[0032] Figure 7 According to some embodiments Figure 2 A schematic diagram of a display system;

[0033] Figure 8 According to some embodiments, Figure 2 The first XR device to generate the first 3D local tracking map (ground Figure 1 ) two-dimensional representation;

[0034] Fig. 9 is an illustration of a flow from a first XR device to a Figure 6 Server upload location Figure 1 Block diagram of

[0035] Fig.10 is a diagram showing some embodiments of the present invention. Figure 6 A schematic diagram of an XR system showing that a second user has initiated a second session using a second XR device of the XR system after the first user has terminated the first session;

[0036] Fig.11A is a diagram showing a method for Fig.10 A block diagram of a new session for a second XR device;

[0037] Fig. 11B is a diagram showing a method for Fig.10 A block diagram of creation of a tracking map for a second XR device;

[0038] Fig. 11C is a diagram showing a method for transmitting data from a server to a server according to some embodiments. Fig.10 A block diagram of a second XR device downloading a canonical map;

[0039] Fig.11D is a block diagram illustrating creation of a persistent coordinate frame (PCF) and XR content attached to the PCF according to some embodiments;

[0040] Fig.12 is to show that according to some embodiments, Fig.10 A second tracking map (map) generated by a second XR device Figure 2 ) A schematic diagram of a positioning attempt to locate the canonical map;

[0041] Fig.13 is a diagram showing the Fig.12 Second tracking map (map Figure 2 ) is a schematic diagram of a positioning attempt to locate to a canonical map, the second tracking map can be further developed and has Figure 2 The XR content associated with the anchor point;

[0042] Figure 14A-14B is a diagram showing the Fig.13 land Figure 2 Schematic diagram of successful positioning on the normative map;

[0043] Fig.15 is a diagram showing that according to some embodiments, Fig.14A The specification map contains one or more anchor points to Fig.13 land Figure 2 Schematic diagram of the normative map generated by

[0044] Fig.16 is a diagram showing some embodiments of the present invention. Fig.15 The canonical map and the map on the second XR device Figure 2 A further extended schematic diagram of

[0045] Fig.17 FIG. 2 is a diagram illustrating uploading a map from a second XR device to a server according to some embodiments. Figure 2 Block diagram of

[0046] Fig.18 is a diagram showing how to convert the ground Figure 2 box plot merged with normative map;

[0047] Fig.19 is a block diagram illustrating transmission of a new canonical map from a server to a first XR device and a second XR device according to some embodiments;

[0048] Fig. 20 is a diagram showing a method according to some embodiments Figure 2 Two-dimensional representation and reference ground Figure 2 A block diagram of a head coordinate frame of a second XR device;

[0049] Fig.21 is a block diagram illustrating in two dimensions adjustments of a head coordinate frame that may occur in six degrees of freedom according to some embodiments;

[0050] Fig. 22 is a block diagram illustrating a canonical map on a second XR device according to some embodiments, wherein sound is relative to ground Figure 2 The anchor point is located;

[0051] Fig.23 and Fig.24 are perspective and block diagrams illustrating use of an XR system when a first user has terminated a first session and the first user has initiated a second session using the XR system in accordance with some embodiments;

[0052] Fig.25 and Fig.26are perspective views and block diagrams illustrating use of an XR system when three users are using the XR system simultaneously in the same session, according to some embodiments;

[0053] Fig. 27 is a block diagram illustrating a two-dimensional view representing latitude and longitude of a canonical map stored on a server according to some embodiments;

[0054] Fig.28 is a schematic diagram illustrating a series of filters for selecting a canonical map for transmission by a server to an XR device according to some embodiments;

[0055] Figures 29A-29C is a block diagram illustrating a system for generating a global feature string for an image and / or map according to some embodiments;

[0056] Fig.29D is a block diagram illustrating a method of training a neural network according to some embodiments;

[0057] Fig.30 is a schematic diagram illustrating an example of a simplified augmented reality (AR) scene according to some embodiments;

[0058] Fig.31 is a sketch of an exemplary simplified AR scene, illustrating exemplary world reconstruction use cases, including visual occlusion, physics-based interaction, and environmental reasoning, in accordance with some embodiments.

[0059] Fig.32 is a schematic diagram illustrating data flow in an AR system according to some embodiments, the AR system being configured to provide an experience of AR content interacting with the physical world;

[0060] Fig.33 is a schematic diagram illustrating an example of an AR display system according to some embodiments;

[0061] Fig.34A is a schematic diagram illustrating an AR display system worn by a user that renders AR content as the user moves through a physical world environment in accordance with some embodiments;

[0062] Fig.34B is a schematic diagram illustrating a viewing optics assembly and accompanying components according to some embodiments;

[0063] Fig.35A is a schematic diagram illustrating an AR system using a world reconstruction system according to some embodiments;

[0064] Fig.35B is a schematic diagram illustrating a navigable world according to some embodiments;

[0065] Fig.36 is an exemplary environment map according to some embodiments;

[0066] Fig.37 is a schematic diagram illustrating an AR system configured to rank and merge multiple environment maps according to some embodiments;

[0067] Fig.38 is a flow chart illustrating a method of selecting a plurality of ranked environment maps according to some embodiments;

[0068] Fig.39 is a diagram showing some embodiments of the present invention. Fig.37 A schematic diagram of an exemplary map ranking portion of an AR system;

[0069] Fig.40A is a schematic diagram showing an example of area attributes of a tracking map (TM) and an environment map in a database according to some embodiments;

[0070] Fig.40B is a diagram showing the determination of a Fig.38 A schematic diagram of an example of a geo-location filtered Tracking Map(TM);

[0071] Fig.41 is a diagram showing some embodiments of the present invention. Fig.38 A schematic diagram of an example of geographic location filtering;

[0072] Fig.42 is a diagram showing some embodiments of the present invention. Fig.38 A schematic diagram of an example of Wi-Fi BSSID filtering;

[0073] Fig.43 is a diagram showing some embodiments of the present invention. Fig.38 A schematic diagram of an example of positioning;

[0074] Fig.44 is a schematic diagram illustrating a room and key frames established for various areas in the room according to some embodiments;

[0075] Fig.45 is a schematic diagram illustrating the establishment of a keyframe-based persistent gesture according to some embodiments;

[0076] Fig.46 is a schematic diagram illustrating the establishment of a persistent coordinate frame (PCF) based on persistent gestures according to some embodiments;

[0077] Fig.47 is a flow chart illustrating a method of establishing and using a PCF according to some embodiments;

[0078] FIG. 48A to FIG. 48I It shows how to create and use Fig.47 A schematic diagram of an example of a PCF;

[0079] Fig.49 is a flow chart illustrating a method of recovering and resetting head posture according to some embodiments; and

[0080] Fig.50 is a block diagram of a machine in the form of a computer that may find application in the system of the present invention according to some embodiments.

[0081] Figure 51-53 is a block diagram illustrating an example of determining a map to use based at least in part on deep keyframe (DKF) analysis in accordance with some embodiments. DETAILED DESCRIPTION

[0082] Described herein are methods and apparatus for providing an X-reality (XR or cross-reality) scene. In order to provide a realistic XR experience to multiple users, the XR system must understand the users' physical environment in order to correctly associate the positions of virtual objects with real objects. The XR system can build an environment map of the scene, which can be created from images and depth information collected by sensors that are part of an XR device worn by a user of the XR system.

[0083] The inventors have recognized and appreciated techniques for operating an XR system to provide XR scenarios for a more immersive user experience by enabling any of multiple users of the XR system to create, store, and retrieve a persistent environment map. Such a persistent environment map may enable one or more functions that enhance the XR experience. For example, a wearable device worn by a user, after being turned on, may retrieve a previously created and stored appropriate environment map from a persistent storage such as cloud storage. The previously stored map may be data about the environment collected using sensors on the user's wearable device at one or more previous times of the user's presence. Retrieving a persistent map may enable the use of a wearable device without the need to scan the physical world using sensors on the wearable device. Alternatively or additionally, a user may similarly retrieve an appropriate environment map when entering a new area of ​​the physical world.

[0084] In a multi-user XR system, a persistent environment map accessed by one user may have been created and stored by another user and / or may have been constructed by aggregating data about the physical world collected by sensors on wearable devices worn by multiple users who have previously been present in at least a portion of the physical world represented by the persistent environment map.

[0085] In addition, sharing data about the physical world between multiple users can enable a shared user experience of virtual content. For example, two XR devices accessing the same persistent environment map can both be positioned relative to the persistent environment map. Once positioned, the user device can render virtual content with a location specified by reference to the persistent environment map by translating the location into a frame or reference maintained by the user device. The user device can use this local reference frame to control the display of the user device to render virtual content in the specified location.

[0086] To support these and other possible functions, the XR system may include components that develop, maintain, and use one or more persistent maps of the environment based on data collected about the physical world from sensors on the user device. These components may be distributed across the XR system, for example, through some operations on the head-mounted portion of the user device. Other components may operate on a computer associated with a user coupled to the head-mounted portion via a local area network or a personal area network. Still others may operate at a remote location, such as at one or more servers accessible via a wide area network.

[0087] For example, these components may include components that can identify information about the physical world collected by one or more user devices that has a sufficient amount of information to be stored as a persistent environment map. Examples of such components, described in more detail below, are Figure 6 The PCF integrated unit 1300 in.

[0088] As another example, these components can include components that can help determine appropriate persistent environment maps that can be obtained and used by the user device. An example of such a component, described in more detail below, is a map ranking component. For example, such a component can receive input from a user device and identify one or more persistent environment maps that may represent areas of the physical world in which the device operates. For example, the map ranking component can help select a persistent environment map to be used by the local device when rendering virtual content, collecting data about the environment, or performing other actions. Alternatively or additionally, the map ranking component can help identify persistent environment maps that are updated as one or more user devices collect additional information about the physical world.

[0089] Other components may be application programming interfaces (APIs). A platform that builds and maintains a persistent map of an environment may provide one or more APIs to enable separately created applications to obtain information about the physical world.

[0090] Other components may be transformations that convert information captured or described about one reference frame to another reference frame. For example, a sensor may be attached to a head-mounted display so that the data read from the sensor indicates the location of an object in the physical world relative to the wearer's head posture. One or more transformations may be applied to relate this position information to a coordinate frame associated with a persistent environment map. Similarly, data indicating where to render virtual objects when expressed in the coordinate frame of a persistent environment map may be transformed one or more times to be placed in the reference frame of a display on the user's head. As described in more detail below, there may be multiple such transformations. These transformations may be divided across components of the XR system so that they can be updated efficiently or applied in a distributed system.

[0091] In some embodiments, map information may be represented in a manner that can be easily shared between users and between distributed components including applications. For example, information about the physical world may be represented as a coordinate frame with one or more anchor points that represent features identified in the physical world. Such maps may be sparse, providing less than all available information about the physical world so that they can be efficiently processed and communicated.

[0092] In some embodiments, a persistent environment map may be constructed based on information collected by multiple user devices. The XR device may construct a separate tracking map (TM) using information collected by sensors of the XR devices worn by users of the system at various locations and times. Each TM may include points, each of which may be associated with a real object in the environment. In addition to potentially providing input to create and maintain a persistent environment map, the TM may also be used to track the movement of users in the scene, thereby enabling the XR system to estimate the head pose of the corresponding user based on the TM.

[0093] This interdependency between the creation of an environment map and the estimation of head pose poses a significant challenge. A lot of processing may be required to create an environment map and estimate head pose at the same time. As objects move in the scene (such as moving a cup on a table) and as the user moves in the scene, processing must be completed quickly because latency makes the XR experience less immersive for the user. On the other hand, XR devices may offer limited computing resources because the weight of XR devices should be light to be worn comfortably by the user. Therefore, more sensors cannot be used to compensate for the lack of computing resources. In addition, more sensors and more computing resources generate heat, which can cause deformation of the XR device.

[0094] The inventors have recognized and appreciated XR scenarios for operating an XR system to provide techniques for a more immersive user experience, such as estimating head pose at a frequency of 1kHz, with low computational resource usage, such as four video graphics array (VGA) cameras operating at 30Hz, one inertial measurement unit (IMU) operating at 1kHz, the computational power of a single Advanced RISC Machine (ARM) core, less than 1GB of memory, and less than 100Mbp of network bandwidth. These techniques involve reducing the processing required to generate and maintain an environment map and estimate head pose, and involve providing and consuming data with low computational overhead.

[0095] These techniques may include hybrid tracking, such that the XR system may utilize both: (1) patch-based tracking of distinguishable points between successive images of an environment (frame-to-frame tracking), and (2) descriptor-based matching of points of interest in a current image to a map of known real-world locations of corresponding points of interest (map-to-frame tracking). In frame-to-frame tracking, the XR system may track specific points of interest (vertices), such as corners, between images captured in a real-world environment. For example, a display system may identify the location of a point of visual interest in a current image that is included in (e.g., located in) a previous image. This identification may be accomplished using, for example, a photometric error minimization process. In map-to-frame tracking, the XR system may access map information indicating the real-world location (e.g., three-dimensional coordinates) of the point of interest and match the point of interest included in the current image to the point of interest indicated in the map information. Information about the point of interest is stored as a descriptor in a map database. The XR system may then compute its pose based on the matched visual features. U.S. patent application Ser. No. 16 / 221,065 describes hybrid tracking and is incorporated herein by reference in its entirety.

[0096] These techniques may include reducing the amount of data processed when constructing a map, such as by constructing a sparse map using a collection of map construction points and keyframes and / or dividing the map into blocks to enable block-by-block updates. The map construction points may be associated with a three-dimensional (3D) world reconstruction of real objects in the environment. The keyframes may include information selected from camera captured data. U.S. Patent Application Serial No. 16 / 520,582 describes determining and / or evaluating a localization map, and is incorporated herein by reference in its entirety.

[0097] These techniques may include creating dynamic maps based on one or more coordinate systems in real space across one or more sessions, generating a persistent coordinate frame (PCF) on a sparse map that can be exposed to an XR application, for example via an application programming interface (API), ranking and map building of multiple maps created by one or more XR devices, using deep keyframe analysis in some embodiments, and recovering and resetting head pose.

[0098] The techniques described herein can be used together or alone with many types of devices and for many types of scenarios, including wearable or portable devices with limited computing resources that provide augmented reality scenarios.

[0099] Coordinate frame

[0100] Methods and apparatus are described herein for providing virtual content using an XR system that is independent of the position of the eyes viewing the virtual content. Traditionally, virtual content is re-rendered upon any movement of the display system. For example, if a user wearing a display system views a virtual representation of a three-dimensional (3D) object on a display and walks around an area where the 3D object appears, the 3D object should be re-rendered for each viewpoint so that the user has the perception that he or she is walking around the object occupying real space. However, re-rendering consumes a large amount of the system's computing resources and causes artifacts due to latency.

[0101] The inventors have recognized and appreciated that head pose (e.g., position and orientation of a user wearing an XR system) can be used to render virtual content that is independent of eye rotation within the user's head. In some embodiments, a dynamic map of a scene can be generated based on multiple coordinate frames in real space across one or more sessions, so that virtual content that interacts with the dynamic map can be robustly rendered, independent of eye rotation within the user's head and / or independent of sensor deformation caused by, for example, heat generated during high-speed, computationally intensive operations. In some embodiments, the configuration of multiple coordinate frames can enable a first XR device worn by a first user and a second XR device worn by a second user to identify a common location in a scene. In some embodiments, the configuration of multiple coordinate frames can enable a first and a second user wearing the first and second XR devices to view virtual content at the same location of a scene.

[0102] In some embodiments, a tracking map may be constructed in a world coordinate frame, which may have a world origin. When the XR device is powered on, the world origin may be the first pose of the XR device. The world origin may be aligned with gravity, allowing developers of XR applications to perform gravity alignment without additional work. Different tracking maps may be constructed in different world coordinate frames, because the tracking map may be captured by the same XR device in different sessions and / or by different XR devices worn by different users. In some embodiments, a session of an XR device may start from when the device is powered on to when the device is turned off. In some embodiments, the XR device may have a head coordinate frame, which may have a head origin. The head origin may be the current pose of the XR device when the image is taken. The difference between the head pose of the world coordinate frame and the head pose of the head coordinate frame may be used to estimate the tracking route.

[0103] In some embodiments, the XR device may have a camera coordinate frame that may have a camera origin. The camera origin may be the current pose of one or more sensors of the XR device. The inventors have recognized and appreciated that the configuration of the camera coordinate frame enables robust display of virtual content that is independent of eye rotation within the user's head. The configuration also enables robust display of virtual content that is independent of sensor deformation, such as due to heat generated during operation.

[0104] In some embodiments, the XR device may have a head unit having a head-mounted frame that a user can fix to their head and may include two waveguides, one in front of each eye of the user. The waveguides may be transparent so that ambient light from real-world objects can be transmitted through the waveguides and the user can see the real-world objects. Each waveguide may send projection light from a projector to a corresponding eye of the user. The projection light may form an image on the retina of the eye. Thus, the retina of the eye receives both the ambient light and the projection light. The user may simultaneously see real-world objects and one or more virtual objects created by the projection light. In some embodiments, the XR device may have sensors that detect real-world objects around the user. These sensors may, for example, be cameras that capture images that can be processed to identify the location of real-world objects.

[0105] In some embodiments, instead of attaching virtual content in a world coordinate frame, the XR system can assign a local coordinate frame to virtual content. Such a configuration enables virtual content to be attached to a more persistent frame location, such as a persistent coordinate frame (PCF). In some embodiments, when a head gesture session ends, the local coordinate frame can be converted to a world coordinate frame. In some embodiments, the world coordinate frame can be the origin of a map of a previous session that has been positioned relative to the current local coordinate frame. When the position of an object changes, the XR device can detect the change in the environment map and determine the movement of the head unit worn by the user relative to the real-world object.

[0106] Figure 1 An XR system 10 is shown according to some embodiments. The XR system may include a first XR device 12 . 1 worn by a first user 14 . 1 , a real object in the form of a table 16 , a network 18 , and a server 20 .

[0107] In the example shown, the first XR device 12.1 includes a head unit 22, a waist pack 24, and a cable connection 26. The first user 14.1 secures the head unit 22 to his head and secures the waist pack 24, which is remote from the head unit 22, to his waist. The cable connection 26 connects the head unit 22 to the waist pack 24. The head unit 22 includes technology for displaying one or more virtual objects to the first user 14.1 while allowing the first user 14.1 to see real objects such as a table 16. The waist pack 24 primarily includes the processing and communication capabilities of the first XR device 12.1. In some embodiments, the processing and communication capabilities may reside in whole or in part in the head unit 22, so that the waist pack 24 may be removed or may be located in another device such as a backpack.

[0108] In the example shown, the waist pack 24 is connected to the network 18 via a wireless connection. The server 20 is connected to the network 18 and maintains data representing local content. The waist pack 24 downloads the data representing the local content from the server 20 via the network 18. The waist pack 24 provides the data to the head unit 22 via a cable connection 26. The head unit 22 may include a display having a light source, such as a laser light source or a light emitting diode (LED) light source, and a waveguide to guide the light.

[0109] In some embodiments, the first user 14.1 may mount the head unit 22 to his head and the waist pack 24 to his waist. The waist pack 24 may download image data from the server 20 via the network 18. The first user 14.1 may see the table 16 through the display of the head unit 22. A projector forming part of the head unit 22 may receive image data from the waist pack 24 and generate light based on the image data. The light may travel through one or more waveguides forming part of the display of the head unit 22. The light may then leave the waveguide and propagate onto the retina of the eye of the first user 14.1. The projector may generate light in a pattern that is replicated on the retina of the eye of the first user 14.1. The light that falls on the retina of the eye of the first user 14.1 may have a selected depth of field so that the first user 14.1 perceives an image at a preselected depth behind the waveguide. In addition, the two eyes of the first user 14.1 may receive slightly different images so that the brain of the first user 14.1 perceives one or more three-dimensional images at a selected distance from the head unit 22. In the example shown, the first user 14.1 perceives virtual content 28 above the table 16. The scale of the virtual content 28 and its position and distance from the first user 14.1 are determined by the data representing the virtual content 28 and the various coordinate frames used to display the virtual content 28 to the first user 14.1.

[0110] In the example shown, the virtual content 28 is not visible from the perspective of the drawing and is visible to the first user 14.1 using the first XR device 12.1. The virtual content 28 may initially reside as a data structure within the visual data and algorithms in the waist pack 24. The data structure may then manifest itself as light when the projector of the head unit 22 generates light based on the data structure. It should be understood that although the virtual content 28 does not exist in the three-dimensional space in front of the first user 14.1, the virtual content 28 is still represented in the three-dimensional space for illustration purposes. Figure 1 Visualizations of computer data in three-dimensional space may be used in this description to illustrate how data structures that facilitate rendering by one or more users relate to one another within the data structures in the waist pack 24 .

[0111] Figure 2 Components of a first XR device 12.1 are shown in accordance with some embodiments. The first XR device 12.1 may include a head unit 22, and various components that form part of the visual data and algorithms, including, for example, a rendering engine 30, various coordinate frames 32, various origin and destination coordinate frames 34, and various origin to destination coordinate frame transformers 36.

[0112] Head unit 22 may include a head mounted frame 40 , a display system 42 , a real object detection camera 44 , a motion tracking camera 46 , and an inertial measurement unit 48 .

[0113] The head-mounted frame 40 may have a Figure 1 The display system 42, real object detection camera 44, motion tracking camera 46 and inertial measurement unit 48 can be mounted to the head mounted frame 40 and thus move with the head mounted frame 40.

[0114] Coordinate system 32 may include local data system 52 , world frame system 54 , head frame system 56 , and camera frame system 58 .

[0115] The local data system 52 may include a data channel 62, a local frame determination routine 64, and local frame storage instructions 66. The data channel 62 may be an internal software routine, a hardware component such as an external cable or radio frequency receiver, or a hybrid component such as an open port. The data channel 62 may be configured to receive image data 68 representing virtual content.

[0116] The local frame determination routine 64 can be connected to the data channel 62. The local frame determination routine 64 can be configured to determine the local coordinate frame 70. In some embodiments, the local frame determination routine can determine the local coordinate frame based on a real-world object or a real-world location. In some embodiments, the local coordinate frame can be based on the origin at the initialization of the head gesture session. In some embodiments, the local coordinate frame can be located at the location where the device is guided, or can be located at a new place when the head gesture is lost during the guided session. In some embodiments, the local coordinate frame can be the origin at the beginning of the head gesture session. In some embodiments, the virtual content can also have an associated content coordinate frame. In some embodiments, the content coordinate frame can be based on the top edge relative to the bottom edge of the browser window, the head or foot of the character, a node on the outer surface of a prism or bounding box surrounding the virtual content, or any other suitable location of a coordinate frame that defines the facing direction of the virtual content and the location where the virtual content is placed (e.g., a node, such as a placement node or an anchor node), etc.

[0117] The local frame storage instructions 66 may be connected to the local frame determination routine 64. Those skilled in the art will appreciate that software modules and routines are "connected" to each other through subroutines, calls, and the like. The local frame storage instructions 66 may store the local coordinate frame 70 as a local coordinate frame 72 within the origin and destination coordinate frame 34. In some embodiments, the origin and destination coordinate frames 34 may be one or more coordinate frames that may be manipulated or transformed to allow virtual content to persist between sessions. In some embodiments, a session may be a time period between startup and shutdown of an XR device. Two sessions may be two startup and shutdown time periods of a single XR device, or startup and shutdown time periods of two different XR devices.

[0118] In some embodiments, the origin and destination coordinate frames 34 may be coordinate frames involved in one or more transformations required to enable the XR device of the first user and the XR device of the second user to recognize a common location. In some embodiments, the destination coordinate frame may be the output of a series of calculations and transformations applied to the target coordinate frame so that the first and second users view virtual content in the same location.

[0119] The rendering engine 30 may be connected to the data channel 62. The rendering engine 30 may receive image data 68 from the data channel 62 so that the rendering engine 30 may render virtual content based at least in part on the image data 68.

[0120] Display system 42 may be connected to rendering engine 30. Display system 42 may include components that transform image data 68 into visible light. The visible light may be formed into two patterns, one for each eye. The visible light may enter Figure 1 The eye of the first user 14.1 may be detected on the retina of the eye of the first user 14.1.

[0121] Real object detection camera 44 may include one or more cameras that can capture images from different sides of head mounted frame 40. Motion tracking camera 46 may include one or more cameras that can capture images on the sides of head mounted frame 40. One set of one or more cameras may be used instead of two sets of one or more cameras representing real object detection camera 44 and motion tracking camera 46. In some embodiments, cameras 44, 46 may capture images.

[0122] In some embodiments, the images captured by the cameras 44, 46 may be computed as one or more key frames (e.g., Fig.11D1, 2 in the keyframes). The keyframe may correspond to a camera pose. The keyframe may include one or more camera images captured at the camera pose. In some embodiments, the XR system may determine that a portion of the camera image captured at the camera pose is useless and therefore does not include the portion in the keyframe. Therefore, using keyframes to align new images with early knowledge of the scene can reduce the use of XR system computing resources. In some embodiments, a keyframe may be an image or image data at a location with a direction / angle. In some embodiments, a keyframe may be a location and direction from which one or more map points can be observed. In some embodiments, a keyframe may be a coordinate frame with an ID. U.S. patent application serial number 15 / 877,359 describes keyframes, and the entire contents of the same are incorporated herein by reference.

[0123] Inertial measurement unit 48 may include multiple devices for detecting motion of head unit 22. Inertial measurement unit 48 may include a gravity sensor, one or more accelerometers, and one or more gyroscopes. The sensors of inertial measurement unit 48, in combination, track motion of head unit 22 in at least three orthogonal directions and around at least three orthogonal axes.

[0124] In the example shown, the world frame system 54 includes a world surface determination routine 78, a world frame determination routine 80, and a world frame storage instruction 82. The world surface determination routine 78 is connected to the real object detection camera 44. The world surface determination routine 78 accepts images and / or keyframes based on images captured by the real object detection camera 44 and processes the images to identify surfaces in the images. A depth sensor (not shown) can determine the distance to the surface. Therefore, these surfaces are represented by data in three dimensions including their size, shape, and distance from the real object detection camera.

[0125] In some embodiments, the world surface determination routine 78 may convert one or more keyframes into a persistent pose (e.g., Fig.11D 1, 2 in the world surface determination routine 78). The persistent gesture may be an output of routine 78. In some embodiments, the persistent gesture may be a coordinate position and / or direction with one or more associated keyframes. In some embodiments, the persistent gesture may be automatically created after the user has traveled a certain distance (e.g., three meters). In some embodiments, the persistent gesture may be used as a reference point during positioning. In some embodiments, the data determined by the world surface determination routine 78 may be stored in a traversable world (e.g., a digital representation of a real object in the physical world that may be stored and updated as the real object in the physical world changes).

[0126] In some embodiments, the navigable world may include or be created from one or more canonical maps, each of which may include a corresponding world coordinate frame. The canonical map may be derived from a tracking map (TM) (e.g., Fig.40A In some embodiments, the XR device may send one or more TMs to the cloud to be merged with additional TMs captured by the XR device at different times or by other XR devices.

[0127] In some embodiments, a persistent frame determination routine 80 may be coupled to the world surface determination routine 78 and may determine a persistent coordinate frame (PCF) 84. The PCF may include a plurality of PCF anchor points (e.g., Fig.11D The PCF anchors 1-5 in the traversable world). New PCF anchors may be determined based on predefined distances allowed between different PCF anchors. The world surface determination routine 78 may store PCF anchors that may constitute PCFs in the traversable world. In some embodiments, the world frame determination routine 80 may convert one or more persistent gestures to PCF anchors when the user travels a predetermined distance (e.g., five meters). In some embodiments, for example, in the traversable world, the PCF anchors may be associated with one or more world coordinate frames 86. In some embodiments, the world coordinate frames 86 may be stored within the origin and destination coordinate frames 34.

[0128] In some embodiments, the virtual content's own content coordinate frame (i.e., a coordinate frame attached to the virtual content independently of its real-world position) can be associated with one or more PCF anchor points. The inventors have recognized and appreciated that associating virtual content with PCF anchor points can give the virtual content a more persistent frame position, as opposed to measuring directly against a world coordinate frame. For example, if a virtual lamp is placed on a table, there may be multiple data points on the table to provide placement input for the relative positioning of the virtual lamp that does not substantially change over time. In contrast, if a world map is created based on a specific orientation and position, and the user changes position or orientation, thereby requiring a new world coordinate frame, the virtual lamp may continue to utilize the same local coordinate frame rather than adjusting to a new world frame that may introduce jitter or position shifts in the appearance of the lamp.

[0129] In some embodiments, persistent frame storage instructions 82 may be connected to persistent frame determination routine 80 to receive persistent coordinate frame 84 from persistent frame determination routine 80. In some embodiments, persistent frame storage instructions 82 may store persistent coordinate frame 84 in a database locally and / or in the cloud (e.g., may depend on security settings).

[0130] The head frame system 56 may include a head frame determination routine 90 and head frame storage instructions 92. The head frame determination routine 90 may be connected to the motion tracking camera 46 and the inertial measurement unit 48. The head frame determination routine 90 may use data from the motion tracking camera 46 and the inertial measurement unit 48 to calculate a head coordinate frame 94. For example, the inertial measurement unit 48 may have a gravity sensor that determines the direction of gravity relative to the head unit 22. The motion tracking camera 46 may continuously capture images that are used by the head frame determination routine 90 to refine the head coordinate frame 94. When Figure 1 When the first user 14.1 in the image moves their head, the head unit 22 moves. The motion tracking camera 46 and the inertial measurement unit 48 may continuously provide data to the head frame determination routine 90 so that the head frame determination routine 90 may update the head coordinate frame 94.

[0131] The head frame storage instructions 92 may be connected to the head frame determination routine 90 to receive the head coordinate frame 94 from the head frame determination routine 90. The head frame storage instructions 92 may store the head coordinate frame 94 as a head coordinate frame 96 in the origin and destination coordinate frames 34. The head frame storage instructions 92 may repeatedly store the updated head coordinate frame 94 as the head coordinate frame 96 when the head frame determination routine 90 recalculates the head coordinate frame 94. In some embodiments, the head coordinate frame may be the position of the wearable XR device 12.1 relative to the local coordinate frame 72.

[0132] The camera frame system 58 may include camera intrinsics 98. The camera intrinsics 98 may include the dimensions of the head unit 22 as a feature of its design and manufacture. The camera intrinsics 98 may be used to calculate a camera coordinate frame 100 stored within the origin and destination coordinate frame 34.

[0133] In some embodiments, the camera coordinate frame 100 may include Figure 1 1. When the left eye moves from left to right or up and down, the pupil position of the left eye is located in the camera coordinate frame 100. In addition, the pupil position of the right eye is located in the camera coordinate frame 100 of the right eye. In some embodiments, the camera coordinate frame 100 may include the position of the camera relative to the local coordinate frame when the image is captured.

[0134] The origin-to-destination coordinate frame transformer 36 may include a local-to-world coordinate transformer 104, a world-to-head coordinate transformer 106, and a head-to-camera coordinate transformer 108. The local-to-world coordinate transformer 104 may receive the local coordinate frame 72 and transform the local coordinate frame 72 to the world coordinate frame 86. The transformation of the local coordinate frame 72 to the world coordinate frame 86 may be represented as a transformation of the local coordinate frame within the world coordinate frame 86 to the world coordinate frame 110.

[0135] The world to head coordinate transformer 106 may transform from the world coordinate frame 86 to the head coordinate frame 96. The world to head coordinate transformer 106 may transform the local coordinate frame transformed to the world coordinate frame 110 to the head coordinate frame 96. The transformation may be represented as a local coordinate frame transformed within the head coordinate frame 96 to the head coordinate frame 112.

[0136] The head-to-camera coordinate transformer 108 may transform from the head coordinate frame 96 to the camera coordinate frame 100. The head-to-camera coordinate transformer 108 may transform the local coordinate frame transformed to the head coordinate frame 112 to the local coordinate frame transformed to the camera coordinate frame 114 within the camera coordinate frame 100. The local coordinate frame transformed to the camera coordinate frame 114 may be input to the rendering engine 30. The rendering engine 30 may render the image data 68 representing the local content 28 based on the local coordinate frame transformed to the camera coordinate frame 114.

[0137] Figure 3 is a spatial representation of the various origin and destination coordinate frames 34. In the figure, a local coordinate frame 72, a world coordinate frame 86, a head coordinate frame 96, and a camera coordinate frame 100 are represented. In some embodiments, the local coordinate frame associated with the XR content 28 can define the position and rotation of the virtual content (e.g., a node and a facing direction can be provided), which can then be transformed into a local and / or world coordinate frame and / or PCF when the virtual content is placed in the real world so that the virtual content can be viewed by the user. Each camera can have its own camera coordinate frame 100 that contains all the pupil positions of one eye. Reference numerals 104A and 106A respectively represent the camera coordinate frames 104A and 106A of the XR content 28. Figure 2 The transformations are performed by the local to world coordinate transformer 104, the world to head coordinate transformer 106 and the head to camera coordinate transformer 108 in FIG.

[0138] Figure 4A camera rendering protocol for transforming from a head coordinate frame to a camera coordinate frame is depicted in accordance with some embodiments. In the example shown, the pupil of a single eye moves from position A to position B. Virtual objects that are to appear stationary will be projected onto a depth plane at one of two positions A or B, depending on the position of the pupil (assuming the camera is configured to use the pupil as its coordinate frame). As a result, using the pupil coordinate frame transformed to the head coordinate frame will result in jittering of stationary virtual objects as the eye moves from position A to position B. This situation is referred to as view-dependent display or projection.

[0139] like Figure 5 As shown, a camera coordinate frame (e.g., CR) is placed and contains all pupil positions, and the object projection will now be consistent regardless of the pupil positions A and B. The head coordinate frame is transformed into the CR frame, which is called a view-independent display or projection. Image re-projection can be applied to the virtual content to account for changes in eye position, however, since the rendering is still in the same position, jitter can be minimized.

[0140] like Figure 6 As shown in FIG. 1 , the second XR device 12.2, which may be in the same scene as the first XR device 12.1, may include a permanent coordinate frame (PCF) integration unit 1300, an application 1302 that generates image data 68, and a frame embedding generator 308. In some embodiments, the map download system 126, the anchor point identification system 128 ... Figure 2 , positioning module 130, canonical map merger 132, canonical map 133 and map publisher 136 are collected as a traversable world unit 1304. The PCF integration unit 1300 can be connected to the traversable world unit 1304 and other components of the second XR device 12.2 to allow the acquisition, generation, use, upload and download of PCF anchors. The coordinate framework composed of PCF anchors can achieve greater persistence in a constantly changing world (for example, a world where people can enter and exit a scene and objects such as doors move relative to the scene), require less storage space and transmission rate, and enable mapping of the scene using individual PCF anchors and their relationships to each other and / or an integrated constellation of PCF anchors.

[0141] In some embodiments, the PCF integration unit 1300 may include a PCF anchor 1306 previously stored in a data storage on a storage unit of the second XR device 12.2, a PCF tracker 1308, a persistent pose acquirer 1310, a PCF checker 1312, a PCF generation system 1314, a coordinate frame calculator 1316, a persistent pose calculator 1318, and three transformers including a tracking map and persistent pose transformer 1320, a persistent pose and PCF transformer 1322, and a PCF and image data transformer 1324.

[0142] In some embodiments, the PCF tracker 1308 may have an on prompt and a off prompt selectable by the application 1302. The application 1302 may be executable by a processor of the second XR device 12.2 to, for example, display virtual content. The application 1302 may have a call to turn on the PCF tracker 1308 via an on prompt. When the PCF tracker 1308 is on, the PCF tracker 1308 may generate PCF anchors. The application 1302 may have a subsequent call to turn off the PCF tracker 1308 via a off prompt. When the PCF tracker 1308 is off, the PCF tracker 1308 terminates PCF generation.

[0143] In some embodiments, the server 20 may include a plurality of persistent gestures 1332 and a plurality of PCF anchor points 1330 that have been previously saved in association with the canonical map 120. The map sender 122 may send the canonical map 120 together with the persistent gestures 1332 and / or the PCF anchor points 1330 to the second XR device 12.2. The persistent gestures 1332 and the PCF anchor points 1330 may be stored on the second XR device 12.2 in association with the canonical map 133. Figure 2 When locating to the standard map 133, you can Figure 2 The persistent gesture 1332 and the PCF anchor point 1330 are stored in association.

[0144] In some embodiments, the persistent posture acquirer 1310 may acquire the Figure 2 The PCF checker 1312 may be connected to the persistent gesture acquirer 1310. The PCF checker 1312 may acquire PCF anchor points from the PCF anchor points 1306 based on the persistent gesture acquired by the persistent gesture acquirer 1310. The PCF anchor points acquired by the PCF checker 1312 may form an initial set of PCF anchor points for image display based on the PCF anchor points.

[0145] In some embodiments, application 1302 may need to generate additional PCF anchor points. For example, if the user moves to an area that has not been previously mapped, application 1302 may open PCF tracker 1308. PCF generation system 1314 may connect to PCF tracker 1308 and generate additional PCF anchor points as the user moves to an area that has not been previously mapped. Figure 2 Start expanding and start based on the ground Figure 2 Generating PCF anchor points. The PCF anchor points generated by the PCF generation system 1314 may form a second set of PCF anchor points that may be used for PCF-based image display.

[0146] The coordinate frame calculator 1316 may be connected to the PCF checker 1312. After the PCF checker 1312 obtains the PCF anchor point, the coordinate frame calculator 1316 may call the head coordinate frame 96 to determine the head pose of the second XR device 12.2. The coordinate frame calculator 1316 may also call the persistent pose calculator 1318. The persistent pose calculator 1318 may be directly or indirectly connected to the frame embedding generator 308. In some embodiments, the frame embedding generator 308 may output a global feature string 316, which may be used as a data representation of an image (e.g., a frame or a keyframe). In some embodiments, an image / frame may be designated as a keyframe after a threshold distance (e.g., 3 meters) from a previous keyframe. The persistent pose calculator 1318 may generate a persistent pose based on multiple (e.g., three) keyframes. In some embodiments, the persistent pose may be substantially an average of the coordinate frames of the multiple keyframes.

[0147] Tracking map and persistent posture changer 1320 can be connected to the ground Figure 2 and persistent posture calculator 1318. Tracking map and persistent posture converter 1320 can Figure 2 Transform to a persistent pose to determine relative to the ground Figure 2 The persistent pose at the origin of .

[0148] Persistent pose and PCF transformer 1322 may be connected to tracking map and persistent pose transformer 1320, and further connected to PCF checker 1312 and PCF generation system 1314. Persistent pose and PCF transformer 1322 may transform the persistent pose (to which the tracking map has been transformed) from PCF checker 1312 and PCF generation system 1314 into a PCF anchor point to determine the PCF relative to the persistent pose.

[0149] PCF and image data transformer 1324 may be connected to persistent gesture and PCF transformer 1322 and data channel 62. PCF and image data transformer 1324 transforms the PCF into image data 68. Rendering engine 30 may be connected to PCF and image data transformer 1324 to display image data 68 to a user relative to a PCF anchor point.

[0150] The PCF integration unit 1300 may store additional PCF anchor points generated using the PCF generation system 1314 within the PCF anchor point 1306. The PCF anchor point 1306 may be stored relative to the persistent gesture. Figure 2 When the map publisher 136 obtains the PCF anchor point 1306 and the persistent posture associated with the PCF anchor point 1306, the map publisher 136 also sends the map to the server 20. Figure 2The associated PCF and persistent gesture. When the map storage routine 118 of the server 20 stores the map Figure 2 When the map storage routine 118 is generated by the second viewing device 12.2, the persistent gesture and PCF anchor point can also be stored. The map merging algorithm 124 can use the map associated with the canonical map 120 and stored in the persistent gesture 1332 and PCF anchor point 1330, respectively. Figure 2 The canonical map 120 is created using the persistent pose and PCF anchors.

[0151] The first XR device 12.1 may include a PCF integrated unit similar to the PCF integrated unit 1300 of the second XR device 12.2. When the map transmitter 122 sends the canonical map 120 to the first XR device 12.1, the map transmitter 122 may send a persistent gesture 1332 and a PCF 1330 associated with the canonical map 120 and originating from the second XR device 12.2. The first XR device 12.1 may store the PCF anchor and the persistent gesture in a data store on a storage device of the first XR device 12.1. The first XR device 12.1 may then utilize the persistent gesture and PCF anchor originating from the second XR device 12.2 for image display relative to the PCF anchor. Additionally or alternatively, the first XR device 12.1 may obtain, generate, use, upload, and download PCF anchors and persistent gestures in a manner similar to the second XR device 12.2 as described above.

[0152] Fig.44 The key frames generated for the room are shown. In the example shown, a first key frame KF1 is generated for the door on the left wall of the room. A second key frame KF2 is generated for the corner area where the floor, left wall and right wall of the room intersect. A third key frame KF3 is generated for the window area on the right wall of the room. A fourth key frame KF4 is generated on the floor of the wall, at the far end of the carpet. A fifth key frame KF5 is generated for the area of ​​the carpet closest to the user.

[0153] Fig.45Persistent gestures PP1 and PP2 are shown in the user environment. In some embodiments, a new persistent gesture is created when the device measures a threshold distance traveled, and / or when an application requests a new persistent gesture. In some embodiments, the threshold distance can be 3 meters, 5 meters, 20 meters, or any other suitable distance. Selecting a smaller threshold distance (e.g., 1m) can result in an increase in computational load because a larger number of PPs can be created and managed compared to a larger threshold distance. Selecting a larger threshold distance (e.g., 40m) can result in increased virtual content placement errors because a smaller number of PPs will be created, which will result in a smaller number of PCFs created, meaning that virtual content attached to a PCF anchor point may be a relatively large distance away from the PCF anchor point (e.g., 30m), and the error increases with increasing distance from the anchor point to the virtual content.

[0154] In some embodiments, a PP may be created at the start of a new session. This initial PP may be considered to be zero and may be visualized as the center of a circle with a radius equal to a threshold distance. When the device reaches the circumference of the circle, and in some embodiments, an application requests a new PP, a new PP may be placed at the device's current location (at the threshold distance). In some embodiments, if the device is able to find an existing PP within the threshold distance from the device's new location, a new PP will not be created at the threshold distance. In some embodiments, when a new PP is created, the device appends one or more closest keyframes to the PP. In some embodiments, the location of the PP relative to the keyframes may be based on the location of the device when the PP was created. In some embodiments, a PP will not be created when the device travels the threshold distance unless an application requests a PP.

[0155] In some embodiments, when an application has virtual content to display to the user, the application can request a PCF from the device. A PCF request from the application can trigger a PP request, and a new PP will be created after the device travels a threshold distance. Fig.45 A first permanent pose PP1 is shown, which may have closest keyframes (eg, KF1, KF2, and KF3) attached. Fig.45 Also shown is a second permanent pose PP2, which may have additional proximal keyframes (eg, KF4 and KF5).

[0156] Fig.46 PCF anchor points are shown that may include PP1 and PP2. As described above, PCFs may be used to display image data relative to the PCFs. In some embodiments, each PCF may have coordinates in another coordinate frame (e.g., a world coordinate frame) and descriptors. In some embodiments, various constellations of PCF anchor points may be combined in a persistent manner to represent the real world, and this requires less data and less data transmission.

[0157] Fig.47 A method 4700 of establishing and using a persistent coordinate frame according to some embodiments is shown. Method 4700 may begin by capturing an image about a scene using one or more sensors of an XR device (action 4702). Multiple cameras may be used, and a camera may generate multiple images. Method 4700 may include: extracting (4704) features from the captured image; generating (action 4706) descriptors for the extracted features; generating (action 4708) key frames based on the descriptors; and generating (action 4710) persistent gestures based on the key frames. In some embodiments, a key frame may be generated based on each image in the image, and a persistent gesture may be generated using an average of the key frames of all images of all cameras. Method 4700 may include generating (action 4712) anchor points based on the persistent gestures. The anchor points may include one or more persistent gestures. Method 4700 may include associating image data of virtual content to be displayed by the XR device with at least one of the anchor points (step 4714).

[0158] Figures 48A to 48I is a diagram illustrating an example of establishing and using a persistent coordinate frame, which may enable data sharing between XR devices. Fig.48A Two users 4802A, 4802B are shown with respective local tracking maps 4804A, 4804B that have not yet been localized to a canonical map. The origin 4806A, 4806B of each user is depicted by a coordinate system (e.g., a world coordinate system) in their respective regions. In some embodiments, TM 4802A may include a persistent pose (PP) 4808A; TM 4802B may include a PP 4808B. Fig.48B It is shown that the XR devices worn by the respective users 4802A, 4802B can create local PCF anchor points 4810A, 4810B based on the PP 4808A, 4908B. Fig.48C It is shown that persistent content 4812 (eg, virtual content) may now be attached to the PCF anchor points 4810A, 4810B. Fig.48D Both devices are shown receiving a canonical map 1814, for example from the cloud. Fig.48E A canonical map 4814 is shown with both devices positioned on the respective devices. Fig.48F Each device is shown creating an attachment (e.g., attachment 4816A, 4816B) to a respective persistent gesture (e.g., PP 4818A, 4818B) on the canonical map between local PCF anchors (e.g., PCF anchors 4810A, 4810B) on the respective device. Figure 48G A persistent gesture snapshot from the canonical map to the local tracking map is shown. As can be seen, the local tracking maps are connected to each other via the persistent gesture. Figure 48H PCF anchor point 4810A on a device worn by user 4802A is shown accessible in a device worn by user 4802B. Fig.48I It is shown that all maps may eventually be merged and the PCFs may be "federated" in a connected constellation.

[0159] Map building and localization

[0160] Figure 6 The first XR device 12.1 is shown in more detail according to some embodiments, as well as the visual data and algorithms of the second XR device 12.2 and the server 20. Although not shown, the first XR device 12.1 may be configured identically to the second XR device 12.2.

[0161] The server 20 may have a map storage routine 118 , a canonical map 120 , a map transmitter 122 , and a map merging algorithm 124 .

[0162] In the example shown, the first XR device 12.1 generates a local tracking map (hereinafter referred to as a “map”). Figure 1 ”), and the map storage routine 118 receives the map from the first XR device 12.1 Figure 1 The map storage routine 118 then stores the map Figure 1 The specification map 120 is stored on the storage device of the server 20 .

[0163] The second XR device 12 . 2 includes a map download system 126 , an anchor point identification system 128 , a positioning module 130 , a canonical map merger 132 , a local content positioning system 134 , and a map publisher 136 .

[0164] In use, the map sender 122 sends the normative map 120 to the second XR device 12 . 2 , and the map download system 126 downloads and stores the normative map 120 from the server 20 as the normative map 133 .

[0165] The anchor point identification system 128 is connected to the world surface determination routine 78. The anchor point identification system 128 identifies anchor points based on objects detected by the world surface determination routine 78. The anchor point identification system 128 generates a second map (map) using the anchor points. Figure 2 As shown in loop 138, the anchor point identification system 128 continues to identify anchor points and continues to update the ground Figure 2 The positions of the anchor points are recorded as three-dimensional data based on data provided by the world surface determination routine 78 . The world surface determination routine 78 receives images from the real object detection camera 44 and depth data from the depth sensor 135 to determine the positions of surfaces and their relative distances from the depth sensor 135 .

[0166] The positioning module 130 is connected to the specification map 133 and the Figure 2 The positioning module 130 repeatedly attempts to Figure 2 The standard map 133 is located. The standard map merger 132 is connected to the standard map 133 and the map Figure 2 When the positioning module 130 sets the ground Figure 2 When the standard map 133 is located, the standard map merger 132 merges the standard map 133 into the map. Figure 2 Then, the map is updated with the missing data included in the canonical map. Figure 2 .

[0167] The local content location system 134 is connected to the ground Figure 2 The local content location system 134 may be, for example, a system in which a user can locate local content at a specific location within a world coordinate frame. The local content then attaches itself to the local content. Figure 2 The local to world coordinate converter 104 converts the local coordinate frame into the world coordinate frame based on the settings of the local content positioning system 134. Figure 2 The functionality of the rendering engine 30, display system 42, and data pipeline 62 are described.

[0168] Map publisher 136 will Figure 2 Upload to the server 20. The map storage routine 118 of the server 20 then Figure 2 Stored in the storage medium of the server 20.

[0169] Map merging algorithm 124 Figure 2 Merge with the canonical map 120. When more than two maps (e.g., three or four maps) have been stored, the map merging algorithm 124 merges all the maps into the canonical map 120 to render the new canonical map 120. The map sender 122 then sends the new canonical map 120 to any and all devices 12.1 and 12.2 located in the area represented by the new canonical map 120. When the devices 12.1 and 12.2 locate their respective maps to the canonical map 120, the canonical map 120 becomes the upgraded map.

[0170] Figure 7 The display system 42 is shown in more detail. The display system 42 comprises a stereo analyser 144 which is connected to the rendering engine 30 and forms part of the visual data and algorithms.

[0171] The display system 42 further includes a left projector 166A and a right projector 166B and a left waveguide 170A and a right waveguide 170B. The left projector 166A and the right projector 166B are connected to a power supply. Each projector 166A and 166B has a corresponding input for image data to be provided to the corresponding projector 166A or 166B. The corresponding projector 166A or 166B generates a two-dimensional pattern of light and emits light therefrom when powered on. The left waveguide 170A and the right waveguide 170B are positioned to receive light from the left projector 166A and the right projector 166B, respectively. The left waveguide 170A and the right waveguide 170B are transparent waveguides.

[0172] In use, the user mounts the head mounted frame 40 to their head. The components of the head mounted frame 40 may, for example, include a strap (not shown) that wraps around the back of the user's head. The left and right waveguides 170A, 170B are then located in front of the user's left and right eyes 220A, 220B.

[0173] The rendering engine 30 inputs the received image data into the stereo analyzer 144. The image data is Figure 1 The three-dimensional image data of the local content 28 in the image data is projected onto a plurality of virtual planes. The stereo analyzer 144 analyzes the image data to determine a left image data set and a right image data set based on the image data for projection onto each depth plane. The left image data set and the right image data set are data sets representing a two-dimensional image that is projected in three dimensions to give a user a sense of depth.

[0174] The stereo analyzer 144 inputs the left image data set and the right image data set to the left projector 166A and the right projector 166B. Then, the left projector 166A and the right projector 166B create the left illumination pattern and the right illumination pattern. The components of the display system 42 are shown in a plan view, but it should be understood that when shown in a front view, the left pattern and the right pattern are two-dimensional patterns. Each light pattern includes a plurality of pixels. For the purpose of illustration, it is shown that the light 224A and 226A from two pixels leave the left projector 166A and enter the left waveguide 170A. The light 224A and 226A are reflected from the side of the left waveguide 170A. It is shown that the light 224A and 226A propagate from left to right in the left waveguide 170A by internal reflection, but it should be understood that the light 224A and 226A also propagate into the paper in a certain direction using a refraction and reflection system.

[0175] Light rays 224A and 226A exit the left optical waveguide 170A through the pupil 228A and then enter the left eye 220A through the pupil 230A of the left eye 220A. Light rays 224A and 226A then fall on the retina 232A of the left eye 220A. In this manner, the left light pattern falls on the retina 232A of the left eye 220A. The perception to the user is that the pixels formed on the retina 232A are what the user perceives as pixels 234A and 236A at a certain distance on the side of the left waveguide 170A opposite the left eye 220A. Depth perception is created by manipulating the focal length of the light.

[0176] In a similar manner, the stereo analyzer 144 inputs the right image data set into the right projector 166B. The right projector 166B transmits a right light pattern, which is represented by pixels in the form of light rays 224B and 226B. Light rays 224B and 226B are reflected within the right waveguide 170B and exit through the pupil 228B. Light rays 224B and 226B then enter through the pupil 230B of the right eye 220B and fall on the retina 232B of the right eye 220B. The pixels of light rays 224B and 226B are perceived as pixels 134B and 236B behind the right waveguide 170B.

[0177] The patterns created on the retinas 232A and 232B are perceived as left and right images, respectively. The left and right images are slightly different from each other due to the function of the stereo analyzer 144. The left and right images are perceived as three-dimensional renderings in the user's mind.

[0178] As mentioned, the left waveguide 170A and the right waveguide 170B are transparent. Light from real objects on the table 16 on the side opposite to the eyes 220A and 220B, such as the left waveguide 170A and the right waveguide 170B, can be projected through the left waveguide 170A and the right waveguide 170B and fall on the retinas 232A and 232B.

[0179] Figure 8 The first XR device 12.1 shows the Figure 1 and local content (content 123 and content 456). Figure 1 A number of anchor points (anchor point a to anchor point d) are included. From the perspective of the first XR device 12.1, anchor point a, for example, has X, Y, and Z coordinates of (0, 0, 0), and anchor point b has X, Y, and Z coordinates of (-1, 0, 0). Content 123 is associated with anchor point a. In this example, content 123 has an X, Y, and Z relationship with respect to anchor point a of (1, 0, 0). Content 456 has a relationship with respect to anchor point b. In this example, content 456 has an X, Y, and Z relationship with respect to anchor point b of (1, 0, 0). Figure 1 It also has an origin (origin 1).

[0180] exist Fig. 9 In the first XR device 12.1, Figure 1 Upload to server 20. Server 20 now has a Figure 1 The first XR device 12.1 has a canonical map which is empty at this stage. For the purposes of discussion, and in some embodiments, the server 20 has a canonical map in addition to the map. Figure 1 No other maps are included. No maps are stored on the second XR device 12.2.

[0181] The first XR device 12.1 also sends its Wi-Fi signature data to the server 20. The server 20 may use the Wi-Fi signature data to determine the approximate location of the first XR device 12.1 based on intelligence gathered from other devices that have connected to the server 20 or other servers in the past along with the recorded GPS locations of such other devices.

[0182] The first XR device 12.1 may now end the first session (see Figure 1 ), and can disconnect from the server 20.

[0183] Fig.10 and Fig.11A The second user 14.2 is shown initiating a second session. Because the first session of the first user 14.1 has ended, the first user 14.1 is shown in dotted lines. The second XR device 12.2 begins recording the object. The server 20 may use various systems with different granularity to determine that the second session of the second XR device 12.2 is in the same vicinity as the first session of the first XR device 12.1. For example, the first XR device 12.1 and the second XR device 12.2 may include Wi-Fi signature data, global positioning system (GPS) location data, GPS data based on Wi-Fi signature data, or any other data indicating location to record their locations. Alternatively, the anchor point identified by the second XR device 12.2 may be displayed relative to the ground. Figure 1 The similarity of the anchor points.

[0184] like Fig. 11B As shown in , the second XR device starts and begins collecting data, such as images 1110 from one or more cameras 44, 46. Fig.11DAs shown in , in some embodiments, an XR device (e.g., a second XR device 12.2) may collect one or more images 1110 and perform image processing to extract one or more features / points of interest 1120. Each feature may be converted to a descriptor 1130. In some embodiments, the descriptor 1130 may be used to describe a keyframe 1140, which may have a position and orientation of an associated image attached. One or more keyframes 1140 may correspond to a single persistent pose 1150, which may be automatically generated after a threshold distance (e.g., 3 meters) from a previous persistent pose 1150. One or more persistent poses 1150 may correspond to a single PCF 1160, which may be automatically generated after a predetermined distance (e.g., every 5 meters). Over time, as the user continues to move around the user's environment and the XR device continues to collect more data (such as image 1110), additional PCF anchors (e.g., PCF anchors 4, 5, and PCF anchor 3) may be created. One or both applications 1180 may run on the XR device and provide virtual content 1170 to the XR device for presentation to the user. The virtual content may have an associated content coordinate frame that may be placed relative to one or more PCF anchor points. Fig. 11B As shown in , the second XR device 12.2 creates three PCF anchor points. In some embodiments, the second XR device 12.2 may attempt to locate one or more canonical maps stored on the server 20. In some embodiments, as Fig. 11C As shown in FIG. 1 , the second XR device 12.2 can download the canonical map 120 from the server 20. Figure 1 Includes anchor points a to d and origin 1. In some embodiments, server 20 may have multiple canonical maps for various locations and may determine that second XR device 12.2 is located in the same vicinity as first XR device 12.1 during the first session and send a canonical map of the vicinity to second XR device 12.2.

[0185] Fig.12 The second XR device 12.2 starts identifying anchor points for generating a map. Figure 2 The second XR device 12.2 recognizes only a single anchor point, PCF 1,2. The X, Y, and Z coordinates of PCF 1,2 of the second XR device 12.2 may be (1, 1, 1). Figure 2 The second XR device 12.2 may have its own origin (origin 2), which may be based on the head pose of device 2 of the current head pose session. In some embodiments, the second XR device 12.2 may immediately attempt to Figure 2 In some embodiments, because the system cannot identify any or sufficient overlap between the two maps, the map Figure 2 It may not be possible to locate the standard map (map Figure 1 ) (i.e., positioning may fail). In some embodiments, the system may locate based on anchor point / PCF comparison between the local map and the canonical map. In some embodiments, the system may locate based on persistent pose comparison between the local map and the canonical map. In some embodiments, the system may locate based on keyframe comparison between the local map and the canonical map.

[0186] Fig.13 The second XR device 12.2 identifies the Figure 2 The other anchor points (PCF 1, 2, PCF 3, PCF 4, 5) after Figure 2 The second XR device 12.2 tries again to Figure 2 Locate to the standard map. Figure 2 has been extended to overlap at least a portion of the canonical map, so the positioning attempt will succeed. Figure 2 The overlap between the and canonical maps can be represented by PCFs / anchors, persistent poses, keyframes, or any other suitable intermediate or derived construct.

[0187] In addition, the second XR device 12.2 has linked content 123 and content 456 to the local Figure 2 The content 123 is associated with PCFs 1, 2 and 3. The content 123 has X, Y and Z coordinates (1, 0, 0) relative to PCFs 1, 2.

[0188] Similarly, relative to ground Figure 2 In PCF 3, the X, Y, and Z coordinates of content 456 are (1, 0, 0).

[0189] Fig.14A and Fig. 14B Shown Figure 2 The overlapped area / volume / section of map 1410 represents the successful localization of the canonical map. Figure 1 and the common parts of the normative map. Figure 2 PCFs 3, 4, and 5 were created before positioning, and the normative map was created before the location Figure 2 Anchor points a and c were created previously, so different anchor points / PCFs are created to represent the same volume in real space (e.g., in different maps).

[0190] like Fig.15 As shown in FIG. 1 , the second XR device 12.2 extends the ground Figure 2 , to include anchor points ad from the canonical map. Including anchor points ad indicates that the Figure 2In some embodiments, the XR system may perform an optimization step to remove duplicate anchor points / PCFs from overlapping areas, such as PCF 1410, PCF 3, and PCF 4, 5. Figure 2 After positioning, the placement of virtual content (such as content 456 and content 123) will be updated. Figure 2 The virtual content appears in the same real-world location relative to the user, despite the change to the content’s PCF attachment, and despite the update to the map. Figure 2 Anchor point.

[0191] like Fig.16 As shown in FIG. , the second XR device 12.2 continues to expand Figure 2 , for example, when the user moves around the real world, the second XR device 12.2 will identify other anchor points (anchor points e, f, g and h). It should also be noted that the ground Figure 1 exist Fig.15 and Fig.16 There are no extensions in .

[0192] refer to Fig.17 , the second XR device 12.2 will be Figure 2 Upload to server 20. Server 20 will Figure 2 With normative Figure 1 In some embodiments, when the session for the second XR device 12.2 ends, the address Figure 2 It can be uploaded to the server 20.

[0193] The canonical map in the server 20 now includes an anchor point i which is not included in the map on the first XR device 12.1. Figure 1 When a third XR device (not shown) uploads a map to the server 20 and the map includes the anchor point i, the canonical map on the server 20 may have been extended to include the anchor point i.

[0194] exist Fig.18 In the example, the server 20 will Figure 2 The server 20 determines the anchor points a to d for the standard map and the map. Figure 2 The server extends the canonical map to include anchor points e through h and Figure 2 The PCFs 1 and 2 of the first XR device 12.1 and the second XR device 12.2 are based on the ground. Figure 1 , and is outdated.

[0195] exist Fig.19In some embodiments, this may occur when the first XR device 12.1 and the second XR device 12.2 attempt to locate during a different or new or subsequent session. The first XR device 12.1 and the second XR device 12.2 proceed as described above to locate their respective local maps (respectively, local Figure 1 peacefully Figure 2 ) to locate the new canonical map.

[0196] like Fig. 20 As shown in FIG. 1 , the head coordinate frame 96 or “head pose” is relative to the ground. Figure 2 In some embodiments, the origin of the map, origin 2, is based on the head pose of the second XR device 12.2 at the start of the session. When anchors are created during a session, they are placed relative to the local coordinate frame origin 2. Figure 2 The anchor point of is used as a persistent coordinate frame relative to the world coordinate frame, where the world coordinate frame can be the local coordinate frame of the previous session (e.g., Figure 8 The land in Figure 1 The origin of 1). Figure 2 The transformation from the world coordinate frame to the head coordinate frame 96 is discussed. Fig. 20 The head coordinate frame 96 shown in FIG. 1 has only two orthogonal axes relative to the ground. Figure 2 The anchor point is at a specific coordinate position and relative to the ground Figure 2 However, it should be understood that the head coordinate frame 96 is relative to the ground. Figure 2 The anchor point of is located in three dimensions and has three orthogonal axes in three-dimensional space.

[0197] exist Fig.21 In the figure, the head coordinate frame 96 is relative to the ground Figure 2 Since the second user 14.2 has moved his head, the head coordinate frame 96 has moved. The user can move his head with six degrees of freedom (6dof). The head coordinate frame 96 can therefore be moved in 6dof (i.e., from its Fig. 20 The previous position in three dimensions, and relative to the ground Figure 2 The anchor point moves around three orthogonal axes). Figure 2The head coordinate frame 96 is adjusted as the real object detection camera 44 and the inertial measurement unit 48 in the head unit 22 detect real objects and motion, respectively. More information about head pose tracking is disclosed in U.S. patent application Ser. No. 16 / 221,065, entitled “Enhanced Pose Determination for Display Device,” and is incorporated herein by reference in its entirety.

[0198] Fig. 22 Sounds can be associated with one or more anchor points. The user can, for example, wear headphones or earphones with stereo sound. The location of the sound through the earphones can be simulated using conventional techniques. The location of the sound can be located at a fixed position so that when the user rotates his head to the left, the location of the sound rotates to the right so that the user perceives the sound as coming from the same location in the real world. In this example, the location of the sound is represented by sound 123 and sound 456. For ease of discussion, Fig. 22 In terms of analysis Fig.16 When the first user 14.1 and the second user 14.2 are in the same room at the same or different times, they perceive the sound 123 and the sound 456 to come from the same location in the real world.

[0199] Fig.23 and Fig.24 Another implementation of the above technology is shown. Figure 1 As described above, the first user 14.1 has initiated a first session. Fig.23 As shown in FIG. 1 , the first user 14.1 has terminated the first session, as shown by the dashed line. At the end of the first session, the first XR device 12.1 Figure 1 Uploaded to the server 20. The first user 14.1 has now initiated a second session at a later time than the first session. Figure 1 The first XR device 12.1 is already stored in the first XR device 12.1, so the first XR device 12.1 does not download the address from the server 20. Figure 1 If you lose your place Figure 1 , the first XR device 12.1 downloads the address from the server 20 Figure 1 Then, the first XR device 12.1 continues to construct Figure 2 Anchor point, positioned to the ground Figure 1 , and further develop the canonical map as described above. Then, as described above, the map Figure 2 Used to associate local content, head coordinate frame, local sound, etc.

[0200] refer to Fig.25 and Fig.26It is also possible that more than one user interacts with the server in the same session. In this example, the first user 14.1 and the second user 14.2 are joined together by a third user 14.3 and a third XR device 12.3. Each XR device 12.1, 12.2 and 12.3 starts generating its own map, i.e., the map Figure 1 ,land Figure 2 peacefully Figure 3 As XR devices 12.1, 12.2 and 12.3 continue to develop Figure 1 , 2 and 3, the map is incrementally uploaded to the server 20. The server 20 merges the Figure 1 , 2 and 3 to form a canonical map. The canonical map is then sent from the server 20 to each of the XR devices 12.1, 12.2 and 12.3.

[0201] Multiple norm maps and keyframes for norm map filtering and for positioning

[0202] Fig. 27 A plurality of canonical maps 120 are shown stored in a data store on a computer readable medium of the server 20. Each canonical map 120 has a plurality of canonical map identifiers associated therewith. These canonical map identifiers include regions on the earth represented by ranges of longitude and latitude. The canonical map identifiers also include global feature strings.

[0203] Since the specification map 120 may exist on the surface of the earth, the specification map 120 is geographically illustrated in a two-dimensional manner. The longitude and latitude are different from each other to uniquely identify the specification map. The server 20 has a location detector that receives location data including longitude and latitude, Wi-Fi fingerprint, and features of an image taken by the corresponding viewing device 12.1 or 12.2. The location detector then stores a location identifier. Not all viewing devices provide longitude and latitude as location identifiers to the server 20. However, viewing devices typically include Wi-Fi fingerprints and image features as location identifiers.

[0204] Fig.28 A canonical map 120 is shown and a series of filters including a neighboring area filter 300, a Wi-Fi fingerprint filter 302, and a keyframe filter 304 that the server 20 executes to determine one or more maps sent to a viewing device, followed by positioning 306 on the viewing device as described above.

[0205] The specification map 120 can have many specification maps, such as thousands of specification maps. In some embodiments, due to bandwidth limitations, it is impossible to send all specification maps to viewing devices.

[0206] In some embodiments, the adjacent area filter 300 may select at least one matching canonical map 120 covering longitude and latitude, which includes the longitude and latitude of the location identifier received from the viewing device, as long as there is at least one map at that longitude and latitude. In some embodiments, the adjacent area filter 300 further selects at least one adjacent canonical map covering longitude and latitude adjacent to the matching canonical map. In some embodiments, the adjacent area filter 300 may select multiple matching canonical maps and multiple adjacent canonical maps. The adjacent area filter 300 may, for example, reduce the number of canonical maps from thousands to, for example, 130 to form a first filtering selection. In some embodiments, the adjacent area filter 300 may be configured to reduce the number of canonical maps 120. In other words, the number of canonical maps entering the adjacent area filter 300 may be greater than the number of canonical maps leaving the adjacent area filter 300.

[0207] In some embodiments, the Wi-Fi fingerprint filter 302 forms a second filter. The Wi-Fi fingerprint filter 302 determines the latitude and longitude based on the Wi-Fi fingerprint received from the viewing device as part of the location identifier. The Wi-Fi fingerprint filter 302 then compares the latitude and longitude from the Wi-Fi fingerprint filter 302 with the latitude and longitude of the normative map 120 to determine one or more normative maps, which form a second filter selection within the first filter selection established by the adjacent area filter 300. The second filter selection can, for example, reduce the number of normative maps to 50 normative maps to form a second selection of normative maps, which is based on the first filter selection outside the second filter selection and does not include any normative maps. Therefore, in this example, the first filter selection includes 130 normative maps, and the second filter selection includes 50 normative maps of the 130 normative maps, but does not include the other 80 normative maps of the 130 normative maps.

[0208] The keyframe filter 304 further reduces the number of canonical maps to, for example, five canonical maps. The keyframe filter 304 can be a computer vision filter that compares the data represented by the image captured by the viewing device with the data representing the canonical map 120. The keyframe filter 304 thus provides a third filtered selection as an output that includes only five canonical maps out of the 50 canonical maps identified after the second filtered selection. The map transmitter 122 then sends one or more canonical maps based on the third filtered selection to the viewing device. In this example and in some embodiments, the viewing device receives five canonical maps and attempts to locate all five canonical maps at 306.

[0209] Figures 29A-29CA frame embedding generator 308 is shown, which in some embodiments may be used within the server 20 (or within one of the viewing devices 12.1 and 12.2) to generate a unique and reduced data representation of an image from an initial size (e.g., 76,800 bytes) to a final size (e.g., 256 bytes / one bit). In some embodiments, the frame embedding generator may be configured to generate a data representation of an image, which may be a keyframe or a frame. In some embodiments, the frame embedding generator 308 may be configured to convert an image at a particular position and orientation into a unique 256-byte string of digits. An image 320 captured by a viewing device may be processed by the viewing device to detect features 324 in the image 320 (e.g., using a deep sparse feature method), and each feature 322 may be represented by a string 326 of 32 digits. The features 322 are collectively shown by a box 310. For example, there may be n=100 features, and each feature is represented by a string of 32 digits.

[0210] In some embodiments, the frame embedding generator 308 may include a multilayer perceptron unit 312 and a maximum (largest) pooling unit 314. In some embodiments, the multilayer perceptron (MLP) unit 312 may include a multilayer perceptron, which may include a neural network. In some embodiments, the MLP unit 312 may be configured to perform matrix multiplication. The multilayer perceptron unit 312 receives a plurality of features 322 of the image 320 and converts each feature into a corresponding string of 256 numbers. In this example, there may be 100 features, and each feature is represented by a byte size of 256 numbers. In this example, a matrix with 100 horizontal rows and 256 vertical columns may be created. Each row has a series of 256 numbers that are of different sizes, some of which are smaller and others are larger. In some embodiments, the output of the MLP may be an n×256 matrix 328, where n represents the number of features extracted from the image.

[0211] In some embodiments, the MLP may include a neural network that may have a training phase (e.g., for determining model parameters) and a use phase. Fig.29DAs shown in , an MLP can be trained. The input training data may include data in groups of three, the group of three including 1) a query image, 2) a positive sample, and 3) a negative sample. The query image can be considered a reference image. The positive sample may include an image similar to the query image. For example, in some embodiments, the similarity may be an object that is the same in both the query and positive sample images but viewed from different angles. In some embodiments, the similarity may be an object that is the same in both the query image and the positive sample image but has an object offset (e.g., left, right, top, bottom) relative to the other image. The negative sample may include an image that is not similar to the query image. For example, in some embodiments, the dissimilarity may be an object in the query image but not in the sample image. In some embodiments, the dissimilarity may be having a large portion of the object in the query image (e.g., >50% or >75%), but only having a small portion of the same object in the negative sample (e.g., <10%, 1%). In some embodiments, features (e.g., sparse features) may be extracted from images in the input training data and may be converted into descriptors. In some embodiments, as described in U.S. Patent Application 16 / 190,948, a deep sparse feature (DSF) process may be used to generate a descriptor (e.g., a DSF descriptor, a feature descriptor). In some embodiments, the DSF descriptor is n×32 dimensional. The descriptor may then be passed through a model / MLP to create a 256-byte output. In some embodiments, the model / MLP may be Figures 29A-29C The 256-byte output from the MLP model can then be sent to a triple boundary loss module (and can only be used during the training phase, not during the use phase of the MLP neural network). In some embodiments, the triple boundary loss module can be configured to minimize the difference between the 256 bytes output from the query image and the 256 bytes output from the positive samples, and to maximize the difference between the 256 bytes output from the query image and the 256 bytes output from the negative samples. In some embodiments, the training phase can include feeding multiple triplet input images into a learning process (e.g., an MLP model) to determine model parameters.

[0212] The max pooling unit 314 analyzes each column to determine the maximum number in the corresponding column. The max pooling unit 314 combines the maximum values ​​of the numbers in each column into a global feature string 316 having 256 numbers. The global feature string 316 is a relatively small number that takes up relatively little memory and is easy to search. Therefore, the image can be searched without analyzing each raw frame from the camera, and it is also cheaper to store 256 bytes instead of a complete frame.

[0213] In some embodiments, Fig.28The keyframe filter 304 in the canonical map 120 may be based on the global feature string 316 and the image captured by the viewing device (e.g., which may be the user's local tracking location). Figure 1 The global feature string 316 of the image of the part) is used to filter the specification map 120. Therefore, Fig. 27 Each of the canonical maps 120 in the canonical map has one or more global feature strings 316 associated therewith. In some embodiments, the global feature strings 316 may be acquired when a viewing device submits an image or feature details to the server 20 and the server 20 processes the image or feature details to generate a global feature string 316 for the canonical map 120. In some embodiments, the server 20 may receive feature details of a live / new / current image captured by the viewing device, the server 20 may generate a global feature string 316 for the live image, and filter the canonical map 120 based on the live global feature string 316. In some embodiments, the global feature strings may be generated on a local viewing device. In some embodiments, the global feature strings may be generated remotely, for example on a cloud or server 20. In some embodiments, the viewing device may capture one or more images, perform Fig.29C The process shown in and appending the global feature string 316 to the local tracking map. In some embodiments, the global feature string 316 can be a data representation of a frame or key frame in a local tracking map or a canonical map on a local device or cloud.

[0214] In some embodiments, the server 20 may send the filtered canonical map to the viewing device along with the global feature string 316 associated with the filtered canonical map. In some embodiments, when the viewing device locates its tracking map as the canonical map, it may do so by matching the global feature string 316 of the local tracking map with the global feature string of the canonical map.

[0215] Ranking and merging maps

[0216] Described herein are methods and apparatus for ranking and merging multiple environment maps (e.g., canonical maps) in an X-reality (XR) system. In order to provide a realistic XR experience to a user, the XR system must understand the user's physical environment in order to correctly associate the positions of virtual objects relative to real objects. Information about the user's actual environment can be obtained from an environment map at the user's location. The environment map can be created based on image and depth information collected using sensors that are part of an XR device worn by a user of the XR system. The environment map can be constructed using information collected by sensors worn by users of the system at various locations and times. Each environment map can include map construction points, each of which is associated with a three-dimensional (3D) world reconstruction of real objects in the environment.

[0217] The inventors have recognized and appreciated that an XR system can provide an enhanced XR experience to multiple users sharing the same world including real and / or virtual content by enabling efficient sharing of environmental maps of the real / physical world collected by multiple users, whether the users are present in the world at the same time or at different times. However, there are significant challenges in providing such a system. Such a system may store multiple maps generated by multiple users and / or the system may store multiple maps generated at different times. For operations that may be performed using previously generated maps (such as, for example, positioning as described above), a significant amount of processing may be required to identify the relevant environmental map of the same world (e.g., the same real-world location) from all environmental maps collected by the XR system. In some embodiments, such as in Figure 8-26 In the embodiments described in , there may be only a small number of maps of the environment that the device can access, for example, for positioning. In some embodiments, such as for example above with respect to Figure 27-29C As described, there may be a large number of maps of the environment accessible to the device. The inventors have recognized and appreciated the need to quickly and accurately map all possible maps of the environment (such as, for example, Fig.28 A technique for ranking the relevance of environmental maps in a universe of all canonical maps 120 in the user's environment. Highly ranked maps can then be selected for further processing, such as rendering virtual objects on a user's display so that they interact realistically with the physical world around the user, or merging the map data collected by the user with stored maps to create a larger or more accurate map.

[0218] In some embodiments, stored maps relevant to the user's task at a location in the physical world may be identified by filtering the stored maps based on a plurality of criteria. Figure 27-29C As described, these criteria may indicate a comparison of a tracking map generated by a wearable device of the user in the location with candidate environment maps stored in a database. The comparison may be performed based on metadata associated with the map, such as a Wi-Fi fingerprint detected by the device generating the map and / or a set of BSSIDs to which the device was connected while forming the map. The comparison may also be performed based on compressed or uncompressed content of the map. Such as, for example Figures 29A-29C As described in , the comparison based on the compressed representation can be performed by comparing vectors calculated from the map content. For example, the comparison based on the uncompressed map can be performed by locating the tracking map within the stored map, or vice versa. Multiple comparisons can be performed in sequence based on the computation time required to reduce the number of candidate maps to be considered, where comparisons involving less computation will be performed earlier in the sequence than other comparisons requiring more computation.

[0219] The technology described herein can be used with or alone many types of devices, including wearable or portable devices with limited computing resources that provide XR scenes, and for many types of scenes. In some embodiments, the technology can be implemented by one or more services that form part of an XR system.

[0220] Fig.30 and Fig.31 Such a scenario is shown. For illustration purposes, an AR system is used as an example of an XR system. Figure 32-3 5 shows an exemplary AR system including one or more processors, memory, sensors, and user interfaces that can operate according to the techniques described herein.

[0221] refer to Fig.30 , depicting an outdoor AR scene 354 in which the user of the AR technology sees a park-like setting 356 of the physical world, which features people, trees, buildings in the background, and a concrete platform 358. In addition to these items, the user of the AR technology also perceives that they "see" a robot statue 357 standing on the physical world concrete platform 358, and a flying cartoon-like avatar character 352 that appears to be the head of a bumblebee, even though these elements (e.g., avatar character 352 and robot statue 357) do not exist in the physical world. Due to the extreme complexity of human visual perception and the nervous system, it is challenging to produce an AR technology that promotes a comfortable, natural feeling, rich presentation of virtual image elements among other virtual or physical world image elements.

[0222] Such an AR scene can be implemented by a system including a world reconstruction component that can build and update a digital representation of the physical world surface around the user. This representation can be used for occlusion rendering, placement of virtual objects, and for virtual character path planning and navigation in physics-based interactions, or for other operations that use information about the physical world. Fig.31 Depicted is another example of an indoor AR scene 400 showing an exemplary world reconstruction use case including visual occlusion 402 , physics-based interaction 404 , and environmental reasoning 406 , in accordance with some embodiments.

[0223] The exemplary scene 400 is a living room with a wall, a bookshelf on one side of the wall, a floor lamp in the corner of the room, a floor, a sofa, and a coffee table on the floor. In addition to these physical items, users of AR technology can also perceive virtual objects, such as images on the wall behind the sofa, birds flying over the door, deer peeking from the bookshelf, and decorations in the form of windmills placed on the coffee table. For images on the wall, AR technology requires not only information about the surface of the wall, but also information about objects and surfaces in the room (such as the shape of the lamp), which will block the image to correctly render the virtual object. For flying birds, AR technology requires information about all objects and surfaces around the room in order to render the birds with realistic physical effects to avoid objects and surfaces or to avoid rebounding when the birds collide. For deer, AR technology requires information about surfaces (such as floors or coffee tables) to calculate the placement of the deer. For the windmill, the system can recognize that it is an object separated from the table and can determine that it is movable, while the corner of the shelf or the corner of the wall can be determined to be stationary. This distinction can be used to determine which parts of the scene are used or updated in each of the various operations.

[0224] The scene can be presented to the user via a system including multiple components, including a user interface that can stimulate one or more user senses (including vision, sound and / or touch). In addition, the system can include one or more sensors that can measure parameters of the physical part of the scene, including the position and / or movement of the user within the physical part of the scene. In addition, the system can include one or more computing devices, and associated computer hardware, such as memory. These components can be integrated into a single device, or can be distributed across multiple interconnected devices. In some embodiments, some or all of these components can be integrated into a wearable device.

[0225] Fig.32 An AR system 502 is depicted that is configured to provide an experience of AR content that interacts with a physical world 506 in accordance with some embodiments. The AR system 502 may include a display 508. In the illustrated embodiment, the display 508 may be worn by a user as part of a headset so that the user may wear the display over their eyes like a pair of goggles or glasses. At least a portion of the display may be transparent so that the user may observe a see-through reality 510. The see-through reality 510 may correspond to a portion of the physical world 506 that is within a current viewpoint of the AR system 502, which may correspond to the user's viewpoint if the user wears a headset that incorporates a display and sensors of the AR system to obtain information about the physical world.

[0226] AR content may also be presented on display 508, overlaid on see-through reality 510. To provide accurate interaction between AR content and see-through reality 510 on display 508, AR system 502 may include sensors 522 configured to capture information about physical world 506.

[0227] Sensors 522 may include one or more depth sensors that output depth maps 512. Each depth map 512 may have a plurality of pixels, each of which may represent a distance from a surface in the physical world 506 in a particular direction relative to the depth sensor. Raw depth data may come from the depth sensor to create a depth map. The depth map may be updated as fast as the depth sensor can form a new image, which may be hundreds or thousands of times per second. However, the data may be noisy and incomplete, and have holes shown as black pixels on the depth map shown.

[0228] The system may include other sensors, such as image sensors. Image sensors may acquire monocular or stereo information, which may be processed to represent the physical world in other ways. For example, images may be processed in world reconstruction component 516 to create a mesh that represents connected parts of objects in the physical world. Metadata about such objects, including, for example, color and surface texture, may similarly be acquired using sensors and stored as part of the world reconstruction.

[0229] The system may also obtain information about the user's head pose relative to the physical world. In some embodiments, the sensor 522 may include an inertial measurement unit that can be used to calculate and / or determine the head pose 514. The head pose 514 for the depth map may indicate, for example, the current viewpoint of the sensor capturing the depth map with six degrees of freedom (6DoF), but the headset 514 may be used for other purposes, such as associating image information with a specific part of the physical world or associating the position of a display worn on the user's head with the physical world. In some embodiments, the head pose information may be derived in other ways than an IMU (such as analyzing objects in the image).

[0230] The world reconstruction component 516 may receive the depth map 512 and head pose 514 and any other data from the sensor and integrate the data into the reconstruction 518. The reconstruction 518 may be more complete and less noisy than the sensor data. The world reconstruction component 516 may update the reconstruction 518 using spatial and temporal averaging of sensor data from multiple viewpoints over time.

[0231] Reconstruction 518 may include a representation of the physical world in one or more data formats, including, for example, voxels, grids, planes, etc. Different formats may represent alternative representations of the same portion of the physical world or may represent different portions of the physical world. In the example shown, on the left side of reconstruction 518, portions of the physical world are presented as global surfaces; on the right side of reconstruction 518, portions of the physical world are presented as grids.

[0232] The reconstruction 518 can be used as a map or processed into a map that provides information about the portion of the physical world represented by the reconstruction. The map can indicate the location of key points and / or structures (such as edges and / or planes). In addition, the map can include information about the user's posture, from which the user can observe points or structures, which in some embodiments can be referred to as keyframes or key bindings. In the illustrated embodiment, the map represents the portion of the user in the physical world where a single wearable device exists. In conjunction with the content that characterizes that portion of the physical world, the map can include metadata. The metadata can, for example, indicate the time when the sensor information used to form the map was captured. Alternatively or in addition, the metadata can indicate the location of the sensor when the information used to form the map was captured. The location can be directly represented, such as using information from a GPS chip, or indirectly represented, such as using a Wi-Fi signature that indicates the strength of the signal received from one or more wireless access points while collecting sensor data, and / or using the BSSID of the wireless access point to which the user device is connected while collecting sensor data.

[0233] Reconstruction 518 can be used for AR functions, such as generating a surface representation of the physical world for occlusion handling or physics-based processing. The surface representation may change as the user moves or objects in the real world change. Aspects of reconstruction 518 can be used, for example, by a component 520 that generates a changing global surface representation in world coordinates, which can be used by other components.

[0234] AR content can be generated based on this information, such as by an AR application 504. The AR application 504 can be, for example, a game program that performs one or more functions, such as visual occlusion, physics-based interaction, and environmental reasoning, based on information about the physical world. It can perform these functions by querying data in different formats from the reconstruction 518 generated by the world reconstruction component 516. In some embodiments, component 520 can be configured to output updates when the representation in the focus area of ​​the physical world changes. For example, the focus area can be set to approximate a portion of the physical world near the system user, such as a portion within the user's field of view, or projected (predicted / determined) as entering the user's field of view.

[0235] The AR application 504 can use this information to generate and update the AR content. The virtual portion of the AR content can be presented on the display 508 in combination with the see-through reality 510, thereby creating a realistic user experience.

[0236] In some embodiments, an AR experience may be provided to a user via a wearable display system. Fig.33 An example of a wearable display system 580 (hereinafter referred to as "system 580") is shown. System 580 includes a head mounted display device 562 (hereinafter referred to as "display device 562"), and various mechanical and electronic modules and systems that support the functions of display device 562. Display device 562 can be coupled to a frame 564, which can be worn by a display system user or viewer 560 (hereinafter referred to as "user 560") and is configured to position display device 562 in front of the eyes of user 560. According to various embodiments, display device 562 can display sequentially. Display device 562 can be monocular or binocular. In some embodiments, display device 562 can be Fig.32 An example of display 508 in FIG.

[0237] In some embodiments, a speaker 566 is coupled to the frame 564 and positioned near the ear canal of the user 560. In some embodiments, another speaker, not shown, is positioned near the other ear canal of the user 560 to provide stereo / plastic sound control. The display device 562 is operably coupled to a local data processing module 570, such as by a wired conductor or wireless connection 568, which can be mounted in a variety of configurations, such as fixedly attached to the frame 564, fixedly attached to a helmet or hat worn by the user 560, embedded in headphones, or otherwise removably attached to the user 560 (e.g., in a backpack configuration, in a belt-coupled configuration).

[0238] The local data processing module 570 may include a processor and digital storage such as non-volatile memory (e.g., flash memory), both of which may be used to facilitate processing, caching, and storage of data. The data may include: a) data captured from a sensor (e.g., which may be operably coupled to the frame 564) or otherwise attached to the user 560, such as an image capture device (such as a camera), a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a radio device, and / or a gyroscope; and / or b) data acquired and / or processed using the remote processing module 572 and / or remote data repository 574, and possibly transferred to the display device 562 after such processing or acquisition.

[0239] The local data processing module 570 may be operably coupled to the remote processing module 572 and the remote data repository 574, respectively, through communication links 576, 578 (such as via wired or wireless communication links), so that these remote modules 572, 574 are operably coupled to each other and can be used as resources for the local data processing module 570. In some embodiments, Fig.32 The world reconstruction component 516 in can be implemented at least in part in the local data processing module 570. For example, the local data processing module 570 can be configured to execute computer-executable instructions to generate a physical world representation based at least in part on at least a portion of the data.

[0240] In some embodiments, processing can be distributed across local and remote processors. For example, for the world reconstruction component 516, local processing can be used to construct a map (e.g., a tracking map) on the user device based on sensor data collected using sensors on the user device. Such maps can be used by applications on the user device. In addition, previously created environmental maps (e.g., world maps and / or specification maps) can be stored in a remote data repository 574. In the case where a suitable stored environmental map is available, it can be used instead of a tracking map or reused outside of a tracking map. The environmental map may be loaded onto the user device to allow the user device to render virtual content without delays associated with scanning locations and building tracking maps based on sensor data obtained during scanning. Alternatively or in addition, the tracking map can be merged with previously stored environmental maps to expand or improve the quality of those environmental maps. The processing of determining whether a suitable previously created environmental map is available and / or merging the tracking map with one or more stored environmental maps can be completed in a local data processing module 570 or a remote processing module 572.

[0241] In some embodiments, the local data processing module 570 may include one or more processors (e.g., a graphics processing unit (GPU)) configured to analyze and process data and / or image information. In some embodiments, the local data processing module 570 may include a single processor (e.g., a single-core or multi-core ARM processor), which will limit the computational budget of the local data processing module 570, but enable a smaller device. In some embodiments, the world reconstruction component 516 can use a computational budget less than a single Advanced RISC Machine (ARM) core to generate a physical world representation in real time on a non-predefined space, so that the remaining computational budget of the single ARM core can be accessed for other uses, such as, for example, extracting a mesh.

[0242] In some embodiments, the remote data repository 574 may include a digital data storage facility that may be available via the Internet or other networking configuration in a "cloud" resource configuration. In some embodiments, all data is stored and all computations are performed in the local data processing module 570, allowing for fully autonomous use from a remote module. In some embodiments, all data is stored and all or most computations are performed in the remote data repository 574, allowing for smaller devices. For example, a world reconstruction may be stored in whole or in part in this repository 574.

[0243] In embodiments where the data is stored remotely and can be accessed over a network, the data can be shared by multiple users of the augmented reality system. For example, a user device can upload their tracking map, which can be stored in a database of an environmental map. In some embodiments, the tracking map upload occurs at the end of a session. In some embodiments, the tracking map upload can occur continuously, semi-continuously, or intermittently at a predefined time, after a predefined time period from a previous upload, or when triggered by an event. Regardless of whether it is based on data from the user device or any other user device, the tracking map uploaded by any user device can be used to expand or improve the previously stored environmental map. Similarly, the environmental map downloaded to the user device can be based on data from the user device or any other user device. In this way, users can easily obtain high-quality environmental maps to improve their experience in the AR system.

[0244] In some embodiments, the local data processing module 570 is operably coupled to a battery 582. In some embodiments, the battery 582 is a removable power source, such as above a counter battery. In other embodiments, the battery 582 is a lithium-ion battery. In some embodiments, the battery 582 includes both an internal lithium-ion battery that can be charged by the user 560 during the non-operating time of the system 580, and a removable battery so that the user 560 can operate the system 580 for a longer period of time without having to connect to a power source to charge the lithium-ion battery, or without having to shut down the system 580 to replace the battery.

[0245] Fig.34AA user 530 is shown wearing an AR display system that renders AR content as the user 530 moves through a physical world environment 532 (hereinafter referred to as "environment 532"). The user 530 places the AR display system at a location 534, and the AR display system records environmental information of a navigable world (e.g., digital representations of real objects in the physical world, which can be stored and updated as the real objects in the physical world change) relative to the location 534 (such as a gesture associated with a map-building feature or directional audio input). The location 534 is aggregated to a data input 536 and processed by at least a navigable world module 538, which can, for example, be processed by a navigable world module 538. Fig.33 The traversable world module 538 may include a world reconstruction component 516. Fig.35B Depicted is the component architecture of a PW according to some embodiments.

[0246] The traversable world module 538 determines, at least in part, where and how the AR content 540 as determined from the data input 536 can be placed in the physical world. The AR content is "placed" in the physical world by presenting both the physical world presentation and the AR content via a user interface, the AR content is rendered as if interacting with objects in the physical world, and the objects in the physical world are presented as if the AR content obscures the user's view of these objects when appropriate. In some embodiments, the AR content can be placed by appropriately selecting a portion of a fixed element 542 (e.g., a table) from a reconstruction (e.g., reconstruction 518) to determine the shape and position of the AR content 540. As an example, the fixed element can be a table, and the virtual content can be positioned so that it appears to be on the table. In some embodiments, the AR content can be placed within a structure in a field of view 544, which can be a current field of view or an estimated future field of view. In some embodiments, the AR content can continue to build a (e.g., grid) model 546 relative to a map of the physical world.

[0247] As depicted, fixed element 542 serves as a proxy (e.g., a digital copy) for any fixed element within the physical world that may be stored in traversable world module 538, such that user 530 may perceive content on fixed element 542 without the system having to map to fixed element 542 each time user 530 sees fixed element 542. Thus, fixed element 542 may be a mapping mesh model from a previous modeling session, or may be determined by a separate user but still stored on traversable world module 538 for future reference by multiple users. Thus, traversable world module 538 may recognize environment 532 from a previously mapped environment and display AR content without requiring user 530's device to first map all or a portion of environment 532, thereby saving computational processes and cycles and avoiding latency of any rendered AR content.

[0248] A mapping grid model 546 of the physical world may be created by the AR display system, and appropriate surfaces and metrics for interacting and displaying AR content 540 may be mapped and stored in the traversable world module 538 for future retrieval by the user 530 or other users without the need for full or partial re-mapping or modeling. In some embodiments, data input 536 is input such as geographic location, user identification, and current activity to indicate to the traversable world module 538 which of one or more fixed elements 542 is available, which AR content 540 was last placed on the fixed element 542, and whether to display the same content (such AR content is "persistent" content regardless of how the user views a particular traversable world model).

[0249] Even in embodiments where objects are considered fixed (e.g., a kitchen table), the traversable world module 538 is updated from time to time to account for the possibility of changes in the physical world. Models of fixed objects may be updated at a very low frequency. Other objects in the physical world may be moving or otherwise not considered fixed (e.g., a kitchen chair). In order to render a realistic AR scene, the AR system may update the positions of these non-fixed objects at a much higher frequency than that used to update fixed objects. In order to be able to accurately track all objects in the physical world, the AR system may obtain information from multiple sensors (including one or more image sensors).

[0250] Fig.34Bis a schematic diagram of viewing optical assembly 548 and accompanying components. In some embodiments, two eye tracking cameras 550 directed at the user's eyes 549 detect metrics of the user's eyes 549, such as eye shape, eyelid occlusion, pupil direction, and glint on the user's eyes 549. In some embodiments, one of the sensors can be a depth sensor 551, such as a time-of-flight sensor, which transmits signals to the world and detects reflections of those signals from nearby objects to determine the distance to a given object. The depth sensor can, for example, quickly determine whether an object has entered the user's field of view due to the movement of those objects or changes in the user's posture. However, information about the location of the object in the user's field of view can be collected alternatively or additionally by other sensors. Depth information can be obtained, for example, from a stereoscopic vision image sensor or a plenoptic sensor.

[0251] In some embodiments, world camera 552 records a view larger than the periphery to map environment 532 and detect inputs that can affect AR content. In some embodiments, world camera 552 and / or camera 553 can be grayscale and / or color image sensors that can output grayscale and / or color image frames at fixed time intervals. Camera 553 can further capture an image of the physical world within the user's field of view at a specific time. Even if the value of the pixels of the frame-based image sensor does not change, its pixels can be repeatedly sampled. Each of world camera 552, camera 553, and depth sensor 551 has a corresponding field of view 554, 555, and 556 to obtain information such as the user's surroundings. Fig.34A Collect data in the physical world scene of the physical world environment 532 depicted in and record the physical world scene.

[0252] Inertial measurement unit 557 can determine the motion and orientation of viewing optical assembly 548. In some embodiments, each component is operably coupled to at least one other component. For example, depth sensor 551 is operably coupled to eye tracking camera 550 to confirm the measured accommodation relative to the actual distance at which the user's eye 549 is looking.

[0253] It should be appreciated that viewing optics assembly 548 may include Fig.34B , and may include components instead of or in addition to the components shown. For example, in some embodiments, viewing optical assembly 548 may include two world cameras 552 instead of four. Alternatively or in addition, cameras 552 and 553 do not need to capture visible light images of their entire fields of view. Viewing optical assembly 548 may include other types of components. In some embodiments, viewing optical assembly 548 may include one or more dynamic vision sensors (DVS) whose pixels may asynchronously respond to relative changes in light intensity exceeding a threshold.

[0254] In some embodiments, based on the time-of-flight information, the viewing optical assembly 548 may not include a depth sensor 551. For example, in some embodiments, the viewing optical assembly 548 may include one or more plenoptic cameras whose pixels may capture light intensity and the angle of incident light, from which depth information may be determined. For example, a plenoptic camera may include an image sensor covered with a transmissive diffraction mask (TDM). Alternatively or in addition, the plenoptic camera may include an image sensor that includes angle-sensitive pixels and / or phase detection autofocus pixels (PDAF) and / or a microlens array (MLA). Such a sensor may be used as a source of depth information instead of or in addition to the depth sensor 551.

[0255] It should also be understood that Fig.34B The configuration of components in is shown as an example. The viewing optics assembly 548 may include components having any suitable configuration that may be configured to provide the user with the maximum field of view practicable for a particular set of components. For example, if the viewing optics assembly 548 has a world camera 552, the world camera may be placed in a central region of the viewing optics assembly rather than on the side.

[0256] Information from the sensor in the viewing optical component 548 can be coupled to one or more processors in the system. The processor can generate data that can be rendered so that the user perceives virtual content that interacts with objects in the physical world. The rendering can be implemented in any suitable manner, including generating image data that depicts both physical and virtual objects. In other embodiments, physical and virtual content can be depicted in a scene by modulating the opacity of the display device that the user browses in the physical world. The opacity can be controlled to create the appearance of virtual objects and also prevent the user from seeing objects in the physical world that are blocked by virtual objects. In some embodiments, when viewed through a user interface, the image data can include only virtual content, which can be modified so that the virtual content is perceived by the user as interacting realistically with the physical world (e.g., clipping content to take into account occlusion). No matter how the content is presented to the user, a model of the physical world is required so that the characteristics of virtual objects that can be affected by physical objects can be correctly calculated, including the shape, position, motion, and visibility of virtual objects. In some embodiments, the model can include a reconstruction of the physical world, such as reconstruction 518.

[0257] The model may be created from data collected from sensors on a user's wearable device. However, in some embodiments, the model may be created from data collected from multiple users, which may be aggregated in a computing device remote from all users (and the data may be in the "cloud").

[0258] The model may be created at least in part by a world reconstruction system, e.g. Fig.35A Described in more detail in Fig.32 The world reconstruction component 516 may include a perception module 660 that can generate, update, and store a representation of a portion of the physical world. In some embodiments, the perception module 660 may represent the portion of the physical world within the reconstruction range of the sensor as a plurality of voxels. Each voxel may correspond to a 3D cube of a predetermined volume in the physical world and include surface information indicating whether a surface exists in the volume represented by the voxel. Voxels may be assigned values ​​indicating whether their corresponding volumes have been determined to include the surface of the physical object, determined to be empty, or have not yet been measured with the sensor and therefore their values ​​are unknown. It should be understood that there is no need to explicitly store values ​​indicating voxels that are determined to be empty or unknown, as the values ​​of the voxels may be stored in the computer memory in any suitable manner, including not storing information for voxels that are determined to be empty or unknown.

[0259] In addition to generating information for the persistent world representation, the perception module 660 can also identify and output indications of changes in the area surrounding the user of the AR system. Such indications of changes can trigger updates to volumetric data stored as part of the persistent world, or trigger other functions, such as triggering the generation of AR content to update the trigger component 604 of the AR content.

[0260] In some embodiments, the perception module 660 can identify changes based on a signed distance function (SDF) model. The perception module 660 can be configured to receive sensor data such as, for example, a depth map 660a and a head pose 660b, and then fuse the sensor data into an SDF model 660c. The depth map 660a can directly provide SDF information, and the image can be processed to obtain the SDF information. The SDF information represents the distance from the sensor used to capture the information. Since those sensors can be part of a wearable unit, the SDF information can represent the physical world from the perspective of the wearable unit and therefore from the perspective of the user. The head pose 660b can enable the SDF information to be related to voxels in the physical world.

[0261] In some embodiments, the perception module 660 can generate, update, and store representations of portions of the physical world within a perception range. The perception range can be determined at least in part based on a reconstruction range of the sensor, which can be determined at least in part based on a limit of an observation range of the sensor. As a specific example, an active depth sensor operating using active IR pulses can reliably operate within a range of distances, thereby creating an observation range of the sensor, which can be from a few centimeters or tens of centimeters to several meters.

[0262] The world reconstruction component 516 may include additional modules that may interact with the perception module 660. In some embodiments, the persistent world module 662 may receive a representation of the physical world based on data acquired by the perception module 660. The persistent world module 662 may also include representations of the physical world in various formats. In some embodiments, the representation of the physical world may be an image, features from an image, descriptors of features, keyframes / keybindings, persistent poses, PCFs, and / or global feature strings. For example, volume metadata 662b such as voxels may be stored, as well as meshes 662c and planes 662d. In some embodiments, other information may be saved, such as a depth map.

[0263] In some embodiments, the perception module 660 may include modules that generate representations for the physical world in various formats, including, for example, grid 660d, plane, and semantic 660e. These modules may generate representations based on data within the perception range of one or more sensors when generating the representation, as well as data captured at previous times and information in the persistent world module 662. In some embodiments, these components may operate with respect to depth information captured using a depth sensor. However, the AR system may include a visual sensor and may generate such a representation by analyzing monocular or binocular visual information.

[0264] In some embodiments, these modules may operate on regions of the physical world. When perception module 660 detects a change in the physical world in a sub-region of the physical world, those modules may be triggered to update the sub-region of the physical world. For example, such a change may be detected by detecting a new surface or other criteria in SDF model 660c (e.g., changing the value of a sufficient number of voxels representing the sub-region).

[0265] The world reconstruction component 516 may include a component 664 that may receive a representation of the physical world from the perception module 660. Information about the physical world may be extracted by these components based on, for example, a usage request from an application. In some embodiments, information may be pushed to the usage component, such as via an indication of a change in a pre-identified area or a change in the representation of the physical world within the perception range. The component 664 may include, for example, a game program and other components that perform processing for visual occlusion, physics-based interaction, and environmental reasoning.

[0266] In response to a query from component 664, perception module 660 can send representations for the physical world in one or more formats. For example, when component 664 indicates that the use is for visual occlusion or physics-based interaction, perception module 660 can send representations of surfaces. When component 664 indicates that the use is for environmental reasoning, perception module 660 can send meshes, planes, and semantics of the physical world.

[0267] In some embodiments, perception module 660 may include a component that formats information to provide component 664. An example of such a component may be raycast component 660f. Using a component (e.g., component 664), for example, information about the physical world may be queried from a particular viewpoint. Raycast component 660f may select from one or more representations of physical world data within the field of view from that viewpoint.

[0268] It should be understood from the above description that the perception module 660 or another component of the AR system can process data to create a 3D representation of a portion of the physical world. The data to be processed can be reduced by picking a portion of the 3D reconstruction volume based at least in part on the camera frustum and / or the depth image, extracting and retaining plane data, capturing, retaining, and updating the 3D reconstruction data in blocks that allow local updates while maintaining neighborhood consistency, providing occlusion data to applications generating such scenes, wherein the occlusion data is derived from a combination of one or more depth data sources, and / or performing multi-stage mesh simplification.

[0269] A world reconstruction system can integrate sensor data from multiple viewpoints of the physical world over time. As a device including a sensor moves, the pose (e.g., position and orientation) of the sensor can be tracked. Since the sensor's frame pose and its relationship to other poses are known, each of these multiple viewpoints of the physical world can be fused together into a single combined reconstruction. By using spatial and temporal averaging (i.e., averaging the data from multiple viewpoints over time) or any other suitable method, the reconstruction can be more complete and less noisy than the original sensor data.

[0270] The reconstruction may contain data of varying complexity, including, for example, raw data (such as real-time depth data), fused volumetric data (such as voxels), and computed data (such as meshes).

[0271] A navigable world (PW) (e.g., PW 538) may include a map of an environment created at least in part based on data reconstructed by a world reconstruction system of AR devices worn by multiple users. During a session of an AR experience, each AR device (such as the one described above in conjunction with the AR device) may be used to create a map of the environment. Fig.33 The wearable device described herein can create a tracking map (TM). In some embodiments, the tracking map can include a sparse map of the head pose of the AR device's scanning environment and information about objects detected within the environment. The AR device can send the tracking map to the cloud to, for example, merge with an environment map selected from environment maps stored in the cloud. In some embodiments, the selected environment map can be sent from the cloud to the AR device for merging.

[0272] The environment map may provide information about the portion of the physical world represented by the reconstruction data that is processed to create the corresponding map. Fig.36 An exemplary environment map 700 according to some embodiments is depicted. The environment map 700 may include a floor plan 706 of reconstructed physical objects in a corresponding physical world represented by points 702. The environment map 700 may also include a tracking route 710 along which a device collecting data about the points 702 travels. The tracking route 710 may include a key binding 704, which may represent a location where an AR device uploads reconstruction data and associated metadata (such as head pose and / or image data). In some embodiments, a key binding may also be referred to as a keyframe. In the example shown, the environment map includes two tracking routes 710a and 710b, which may indicate that the environment map 700 is created by merging two tracking maps: one is created along the tracking route 710a; the other is created along the tracking route 710b. In some embodiments, the two tracking maps may be created by two different AR devices at the same time, by two different AR devices at different times, or by the same AR device at different times. In some embodiments, the two tracking maps may be created by one AR device and uploaded to the cloud through two network access points, each of which may be accessible within a specified area.

[0273] The environment map may be in any of a variety of formats depending on, for example, where the environment map is stored, including, for example, local storage of the AR device and remote storage. For example, the map in the remote storage may have a higher resolution than the map in the local storage. In order to send the higher resolution map from the remote storage to the local storage, the map may be downsampled or otherwise converted to an appropriate format. In some embodiments, slices or portions of the high resolution map from the remote storage are sent to the local storage, where the slices or portions are not downsampled.

[0274] When a new tracking map is created, a database of environment maps may be updated. To determine which of a potentially large number of environment maps in the database is to be updated, the updating may include effectively selecting one or more environment maps stored in the database that are related to the new tracking map. The selected one or more environment maps may be ranked by relevance, and one or more of the highest ranked maps may be selected for processing to merge the higher ranked selected environment maps with the new tracking map to create one or more updated environment maps.

[0275] Fig.37An AR system 800 configured to rank and merge one or more maps of an environment is depicted in accordance with some embodiments. The AR system may include a traversable world model 802 for an AR device. Information to populate the traversable world model 802 may come from sensors on the AR device, which may include computer executable instructions stored in a processor 804 (e.g., a local data processing module 570), which may perform some or all of the processing to convert the sensor data into a map. Such a map may be a tracking map, as the tracking map may be constructed as the AR device is operating in an area while collecting sensor data. Figure 1 In addition, regional attributes may be provided to indicate the area represented by the tracking map. These regional attributes may be geographic location identifiers, such as coordinates expressed as latitude and longitude, or IDs used by the AR system to represent locations. Alternatively or in addition, regional attributes may be measured characteristics that have a high probability of being unique to the area. Regional attributes may, for example, be derived from parameters of wireless networks detected in the area. In some embodiments, regional attributes may be associated with a unique address of an access point that the AR system is nearby and / or connected to. For example, regional attributes may be associated with a MAC address or basic service set identifier (BSSID) of a 5G base station / router, a Wi-Fi router, or the like.

[0276] exist Fig.37 In the example of FIG. 8 , the tracking map can be merged with other maps of the environment. The map ranking portion 806 receives the tracking map from the device PW 802 and communicates with the map database 808 to select and rank the environment map selected from the map database 808. The selected map with a higher ranking is sent to the map merging portion 810.

[0277] The map merging portion 810 may perform a merging process on the maps sent from the map ranking portion 806. The merging process may require merging the tracking map with some or all of the ranking maps and sending the new merged map to the traversable world model 812. The traversable world model 812 may be a cloud model that may be shared by multiple AR devices. The traversable world model 812 may store or otherwise access those environment maps in the map database 808. In some embodiments, when a previously calculated environment map is updated, a previous version of the map may be deleted to delete outdated maps from the database. In some embodiments, when a previously calculated environment map is updated, a previous version of the map may be archived, thereby enabling the acquisition / viewing of a previous version of the environment. In some embodiments, permissions may be set so that only an AR system with certain read / write access rights can trigger the deletion / archiving of a previous version of the map.

[0278] These environment maps created from tracking maps provided by one or more AR devices / systems can be accessed by AR devices in the AR system. The map ranking portion 806 can also be used to provide environment maps to AR devices. An AR device can send a message requesting an environment map for its current location, and the map ranking portion 806 can be used to select and rank environment maps relevant to the requesting device.

[0279] In some embodiments, the AR system 800 may include a downsampling portion 814 configured to receive a merged map from the cloud PW 812. The merged map received from the cloud PW 812 may be in a storage format for the cloud, which may include dense information. The downsampling portion 814 may be configured to downsample the cloud format map to a format suitable for storage on the AR device. The device format map may be sparse to accommodate the limited local computing power and storage space of the AR device.

[0280] U.S. Provisional Patent Application Serial No. 62 / 812,935, entitled "Merging a Plurality of Individually Mapped Environments," describes an exemplary method and apparatus for merging environment maps. A copy of the filing version of U.S. Application Serial No. 62 / 812,935 is attached as an attachment and is incorporated herein by reference in its entirety for all purposes. The techniques described in that application can be used to merge environment maps.

[0281] Fig.38 9 is a flow chart illustrating a method 900 of selecting one or more ranked environment maps according to some embodiments. In the illustrated embodiment, the ranking is performed on the AR device of the user who is creating the tracking map. Thus, the tracking map can be used to rank the environment maps. In embodiments where the tracking map is not available, some or all of the selection and ranking of the environment maps that do not explicitly rely on the tracking map can be used.

[0282] Method 900 may begin with act 902, where a set of maps in a database of environmental maps (sometimes referred to as canonical maps) located near the location where the tracking map is formed may be accessed and then filtered for ranking. Additionally, at act 902, at least one area attribute of the area in which the user's AR device is operating is determined. In a scenario where the user's AR device is constructing a tracking map, the area attribute may correspond to the area on which the tracking map is created. As a specific example, the area attribute may be calculated based on a received signal from an access point to a computer network while the AR device is calculating the tracking map.

[0283] Fig.39An exemplary map ranking portion 806 of the AR system 800 is depicted in accordance with some embodiments. The map ranking portion 806 may be a cloud computing environment, as it may include a portion of the AR device and a portion of a remote computing system such as the cloud. The map ranking portion 806 may be configured to perform at least a portion of the method 900.

[0284] Fig.40A Examples of area attributes AA1-AA8 of a tracking map (TM) 1102 and environment maps CM1-CM4 in a database according to some embodiments are depicted. As shown, the environment map can be associated with multiple area attributes. The area attributes AA1-AA8 may include parameters of a wireless network detected by the AR device computing the tracking map 1102, such as a basic service set identifier (BSSID) of the network to which the AR device is connected and / or the strength of a received signal to an access point of the wireless network through, for example, a network tower 1104. The parameters of the wireless network may conform to protocols including Wi-Fi and 5G NR. In Fig. 9 In the example shown in , the region attribute is a fingerprint of the area in which the user AR device collects sensor data to form a tracking map.

[0285] Fig.40B An example of a determined geographic location 1106 of a tracking map 1102 according to some embodiments is depicted. In the example shown, the determined geographic location 1106 includes a centroid point 1110 and an area 1108 surrounding the centroid point. It should be understood that the determination of the geographic location of the present application is not limited to the format shown. The determined geographic location can have any suitable format, including, for example, different area shapes. In this example, the geographic location is determined from the area attributes using a database that associates area attributes with the geographic location. Databases are commercially available, for example, a database that associates Wi-Fi fingerprints with locations expressed as latitude and longitude and can be used for this operation.

[0286] exist Fig.38 In an embodiment, the map database containing environmental maps may also include location data for those maps, including the latitudes and longitudes covered by the maps. Processing at action 902 may require selecting a set of environmental maps from the database that cover the same latitudes and longitudes determined for the regional attributes of the tracking map.

[0287] Action 904 is a first filtering of the set of environment maps accessed in action 902. In action 902, environment maps are retained in the set based on proximity to the geographic location of the tracking map. This filtering step may be performed by comparing the latitude and longitude associated with the tracking map and the environment map in the set.

[0288] Fig.41An example of action 904 according to some embodiments is depicted. Each area attribute may have a corresponding geographic location 1202. The set of environment maps may include an environment map having at least one area attribute having a geographic location that overlaps with the determined geographic location of the tracking map. In the example shown, a set of identified environment maps includes environment maps CM1, CM2, and CM4, each of which has at least one area attribute having a geographic location that overlaps with the determined geographic location of the tracking map 1102. CM3, which is associated with area attribute AA6, is not included in the set because it is outside the determined geographic location of the tracking map.

[0289] Other filtering steps may also be performed on the group of environmental maps to reduce / rank the number of environmental maps that are ultimately processed in the group (such as for map merging or providing navigable world information to a user device). Method 900 may include filtering (action 906) the group of environmental maps based on the similarity of one or more identifiers of network access points associated with the tracking map and the environmental map of the group of environmental maps. During the formation of the map, the device that collects sensor data to generate the map may be connected to the network through a network access point (such as through Wi-Fi or a similar wireless communication protocol). The access point may be identified by a BSSID. When the user device moves through the area where the data is collected to form a map, the user device may be connected to multiple different access points. Similarly, when multiple devices provide information to form a map, the device may have been connected through different access points, so for this reason, multiple access points may also be used when forming the map. Therefore, there may be multiple access points associated with the map, and the group of access points may be an indication of the map location.

[0290] In some embodiments, filtering the set of environment maps based on similarity of one or more identifiers of the network access points may include retaining in the set of environment maps an environment map having a highest Jaccard similarity to at least one area attribute of the tracking map based on the one or more identifiers of the network access points. Fig.42 An example of action 906 according to some embodiments is depicted. In the example shown, a network identifier associated with area attribute AA7 may be determined as an identifier of tracking map 1102. The set of environment maps after action 906 includes: environment map CM2, which may have an area attribute within a higher Jaccard similarity to AA7; and environment map CM4, which also includes area attribute AA7. Environment map CM1 is not included in the set because it has the lowest Jaccard similarity to AA7.

[0291] The processing of actions 902-906 can be performed based on metadata associated with the map without actually accessing the content of the map stored in the map database. Other processing may involve accessing the content of the map. Action 908 indicates accessing the environment map retained in the subset after filtering based on the metadata. It should be understood that if subsequent operations can be performed on the accessed content, this action can be performed earlier or later in the process.

[0292] Method 900 may include filtering (act 910) a set of environment maps based on similarity of metrics representing the content of the tracking map and the environment map of the set of environment maps. The metrics representing the content of the tracking map and the environment map may include vectors of values ​​calculated from the content of the maps. The metrics may be calculated from the maps retrieved at act 908, or may be pre-calculated and stored as metadata associated with those maps. In some embodiments, filtering the set of environment maps based on similarity of metrics representing the content of the tracking map and the environment map of the set of environment maps may include retaining in the set of environment maps an environment map having a minimum vector distance between a feature vector of the tracking map and a vector representing an environment map in the set of environment maps.

[0293] Method 900 may include further filtering (action 912) a set of environment maps based on a degree of match between a portion of the tracking map and a portion of an environment map of the set of environment maps. The degree of match may be determined as part of a positioning process. As a non-limiting example, positioning may be performed by identifying critical points in the tracking map and the environment map that are sufficiently similar to the same portion of the physical world that they may represent. In some embodiments, the key points may be features, feature descriptors, keyframes / key bindings, persistent poses, and / or PCFs. Then, a set of critical points in the tracking map may be aligned to produce an optimal fit with the set of critical points in the environment map. The mean square distance between corresponding critical points may be calculated and, if below a threshold for a particular area of ​​the tracking map, used as an indication that the tracking map and the environment map represent the same area of ​​the physical world.

[0294] In some embodiments, filtering a set of environment maps based on a degree of match between a portion of a tracking map and a portion of an environment map of the set of environment maps may include: calculating a volume of the physical world represented by the tracking map, which tracking map is also represented in an environment map of the set of environment maps; and retaining in the set of environment maps an environment map having a larger calculated volume than an environment map filtered from the set. Fig.43An example of action 912 is depicted in accordance with some embodiments. In the example shown, a set of environment maps following action 912 includes environment map CM4 having regions 1402 that match regions of tracking map 1102. Environment map CM1 is not included in the set because it does not have regions that match regions of tracking map 1102.

[0295] In some embodiments, the set of environment maps may be filtered in the order of action 906, action 910, and action 912. In some embodiments, the set of environment maps may be filtered based on action 906, action 910, and action 912, which may be performed in the order of processing required to perform the filtering, from lowest to highest.

[0296] Method 900 may include loading (act 914 ) the set of environment maps and data.

[0297] In the example shown, the user database stores an area identifier indicating an area where the AR device is used. The area identifier may be an area attribute that may include parameters of a wireless network detected by the AR device during use. The map database stores multiple environment maps constructed from data provided by the AR device and associated metadata. The associated metadata may include an area identifier derived from the area identifier of the AR device providing the data, and the environment map is constructed from the data. The AR device may send a message to the PW module indicating that a new tracking map has been created or is being created. The PW module calculates an area identifier for the AR device and updates the user database based on the received parameters and / or the calculated area identifier. The PW module may also determine the area identifier associated with the AR device requesting the environment map, identify the group of environment maps from the map database based on the area identifier, filter the group of environment maps, and send a filtered set of environment maps to the AR device. In some embodiments, the PW module can filter the set of environmental maps based on one or more criteria including, for example, the geographic location of the tracking map, the similarity of one or more identifiers of network access points associated with the tracking map and the environmental maps of the set of environmental maps, the similarity of metrics representing the contents of the tracking map and the environmental maps of the set of environmental maps, and the degree of match between a portion of the tracking map and a portion of the environmental maps of the set of environmental maps.

[0298] Having thus described several aspects of some embodiments, it is to be appreciated that various alterations, modifications, and improvements will readily occur to those skilled in the art.

[0299] As an example, embodiments are described in conjunction with an augmented (AR) environment. It should be understood that some or all of the techniques described herein may be applied in an MR environment or more generally in other XR environments and VR environments.

[0300] As another example, embodiments are described in conjunction with devices such as wearable devices. It should be understood that some or all of the techniques described herein may be implemented via any suitable combination of a network (such as the cloud), a discrete application, and / or a device, network, and a discrete application.

[0301] also, Fig.38 Examples of criteria that can be used to filter candidate maps to produce a set of highly ranked maps are provided. Other criteria can be used instead of or in addition to the criteria described. For example, if multiple candidate maps have similar values ​​for a metric used to filter out less desirable maps, the characteristics of the candidate maps can be used to determine which maps are retained as candidate maps or filtered out. For example, larger or more densely populated candidate maps can be prioritized over smaller candidate maps. In some embodiments, Figure 27-28 Can describe Figure 37-39 All or part of the systems and methods described in.

[0302] Figure 51-53 An example of determining a map to use based at least in part on deep keyframe (DKF) analysis in accordance with some embodiments is described. DKF analysis can be part of a process of determining an environment map to be used by an XR device and / or determining a current head pose for a user (e.g., the position and / or orientation of the user's head) based on a TM generated by the XR device. In some embodiments, the DKF analysis employs a suitable deep learning technique to predict or otherwise determine an appropriate environment map to use by providing a keyframe of the TM as input to a deep learning architecture such as a neural network. In some embodiments, the DKF analysis is used as part of a process of ranking environment maps, also referred to as canonical maps (CMs). For example, the CMs can be ranked based on a degree of similarity of the DKF descriptors between the TM and each CM analyzed as a candidate. As in Fig.53 As shown in the example of , the DKF analysis can be part of a hierarchical (e.g., multi-layer) analysis for selecting a CM, in which geolocation filtering is performed (e.g., first) as a top-level analysis, Wi-Fi BSSID filtering is performed as a next layer, and then DKF analysis is performed as a third (e.g., final) layer to rank the CMs. The highest ranked CM or set of CMs can be provided to the XR device to be used and / or further analyzed. The ranking analysis can be performed in the cloud (e.g., using a server such as Figure 51-53 At each layer, the set of candidate CMs may be filtered (e.g., reduced) until a single map (or a subset of maps) remains after the underlying DKF-based analysis. Fig.51 The map ranking module shown in can perform the ranking and pass the results to the map merging module as described above.

[0303] U.S. Patent Application Serial No. 16 / 520,582 and U.S. Provisional Patent Application Serial No. 62 / 702,829 are incorporated herein by reference and describe further details of map sharing based on map quality, common visibility graphical map hygiene, and creation of localizable maps from camera images.

[0304] Head pose tracking recovery and reset

[0305] Fig.49 Aspects of a viewing method for recovering and / or resetting a head pose according to some embodiments are shown. In the example shown, at action 1400, the viewing device is powered on. At action 1410, in response to the power on, a new session is initiated. In some embodiments, the new session may include establishing a head pose. One or more capture devices on a head mounted frame fixed to the user's head capture the surface of the environment by first capturing an image of the environment and then determining the surface from the image. The surface data is also combined with data from a gravity sensor to establish the head pose.

[0306] At action 1420, the processor of the viewing device enters a routine for tracking head pose. As the user moves their head to determine the orientation of the head mounted frame relative to the surface, the capture device continues to capture the surface of the environment.

[0307] At action 1430, the processor determines whether the head pose has been lost. The head pose may be lost due to "edge" situations, such as too many reflective surfaces, low light, blank walls, outdoors, etc. that can result in low feature acquisition; or due to dynamic situations such as a crowd of people moving and forming part of a map. The routine at 1430 allows a certain amount of time to pass, such as 10 seconds, to allow sufficient time to determine whether the head pose has been lost. If the head pose has not been lost, the processor returns to 1420 and once again enters tracking of the head pose.

[0308] If the head pose has been lost at action 1430, the processor enters a routine to recover the head pose at 1440. If the head pose is lost due to low light, a message such as the following message will be displayed to the user via the display of the viewing device:

[0309] The system is detecting low light conditions. Please move to a better lit area.

[0310] The system will continue to monitor whether sufficient light is available and whether the head pose can be recovered. The system may alternatively determine that low texture of the surface is causing the head pose to be lost, in which case the following prompt is given to the user in the display as a suggestion to improve surface capture:

[0311] The system cannot detect enough surfaces with fine textures. Please move to an area with less rough surface textures and finer textures.

[0312] At action 1450, the processor enters a routine to determine whether head posture recovery has failed. If head posture recovery has not failed (i.e., head posture recovery has succeeded), the processor returns to action 1420 by re-entering the tracking of the head posture. If head posture recovery has failed, the processor returns to action 1410 to establish a new session. As part of the new session, all cached data is invalidated, and the head posture is re-established thereafter.

[0313] Fig.50 A schematic diagram of a machine in the exemplary form of a computer system 1900 is shown according to some embodiments, within which a set of instructions may be executed to cause the machine to perform any one or more of the methodologies discussed herein. In alternative embodiments, the machine operates as a standalone device or may be connected (e.g., networked) to other machines. Furthermore, while only a single machine is shown, the term "machine" should also be understood to include any collection of machines that individually or collectively execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

[0314] The exemplary computer system 1900 includes a processor 1902 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), or both), a main memory 1904 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM)), and a static memory 1906 (e.g., flash memory, static random access memory (SRAM), etc.) that communicate with each other via a bus 1908.

[0315] The computer system 1900 may further include a disk drive unit 1916 and a network interface device 1920 .

[0316] The disk drive unit 1916 includes a machine-readable medium 1922 on which is stored one or more sets of instructions 1924 (e.g., software) embodying any one or more of the methods or functions described herein. During execution of the software by the computer system 1900, the software may also reside, in whole or in part, within the main memory 1904 and / or within the processor 1902, the main memory 1904 and the processor 1902 also constituting machine-readable media.

[0317] Software may also be sent or received over network 18 via network interface device 1920 .

[0318] The computer system 1900 includes a driver chip 1950 for driving the projector to generate light. The driver chip 1950 includes its own data storage 1960 and its own processor 1962.

[0319] Although the machine-readable medium 1922 is shown as a single medium in the exemplary embodiment, the term "machine-readable medium" should be understood to include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) that store one or more sets of instructions. The term "machine-readable medium" should also be considered to include any medium that can store, encode, or carry a set of instructions for execution by a machine and cause the machine to perform any one or more of the methods of the present invention. Therefore, the term "machine-readable medium" should be considered to include, but is not limited to, solid-state memories, optical and magnetic media, and carrier signals.

[0320] Having thus described several aspects of some embodiments, it is to be appreciated that various alterations, modifications, and improvements will readily occur to those skilled in the art.

[0321] As an example, embodiments are described in conjunction with an augmented (AR) environment. It should be understood that some or all of the techniques described herein may be applied in an MR environment or more generally in other XR environments and VR environments.

[0322] As another example, embodiments are described in conjunction with devices such as wearable devices. It should be understood that some or all of the techniques described herein may be implemented via any suitable combination of a network (such as the cloud), a discrete application, and / or a device, network, and a discrete application.

[0323] also, Fig.38 Examples of criteria that can be used to filter candidate maps to produce a set of highly ranked maps are provided. Other criteria can be used instead of or in addition to the described criteria. For example, if multiple candidate maps have similar values ​​for a metric used to filter out less desirable maps, the characteristics of the candidate maps can be used to determine which maps are retained as candidate maps or filtered out. For example, larger or more dense candidate maps can be prioritized over smaller candidate maps.

[0324] Such changes, modifications and improvements are intended to be a part of the present disclosure and are intended to be within the spirit and scope of the present disclosure. In addition, although the advantages of the present disclosure are indicated, it should be understood that not every embodiment of the present disclosure will include each described advantage. In some cases, some embodiments may not implement any of the features described as advantageous herein. Therefore, the foregoing description and accompanying drawings are provided only as examples.

[0325] The above-mentioned embodiments of the present disclosure can be implemented in any of a variety of ways. For example, the embodiments can be implemented using hardware, software, or a combination thereof. When implemented in software, the software code can be executed on any suitable processor or processor set, whether provided in a single computer or distributed in multiple computers. Such a processor can be implemented as an integrated circuit, with one or more processors in an integrated circuit component, including commercially available integrated circuit components known in the art, with names such as CPU chips, GPU chips, microprocessors, microcontrollers, or coprocessors. In some embodiments, the processor can be implemented in a custom circuit (such as an ASIC) or in a semi-custom circuit generated by configuring a programmable logic device. As another alternative, the processor can be part of a larger circuit or semiconductor device, whether commercially available, semi-custom or custom. As a specific example, some commercially available microprocessors have multiple cores, so that one or a subset of these cores can constitute a processor. However, a circuit of any appropriate format can be used to implement the processor.

[0326] Furthermore, it should be understood that a computer may be embodied in any of a variety of forms, such as a rack-mounted computer, a desktop computer, a laptop computer, or a tablet computer. Additionally, a computer may be embedded in a device that is not generally considered a computer but has suitable processing capabilities, including a personal digital assistant (PDA), a smart phone, or any other suitable portable or fixed electronic device.

[0327] In addition, the computer may have one or more input and output devices. These devices may be used in particular for presenting a user interface. Examples of output devices that may be used to provide a user interface include a printer or display screen for visual presentation output, and a speaker or other sound generating device for auditory presentation output. Examples of input devices that may be used for a user interface include a keyboard and a pointing device, such as a mouse, a touch pad, and a digitized tablet computer. As another example, a computer may receive input information by voice recognition or other audible formats. In the illustrated embodiment, the input / output device is shown as being physically separated from the computing device. However, in some embodiments, the input and / or output device may be physically integrated into the same unit as other elements of a processor or computing device. For example, a keyboard may be implemented as a soft keyboard on a touch screen. In some embodiments, the input / output device may be completely disconnected from the computing device and functionally integrated by a wireless connection.

[0328] Such computers may be interconnected by one or more networks of any suitable form, including as a local area network or a wide area network such as an enterprise network or the Internet. Such a network may be based on any suitable technology and may operate according to any suitable protocol, and may include a wireless network, a wired network, or a fiber optic network.

[0329] In addition, the various methods or processes outlined herein may be encoded as software that can be executed on one or more processors using any of a variety of operating systems or platforms. In addition, such software may be written using any of a variety of suitable programming languages ​​and / or programming or scripting tools, and may also be compiled into executable machine language code or intermediate code that is executed on a framework or virtual machine.

[0330] In this regard, the present disclosure may be embodied as a computer-readable storage medium (or multiple computer-readable media) (e.g., a computer memory, one or more floppy disks, compact discs (CDs), optical disks, digital video discs (DVDs), tapes, flash memory, field programmable gate arrays or other semiconductor devices or other tangible computer storage media) encoded with one or more programs, which when executed on one or more computers or other processors will perform methods for implementing the various embodiments of the present disclosure discussed above. It is apparent from the foregoing examples that the computer-readable storage medium can retain information for a sufficient time to provide computer-executable instructions in a non-transient form. Such one or more computer-readable storage media may be removable so that one or more programs stored thereon can be loaded onto one or more different computers or other processors to implement various aspects of the present disclosure as described above. As used herein, the term "computer-readable storage medium" covers only computer-readable media that can be considered as an article of manufacture (i.e., an article of manufacture) or a machine. In some embodiments, the present disclosure may be embodied as a computer-readable medium other than a computer-readable storage medium, such as a propagating signal.

[0331] The term "program" or "software" is used herein in a general sense to refer to computer code or a computer executable instruction set that can be used to program a computer or other processor to implement various aspects of the present disclosure as described above. In addition, it should be understood that according to one aspect of this embodiment, one or more computer programs that perform the methods of the present disclosure when executed do not need to reside on a single computer or processor, but can be distributed in a modular manner between multiple different computers or processors to implement various aspects of the present disclosure.

[0332] Computer executable instructions can have many forms, such as program modules that are executed by one or more computers or other devices. Typically, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Typically, in various embodiments, the functionality of program modules can be combined or distributed as needed.

[0333] In addition, the data structure can be stored in a computer-readable medium in any suitable form. To simplify the description, the data structure can be shown to have fields that are related by position in the data structure. Similarly, such relationships can be achieved by allocating storage for the fields by conveying the positions in the computer-readable medium that communicate the relationships between the fields. However, any suitable mechanism can be used to establish the relationship between the information in the fields of the data structure, including by using pointers, tags, or other mechanisms that establish relationships between data elements.

[0334] Various aspects of the present disclosure may be used alone, in combination, or in various arrangements not specifically discussed in the foregoing embodiments, and therefore, are not limited in their application to the details and arrangements of components set forth in the foregoing description or shown in the accompanying drawings. For example, aspects described in one embodiment may be combined in any manner with aspects described in other embodiments.

[0335] In addition, the present disclosure may be embodied as a method, an example of which has been provided. The actions performed as part of the method may be ordered in any suitable manner. Thus, embodiments may be constructed in which the actions are performed in an order different from that shown, and even if shown as sequential actions in an illustrative embodiment, the actions may include performing some actions simultaneously.

[0336] The use of ordinal terms such as "first", "second", "third" and the like in the claims to modify claim elements does not in itself indicate any priority, precedence or order of one claim element relative to another sequential or temporal order of performing the method actions, but serves merely as a marker to distinguish one claim element having a certain name from another element having the same name (but used in ordinal numbers) to distinguish the claim elements.

[0337] In addition, the words and terms used herein are for descriptive purposes and should not be regarded as limiting. The use of "includes," "comprising," or "having," "including," "involving," and variations thereof herein is intended to encompass the items listed thereafter and their equivalents as well as other items.

Claims

1. A user-portable electronic system comprising: one or more sensors configured to capture images of one or more physical objects in a scene, the images being in a first coordinate frame; an application configured to execute computer-executable instructions to render virtual content in the scene, wherein the application displays the virtual content in a second coordinate frame different from the first coordinate frame such that display of the virtual content is independent of eye rotation within the user's head and / or deformation of the one or more sensors; and At least one processor configured to execute computer executable instructions to provide the virtual content to the application, wherein the computer executable instructions include the following instructions: determining a local coordinate frame for the virtual content based at least in part on the one or more physical objects in the scene; and Image data regarding the virtual content in the local coordinate frame is transformed into image data regarding the virtual content in the second coordinate frame.

2. An XR system, comprising: a map storage routine that stores a first map, the first map being a canonical map having a plurality of anchor points, each anchor point of the first map having a set of coordinates; a real object detection device positioned to detect the position of a real object; an anchor point identification system connected to the real object detection device to detect anchor points of a second map based on the position of the real object, each anchor point of the second map having a set of coordinates; as well as A positioning module connected to the canonical map and the second map and executable to position the second map to the canonical map by matching a first anchor point of the second map with a first anchor point of the canonical map and matching a second anchor point of the second map with a second anchor point of the canonical map.

3. An XR system, comprising: A server having: processor; a computer readable medium coupled to said processor; a plurality of specification maps on said computer readable medium; a respective authority map identifier on the computer-readable medium associated with each respective authority map, the authority map identifiers being different from one another to uniquely identify the authority map; a location detector located on the computer-readable medium and executable by the processor to receive and store a location identifier from an XR device; a first filter located on the computer-readable medium and executable by the processor to compare the location identifier to the canonical map identifier to determine one or more canonical maps forming a first filtered selection; as well as A map transmitter, located on the computer-readable medium and executable by the processor to transmit one or more of the canonical maps to the XR device based on the first filtered selection.

4. An XR system, comprising: processor; a computer readable medium coupled to said processor; a multi-layer perception unit on the computer-readable medium and executable by the processor that receives a plurality of features of an image and converts each feature into a corresponding digital string; and A maximum pooling unit on the computer-readable medium and executable by the processor combines the maximum values ​​of each digital string into a global feature string representing the image.

5. A method of operating a computing system for identifying one or more environmental maps stored in a database to merge with a tracking map calculated based on sensor data collected by a device worn by a user, wherein: The device receives a signal of an access point of a computer network while calculating the tracking map, and the method includes: determining at least one area attribute of the tracking map based on communication characteristics with the access point; determining a geographic location of the tracking map based on the at least one regional attribute; identifying a set of environmental maps stored in the database corresponding to the determined geographic location; filtering the set of environmental maps based on similarity of one or more identifiers of network access points associated with the tracking map and the environmental map in the set of environmental maps; filtering the set of environment maps based on similarity of metrics representing content of the tracking map and the environment maps in the set of environment maps; The set of environment maps is filtered based on a degree of match between a portion of the tracking map and a portion of the environment map in the set of environment maps.

6. A cloud computing environment for an augmented reality system, the augmented reality system being configured to communicate with a plurality of user devices including sensors, the cloud computing environment comprising: The user database stores area identifications indicating areas in which the plurality of user devices are used, the area identifications including parameters of wireless networks detected by the user devices in use; a map database storing a plurality of maps of an environment constructed from data provided by the plurality of user devices and associated metadata, the associated metadata including region identifications derived from region identifications of the plurality of user devices providing data from which the maps were constructed, the region identifications including parameters of wireless networks detected by the user devices providing data from which the maps were constructed; A non-transitory computer storage medium storing computer executable instructions that, when executed by at least one processor in the cloud computing environment: receiving messages from the plurality of user equipments including parameters of wireless networks detected by the user equipments, calculating area identifiers for the user equipments, and updating the user database based on the received parameters and / or the calculated area identifiers; as well as receiving requests for environment maps from the plurality of user devices, determining a region identifier associated with the user devices requesting environment maps, identifying a plurality of sets of environment maps from the map database based at least in part on the region identifier, filtering the plurality of sets of environment maps, and sending the filtered plurality of sets of environment maps to the user devices, Therein, filtering a set of environment maps is based on similarity of parameters of a wireless network detected by a user device from which the request for an environment map originates and parameters of wireless networks in the map database for the environment map in the set of environment maps.

7. An XR system comprising: a real object detection device that detects multiple surfaces of a real-world object; an anchor recognition system connected to the real object detection device to generate a map based on the real world objects; a PCF generation system that generates a first PCF based on the map and associates the first PCF with the map; a first storage medium and a second storage medium on the first XR device and the second XR device, respectively; and At least a first processor and a second processor of the first XR device and the second XR device store the first PCF in a first storage medium of the first XR device and a second storage medium of the second XR device, respectively.

8. A viewing method, comprising: detecting a plurality of surfaces of a real-world object using at least one detection device; generating, using at least one processor, a map based on the real-world objects; Using at least one processor, generate a first PCF based on the map; associating, with the at least one processor, the first PCF with the map; as well as The first PCF is stored in a first storage medium of the first XR device and a second storage medium of the second XR device, respectively, using at least a first processor of the first XR device and a second processor of the second XR device.

9. An XR system comprising: A first XR device comprising: The first XR device processor; a first XR device storage device connected to the first XR device processor; a set of instructions on the first XR device processor, comprising: a download system executable by the first XR device processor to download a persistent gesture from a server; a PCF fetcher executable by the first XR device processor to fetch a PCF based on the persistent gesture from the first storage device of the first XR device; and A coordinate frame calculator executable by the first XR device processor to calculate a coordinate frame based on the PCF retrieved from the first storage device.

10. A viewing method, comprising: Downloading, using a first XR device processor of the first XR device, the persistent gesture from the server; Retrieve, using the first XR device processor, a PCF based on the persistent gesture from the first storage device of the first XR device; as well as A coordinate frame is calculated, using the first XR device processor, based on the PCF retrieved from the first storage device.

Images