Cross Reality System

Abstract

A cross-reality system that provides an immersive user experience by storing persistent spatial information about the physical world, where one or more user devices can access the persistent spatial information to determine their location in the physical world, and applications can access the persistent spatial information to specify the location of virtual objects in the physical world. The persistent spatial information enables users to have shared virtual and physical experiences when interacting with the cross-reality system. Additionally, the persistent spatial information can be used in a map of the physical world, enabling one or more devices to access and locate in a previously stored map, reducing the need to map the physical space before using the cross-reality system therein. The persistent spatial information can be stored as a persistent coordinate frame, which can include a transformation relative to a reference orientation and information derived from an image at a location corresponding to the persistent coordinate frame.

Description

[0001] Cross - Reference to Related Applications

[0002] This patent application claims the benefit and priority of U.S. Provisional Patent Application Serial No. 62 / 742,237, filed on October 5, 2018, and entitled "Coordinate Frame Processing Augmented Reality", which is hereby incorporated by reference in its entirety. This patent application also claims the benefit and priority of U.S. Provisional Patent Application Serial No. 62 / 812,935, filed on March 1, 2019, and entitled "Merging a Plurality of Individually Mapped Environments", which is hereby incorporated by reference in its entirety. U.S. Provisional Patent Application Serial No. 62 / 815,955, filed on March 8, 2019, and entitled "Viewing Device or Viewing Devices Having One or More Coordinate Frame Transformers", which is hereby incorporated by reference in its entirety. This patent application also claims the benefit and priority of U.S. Provisional Patent Application Serial No. 62 / 868,786, filed on June 28, 2019, and entitled "Ranking and Merging a Plurality of Environment Maps", which is hereby incorporated by reference in its entirety. This patent application also claims the benefit and priority of U.S. Provisional Patent Application Serial No. 62 / 870,954, filed on July 5, 2019, and entitled "Ranking and Merging a Plurality of Environment Maps", which is hereby incorporated by reference in its entirety. This patent application also claims the benefit and priority of U.S. Provisional Patent Application Serial No. 62 / 884,109, filed on August 7, 2019, and entitled "A Viewing System", which is hereby incorporated by reference in its entirety. Technical Field

[0003] This application generally relates to cross - reality systems. Background Art

[0004] A computer can control a human user interface to create an X Reality (XR) environment, in which some or all of the XR environment perceived by the user is generated by the computer. These XR environments can be virtual reality (VR), augmented reality (AR), and mixed reality (MR) environments, and some or all of these XR environments can be generated by the computer using in part data that describes the environment. For example, the data can describe virtual objects that can be rendered in a manner that the user senses or perceives as being part of the physical world and with which the user can interact. Since the data is rendered and presented via a user interface device such as, for example, a head-mounted display device, the user can experience these virtual objects. The data can be shown to the user or can control the audio played to the user or can control a haptic (or tactile) interface so that the user can experience the sensation of feeling the touch of a virtual object as the user senses or perceives it.

[0005] XR systems can be used for many applications across the fields of scientific visualization, medical training, engineering design and prototyping, telemanipulation and telepresence, and personal entertainment. Compared with VR, AR and MR include one or more virtual objects related to real objects in the physical world. The experience of virtual objects interacting with real objects significantly enhances the user's enjoyment of using XR systems and also opens the door to a variety of applications for presenting information about how to change the reality of the physical world in an easy-to-understand manner.

[0006] To realistically render virtual content, an XR system can establish a representation of the physical world around the user of the system. For example, this representation can be constructed by processing images acquired using sensors on a wearable device that forms part of the XR system. In such a system, the user can perform an initialization routine by looking around the room or other physical environment in which the user intends to use the XR system until the system has obtained enough information to construct a representation of that environment. As the system operates and the user moves in the environment or moves to other environments, the sensors on the wearable device can acquire additional information to extend or update the representation of the physical world. Summary of the Invention

[0007] Some embodiments relate to an electronic system, comprising: one or more sensors configured to capture information about a three-dimensional (3D) environment. The captured information includes a plurality of images. The electronic system includes at least one processor configured to execute computer-executable instructions to generate a map of at least a portion of the 3D environment based on the plurality of images. The computer-executable instructions further include instructions for: identifying a plurality of features in the plurality of images; selecting a plurality of key frames from the plurality of images, the selection being at least partially based on the plurality of features of the selected key frames; generating one or more coordinate frames at least partially based on the identified features of the selected key frames; and storing the one or more coordinate frames as one or more persistent coordinate frames in association with the map of the 3D environment.

[0008] In some embodiments, the one or more sensors include a plurality of pixel circuits arranged in a two-dimensional array such that each image of the plurality of images includes a plurality of pixels. Each feature corresponds to a plurality of pixels.

[0009] In some embodiments, identifying a plurality of features in the plurality of images includes: selecting, based on a measure of similarity to a group of pixels depicting a persistent object portion, a number of groups of pixels less than a predetermined maximum value as the identified features.

[0010] In some embodiments, storing the one or more coordinate frames includes storing, for each of the one or more coordinate frames:

[0011] A descriptor that represents at least a subset of the features in the selected key frames from which the coordinate frame is generated.

[0012] In some embodiments, storing the one or more coordinate frames includes storing, for each of the one or more coordinate frames: at least a subset of the features in the selected key frames from which the coordinate frame is generated.

[0013] In some embodiments, storing the one or more coordinate frames includes storing, for each of the one or more coordinate frames: a transformation between the coordinate frame of the map of the 3D environment and the persistent coordinate frame; and geographic information that indicates the position of the selected key frames within the 3D environment from which the coordinate frame is generated.

[0014] In some embodiments, the geographic information includes a WiFi fingerprint of the location.

[0015] In some embodiments, the computer-executable instructions include instructions for calculating a feature descriptor for each feature using an artificial neural network.

[0016] In some embodiments, the first artificial neural network is the first artificial neural network. The computer-executable instructions include instructions for implementing a second artificial neural network that is configured to compute a frame descriptor representing a keyframe based at least in part on a feature descriptor computed for an identified feature in the keyframe.

[0017] In some embodiments, the computer-executable instructions further include: an application programming interface configured to provide information characterizing a persistent coordinate frame in one or more persistent coordinate frames to an application executing on a portable electronic system; instructions for refining the map of the 3D environment based on a second plurality of images; adjusting one or more of the persistent coordinate frames at least in part based on the second plurality of images; instructions for providing a notification of the adjusted persistent coordinate frame via the application programming interface.

[0018] In some embodiments, adjusting one or more of the persistent coordinate frames includes adjusting the translation and rotation of one or more of the persistent coordinate frames relative to an origin of the map of the 3D environment.

[0019] In some embodiments, the electronic system includes a wearable device, and the one or more sensors are mounted on the wearable device. The map is a tracking map computed on the wearable device. The origin of the map is determined based on the location where the device is powered on.

[0020] In some embodiments, the electronic system includes a wearable device, and the one or more sensors are mounted on the wearable device. The computer-executable instructions further include instructions for: tracking the movement of the portable device; and based on the tracked movement indicating that the movement of the wearable device exceeds a threshold distance, controlling the execution timing of instructions for generating one or more coordinate frames and / or instructions for storing one or more persistent coordinate frames, wherein the threshold distance is between two meters and twenty meters.

[0021] Some embodiments relate to a method of operating an electronic system to render virtual content in a 3D environment including a portable device. The method includes using one or more processors to: maintain a coordinate frame local to the portable device on the portable device based on outputs of one or more sensors on the portable device; obtain a stored coordinate frame according to stored spatial information about the 3D environment; calculate a transformation between the coordinate frame local to the portable device and the obtained stored coordinate frame; receive a specification of a virtual object and a position of the virtual object relative to a selected stored coordinate frame, the virtual object having a coordinate frame local to the virtual object; and render the virtual object on a display of the portable device at a position determined at least in part based on the calculated transformation and the received position of the virtual object.

[0022] In some embodiments, obtaining the stored coordinate frame includes: obtaining the coordinate frame through an application programming interface (API).

[0023] In some embodiments, the portable device includes a first portable device including a first processor of the one or more processors. The system further includes a second portable device including a second processor of the one or more processors; wherein the processors on each of the first device and the second device: obtain the same stored coordinate frame; calculate a transformation between the respective device-local coordinate frame and the same obtained stored coordinate frame; receive the specification of the virtual object; and render the virtual object on the respective display.

[0024] In some embodiments, each of the first device and the second device includes: a camera configured to output a plurality of camera images; a keyframe generator configured to transform the plurality of camera images into a plurality of keyframes; a persistent pose calculator configured to generate a persistent pose by averaging the plurality of keyframes; a tracking map and persistent pose transformer configured to transform a tracking map into the persistent pose to determine the persistent pose relative to an origin of the tracking map; a persistent pose and persistent coordinate frame (PCF) transformer configured to transform the persistent pose into a PCF; and a map publisher configured to send spatial information including the PCF to a server.

[0025] In some embodiments, the method includes: executing an application to generate the specification of the virtual object and the position of the virtual object relative to a selected stored coordinate frame.

[0026] In some embodiments, maintaining a coordinate frame local to the portable device on the portable device includes, for each of the first portable device and the second portable device: capturing, from one or more sensors of the portable device, a plurality of images of the 3D environment; calculating, at least in part based on the plurality of images, one or more persistent poses; and generating, at least in part based on the calculated one or more persistent poses, spatial information about the 3D environment. The method further includes, for each of the first portable device and the second portable device: sending the generated spatial information to a remote server; and obtaining the stored coordinate frame includes receiving the stored coordinate frame from the remote server.

[0027] In some embodiments, calculating the one or more persistent poses at least in part based on the plurality of images includes: extracting one or more features from each of the plurality of images; generating a descriptor for each of the one or more features; generating, at least in part based on the descriptor, a key frame for each of the plurality of images; and generating the one or more persistent poses at least in part based on the one or more key frames.

[0028] In some embodiments, generating the one or more persistent poses includes selectively generating a persistent pose based on the portable device traveling a predetermined distance from its position in other persistent poses.

[0029] In some embodiments, each of the first device and the second device includes: a download system configured to download the stored coordinate frame from a server.

[0030] Some embodiments relate to an electronic system for maintaining persistent spatial information about a 3D environment for rendering virtual content on each of a plurality of portable devices. The electronic system includes: a networked computing device. The networked computing device includes: at least one processor; at least one storage device connected to the processor; a map storage routine executable by the at least one processor to receive a plurality of maps from a portable device among the plurality of portable devices and store map information on the at least one storage device, wherein each of the plurality of received maps includes at least one coordinate frame; and a map transmitter executable by the at least one processor to: receive position information from a portable device among the plurality of portable devices; select one or more maps from the stored maps; and send information from the selected one or more maps to the portable device among the plurality of portable devices, wherein the sent information includes the coordinate frame of the map in the selected one or more maps.

[0031] In some embodiments, the coordinate frame includes a computer data structure. The computer data structure includes: a coordinate frame that includes information representing multiple features of an object in the 3D environment.

[0032] In some embodiments, the information representing the multiple features includes: a descriptor representing a region of the 3D environment.

[0033] In some embodiments, each coordinate frame of the at least one coordinate frame includes: persistent points characterized by features detected in sensor data representing the 3D environment.

[0034] In some embodiments, each coordinate frame of the at least one coordinate frame includes a persistent pose.

[0035] In some embodiments, each coordinate frame of the at least one coordinate frame includes a persistent coordinate frame.

[0036] The foregoing summary is provided by way of illustration and not by way of limitation. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings are not necessarily to scale. In the drawings, each identical or nearly identical component that is illustrated in various drawings is represented by a like numeral. For clarity, not every component may be labeled in every drawing. In the drawings:

[0038] Figure 1 is a schematic diagram showing an example of a simplified augmented reality (AR) scene according to some embodiments;

[0039] Figure 2 is a schematic diagram of an exemplary simplified AR scene according to some embodiments, showing an exemplary usage scenario of an XR system;

[0040] Figure 3 is a schematic diagram showing a data flow for a single user in an AR system according to some embodiments, the AR system being configured to provide the user with an experience of interacting with physical world with AR content;

[0041] Figure 4 is a schematic diagram showing an exemplary AR display system according to some embodiments, the exemplary AR display system displaying virtual content for a single user;

[0042] Figure 5A is a schematic diagram according to some embodiments showing that when a user wears an AR display system, the AR display system renders AR content as the user moves through a physical world environment;

[0043] Figure 5Bis a schematic diagram showing a viewing optical component and associated components according to some embodiments;

[0044] Fig. 6A is a schematic diagram showing an AR system using a world reconstruction system according to some embodiments;

[0045] Figure 6B is a schematic diagram showing components of an AR system that maintains a model of a traversable world according to some embodiments.

[0046] Figure 7 is a schematic diagram of a tracking map formed by a path traversed by a device through the physical world.

[0047] Figure 8 is a schematic diagram showing a user of a cross-reality (XR) system that perceives virtual content according to some embodiments;

[0048] Fig. 9 is of an XR system that transforms between coordinate systems according to some embodiments Figure 8 block diagram of components of a first XR device;

[0049] Fig.10 is a schematic diagram showing an exemplary transformation of an origin coordinate frame to a destination coordinate frame for proper rendering of local XR content according to some embodiments;

[0050] Fig.11 is a top view plan showing a pupil-based coordinate frame according to some embodiments;

[0051] Fig.12 is a top view plan showing a camera coordinate frame including all pupil positions according to some embodiments;

[0052] Fig.13 is of a display system according to some embodiments Fig. 9 schematic diagram;

[0053] Fig.14 is a block diagram showing the creation of a persistent coordinate frame (PCF) and the attachment of XR content to the PCF according to some embodiments;

[0054] Fig.15 is a flowchart showing a method of establishing and using a PCF according to some embodiments;

[0055] Fig.16 is of an XR system including a second XR device according to some embodiments Figure 8 block diagram;

[0056] Fig.17 is a schematic diagram showing a room and key frames established for various regions in the room according to some embodiments;

[0057] Fig.18 is a schematic diagram showing the establishment of key-frame based persistent poses according to some embodiments;

[0058] Fig.19 is a schematic diagram showing the establishment of a persistent coordinate frame (PCF) based on persistent poses according to some embodiments;

[0059] FIG. 20A to FIG. 20C is a schematic diagram showing an example of creating a PCF according to some embodiments;

[0060] Fig.21 is a block diagram showing a system for generating global descriptors for a single image and / or map according to some embodiments;

[0061] Fig. 22 is a flowchart showing a method for calculating an image descriptor according to some embodiments;

[0062] Fig.23 is a flowchart showing a localization method using an image descriptor according to some embodiments;

[0063] Fig.24 is a flowchart showing a method for training a neural network according to some embodiments;

[0064] Fig.25 is a block diagram showing a method for training a neural network according to some embodiments;

[0065] Fig.26 is a schematic diagram showing an AR system configured to rank and merge multiple environmental maps according to some embodiments;

[0066] Fig. 27 is a simplified block diagram showing multiple canonical maps stored on a remote storage medium according to some embodiments;

[0067] Fig.28 is a schematic diagram showing a method for selecting a canonical map to, for example, locate a new tracking map in one or more canonical maps and / or obtain a PCF from the canonical map according to some embodiments;

[0068] Fig.29 is a flowchart showing a method for selecting multiple ranked environmental maps according to some embodiments;

[0069] Fig.30 is a schematic diagram showing an Fig.26 exemplary map ranking section of an AR system according to some embodiments;

[0070] Fig.31AIs a schematic diagram showing an example of the area attributes of a tracking map (TM) and an environmental map in a database according to some embodiments;

[0071] Fig.31B Is a schematic diagram showing an example of determining the geographical location of a tracking map (TM) for geographical location filtering according to some embodiments; Fig.29 for

[0072] Fig.32 Is a schematic diagram showing an example of Fig.29 geographical location filtering according to some embodiments;

[0073] Fig.33 Is a schematic diagram showing an example of Fig.29 Wi-Fi BSSID filtering according to some embodiments;

[0074] Fig.34 Is a schematic diagram showing an example of Fig.29 positioning according to some embodiments;

[0075] Fig.35 And 36 Is a block diagram of an XR system configured to rank and merge multiple environmental maps according to some embodiments.

[0076] Fig.37 Is a block diagram showing a method of creating an environmental map of the physical world in a canonical form according to some embodiments;

[0077] Fig.38A And 38B Is a schematic diagram showing an environmental map created in a canonical form by updating the Figure 7 tracking map with a new tracking map according to some embodiments.

[0078] Figures 39A to 39F Is a schematic diagram showing an example of a merged map according to some embodiments;

[0079] Fig.40 Is a two-dimensional representation of a three-dimensional first local tracking map (ground Fig. 9 ) that can be generated by a first XR device according to some embodiments; Figure 1 )

[0080] Fig.41 Is a schematic diagram showing uploading the ground Fig. 9 from a first XR device to a Figure 1 server according to some embodiments;

[0081] Fig.42 Is a schematic diagram showing an example of Fig.16Schematic diagram of an XR system, showing that after a first user has terminated a first session, a second user has initiated a second session using a second XR device of the XR system;

[0082] Fig.43A Is a block diagram showing a new session of a second XR device according to some embodiments for Fig.42 ;

[0083] Fig.43B Is a block diagram showing the creation of a tracking map of a second XR device according to some embodiments for Fig.42 ;

[0084] Fig.43C Is a block diagram showing the download of a canonical map from a server to a second XR device according to some embodiments for Fig.42 ;

[0085] Fig.44 Is a schematic diagram showing a positioning attempt to position a second tracking map (ground Fig.42 ) that can be generated by a second XR device to a canonical map according to some embodiments; Figure 2 )

[0086] Fig.45 Is a schematic diagram showing a positioning attempt to position a second tracking map (ground Fig.44 ) that can be further developed and has XR content associated with the PCF of the ground Figure 2 ) to a canonical map according to some embodiments; Figure 2 ;

[0087] FIG. 46A to FIG. 46B Is a schematic diagram showing the successful positioning of the ground Fig.45 to a canonical map according to some embodiments; Figure 2 )

[0088] Fig.47 Is a schematic diagram showing a canonical map generated by including one or more PCFs of a canonical map from Fig.46A into the ground Fig.45 according to some embodiments; Figure 2 ;

[0089] Fig.48 Is a schematic diagram showing a canonical map according to some embodiments and a further expansion of the ground Fig.47 on a second XR device; Figure 2 ;

[0090] Fig.49 Is a block diagram showing the upload of the ground Figure 2 from a second XR device to a server according to some embodiments;

[0091] Fig.50 is a block diagram showing the merging of a local Figure 2 with a canonical map according to some embodiments;

[0092] Fig.51 is a block diagram showing the transmission of a new canonical map from a server to a first XR device and a second XR device according to some embodiments;

[0093] Fig.52 is a block diagram showing the local Figure 2 two-dimensional representation and the head coordinate frame of a second XR device with respect to a reference local Figure 2 according to some embodiments;

[0094] Fig.53 is a block diagram showing the adjustment of the head coordinate frame that can occur in six degrees of freedom in a two-dimensional manner according to some embodiments;

[0095] Fig.54 is a block diagram showing the canonical map on a second XR device according to some embodiments, where sound is positioned with respect to the local Figure 2 PCF;

[0096] Fig.55 and Fig.56 are a perspective view and a block diagram showing the use of an XR system when a first user has terminated a first session and the first user has initiated a second session using the XR system according to some embodiments;

[0097] Fig.57 and Fig.58 are a perspective view and a block diagram showing the use of an XR system when three users simultaneously use the XR system in the same session according to some embodiments;

[0098] Fig.59 is a flowchart showing a method for restoring and resetting a head pose according to some embodiments; and

[0099] Fig.60 is a block diagram of a machine in computer form that can find application in the system of the present invention according to some embodiments. Detailed Description

[0100] Described herein are methods and apparatuses for providing X Reality (XR or Cross Reality) scenarios. To provide a realistic XR experience to multiple users, the XR system must understand the physical environment of the users in order to correctly associate the positions of virtual objects related to actual objects. The XR system can construct an environmental map of the scenario, which can be created from images and / or depth information collected by sensors that are part of XR devices worn by the users of the XR system.

[0101] The inventors have recognized and understood that it may be beneficial to have an XR system where each XR device develops a local map of its physical environment by integrating information from one or more images collected during a scan at a single point in time. In some embodiments, the coordinate system of the map is tied to the orientation of the device when the scan is initialized. This orientation can change moment to moment as the user interacts with the XR system, whether different instances in time are associated with different users (each with their own wearable device with sensors for the scanned environment), or with the same user using the same device at different times. The inventors have recognized and understood techniques for operating an XR system based on persistent spatial information that overcome limitations of XR systems where each user device relies only on spatial information collected relative to an orientation that varies for different user situations (e.g., time snapshots) or sessions (e.g., the time between power on and power off) of the system. These techniques can, for example, provide a more computationally efficient and immersive XR experience for single or multiple users by enabling any of a plurality of users of the XR system to create, store, and retrieve persistent spatial information.

[0102] Persistent spatial information can be represented by a persistent map, which can enable one or more functions that enhance the XR experience. The persistent map can be stored in a remote storage medium (e.g., the cloud). For example, a wearable device worn by a user can retrieve a previously created and stored appropriate stored map from persistent storage such as cloud storage after being powered on. The previously stored map may be based on data about the environment collected using sensors on the user's wearable device during a previous session. Retrieving the stored map can enable use of the wearable device without scanning the physical world using the sensors on the wearable device. Alternatively or additionally, the system / device can similarly retrieve an appropriate stored map when entering a new area of the physical world.

[0103] The stored map can be represented in a canonical form that each XR device can relate to its local reference frame. In a multi-device XR system, a stored map accessed by one device may have been created and stored by another device and / or may be constructed by aggregating data about the physical world collected by sensors on multiple wearable devices that have previously been present in at least a portion of the physical world represented by the stored map.

[0104] In addition, sharing data about the physical world among multiple devices can enable a shared user experience of virtual content. For example, two XR devices accessing the same stored map can both be located relative to the stored map. Once located, the user device can render virtual content with a position specified by referencing the stored map by translating the position into a frame or reference maintained by the user device. The user device can use this local reference frame to control the display of the user device to render virtual content at the specified position.

[0105] To support these and other functions, an XR system can include components that develop, maintain, and use persistent spatial information (including one or more stored maps) based on data about the physical world collected by sensors on user devices. These components can be distributed across the XR system, for example, by some operations on the head-mounted portion of the user device. Other components can operate on a computer and be associated with a user coupled to the head-mounted portion via a local area network or a personal area network. Still others can operate at a remote location (such as at one or more servers accessible via a wide area network).

[0106] These components may include, for example, components that can identify information from the information about the physical world collected by one or more user devices that is of sufficient quality to be stored as or in a persistent map. An example of such a component, described in more detail below, is a map merging component. Such a component can, for example, receive input from a user device and determine the fitness of portions of the input to be used to update the persistent map. The map merging component can, for example, divide a local map created by the user device into multiple portions, determine the mergability of one or more portions with the persistent map, and merge portions that meet the qualified mergability criteria into the persistent map. The map merging component can, for example, also promote portions that are not merged with the persistent map to separate persistent maps.

[0107] As another example, these components can include components that can help determine an appropriate persistent map that can be accessed and used by a user device. An example of such a component, described in more detail below, is a map ranking component. For example, such a component can receive input from a user device and identify one or more persistent maps that may represent the area of the physical world in which the device operates. For example, the map ranking component can help select the persistent map to be used by the local device when rendering virtual content, collecting data about the environment, or performing other actions. Alternatively or additionally, the map ranking component can help identify persistent maps that are updated as one or more user devices collect additional information about the physical world.

[0108] Other components can determine the transformation that converts information captured or described with respect to one reference frame to another reference frame. For example, a sensor can be attached to a head-mounted display such that the data read from the sensor indicates the position of an object in the physical world relative to the pose of the wearer's head. One or more transformations can be applied to relate the position information to the coordinate frame associated with the persistent environment map. Similarly, data indicating where to render a virtual object when expressed in the coordinate frame of the persistent environment map can be transformed one or more times to be in the reference frame of the user's head-mounted display. As described in more detail below, there may be multiple such transformations. These transformations can be partitioned across the components of the XR system such that they can be updated efficiently or applied in a distributed system.

[0109] In some embodiments, a persistent map can be constructed based on information collected by multiple user devices. The XR device can capture local space information and use the information collected by the sensors of each XR device at various locations and times to construct a separate tracking map. Each tracking map can include points, and each point can be associated with a feature of a real object that can include multiple features. In addition to potentially providing input to create and maintain the persistent map, the tracking map can also be used to track the movement of the user in the scene, enabling the XR system to estimate the head pose of the corresponding user based on the tracking map.

[0110] This interdependence between map creation and head pose estimation poses a significant challenge. Considerable processing may be required to create the map and simultaneously estimate the head pose. As objects move in the scene (e.g., moving a cup on a table) and as the user moves in the scene, the processing must be completed quickly because latency reduces the realism of the XR experience for the user. On the other hand, the XR device can provide limited computational resources because the XR device should be lightweight for the user to wear comfortably. More sensors cannot be employed to compensate for the lack of computational resources because adding sensors would also undesirably increase the weight. Additionally, more sensors or more computational resources generate heat, which can cause deformation of the XR device.

[0111] The inventors have recognized and understood the XR scenarios of techniques for operating an XR system to provide a more immersive user experience, such as estimating the head pose at a frequency of 1 kHz, using a lower rate of computational resources associated with the XR device, such as four video graphics array (VGA) cameras that can be configured to operate at 30 Hz, one inertial measurement unit (IMU) operating at 1 kHz, the computational power of a single advanced RISC machine (ARM) core, less than 1 GB of memory, and less than 100 Mbp of network bandwidth. These techniques involve reducing the processing required to generate and maintain the map and estimate the head pose, and involve providing and consuming data with low computational overhead.

[0112] These techniques can include hybrid tracking such that the XR system can utilize both: (1) patch-based tracking of distinguishable points between successive images of the environment (e.g., frame-to-frame tracking), and (2) matching of points of interest in the current image to a descriptor-based map of the known ground truth locations of corresponding points of interest (e.g., map-to-frame tracking). In frame-to-frame tracking, the XR system can track specific points of interest (e.g., corner points) between images captured in the real-world environment, such as corner points. For example, the display system can identify the location of a visual point of interest in the current image that was included in (e.g., located in) a previous image. This identification can be done using, for example, a photometric error minimization process. In map-to-frame tracking, the XR system can access map information that indicates the ground truth locations of points of interest and match the points of interest included in the current image to the points of interest indicated in the map information. Information about the points of interest can be stored in the map database as descriptors. The XR system can compute its pose based on the matched visual features. U.S. Patent Application Serial No. 16 / 221,065 describes hybrid tracking and is incorporated herein by reference in its entirety.

[0113] These techniques can include reducing the amount of data processed when constructing a map, such as by constructing a sparse map using a set of map construction points and key frames and / or partitioning the map into chunks to enable chunk-by-chunk updates. The map construction points can be associated with points of interest in the environment. The key frames can include information selected from the data captured by the camera. U.S. Patent Application Serial No. 16 / 520,582 describes determining and / or evaluating a localization map and is incorporated herein by reference in its entirety.

[0114] In some embodiments, persistent spatial information can be represented in a manner that can be easily shared between users and between distributed components including applications. For example, information about the physical world can be represented as a Persistent Coordinate Frame (PCF). The PCF can be defined based on one or more points that represent features identified in the physical world. The features can be selected such that they are likely to be the same between user sessions of the XR system. The PCF may be sparse, providing less information than all the available information about the physical world so that they can be efficiently processed and transferred. Techniques for processing persistent spatial information can include: creating a dynamic map based on one or more coordinate systems in real space spanning one or more sessions and generating a Persistent Coordinate Frame (PCF) on the sparse map, which can be exposed to XR applications via, for example, an Application Programming Interface (API). These functions can be supported by techniques for ranking and merging multiple maps created by one or more XR devices. Persistent spatial information can also enable fast recovery and reset of the head pose on each of one or more XR devices in a computationally efficient manner.

[0115] In addition, these techniques can enable efficient comparison of spatial information. In some embodiments, an image frame can be represented by a digital descriptor. The descriptor can be computed by a transformation that maps a set of features identified in the image to the descriptor. The transformation can be performed in a trained neural network. In some embodiments, the set of features provided as input to the neural network can be a filtered set of features extracted from the image using techniques such as preferentially selecting features that are likely to be persistent.

[0116] Representing the image frame as a descriptor enables, for example, efficient matching of new image information with stored image information. The XR system can be stored along with persistent map descriptors for one or more frames underlying the persistent map. A local image frame acquired by a user device can similarly be transformed into such a descriptor. By selecting stored maps having descriptors similar to the descriptor of the local image frame, one or more persistent maps that may represent the same physical space as the user device can be selected with relatively little processing. In some embodiments, descriptors can be computed for key frames in the local map and the persistent map, thereby further reducing the processing when comparing maps. For example, this efficient comparison can be used to simplify finding a persistent map to load into a local device or finding a persistent map to update based on image information acquired using the local device.

[0117] The techniques described herein can be used together or separately with many types of devices and for many types of scenarios, including wearable or portable devices with limited computational resources that provide augmented or mixed reality scenarios. In some embodiments, the techniques can be implemented by one or more services that form part of the XR system.

[0118] AR System Overview

[0119] Figure 1 and Figure 2 Scenes with virtual content are shown that are displayed together with a portion of the physical world. For illustrative purposes, an AR system is used as an example of an XR system. Figure 3-6B An exemplary AR system is shown that includes one or more processors, a memory, sensors, and a user interface that can operate according to the techniques described herein.

[0120] Reference Figure 1, depicts an outdoor AR scene 354, where a user of AR technology sees a park-like setting 356 of the physical world, characterized by people, trees, buildings in the background, and a concrete platform 358. In addition to these items, the user of AR technology also perceives that they "see" a robotic statue 357 standing on the concrete platform 358 of the physical world, and a flying, cartoon-like avatar character 352 that appears to be the head of a bumblebee, even though these elements (e.g., avatar character 352 and robotic statue 357) do not exist in the physical world. Due to the extreme complexity of human visual perception and the nervous system, it is challenging to develop AR technology that promotes a comfortable, natural feeling and rich presentation of virtual image elements among other virtual or physical world image elements.

[0121] Such an AR scene can be achieved by a system that builds a map of the physical world based on tracking information, enabling users to place AR content in the physical world, determine the location to place the AR content in the map of the physical world, retain the AR scene so that the placed AR content can be reloaded and displayed in the physical world during, for example, different AR experience sessions, and enable multiple users to share the AR experience. The system can build and update a digital representation of the surface of the physical world around the user. This representation can be used to render virtual content to appear to be fully or partially occluded by physical objects between the user and the rendering location of the virtual content, for placing virtual objects in physics-based interactions, and for virtual character path planning and navigation, or for other operations that use information about the physical world.

[0122] Figure 2 Depicts another example of an indoor AR scene 400 according to some embodiments, which shows an exemplary usage scenario of an XR system. The exemplary scene 400 is a living room with walls, a bookshelf on one side of the wall, a floor lamp at the corner of the room, a floor, a sofa, and a coffee table on the floor. In addition to these physical items, a user of AR technology can also perceive virtual objects, such as an image on the wall behind the sofa, a bird flying through the door, a deer peeking out from the bookshelf, and an ornament in the form of a windmill placed on the coffee table.

[0123] For an image on a wall, AR technology requires not only information about the wall surface but also information about objects and surfaces within the room (such as the shape of a lamp), which will occlude the image to correctly render virtual objects. For a flying bird, AR technology requires information about all objects and surfaces around the room in order to render the bird with realistic physics, to avoid objects and surfaces or to bounce when the bird collides. For a deer, AR technology requires information about surfaces (such as the floor or a coffee table) to calculate the placement of the deer. For a windmill, the system can identify that it is an object separate from the table and can determine that it is movable, while the corner of a shelf or the corner of a wall can be determined to be stationary. This distinction can be used to determine which parts of the scene to use or update in each of various operations.

[0124] Virtual objects can be placed in a previous AR experience session. When a new AR experience session starts in the living room, AR technology needs to accurately display the virtual objects in the positions where they were previously placed and are actually visible from different viewpoints. For example, the windmill should be shown as standing on a book, rather than floating above the table in a different position without the book. This floating can occur if the position of the user in the new AR experience session is not accurately located in the living room. As another example, if the user views the windmill from a different viewpoint than the one from which it was placed, AR technology needs to display the corresponding side of the windmill.

[0125] A scene can be presented to a user via a system including multiple components, the multiple components including a user interface that can stimulate one or more of the user's senses (such as vision, sound, and / or touch). Additionally, the system can include one or more sensors that can measure parameters of the physical parts of the scene, including the position and / or movement of the user within the physical parts of the scene. Further, the system can include one or more computing devices, as well as associated computer hardware such as memory. These components can be integrated into a single device or can be distributed across multiple interconnected devices. In some embodiments, some or all of these components can be integrated into a wearable device.

[0126] Figure 3FIG. 0 depicts an AR system 502 according to some embodiments, which is configured to provide an experience of AR content interacting with the physical world 506. The AR system 502 may include a display 508. In the illustrated embodiment, the display 508 may be worn by a user as part of a head-mounted headset such that the user can wear the display over their eyes like a pair of goggles or glasses. At least a portion of the display may be transparent such that the user can observe see-through reality 510. The see-through reality 510 may correspond to the portion of the physical world 506 that is within the current viewpoint of the AR system 502, which may correspond to the user's viewpoint in the case where the user wears a head-mounted headset incorporating the display and sensors of the AR system to obtain information about the physical world.

[0127] AR content may also be presented on the display 508, overlaying the see-through reality 510. To provide accurate interaction between the AR content and the see-through reality 510 on the display 508, the AR system 502 may include sensors 522 configured to capture information about the physical world 506.

[0128] The sensors 522 may include one or more depth sensors that output depth maps 512. Each depth map 512 may have a plurality of pixels, each of which may represent the distance to a surface in the physical world 506 in a particular direction relative to the depth sensor. Raw depth data may be received from the depth sensors to create the depth maps. The depth maps may be updated as fast as the depth sensors can form new images, which may be hundreds or thousands of times per second. However, the data may be noisy and incomplete and have holes shown as black pixels on the illustrated depth maps.

[0129] The system may include other sensors, such as image sensors. The image sensors may acquire monocular or stereo information, which may be processed to otherwise represent the physical world. For example, the images may be processed in a world reconstruction component 516 to create a mesh that represents connected portions of objects in the physical world. Metadata about such objects (including, for example, color and surface texture) may similarly be acquired using sensors and stored as part of the world reconstruction.

[0130] The system can also obtain information about the user's head pose (or "pose") relative to the physical world. In some embodiments, the head pose tracking component of the system can be used to calculate the head pose in real time. The head pose tracking component can represent the user's head pose in a coordinate system with six degrees of freedom, which includes, for example, translation along three vertical axes (e.g., forward / backward, up / down, left / right) and rotation around these three vertical axes (e.g., pitch, yaw, and roll). In some embodiments, the sensor 522 can include an inertial measurement unit that can be used to calculate and / or determine the head pose 514. The head pose 514 for the depth map can indicate, for example, the current viewing point of the sensor that captures the depth map in six degrees of freedom, but the head-mounted headset 514 can be used for other purposes, such as associating image information with a specific part of the physical world or associating the position of a display worn on the user's head with the physical world. In some embodiments, the head pose information can be derived in other ways different from the IMU (such as analyzing objects in an image).

[0131] In some embodiments, the head pose information can be derived in other ways different from the IMU (such as analyzing objects in an image). For example, the head pose tracking component can calculate the relative position and orientation of the AR device relative to a physical object based on visual information captured by a camera and inertial information captured by the IMU. The head pose tracking component can then calculate the head pose of the AR device, for example, by comparing the calculated relative position and orientation of the AR device relative to the physical object with the features of the physical object. In some embodiments, this comparison can be performed by identifying features in the images captured by one or more sensors 522 that are stable over time so that the change in the position of these features in the images captured over time can be correlated with the change in the user's head pose.

[0132] In some embodiments, the AR device can build a map based on feature points identified in consecutive images of a series of image frames captured as the user moves the AR device throughout the physical world. Although each image frame can be taken from a different pose as the user moves, the system can adjust the orientation of the features of each consecutive image frame to match the orientation of the initial image frame by matching the features of the consecutive image frames with the previously captured image frames. The translation of the consecutive image frames such that points representing the same feature will match the corresponding feature points in the previously collected image frames can be used to align each consecutive image frame to match the orientation of the previously processed image frame. The frames in the generated map can have a common orientation established when the first image frame was added to the map. The map has multiple sets of feature points in a common reference frame and can be used to determine the user's pose in the physical world by matching the features in the current image frame with the map. In some embodiments, the map can be referred to as a tracking map.

[0133] In addition to being able to track the user's pose in the environment, the map can also enable other components of the system, such as the world reconstruction component 516, to determine the position of physical objects relative to the user. The world reconstruction component 516 can receive the depth map 512, the head pose 514, and any other data from the sensors and integrate this data into the reconstruction 518. The reconstruction 518 can be more complete and less noisy than the sensor data. The world reconstruction component 516 can update the reconstruction 518 using the spatial and temporal averaging of sensor data from multiple viewpoints over time.

[0134] The reconstruction 518 can include a representation of the physical world in one or more data formats, including, for example, voxels, meshes, planes, etc. Different formats can represent alternative representations of the same part of the physical world or can represent different parts of the physical world. In the example shown, on the left side of the reconstruction 518, a portion of the physical world is presented as a global surface; on the right side of the reconstruction 518, a portion of the physical world is presented as a mesh.

[0135] In some embodiments, the map maintained by the head pose component 514 may be sparse relative to other maps of the physical world that may be maintained. The sparse map may indicate the locations of areas of interest and / or structures (e.g., corners or edges), rather than providing information about the locations of surfaces and possibly other features. In some embodiments, the map may include image frames captured by the sensor 522. These frames may be reduced to features that can represent areas of interest and / or structures. In conjunction with each frame, information about the pose of the user from whom the frame was acquired may also be stored as part of the map. In some embodiments, not every image acquired by the sensor may be stored or may not be stored. In some embodiments, when images are collected by the sensor, the system may process the images and select a subset of the image frames for further computation. The selection may be based on one or more criteria that limit the addition of information but ensure that the map contains useful information. The system may add a new image frame to the map, for example, based on an overlap with a previous image frame that has already been added to the map or based on an image frame that contains a sufficient number of features that are determined to likely represent stationary objects. In some embodiments, the selected image frame or a group of features from the selected image frame may be used as a key frame for the map, which is used to provide spatial information.

[0136] The AR system 502 may integrate sensor data over time from multiple viewpoints of the physical world. As the device including the sensor moves, the pose of the sensor (e.g., location and orientation) may be tracked. Since the frame pose of the sensor and its relationship to other poses are known, each of these multiple viewpoints of the physical world may be fused together to form a single combined reconstruction of the physical world, which may be used as an abstract layer of the map and provide spatial information. By using spatial and temporal averaging (i.e., averaging data from multiple viewpoints over time) or any other suitable method, the reconstruction may be more complete and less noisy than the original sensor data.

[0137] In Figure 3 the illustrated embodiment, the map represents the portion of the physical world of a user in which there is a single wearable device. In that case, the head pose associated with the frame in the map may be represented as a local head pose, indicating the orientation relative to the initial orientation of the single device at the start of the session. For example, the head pose may be tracked relative to the initial head pose when the device is turned on or otherwise operated to scan the environment to establish a representation of that environment.

[0138] In combination with the content representing the physical world, a map may include metadata. The metadata may, for example, indicate the time when sensor information captured for forming the map was captured. Alternatively or additionally, the metadata may indicate the location of the sensor when the information for forming the map was captured. The location may be represented directly, such as using information from a GPS chip, or indirectly, such as using a Wi-Fi signature, which indicates the strength of signals received from one or more wireless access points while collecting sensor data, and / or the BSSID of the wireless access point to which the user device was connected while collecting sensor data.

[0139] The reconstruction 518 can be used for AR functions, such as generating a surface representation of the physical world for occlusion handling or physics-based processing. This surface representation may change as the user moves or objects in the real world change. Aspects of the reconstruction 518 can be used, for example, by a component 520 that generates a changing global surface representation in world coordinates, which can be used by other components.

[0140] Based on this information, AR content can be generated, such as by an AR application 504. The AR application 504 can be, for example, a game program that performs one or more functions based on information about the physical world, such as visual occlusion, physics-based interaction, and environmental reasoning. It can perform these functions by querying different formats of data from the reconstruction 518 generated by the world reconstruction component 516. In some embodiments, the component 520 can be configured to output an update when the representation in the region of interest of the physical world changes. For example, the region of interest can be set to approximate a part of the physical world near the user of the system, such as the part within the user's field of view, or projected (predicted / determined) to enter the user's field of view.

[0141] The AR application 504 can use this information to generate and update AR content. The virtual part of the AR content can be presented on the display 508 in combination with the perspective reality 510, thus creating a realistic user experience.

[0142] In some embodiments, an AR experience can be provided to a user via an XR device, which can be a wearable display device and can be part of a system that can include remote processing and / or remote data storage and / or, in some embodiments, other wearable display devices worn by other users. For simplicity of illustration, Figure 4An example of a system 580 (hereinafter referred to as "system 580") including a single wearable device is shown. System 580 includes a head-mounted display device 562 (hereinafter referred to as "display device 562"), as well as various mechanical and electronic modules and systems that support the functions of display device 562. Display device 562 may be coupled to a frame 564 that can be worn by a display system user or viewer 560 (hereinafter referred to as "user 560") and is configured to position display device 562 in front of the eyes of user 560. According to various embodiments, display device 562 may display sequentially. Display device 562 may be monocular or binocular. In some embodiments, display device 562 may be Figure 3 an example of the display 508 in

[0143] In some embodiments, a speaker 566 is coupled to frame 564 and positioned near the ear canal of user 560. In some embodiments, another speaker (not shown) is positioned near the other ear canal of user 560 to provide stereo / plastic sound control. Display device 562 is operatively coupled to a local data processing module 570, such as via a wired wire or a wireless connection 568, and local data processing module 570 may be installed in various configurations, such as fixedly attached to frame 564, fixedly attached to a helmet or hat worn by user 560, embedded in headphones, or otherwise removably attached to user 560 (e.g., in a backpack configuration, in a belt-coupled configuration).

[0144] Local data processing module 570 may include a processor and a digital memory such as a non-volatile memory (e.g., flash memory), both of which may be used to assist in the processing, caching, and storage of data. The data includes: a) data captured from sensors (e.g., that may be operatively coupled to frame 564) or otherwise attached to user 560, such as an image capture device (such as a camera), a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a radio device, and / or a gyroscope; and / or b) data obtained and / or processed using remote processing module 572 and / or remote data repository 574 and possibly passed to display device 562 after such processing or acquisition.

[0145] In some embodiments, the wearable device may communicate with remote components. Local data processing module 570 may be operatively coupled to remote processing module 572 and remote data repository 574 via communication links 576, 578 (such as via a wired or wireless communication link), respectively, such that these remote modules 572, 574 are operatively coupled to each other and can be used as resources for local data processing module 570. In some embodiments, the above-mentioned head pose tracking component may be at least partially implemented in local data processing module 570. In some embodiments, Figure 3The world reconstruction component 516 therein may be implemented at least partially in the local data processing module 570. For example, the local data processing module 570 may be configured to execute computer-executable instructions to generate a map and / or a physical world representation based at least in part on at least a portion of the data.

[0146] In some embodiments, the processing may be distributed across local and remote processors. For example, local processing may be used to construct a map (e.g., a tracking map) on the user device based on sensor data collected using sensors on the user device. Such a map may be used by an application on the user device. Additionally, previously created maps (e.g., canonical maps) may be stored in the remote data repository 574. When a suitable stored or persistent map is available, it may be used in place of or in addition to the tracking map created locally on the device. In some embodiments, the tracking map may be aligned to the stored map such that a correspondence is established between the tracking map and the canonical map, where the tracking map may be oriented relative to the position of the wearable device when the user turns on the system, and the canonical map may be oriented relative to one or more persistent features. In some embodiments, the persistent map may be loaded on the user device to allow the user device to render virtual content without the latency associated with scanning for a location, thereby constructing a tracking map of the user's entire environment based on sensor data acquired during the scan. In some embodiments, the user device may access a remote persistent map (e.g., stored in the cloud) without downloading the persistent map on the user device.

[0147] Alternatively or additionally, the tracking map may be merged with previously stored maps to extend or improve the quality of those maps. The process of determining whether a suitable previously created environmental map is available and / or merging the tracking map with one or more stored environmental maps may be done in the local data processing module 570 or the remote processing module 572.

[0148] In some embodiments, the local data processing module 570 may include one or more processors (e.g., a graphics processing unit (GPU)) configured to analyze and process data and / or image information. In some embodiments, the local data processing module 570 may include a single processor (e.g., a single-core or multi-core ARM processor), which would limit the computational budget of the local data processing module 570 but enable a smaller device. In some embodiments, the world reconstruction component 516 may use less than the computational budget of a single advanced RISC machine (ARM) core to generate a physical world representation in real time over a non-predefined space such that the remaining computational budget of a single ARM core may be accessed for other uses, such as for example extracting a mesh.

[0149] In some embodiments, the remote data repository 574 can include a digital data storage facility that can be made available via the Internet or other networking configurations in a "cloud" resource configuration. In some embodiments, all data is stored in and all computations are performed in the local data processing module 570, allowing for fully autonomous use from remote modules. In some embodiments, all data is stored and all or most computations are performed in the remote data repository 574, allowing for smaller devices. For example, world reconstruction can be stored in whole or in part in the repository 574.

[0150] In embodiments where data is stored remotely and accessible via a network, the data can be shared by multiple users of the augmented reality system. For example, user devices can upload their tracked maps to augment a database of environmental maps. In some embodiments, the tracked map upload occurs at the end of a user session with the wearable device. In some embodiments, the tracked map upload can occur continuously, semi - continuously, intermittently at a predefined time, after a predefined period from a previous upload, or when triggered by an event. Regardless of whether based on data from that user device or any other user device, the tracked maps uploaded by any user device can be used to extend or improve previously stored maps. Similarly, the persistent maps downloaded to a user device can be based on data from that user device or any other user device. In this way, users can easily obtain high - quality environmental maps to improve their experience in the AR system.

[0151] In some embodiments, the local data processing module 570 is operatively coupled to a battery 582. In some embodiments, the battery 582 is a removable power source, such as over a counter battery. In other embodiments, the battery 582 is a lithium - ion battery. In some embodiments, the battery 582 includes both an internal lithium - ion battery that can be charged by the user 560 during non - operating times of the system 580 and a removable battery, such that the user 560 can operate the system 580 for longer periods of time without having to connect to a power source to charge the lithium - ion battery or without having to turn off the system 580 to replace the battery.

[0152] Figure 5AShows user 530 wearing an AR display system that renders AR content as user 530 moves through a physical world environment 532 (hereinafter referred to as "environment 532"). Information captured by the AR system along the user's movement path can be processed into one or more tracking maps. User 530 positions the AR display system at location 534, and the AR display system records environmental information of the traversable world relative to location 534 (e.g., digital representations of real objects in the physical world, which can be stored and updated as real objects change in the physical world). This information can be combined with images, features, directional audio inputs, or other desired data and stored as a pose. Location 534 is aggregated into data input 536, for example, as part of a tracking map, and processed by at least traversable world module 538, which can be implemented, for example, by processing on remote processing module 572 in Figure 4 . In some embodiments, traversable world module 538 can include head pose component 514 and world reconstruction component 516 such that the processed information can be combined with other information related to physical objects used in rendering virtual content to indicate the position of the objects in the physical world.

[0153] Traversable world module 538 at least partially determines the location and manner in which AR content 540, as determined from data input 536, can be placed in the physical world. The AR content is "placed" in the physical world by presenting both the physical world rendering and the AR content via a user interface, with the AR content rendered as if interacting with objects in the physical world and the objects in the physical world presented as if the AR content obscures the user's view of these objects when appropriate. In some embodiments, the shape and location of AR content 540 can be determined by appropriately selecting parts of a fixed element 542 (e.g., a table) from a reconstruction (e.g., reconstruction 518) to place the AR content. As an example, the fixed element can be a table, and the virtual content can be positioned such that it appears to be on the table. In some embodiments, the AR content can be placed within a structure in the field of view 544, which can be the current field of view or an estimated future field of view. In some embodiments, the AR content can persist relative to a model 546 (e.g., a mesh) of the physical world.

[0154] As depicted, the fixed element 542 serves as a proxy (e.g., a digital copy) for any fixed element within the physical world that can be stored in the traversable world module 538, such that the user 530 can perceive the content on the fixed element 542 without the system having to map build to the fixed element 542 every time the user 530 views the fixed element 542. Thus, the fixed element 542 can be a mesh model from a previous modeling session or can be determined by a separate user but still stored by the traversable world module 538 for future reference by multiple users. Accordingly, the traversable world module 538 can identify the environment 532 from a previously map built environment and display AR content without the user 530's device first having to map build all or a portion of the environment 532, thus saving computational processes and cycles and avoiding latency in any rendered AR content.

[0155] A mesh model 546 of the physical world can be created by the AR display system, and the appropriate surfaces and metrics for interacting with and displaying the AR content 540 can be stored by the traversable world module 538 for future access by the user 530 or other users without having to recreate the model completely or partially. In some embodiments, the data input 536 is an input such as a geographical location, user identification, and current activity to indicate to the traversable world module 538 which fixed element 542 among one or more fixed elements is available, which AR content 540 was last placed on the fixed element 542, and whether to display the same content (such AR content is "persistent" content regardless of how the user views a particular traversable world model).

[0156] Even in embodiments where an object is considered fixed (e.g., a kitchen table), the traversable world module 538 can update the models of those objects in the physical world model from time to time to account for the possibility of changes in the physical world. The models of fixed objects may be updated at a very low frequency. Other objects in the physical world may be moving or otherwise not considered fixed (e.g., kitchen chairs). To render an AR scene with a realistic feel, the AR system can update the positions of these non - fixed objects at a much higher frequency than that used to update fixed objects. To be able to accurately track all objects in the physical world, the AR system can obtain information from multiple sensors, including one or more image sensors.

[0157] Figure 5B is a schematic diagram of the viewing optical component 548 and accompanying components. In some embodiments, two eye - tracking cameras 550 that point at the user's eyes 549 detect metrics of the user's eyes 549, such as the eye shape, eyelid occlusion, pupil direction, and blink on the user's eyes 549.

[0158] In some embodiments, one of the sensors can be a depth sensor 551, such as a time-of-flight sensor, which emits signals into the world and detects the reflections of those signals from nearby objects to determine the distance to a given object. The depth sensor can, for example, quickly determine whether an object has entered the user's field of view due to the movement of those objects or changes in the user's pose. However, information about the position of an object in the user's field of view can alternatively or additionally be collected by other sensors. Depth information can, for example, be obtained from a stereo vision image sensor or a plenoptic sensor.

[0159] In some embodiments, the world camera 552 records a view larger than the periphery to map the environment 532 and / or otherwise create a model of the environment 532, and detect inputs that can affect the AR content. In some embodiments, the world camera 552 and / or the camera 553 can be grayscale and / or color image sensors, which can output grayscale and / or color image frames at fixed time intervals. The camera 553 can further capture an image of the physical world within the user's field of view at a particular time. Sampling of its pixels can be repeated even if the values of the pixels of the frame-based image sensor do not change. Each of the world camera 552, the camera 553, and the depth sensor 551 has a corresponding field of view 554, 555, and 556 to collect data from and record a physical world scene such as the physical world environment 532 depicted in Figure 5A the physical world scene.

[0160] The inertial measurement unit 557 can determine the movement and orientation of the viewing optics 548. In some embodiments, each component is operatively coupled to at least one other component. For example, the depth sensor 551 is operatively coupled to the eye tracking camera 550 to confirm the measured accommodation relative to the actual distance at which the user's eye 549 is gazing.

[0161] It should be understood that the viewing optics 548 can include Figure 5B some of the components shown in, and can include components instead of or in addition to the components shown. For example, in some embodiments, the viewing optics 548 can include two world cameras 552 instead of four. Alternatively or additionally, the cameras 552 and 553 do not need to capture visible light images of their entire fields of view. The viewing optics 548 can include other types of components. In some embodiments, the viewing optics 548 can include one or more dynamic vision sensors (DVSs), the pixels of which can respond asynchronously to relative changes in light intensity above a threshold.

[0162] In some embodiments, based on time-of-flight information, the viewing optical assembly 548 may not include a depth sensor 551. For example, in some embodiments, the viewing optical assembly 548 may include one or more plenoptic cameras, the pixels of which may capture light intensity and the angle of incident light, whereby depth information may be determined. For example, a plenoptic camera may include an image sensor covered with a transmissive diffraction mask (TDM). Alternatively or additionally, a plenoptic camera may include an image sensor that includes angle-sensitive pixels and / or phase detection autofocus pixels (PDAF) and / or a microlens array (MLA). Instead of or in addition to the depth sensor 551, such sensors may be used as a source of depth information.

[0163] It should also be understood that Figure 5B the configuration of the components in is provided as an example. The viewing optical assembly 548 may include components having any suitable configuration, which may be set to provide the user with the maximum field of view feasible for a particular set of components. For example, if the viewing optical assembly 548 has a world camera 552, the world camera may be placed in the central region of the viewing optical assembly rather than the side.

[0164] Information from sensors in the viewing optical assembly 548 may be coupled to one or more processors in the system. The processor may generate data that can be rendered to enable the user to perceive interacting with objects in the physical world. The rendering may be implemented in any suitable manner, including generating image data depicting both physical and virtual objects. In other embodiments, physical and virtual content may be depicted in a scene by modulating the opacity of a display device that the user views in the physical world. The opacity may be controlled to create the appearance of virtual objects and also to prevent the user from seeing objects in the physical world that are occluded by the virtual objects. In some embodiments, when viewed through a user interface, the image data may include only virtual content, which may be modified such that the virtual content is perceived by the user as realistically interacting with the physical world (e.g., clipping the content to account for occlusion).

[0165] The location at which display content on the viewing optical assembly 548 is produced to create the impression that an object is located at a particular position may depend on the physical properties of the viewing optical assembly. Additionally, the pose of the user's head relative to the physical world and the direction of the user's eye gaze may affect the position at which the display content in the physical world will appear at a particular location on the viewing optical assembly. The sensors as described above may collect this information and / or provide information from which this information may be calculated such that a processor receiving the sensor input may calculate the position at which an object should be rendered on the viewing optical assembly 548 to create the desired appearance for the user.

[0166] Regardless of how content is presented to the user, a model of the physical world can be used such that characteristics of virtual objects that can be affected by physical objects can be correctly calculated, including the shape, position, motion, and visibility of the virtual objects. In some embodiments, the model can include a reconstruction of the physical world, such as reconstruction 518.

[0167] The model can be created based on data collected from sensors on the user's wearable device. However, in some embodiments, the model can be created from data collected from multiple users, and the data can be aggregated in a computing device that is remote from all users (and the data can be "in the cloud").

[0168] The model can be created at least in part by a world reconstruction system, such as, for example, Fig. 6A more particularly depicted in Figure 3 world reconstruction component 516. World reconstruction component 516 can include a perception module 660 that can generate, update, and store a representation of a portion of the physical world. In some embodiments, perception module 660 can represent the portion of the physical world within the reconstruction range of the sensors as a plurality of voxels. Each voxel can correspond to a 3D cube of a predetermined volume in the physical world and include surface information that indicates whether there is a surface within the volume represented by the voxel. A value can be assigned to the voxel that indicates whether its corresponding volume has been determined to include a surface of a physical object, determined to be empty, or not yet measured by the sensors and thus its value is unknown. It should be understood that it is not necessary to explicitly store the values of voxels determined to be empty or unknown, as the values of the voxels can be stored in computer memory in any suitable manner, including not storing information for voxels determined to be empty or unknown.

[0169] In addition to generating information for the persistent world representation, perception module 660 can also identify and output an indication of a change in the area around the user of the AR system. This indication of change can trigger an update to the volumetric data stored as part of the persistent world, or trigger other functions, such as triggering component 604 that generates AR content to update the AR content.

[0170] In some embodiments, the perception module 660 may identify changes based on a Signed Distance Function (SDF) model. The perception module 660 may be configured to receive sensor data such as, for example, a depth map 660a and a head pose 660b, and then fuse the sensor data into the SDF model 660c. The depth map 660a may directly provide SDF information, and the image may be processed to obtain SDF information. The SDF information represents the distance from the sensor used to capture the information. Since those sensors may be part of a wearable unit, the SDF information may represent the physical world from the perspective of the wearable unit and thus from the user's perspective. The head pose 660b may enable the SDF information to be correlated with voxels in the physical world.

[0171] In some embodiments, the perception module 660 may generate, update, and store a representation of a portion of the physical world within the perception range. The perception range may be determined at least in part based on the reconstruction range of the sensor, which may be determined at least in part based on the limitations of the observation range of the sensor. As a specific example, an active depth sensor operating with active IR pulses may operate reliably within a certain distance range, thereby creating an observation range of the sensor that may range from a few centimeters or tens of centimeters to several meters.

[0172] The world reconstruction component 516 may include additional modules that may interact with the perception module 660. In some embodiments, the persistent world module 662 may receive a representation of the physical world based on data acquired by the perception module 660. The persistent world module 662 may also include representations of the physical world in various formats. For example, volumetric metadata 662b such as voxels, as well as a mesh 662c and a plane 662d may be stored. In some embodiments, other information such as depth maps may be saved.

[0173] In some embodiments, a representation of the physical world (such as Fig. 6A the representation shown) may provide relatively dense information about the physical world compared to a sparse map (such as the feature point-based tracking map described above).

[0174] In some embodiments, the perception module 660 may include modules that generate representations of the physical world in various formats, including, for example, a mesh 660d, planes, and semantics 660e. Representations of the physical world may be stored across local and remote storage media. Depending on, for example, the location of the storage media, the representations of the physical world may be described in different coordinate frames. For example, a representation of the physical world stored in a device may be described in a coordinate frame that is relatively local to the device. A representation of the physical world may have a counterpart stored in the cloud. The counterpart in the cloud may be described in a coordinate frame that can be shared by all devices in the XR system.

[0175] In some embodiments, these modules may generate the representations based on data within the sensing range of one or more sensors at the time of generating the representation, data captured at a previous time, and information in the persistent world module 662. In some embodiments, these components may operate on depth information captured by a depth sensor. However, an AR system may include visual sensors and may generate such representations by analyzing monocular or binocular visual information.

[0176] In some embodiments, these modules may operate on regions of the physical world. When the perception module 660 detects a change in the physical world in a sub-region of the physical world, those modules may be triggered to update the sub-region of the physical world. For example, such a change may be detected by detecting a new surface in the SDF model 660c or other criteria (such as changing the values of a sufficient number of voxels representing the sub-region).

[0177] The world reconstruction component 516 may include a component 664 that can receive a representation of the physical world from the perception module 660. Information about the physical world may be extracted by these components according to, for example, a usage request from an application. In some embodiments, information may be pushed to the usage component, such as via an indication of a change in a pre-identified region or a change in the representation of the physical world within the sensing range. The component 664 may include, for example, a game program and other components that perform processing for visual occlusion, physics-based interaction, and environmental reasoning.

[0178] In response to a query from the component 664, the perception module 660 may send a representation of the physical world in one or more formats. For example, when the component 664 indicates that the usage is for visual occlusion or physics-based interaction, the perception module 660 may send a representation of the surface. When the component 664 indicates that the usage is for environmental reasoning, the perception module 660 may send a mesh, planes, and semantics of the physical world.

[0179] In some embodiments, the perception module 660 may include components that format information to provide component 664. An example of such a component may be the raycasting component 660f. Using a component (e.g., component 664) may query information about the physical world from a particular viewpoint, for example. The raycasting component 660f may be selected from one or more representations of physical world data within the field of view from that viewpoint.

[0180] It should be understood from the above description that the perception module 660 or another component of the AR system may process data to create a 3D representation of a portion of the physical world. The data to be processed may be reduced by: culling portions of the 3D reconstruction volume based at least in part on the camera frustum and / or depth image; extracting and retaining planar data; capturing, retaining, and updating 3D reconstruction data in chunks that allow for local updates while maintaining neighborhood consistency; providing occlusion data to an application that generates such a scene, where the occlusion data is derived from a combination of one or more depth data sources; and / or performing multi-stage mesh simplification. The reconstruction may include data of varying levels of complexity, including for example raw data (e.g., real-time depth data), fused volume data (e.g., voxels), and computed data (e.g., meshes).

[0181] In some embodiments, components of the traversable world model may be distributed, where some portions are executed locally on the XR device and some portions are executed remotely, such as on a network-connected server or in the cloud. The distribution of information processing and storage between the local XR device and the cloud may affect the functionality and user experience of the XR system. For example, reducing the processing on the local device by offloading processing to the cloud may extend battery life and reduce heat generated on the local device. However, offloading too much processing to the cloud may introduce undesirable latency, which results in an unacceptable user experience.

[0182] Figure 6B Depicted is a distributed component architecture 600 configured for spatial computing according to some embodiments. The distributed component architecture 600 may include a traversable world component 602 (e.g., Figure 5A PW 538 therein), Lumin OS 604, API 606, SDK 608, and application 610. Lumin OS 604 may include a Linux-based kernel with custom drivers compatible with the XR device. The API 606 may include an application programming interface that permits XR applications (e.g., application 610) to access the spatial computing features of the XR device. The SDK 608 may include a software development kit that allows for the creation of XR applications.

[0183] One or more components in architecture 600 can create and maintain a model of a traversable world. In this example, sensor data is collected on a local device. Processing of this sensor data can be performed partially locally on the XR device and partially in the cloud. PW 538 can include an environmental map created at least in part based on data captured by AR devices worn by multiple users. During a session of an AR experience, individual AR devices (such as the wearable device described above in connection with Figure 4 can create a tracking map, which is a type of map.

[0184] In some embodiments, the device can include components for building sparse and dense maps. The tracking map can be used as the sparse map and can include information about the head pose of the AR device scanning the environment and objects detected within the environment at each head pose. Those head poses can be maintained locally for each device. For example, the head pose on each device can be relative to the initial head pose when the device started its session. As a result, each tracking map can be local to the device that created it. The dense map can include surface information, which can be represented by a mesh or depth information. Alternatively or additionally, the dense map can include higher-level information derived from the surface or depth information, such as the location and / or characteristics of planes and / or other objects.

[0185] In some embodiments, the creation of the dense map can be independent of the creation of the sparse map. For example, the creation of the dense map and the sparse map can be performed in separate processing pipelines within the AR system. For example, separate processing can enable different types of maps to be generated or processed at different rates. For example, the sparse map may be refreshed at a faster rate than the dense map. However, in some embodiments, even if performed in different pipelines, the processing of the dense map and the sparse map may be related. For example, a change in the physical world revealed in the sparse map can trigger an update of the dense map, and vice versa. Additionally, even if created independently, these maps can be used together. For example, the coordinate system derived from the sparse map can be used to define the location and / or orientation of objects in the dense map.

[0186] Sparse maps and / or dense maps can be persistently saved for reuse by the same device and / or shared with other devices. Such persistence can be achieved by storing the information in the cloud. An AR device can send a tracking map to the cloud, for example, to be merged with an environmental map selected from a persistent map previously stored in the cloud. In some embodiments, the selected persistent map can be sent from the cloud to the AR device for merging. In some embodiments, a persistent map can be oriented relative to one or more persistent coordinate frames. Such maps can be used as canonical maps because they can be used by any of a plurality of devices. In some embodiments, a model of the traversable world can include or be created from one or more canonical maps. Even if some operations are performed based on a device-local coordinate frame, the device can use a canonical map by determining the transformation between the device-local coordinate frame and the canonical map.

[0187] A canonical map can originate from a tracking map (TM) (e.g., Fig.31A TM 1102 in ) which can be promoted to a canonical map. The canonical map can be persistently saved so that a device accessing the canonical map can use the information in the canonical map to determine the position of an object represented in the canonical map in the physical world around the device once it determines the transformation between its local coordinate system and the coordinate system of the canonical map. In some embodiments, the TM can be a head pose sparse map created by an XR device. In some embodiments, a canonical map can be created when an XR device sends one or more TMs to a cloud server for merging with additional TMs captured by the XR device at different times or by other XR devices.

[0188] A canonical map or other map can provide information about various parts of the physical world represented by the data processed to create the corresponding map. Figure 7 Depicts an exemplary tracking map 700 according to some embodiments. The tracking map 700 can provide a floor plan 706 of physical objects in a corresponding physical world represented by points 702. In some embodiments, the map points 702 can represent features of a physical object that can include multiple features. For example, each corner of a table can be a feature represented by a point on the map. These features can be derived by processing an image, for example, an image can be acquired with a sensor of a wearable device in an augmented reality system. For example, features can be derived by processing an image frame output by the sensor to identify features based on large gradients or other suitable criteria in the image. Further processing may limit the number of features in each frame. For example, the processing can select features that are likely to represent persistent objects. One or more heuristics can be applied to this selection.

[0189] The tracking map 700 may include data about points 702 collected by the device. For each image frame with data points included in the tracking map, a pose may be stored. The pose may represent the orientation from which the image frame was captured such that feature points within each image frame may be spatially correlated. The pose may be determined by positioning information such as may be derived by sensors (such as IMU sensors) on the wearable device. Alternatively or additionally, the pose may be determined by matching the image frame to other image frames depicting overlapping portions of the physical world. By finding such position correlations, which may be achieved by matching a subset of feature points in two frames, the relative pose between the two frames may be calculated. The relative pose may be sufficient for the tracking map since the map may be relative to a coordinate system local to the device based on the initial pose of the device when starting to build the tracking map.

[0190] Not all feature points and image frames collected by the device may be retained as part of the tracking map since much of the information collected with sensors is likely to be redundant. Instead, only certain frames may be added to the map. Those frames may be selected based on one or more criteria such as the degree of overlap with image frames already present in the map, the number of new features they contain, or a quality metric of the features in the frame. Image frames not added to the tracking map may be discarded or used to modify the positions of features. As another alternative, all or most image frames represented as a set of features may be retained, but a subset of those frames may be designated as key frames for further processing.

[0191] Key frames may be processed to produce a key rig 704. Key frames may be processed to produce a three-dimensional set of feature points and saved as the key rig 704. For example, such processing may require comparing image frames simultaneously obtained from two cameras to stereoscopically determine the 3D positions of feature points. Metadata may be associated with these key frames and / or key rigs (such as poses).

[0192] The environment map may have any of a variety of formats depending on, for example, the storage location of the environment map, which includes, for example, local storage and remote storage of the AR device. For example, on a wearable device with limited memory, the map in remote storage may have a higher resolution than the map in local storage. To send a higher resolution map from remote storage to local storage, the map may be downsampled or otherwise converted to an appropriate format, such as by reducing the number of poses for each region of the physical world stored in the map and / or the number of feature points stored for each pose. In some embodiments, slices or portions of the high-resolution map from remote storage may be sent to local storage where the slices or portions are not downsampled.

[0193] When creating a new tracking map, the database of the environmental map can be updated. To determine which of the potentially very large number of environmental maps in the database will be updated, the update can include effectively selecting one or more environmental maps stored in the database that are related to the new tracking map. The selected one or more environmental maps can be ranked by relevance, and one or more of the highest-ranked maps can be selected for processing to merge the higher-ranked selected environmental maps with the new tracking map to create one or more updated environmental maps. When the new tracking map represents a portion of the physical world for which there is no pre-existing environmental map to update, the tracking map can be stored in the database as a new environmental map.

[0194] Watch standalone display

[0195] Methods and apparatuses for providing virtual content using an XR system independent of the position of eyes viewing the virtual content are described herein. Conventionally, virtual content is re-rendered upon any movement of the display system. For example, if a user wearing a display system views a virtual representation of a three-dimensional (3D) object on a display and walks around the area where the 3D object appears, the 3D object should be re-rendered for each viewing point so that the user has the feeling that he or she is walking around an object that occupies real space. However, re-rendering consumes a large amount of computational resources of the system and results in artifacts due to latency.

[0196] The inventors have recognized and understood that head pose (e.g., the position and orientation of a user wearing an XR system) can be used to render virtual content independent of eye rotation within the user's head. In some embodiments, a dynamic map of a scene can be generated based on multiple coordinate frames in real space across one or more sessions, such that virtual content interacting with the dynamic map can be robustly rendered independent of eye rotation within the user's head and / or independent of sensor distortion caused by, for example, heat generated during computationally intensive operations at high speed. In some embodiments, the configuration of the multiple coordinate frames can enable a first XR device worn by a first user and a second XR device worn by a second user to identify a common location in the scene. In some embodiments, the configuration of the multiple coordinate frames can enable a user wearing an XR device to view virtual content at the same location in the scene.

[0197] In some embodiments, a tracking map may be constructed in a world coordinate frame that may have a world origin. When the XR device is powered on, the world origin may be the first pose of the XR device. The world origin may be aligned with gravity, such that developers of XR applications can perform gravity alignment without additional work. Different tracking maps may be constructed in different world coordinate frames, as the tracking maps may be captured by the same XR device in different sessions and / or by different XR devices worn by different users. In some embodiments, a session of the XR device may start from when the device is powered on to when the device is powered off. In some embodiments, the XR device may have a head coordinate frame that may have a head origin. The head origin may be the current pose of the XR device when an image is captured. The difference between the head pose of the world coordinate frame and the head pose of the head coordinate frame may be used to estimate a tracking route.

[0198] In some embodiments, the XR device may have a camera coordinate frame that may have a camera origin. The camera origin may be the current pose of one or more sensors of the XR device. The inventors have recognized and understood that the configuration of the camera coordinate frame enables the robust display of virtual content independent of the rotation of the eyes within the user's head. This configuration also enables the robust display of virtual content independent of, for example, sensor deformations due to heat generated during operation.

[0199] In some embodiments, the XR device may have a head unit having a head-mounted frame that a user can fix to their head and may include two waveguides, one in front of each eye of the user. The waveguides may be transparent such that ambient light from objects in the real world can transmit through the waveguides and the user can see the objects in the real world. Each waveguide may send projection light from a projector to the corresponding eye of the user. The projection light may form an image on the retina of the eye. Thus, the retina of the eye receives ambient light and projection light. The user may simultaneously see objects in the real world as well as one or more virtual objects created by the projection light. In some embodiments, the XR device may have sensors that detect real-world objects around the user. These sensors may be, for example, cameras that capture images that can be processed to identify the positions of real-world objects.

[0200] In some embodiments, instead of attaching virtual content to the world coordinate frame, the XR system may assign a coordinate frame to the virtual content. Such a configuration enables the description of virtual content without having to consider where the virtual content is rendered to the user, but the virtual content may be attached to a more persistent frame location, such as being rendered at a specified location with respect to, for example Figures 14 to 20CThe described Persistent Coordinate Frame (PCF). When the position of an object changes, the XR device can detect the change in the environmental map and determine the movement of the head unit worn by the user relative to real-world objects.

[0201] Figure 8 Shown is a user experiencing virtual content rendered by XR system 10 in a physical environment. The XR system can include a first XR device 12.1 worn by a first user 14.1, a network 18, and a server 20. The user 14.1 is in a physical environment having a real object in the form of a table 16.

[0202] In the example shown, the first XR device 12.1 includes a head unit 22, a waist pack 24, and a cable connection 26. The first user 14.1 secures the head unit 22 to their head and secures the waist pack 24, which is remote from the head unit 22, to their waist. The cable connection 26 connects the head unit 22 to the waist pack 24. The head unit 22 includes technology for displaying one or more virtual objects to the first user 14.1 while allowing the first user 14.1 to see real objects such as the table 16. The waist pack 24 primarily includes the processing and communication capabilities of the first XR device 12.1. In some embodiments, the processing and communication capabilities may reside wholly or partially in the head unit 22, such that the waist pack 24 can be removed or can be located in another device such as a backpack.

[0203] In the example shown, the waist pack 24 is connected to the network 18 via a wireless connection. The server 20 is connected to the network 18 and maintains data representative of local content. The waist pack 24 downloads data representative of local content from the server 20 via the network 18. The waist pack 24 provides data to the head unit 22 via the cable connection 26. The head unit 22 can include a display having a light source (e.g., a laser light source or a light emitting diode (LED) light source), and a waveguide for directing the light.

[0204] In some embodiments, a first user 14.1 may attach a head unit 22 to their head and a hip pack 24 to their waist. The hip pack 24 may download image data from a server 20 via a network 18. The first user 14.1 may see a table 16 via a display of the head unit 22. A projector forming part of the head unit 22 may receive the image data from the hip pack 24 and generate light based on the image data. The light may travel through one or more waveguides forming part of the display of the head unit 22. The light may then exit the waveguide and propagate onto the retina of the first user 14.1's eye. The projector may generate light in a pattern replicated on the retina of the first user 14.1's eye. The light falling on the retina of the first user 14.1's eye may have a selected depth of field such that the first user 14.1 perceives an image at a preselected depth behind the waveguide. Additionally, the two eyes of the first user 14.1 may receive slightly different images such that the first user 14.1's brain perceives one or more three-dimensional images at a selected distance from the head unit 22. In the example shown, the first user 14.1 perceives virtual content 28 above the table 16. The scale of the virtual content 28 and its position and distance from the first user 14.1 are determined by the data representing the virtual content 28 and various coordinate frames used to display the virtual content 28 to the first user 14.1.

[0205] In the example shown, the virtual content 28 is not visible from the perspective of the drawing and is visible to the first user 14.1 via the first XR device 12.1. The virtual content 28 may initially reside as a data structure within the visual data and algorithms in the hip pack 24. Then, when the projector of the head unit 22 generates light based on the data structure, the data structure may manifest itself as light. It should be understood that although the virtual content 28 does not exist in the three-dimensional space in front of the first user 14.1, the virtual content 28 still represents in the three-dimensional space Figure 1 to illustrate the perception of the wearer of the head unit 22. Visualization of computer data in three-dimensional space may be used in this description to show how data structures perceived by one or more users to contribute to rendering are related to each other within the data structures in the hip pack 24.

[0206] Fig. 9 Illustrates components of a first XR device 12.1 according to some embodiments. The first XR device 12.1 may include a head unit 22 and various components forming part of the visual data and algorithms, including for example a rendering engine 30, various coordinate frames 32, various origin and destination coordinate frames 34, and various origin-to-destination coordinate frame transformers 36. The various coordinate systems may be based on the intrinsic properties of the XR device or may be determined by reference to other information, such as the persistent pose or persistent coordinate system described herein.

[0207] The head unit 22 may include a head-mounted frame 40, a display system 42, a real object detection camera 44, a motion tracking camera 46, and an inertial measurement unit 48.

[0208] The head-mounted frame 40 may have a shape that can be fixed to Figure 8 the head of the first user 14.1. The display system 42, the real object detection camera 44, the motion tracking camera 46, and the inertial measurement unit 48 may be mounted to the head-mounted frame 40 and thus move with the head-mounted frame 40.

[0209] The coordinate system 32 may include a local data system 52, a world frame system 54, a head frame system 56, and a camera frame system 58.

[0210] The local data system 52 may include a data channel 62, a local frame determination routine 64, and local frame storage instructions 66. The data channel 62 may be an internal software routine, such as a hardware component of an external cable or radio frequency receiver, or a hybrid component such as an open port. The data channel 62 may be configured to receive image data 68 representing virtual content.

[0211] The local frame determination routine 64 may be connected to the data channel 62. The local frame determination routine 64 may be configured to determine a local coordinate frame 70. In some embodiments, the local frame determination routine may determine the local coordinate frame based on a real-world object or a real-world location. In some embodiments, the local coordinate frame may be based on the top edge relative to the bottom edge of a browser window, the head or feet of a character, nodes on the outer surface of a prism or bounding box enclosing virtual content, or any other suitable location of a coordinate frame that defines the orientation of placement of the virtual content and the location where the virtual content is placed (e.g., nodes such as placement nodes or PCF nodes).

[0212] The local frame storage instructions 66 may be connected to the local frame determination routine 64. Those skilled in the art will understand that software modules and routines are "connected" to each other through subroutines, calls, etc. The local frame storage instructions 66 may store the local coordinate frame 70 as a local coordinate frame 72 within the origin and destination coordinate frame 34. In some embodiments, the origin and destination coordinate frame 34 may be one or more coordinate frames that can be manipulated or transformed to make the virtual content persistent between sessions. In some embodiments, a session may be the time period between the startup and shutdown of an XR device. Two sessions may be two startup and shutdown time periods of a single XR device, or the startup and shutdown time periods of two different XR devices.

[0213] In some embodiments, the origin and destination coordinate frames 34 can be the coordinate frames involved in one or more transformations required for the XR devices of the first user and the second user to identify a common location. In some embodiments, the destination coordinate frame can be the output of a series of calculations and transformations applied to the target coordinate frame so that the first and second users view virtual content in the same location.

[0214] The rendering engine 30 can be connected to the data channel 62. The rendering engine 30 can receive the image data 68 from the data channel 62 such that the rendering engine 30 can render virtual content at least in part based on the image data 68.

[0215] The display system 42 can be connected to the rendering engine 30. The display system 42 can include components that transform the image data 68 into visible light. The visible light can form two patterns, one for each eye. The visible light can enter Figure 8 the eyes of the first user 14.1 and can be detected on the retinas of the eyes of the first user 14.1.

[0216] The real object detection camera 44 can include one or more cameras that can capture images from different sides of the head-mounted frame 40. The motion tracking camera 46 can include one or more cameras that can capture images on the sides of the head-mounted frame 40. A single set of one or more cameras can be used instead of the two sets of one or more cameras representing the real object detection camera 44 and the motion tracking camera 46. In some embodiments, the cameras 44, 46 can capture images. As described above, these cameras can collect data for constructing a tracking map.

[0217] The inertial measurement unit 48 can include multiple devices for detecting the motion of the head unit 22. The inertial measurement unit 48 can include a gravity sensor, one or more accelerometers, and one or more gyroscopes. The sensors of the inertial measurement unit 48 collectively track the motion of the head unit 22 in at least three orthogonal directions and about at least three orthogonal axes.

[0218] In the example shown, the world frame system 54 includes a world surface determination routine 78, a world frame determination routine 80, and world frame storage instructions 82. The world surface determination routine 78 is connected to the real object detection camera 44. The world surface determination routine 78 receives images and / or key frames based on the images captured by the real object detection camera 44 and processes the images to identify the surfaces in the images. A depth sensor (not shown) can determine the distance to the surfaces. Thus, these surfaces are represented by data in three dimensions including their size, shape, and distance from the real object detection camera.

[0219] In some embodiments, the world coordinate frame 84 may be based on the origin at the time of initializing the head pose session. In some embodiments, the world coordinate frame may be located at the position where the device is started, or if the head pose is lost during the startup session, the world coordinate frame may be located at a new location. In some embodiments, the world coordinate frame may be the origin at the start of the head pose session.

[0220] In the example shown, the world frame determination routine 80 is connected to the world surface determination routine 78 and determines the world coordinate frame 84 based on the position of the surface determined by the world surface determination routine 78. The world frame storage instruction 82 is connected to the world frame determination routine 80 to receive the world coordinate frame 84 from the world frame determination routine 80. The world frame storage instruction 82 stores the world coordinate frame 84 as the world coordinate frame 86 within the origin and destination coordinate frame 34.

[0221] The head frame system 56 may include a head frame determination routine 90 and head frame storage instructions 92. The head frame determination routine 90 may be connected to the motion tracking camera 46 and the inertial measurement unit 48. The head frame determination routine 90 may use data from the motion tracking camera 46 and the inertial measurement unit 48 to calculate the head coordinate frame 94. For example, the inertial measurement unit 48 may have a gravity sensor that determines the direction of gravity relative to the head unit 22. The motion tracking camera 46 may continuously capture images that the head frame determination routine 90 uses to refine the head coordinate frame 94. When Figure 8 the first user 14.1 in

[0222] moves their head, the head unit 22 moves. The motion tracking camera 46 and the inertial measurement unit 48 may continuously provide data to the head frame determination routine 90 such that the head frame determination routine 90 can update the head coordinate frame 94.

[0223] The camera frame system 58 may include camera intrinsic characteristics 98. The camera intrinsic characteristics 98 may include the dimensions of the head unit 22 as its design and manufacturing features. The camera intrinsic characteristics 98 can be used to calculate the camera coordinate frame 100 stored within the origin and destination coordinate frame 34.

[0224] In some embodiments, the camera coordinate frame 100 may include Figure 8 all pupil positions of the left eye of the first user 14.1 in

[0225] When the left eye moves from left to right or up and down, the pupil position of the left eye is within the camera coordinate frame 100. Additionally, the pupil position of the right eye is within the camera coordinate frame 100 of the right eye. In some embodiments, the camera coordinate frame 100 may include the position of the camera relative to the local coordinate frame when an image is captured.

[0226] The origin-to-destination coordinate frame transformer 36 may include a local-to-world coordinate transformer 104, a world-to-head coordinate transformer 106, and a head-to-camera coordinate transformer 108. The local-to-world coordinate transformer 104 may receive the local coordinate frame 72 and transform the local coordinate frame 72 into the world coordinate frame 86. The transformation of the local coordinate frame 72 to the world coordinate frame 86 may be represented as the local coordinate frame transformed into the world coordinate frame 110 within the world coordinate frame 86.

[0227] The world-to-head coordinate transformer 106 may transform from the world coordinate frame 86 to the head coordinate frame 96. The world-to-head coordinate transformer 106 may transform the local coordinate frame transformed into the world coordinate frame 110 into the head coordinate frame 96. This transformation may be represented as the local coordinate frame transformed into the head coordinate frame 112 within the head coordinate frame 96.

[0228] Fig.10 is a spatial representation of the various origin and destination coordinate frames 34. The local coordinate frame 72, the world coordinate frame 86, the head coordinate frame 96, and the camera coordinate frame 100 are represented in this figure. In some embodiments, when virtual content is placed in the real world so that the user can view the virtual content, the local coordinate frame associated with the XR content 28 may have a position and rotation relative to the local and / or world coordinate frames and / or PCF (e.g., nodes and orientation directions may be provided). Each camera may have its own camera coordinate frame 100 that includes all pupil positions of one eye. Reference numerals 104A and 106A respectively represent by Fig. 9The transformations performed by the local-to-world coordinate transformer 104, the world-to-head coordinate transformer 106, and the head-to-camera coordinate transformer 108 in

[0229] Fig.11 Depicts a camera rendering protocol for transforming from a head coordinate frame to a camera coordinate frame according to some embodiments. In the example shown, the pupil of a single eye moves from position A to position B. The virtual object to appear stationary will depend on the projection of the pupil's position onto the depth plane at one of two positions A or B (assuming the camera is configured to use a pupil-based coordinate frame). As a result, when the eye moves from position A to position B, using the pupil coordinate frame transformed to the head coordinate frame will result in jitter of the stationary virtual object. This situation is called view-dependent display or projection.

[0230] As Fig.12 shown, the camera coordinate frame (e.g., CR) is placed and encompasses all pupil positions, and now the object projection will be consistent regardless of the pupil positions A and B. The head coordinate frame is transformed to the CR frame, which is called view-independent display or projection. Image reprojection can be applied to the virtual content to address changes in eye position. However, since the rendering remains in the same position, jitter can be minimized.

[0231] Fig.13 The display system 42 is shown in more detail. The display system 42 includes a stereo analyzer 144 that is connected to the rendering engine 30 and forms part of the visual data and algorithms.

[0232] The display system 42 further includes a left projector 166A and a right projector 166B, and a left waveguide 170A and a right waveguide 170B. The left projector 166A and the right projector 166B are connected to a power source. Each projector 166A and 166B has a corresponding input for the image data to be provided to the respective projector 166A or 166B. The respective projector 166A or 166B generates and emits light in a two-dimensional pattern when powered on. The left waveguide 170A and the right waveguide 170B are positioned to receive light from the left projector 166A and the right projector 166B, respectively. The left waveguide 170A and the right waveguide 170B are transparent waveguides.

[0233] In use, the user mounts the head-mounted frame 40 on their head. The components of the head-mounted frame 40 may include, for example, a strap (not shown) that wraps around the back of the user's head. The left waveguide 170A and the right waveguide 170B are then positioned in front of the user's left eye 220A and right eye 220B.

[0234] The rendering engine 30 inputs the image data it receives into the stereo analyzer 144. This image data is Figure 8Three-dimensional image data of the local content 28. The image data is projected onto a plurality of virtual planes. The stereoscopic analyzer 144 analyzes the image data to determine a left image data set and a right image data set based on the image data used for projection onto each depth plane. The left image data set and the right image data set are data sets representing two-dimensional images that are projected in three dimensions to give the user a sense of depth.

[0235] The stereoscopic analyzer 144 inputs the left image data set and the right image data set into the left projector 166A and the right projector 166B. Then, the left projector 166A and the right projector 166B create a left illumination pattern and a right illumination pattern. The components of the display system 42 are shown in a plan view, but it should be understood that when shown in a front view, the left pattern and the right pattern are two-dimensional patterns. Each light pattern includes a plurality of pixels. For illustrative purposes, light rays 224A and 226A from two pixels are shown leaving the left projector 166A and entering the left waveguide 170A. The light rays 224A and 226A are reflected from the side surfaces of the left waveguide 170A. The light rays 224A and 226A are shown propagating from left to right within the left waveguide 170A by total internal reflection, but it should be understood that the light rays 224A and 226A also propagate in a certain direction into the paper using a refraction and reflection system.

[0236] The light rays 224A and 226A leave the left optical waveguide 170A through the pupil 228A and then enter the left eye 220A through the pupil 230A of the left eye 220A. Then, the light rays 224A and 226A fall on the retina 232A of the left eye 220A. In this way, the left light pattern falls on the retina 232A of the left eye 220A. The user has the perception that the pixels formed on the retina 232A are pixels 234A and 236A that the user perceives to be at a certain distance on the side of the left waveguide 170A opposite the left eye 220A. Depth perception is created by manipulating the focal length of the light.

[0237] In a similar manner, the stereoscopic analyzer 144 inputs the right image data set into the right projector 166B. The right projector 166B projects a right light pattern, which is represented by pixels in the form of light rays 224B and 226B. The light rays 224B and 226B are reflected within the right waveguide 170B and exit through the pupil 228B. The light rays 224B and 226B then enter and fall on the retina 232B of the right eye 220B through the pupil 230B of the right eye 220B. The pixels of the light rays 224B and 226B are perceived as pixels 134B and 236B behind the right waveguide 170B.

[0238] The patterns created on retinas 232A and 232B are perceived as a left image and a right image, respectively. Due to the function of the stereoscopic analyzer 144, the left image and the right image are slightly different from each other. The left image and the right image are perceived as a three-dimensional rendering in the user's mind.

[0239] As mentioned, the left waveguide 170A and the right waveguide 170B are transparent. Light from real objects on the table 16 on the sides of the left waveguide 170A and the right waveguide 170B opposite the eyes 220A and 220B can be projected through the left waveguide 170A and the right waveguide 170B and fall on the retinas 232A and 232B.

[0240] Persistent Coordinate Frame (PCF)

[0241] This document describes methods and apparatuses for providing spatial persistence between user instances in a shared space. Without spatial persistence, virtual content placed by a user in the physical world during a session may not be present or may be misplaced in the user's view in different sessions. Without spatial persistence, virtual content placed by one user in the physical world may not be present or may be misaligned in the view of a second user, even if the second user intends to share the same physical space experience as the first user.

[0242] The inventors have recognized and understood that spatial persistence can be provided through a Persistent Coordinate Frame (PCF). The PCF can be defined based on one or more points that represent features (e.g., corners, edges) identified in the physical world. The features can be selected such that they appear the same from one user instance of the XR system to another.

[0243] In addition, during tracking, drift that causes the calculated tracking path (e.g., camera trajectory) to deviate from the actual tracking path results in misalignment of the position of virtual content when rendering relative to a local map based only on the tracking map. As the XR device collects more information about the scene over time, the tracking map of the space can be refined to correct the drift. However, if virtual content is placed on a real object and saved relative to the device's world coordinate frame derived from the tracking map before the map is refined, the virtual content may be displaced as if the real object has moved during the map refinement. The PCF can be updated based on the map refinement because the PCF is defined based on features and is updated as the features move during the map refinement.

[0244] The PCF can include six degrees of freedom for translation and rotation with respect to the map coordinate system. The PCF can be stored in a local storage medium and / or a remote storage medium. Depending on, for example, the storage location, the translation and rotation of the PCF can be calculated with respect to the map coordinate system. For example, the PCF used locally by a device may have translation and rotation with respect to the device's world coordinate frame. The PCF in the cloud may have translation and rotation with respect to the canonical coordinate frame of the canonical map.

[0245] The PCF can provide a sparse representation of the physical world, providing less information about the physical world than all available information, such that they can be processed and transferred efficiently. Techniques for processing persistent spatial information can include creating a dynamic map based on one or more coordinate systems in the real space spanning one or more sessions, generating a persistent coordinate frame (PCF) on the sparse map, which can be exposed to XR applications through, for example, an application programming interface (API).

[0246] Fig.14 is a block diagram showing the creation of a persistent coordinate frame (PCF) and the addition of XR content to the PCF according to some embodiments. Each block can represent digital information stored in a computer memory. In the case of application 1180, the data can represent computer-executable instructions. In the case of virtual content 1170, the digital information can define, for example, virtual objects specified by application 1180. In the case of other blocks, the digital information can characterize certain aspects of the physical world.

[0247] In the illustrated embodiment, one or more PCFs are created based on images captured by sensors on a wearable device. In Fig.14 the embodiment, the sensors are visual image cameras. These cameras can be the same cameras as those used to form the tracking map. Thus, some of the processing suggested by Fig.14 can be performed as part of updating the tracking map. However, Fig.14 shows that information providing persistence is generated in addition to the tracking map.

[0248] To derive a 3D PCF, two images 1110 from two cameras installed on the wearable device in a configuration capable of performing stereoscopic image analysis are processed together. Fig.14 shows Image 1 and Image 2, each of Image 1 and Image 2 being from one of the cameras. For simplicity, a single image from each camera is shown. However, each camera can output a stream of image frames, and the Fig.14 processing can be performed for multiple image frames in the stream.

[0249] Thus, Image 1 and Image 2 can be respectively one frame in a sequence of image frames. The processing can be repeated for consecutive image frames in the sequence Fig.14 The processing shown until an image frame containing feature points provides a suitable image to form persistent spatial information based on the image. Alternatively or additionally, when the user moves such that the user is no longer close enough to a previously identified PCF to reliably use the PCF to determine a position relative to the physical world, the Fig.14 processing may be repeated. For example, the XR system may maintain the current PCF for the user. When the distance exceeds a threshold, the system may switch to a new current PCF closer to the user, which may be generated using an image frame obtained at the user's current position according to the Fig.14 process.

[0250] Even when generating a single PCF, a stream of image frames may be processed to identify an image frame depicting content in the physical world, which may be stable and may be easily recognizable by a device near the region of the physical world depicted in the image frame. In the Fig.14 embodiment, the processing begins with the identification of feature 1120 in the image. For example, a feature may be identified by looking for positions in the image that exceed a threshold or the gradient of other features, which may correspond to, for example, the corners of an object. In the illustrated embodiment, the feature is a point, but other recognizable features, such as edges, may alternatively or additionally be used.

[0251] In the illustrated embodiment, a fixed number N of features 1120 are selected for further processing. Those feature points may be selected based on one or more criteria, such as the magnitude of the gradient or proximity to other feature points. Alternatively or additionally, feature points may be tentatively selected, for example, based on characteristics that imply the feature points are persistent. For example, a heuristic may be defined based on the characteristics of feature points that may correspond to the corners of a window or a door or a large piece of furniture. Such a heuristic may take into account the feature points themselves and the things around them. As a specific example, the number of feature points in each image may be between 100 and 500 or between 150 and 250, such as 200.

[0252] Regardless of the number of feature points selected, descriptors 1130 may be calculated for the feature points. In this example, a descriptor is calculated for each selected feature point, but descriptors may be calculated for groups of feature points or subsets of feature points or all features within an image. The descriptors characterize the feature points such that feature points representing the same object in the physical world are assigned similar descriptors. The descriptors may enable the alignment of two frames, such as may occur when one map is positioned relative to another map. Instead of searching for the relative orientation of the frames that minimizes the distance between the feature points of two images, the initial alignment of the two frames may be performed by identifying feature points with similar descriptors. The alignment of the image frames may be based on the aligned points with similar descriptors, which may require less processing compared to calculating the alignment of all feature points in the image.

[0253] A descriptor can be calculated as a mapping from feature points to the descriptor, or in some embodiments, as a mapping from a patch of the image around the feature points to the descriptor. The descriptor can be a numerical quantity. U.S. Patent Application 16 / 190,948 describes the calculated descriptor of feature points, and its entire content is incorporated herein by reference.

[0254] In Fig.14 the example of, a descriptor 1130 is calculated for each feature point in each image frame. Based on the descriptor and / or the feature points and / or the image itself, an image frame can be identified as a key frame 1140. In the illustrated embodiment, a key frame is an image frame that meets a certain criterion, and then that image frame is selected for further processing. For example, when creating a tracking map, an image frame that can add meaningful information to the map can be selected as a key frame to be integrated into the map. On the other hand, an image frame that substantially overlaps with an area where an image frame has already been integrated into the map can be discarded so that they do not become key frames. Alternatively or additionally, key frames can be selected based on the number and / or type of feature points in the image frame. In Fig.14 the embodiment of, a key frame 1150 selected to be included in the tracking map can also be regarded as a key frame for determining the PCF, but different or additional criteria for selecting the key frame for generating the PCF can be used.

[0255] Although Fig.14 it is shown that key frames are used for further processing, the information obtained from the images can be processed in other forms. For example, feature points in a key assembly can be processed alternatively or additionally. Also, although key frames are described as being derived from a single image frame, there does not have to be a one-to-one relationship between the key frames and the acquired image frames. For example, a key frame can be obtained from multiple image frames, such as by stitching or aggregating the image frames together so that only the features that appear in multiple images are retained in the key frame.

[0256] A key frame can include image information and / or metadata associated with the image information. In some embodiments, the information obtained by cameras 44, 46 ( Fig. 9)The captured images are computed as one or more keyframes (e.g., keyframe 1, 2). In some embodiments, a keyframe may include a camera pose. In some embodiments, a keyframe may include one or more camera images captured at the camera pose. In some embodiments, the XR system may determine that a portion of the camera image captured at the camera pose is useless and thus does not include that portion in the keyframe. Thus, aligning new images with an earlier perception of the scene using keyframes can reduce the use of computing resources of the XR system. In some embodiments, a keyframe may include an image and / or image data at a location with a direction / angle. In some embodiments, a keyframe may include a location and orientation where one or more map points can be observed. In some embodiments, a keyframe may include a coordinate frame with an ID. U.S. Patent Application No. 15 / 877,359 describes keyframes, the entire contents of which are incorporated herein by reference.

[0257] Some or all of the keyframes 1140 may be selected for further processing, such as generating a persistent pose 1150 for the keyframe. This selection may be based on the characteristics of all feature points or a subset thereof in the image frame. These characteristics may be determined by processing the descriptors, features, and / or the image frame itself. As a specific example, the selection may be based on a cluster of feature points identified as likely related to a persistent object.

[0258] Each keyframe is associated with the pose of the camera that acquired the keyframe. For the keyframes selected for processing into persistent poses, this pose information may be saved along with other metadata about the keyframe, such as the WiFi fingerprint and / or GPS coordinates at the time of acquisition and / or at the acquisition location.

[0259] A persistent pose is a source of information that the device can use to orient itself relative to previously acquired information about the physical world. For example, if the keyframe from which the persistent pose was created is incorporated into a map of the physical world, the device can use a sufficient number of feature points in the keyframe associated with the persistent pose to orient itself relative to that persistent pose. The device can align its current image of the surrounding environment with the persistent pose. This alignment may be based on matching the current image with the image 1110, features 1120, and / or descriptors 1130 that gave rise to the persistent pose, or any subset of that image or those features or descriptors. In some embodiments, the current image frame that matches the persistent pose may be another keyframe that has been incorporated into the device's tracking map.

[0260] The information about the persistent pose may be stored in a format that facilitates sharing among multiple applications, which may execute on the same or different devices. In Fig.14In an example, some or all of the persistent poses may be reflected as a Persistent Coordinate Frame (PCF) 1160. Similar to the persistent poses, the PCF may be associated with a map and may include a set of features or other information that the device can use to determine its orientation relative to the PCF. The PCF may include a transformation that defines a transformation relative to the origin of the map such that by associating its position with the PCF, the device can determine its position relative to any object in the physical world reflected in the map.

[0261] Since the PCF provides a mechanism for determining positions relative to physical objects, an application (such as application 1180) can define the positions of virtual objects relative to one or more PCFs, which serve as anchors for virtual content 1170. For example, Fig.14 shows that App 1 has associated its virtual content 2 with PCFs 1, 2. Similarly, App 2 has associated its virtual content 3 with PCFs 1, 2. It is also shown that App 1 has associated its virtual content 1 with PCFs 4, 5, and it is shown that App 2 has associated its virtual content 4 with PCF 3. In some embodiments, PCF 3 may be based on Image 3 (not shown), and PCFs 4, 5 may be based on Image 4 and Image 5 (not shown), similar to how PCFs 1, 2 are based on Image 1 and Image 2. When rendering this virtual content, the device can apply one or more transformations to calculate information such as the position of the virtual content relative to the device's display and / or the position of the physical object relative to the desired position of the virtual content. Using the PCF as a reference can simplify such calculations.

[0262] In some embodiments, the persistent pose may be a coordinate position and / or orientation with one or more associated key frames. In some embodiments, a persistent pose may be automatically created after the user has traveled a certain distance (such as three meters). In some embodiments, the persistent pose may be used as a reference point during positioning. In some embodiments, the persistent pose may be stored in the traversable world (e.g., traversable world module 538).

[0263] In some embodiments, a new PCF may be determined based on a predetermined distance allowed between adjacent PCFs. In some embodiments, when the user travels a predetermined distance (such as five meters), one or more persistent poses may be calculated into the PCF. In some embodiments, the PCF may be associated with one or more world coordinate frames and / or canonical coordinate frames in the traversable world, for example. In some embodiments, depending on, for example, security settings, the PCF may be stored in a local database and / or a remote database.

[0264] Fig.15Method 4700 for establishing and using a persistent coordinate framework according to some embodiments is shown. Method 4700 may begin with capturing (act 4702) an image of a scene (e.g., images 1 and 2 in Fig.14 ) using one or more sensors of an XR device. Multiple cameras may be used, and one camera may generate multiple images, e.g., in the form of a stream.

[0265] Method 4700 may include extracting (4704) points of interest from the captured image (e.g., map point 702 in Figure 7 ), feature 1120 in Fig.14 ), generating (act 4706) a descriptor of the extracted points of interest (e.g., descriptor 1130 in Fig.14 ), and generating (act 4708) a keyframe (e.g., keyframe 1140) based on the descriptor. In some embodiments, the method may compare the points of interest in the keyframes and form pairs of keyframes that share a predetermined amount of points of interest. The method may use each pair of keyframes to reconstruct a portion of the physical world. The mapped portion of the physical world may be saved as a 3D feature (e.g., key assembly 704 in Figure 7 ). In some embodiments, a selected portion of the pair of keyframes may be used to build the 3D feature. In some embodiments, the result of the mapping may be selectively saved. Keyframes not used to build the 3D feature may be associated with the 3D feature by pose, e.g., representing the distance between keyframes using the covariance matrix between the poses of the keyframes. In some embodiments, pairs of keyframes may be selected to build the 3D feature such that the distance between each built 3D feature is within a predetermined distance, which may be determined to balance the required computational amount and the accuracy level of the resulting model. Such a method can provide an XR system with a model of the physical world having a data amount suitable for performing efficient and accurate calculations. In some embodiments, the covariance matrix of two images may include the covariance between the poses (e.g., six degrees of freedom) of the two images.

[0266] Method 4700 may include generating (act 4710) a persistent pose based on the keyframes. In some embodiments, the method may include generating a persistent pose based on the 3D feature reconstructed from the pair of keyframes. In some embodiments, the persistent pose may be attached to the 3D feature. In some embodiments, the persistent pose may include the pose of the keyframe used to build the 3D feature. In some embodiments, the persistent pose may include the average pose of the keyframes used to build the 3D feature. In some embodiments, a persistent pose may be generated such that the distance between adjacent persistent poses is within a predetermined value, e.g., in the range of one meter to five meters, any value in between, or any other appropriate value. In some embodiments, the distance between adjacent persistent poses may be represented by the covariance matrix of the adjacent persistent poses.

[0267] Method 4700 may include generating (action 4712) a PCF based on a persistent pose. In some embodiments, the PCF may be attached to a 3D feature. In some embodiments, the PCF may be associated with one or more persistent poses. In some embodiments, the PCF may include the pose of one of the associated persistent poses. In some embodiments, the PCF may include the average pose of the poses of the associated persistent poses. In some embodiments, the PCF may be generated such that the distance between adjacent PCFs is within a predetermined value, such as in the range of three meters to ten meters, any value between the two, or any other suitable value. In some embodiments, the distance between adjacent PCFs may be represented by the covariance matrix of the adjacent PCFs. In some embodiments, the PCF may be exposed to the XR application via, for example, an application programming interface (API), such that the XR application can access the model of the physical world through the PCF without accessing the model itself.

[0268] Method 4700 may include associating (action 4714) the image data of the virtual object to be displayed by the XR device with at least one of the PCFs. In some embodiments, the method may include calculating the translation and orientation of the virtual object relative to the associated PCF. It should be understood that it is not necessary to associate the virtual object with the PCF generated by the device that places the virtual object. For example, the device may obtain the saved PCF in the canonical map in the cloud and associate the virtual object with the obtained PCF. It should be understood that when the PCF is adjusted over time, the virtual object may move together with the associated PCF.

[0269] Fig.16 Visual data and algorithms of a first XR device 12.1, a second XR device 12.2, and a server 20 according to some embodiments are shown. Fig.16 The components shown therein may operate to perform some or all of the operations associated with generating, updating, and / or using spatial information (such as persistent poses, persistent coordinate frames, tracking maps, or canonical maps) as described herein. Although not shown, the first XR device 12.1 may be configured to be the same as the second XR device 12.2. The server 20 may have a map storage routine 118, a canonical map 120, a map transmitter 122, and a map merging algorithm 124.

[0270] The second XR device 12.2 that may be in the same scene as the first XR device 12.1 may include a permanent coordinate frame (PCF) integration unit 1300, an application 1302 that generates image data 68 that can be used to render virtual objects, and a frame embedding generator 308 (see Fig.21 ). In some embodiments, a map download system 126, a PCF recognition system 128, the ground Figure 2 The positioning module 130, the canonical map merger 132, the canonical map 133, and the map publisher 136 are assembled into the traversable world unit 1304. The PCF integration unit 1300 can be connected to the traversable world unit 1304 and other components of the second XR device 12.2 to allow for the acquisition, generation, use, upload, and download of PCFs.

[0271] Maps including PCFs can achieve more persistence in a changing world. In some embodiments, the positioning, such as tracking a map of matching features such as images, can include selecting features representing persistent content from a map constituted by PCFs, which enables fast matching and / or positioning. For example, in a world where people enter and exit a scene and objects such as doors move relative to the scene, less storage space and transmission rate are required, and it is possible to map the scene using individual PCFs and their relationships to each other (e.g., the integrated constellation of PCFs).

[0272] In some embodiments, the PCF integration unit 1300 can include a PCF 1306, a PCF tracker 1308, a persistent pose acquirer 1310, a PCF inspector 1312, a PCF generation system 1314, a coordinate frame calculator 1316, a persistent pose calculator 1318, and three transformers including a tracking map and persistent pose transformer 1320, a persistent pose and PCF transformer 1322, and a PCF and image data transformer 1324, which are previously stored in the data storage of the storage unit of the second XR device 12.2.

[0273] In some embodiments, the PCF tracker 1308 can have an open prompt and a close prompt that can be selected by the application 1302. The application 1302 can be executable by the processor of the second XR device 12.2 to, for example, display virtual content. The application 1302 can have a call to open the PCF tracker 1308 via the open prompt. When the PCF tracker 1308 is open, the PCF tracker 1308 can generate a PCF. The application 1302 can have a subsequent call to close the PCF tracker 1308 via the close prompt. When the PCF tracker 1308 is closed, the PCF tracker 1308 terminates PCF generation.

[0274] In some embodiments, the server 20 can include a plurality of persistent poses 1332 and a plurality of PCFs 1330 that have been previously saved in association with the canonical map 120. The map transmitter 122 can send the canonical map 120 together with the persistent poses 1332 and / or PCFs 1330 to the second XR device 12.2. The persistent poses 1332 and PCFs 1330 can be stored on the second XR device 12.2 in association with the canonical map 133. When Figure 2When located to the canonical map 133, it can store the persistent pose 1332 and the PCF 1330 in association with the ground. Figure 2 Associate and store the persistent pose 1332 and the PCF 1330.

[0275] In some embodiments, the persistent pose acquirer 1310 can acquire the persistent pose associated with the ground. Figure 2 The PCF checker 1312 can be connected to the persistent pose acquirer 1310. The PCF checker 1312 can obtain the PCF from the PCF 1306 based on the persistent pose obtained by the persistent pose acquirer 1310. The PCF obtained by the PCF checker 1312 can form an initial group of PCFs for image display based on the PCF.

[0276] In some embodiments, the application 1302 may need to generate additional PCFs. For example, if the user moves to an area that has not been mapped before, the application 1302 can open the PCF tracker 1308. The PCF generation system 1314 can be connected to the PCF tracker 1308 and start generating PCFs based on the ground as it starts to expand. Figure 2 Start expanding and start generating PCFs based on the ground. Figure 2 The PCFs generated by the PCF generation system 1314 can form a second group of PCFs, which can be used for image display based on the PCF.

[0277] The coordinate frame calculator 1316 can be connected to the PCF checker 1312. After the PCF checker 1312 obtains the PCF, the coordinate frame calculator 1316 can call the head coordinate frame 96 to determine the head pose of the second XR device 12.2. The coordinate frame calculator 1316 can also call the persistent pose calculator 1318. The persistent pose calculator 1318 can be directly or indirectly connected to the frame embedding generator 308. In some embodiments, an image / frame can be specified as a key frame after traveling a threshold distance (e.g., 3 meters) from a previous key frame. The persistent pose calculator 1318 can generate a persistent pose based on multiple (e.g., three) key frames. In some embodiments, the persistent pose can be substantially the average of the coordinate frames of multiple key frames.

[0278] The tracking map and persistent pose transformer 1320 can be connected to the ground and the persistent pose calculator 1318. Figure 2 The tracking map and persistent pose transformer 1320 can transform the ground into a persistent pose to determine the persistent pose at the origin relative to the ground. Figure 2 Transform into a persistent pose to determine the persistent pose at the origin relative to the ground. Figure 2 The persistent pose at the origin relative to the ground.

[0279] The Persistent Pose and PCF Transformer 1322 can be connected to the Tracking Map and Persistent Pose Transformer 1320 and further connected to the PCF Inspector 1312 and the PCF Generation System 1314. The Persistent Pose and PCF Transformer 1322 can transform the persistent pose (to which the tracking map has been transformed) from the PCF Inspector 1312 and the PCF Generation System 1314 into a PCF to determine the PCF relative to the persistent pose.

[0280] The PCF and Image Data Transformer 1324 can be connected to the Persistent Pose and PCF Transformer 1322 and the data channel 62. The PCF and Image Data Transformer 1324 transforms the PCF into image data 68. The rendering engine 30 can be connected to the PCF and Image Data Transformer 1324 to display the image data 68 to the user relative to the PCF.

[0281] The PCF Integration Unit 1300 can store additional PCFs generated by the PCF Generation System 1314 in the PCF 1306. The PCF 1306 can be stored relative to the persistent pose. When the map publisher 136 sends the map to the server 20, Figure 2 the map publisher 136 can obtain the PCF 1306 and the persistent pose associated with the PCF 1306, and the map publisher 136 also sends the PCF and the persistent pose associated with the map to the server 20. Figure 2 When the map storage routine 118 of the server 20 stores the map, Figure 2 the map storage routine 118 can also store the persistent pose and the PCF generated by the second viewing device 12.2. The map merging algorithm 124 can use the persistent pose and the PCF of the map associated with the canonical map 120 and stored separately in the persistent pose 1332 and the PCF 1330 to create the canonical map 120. Figure 2

[0282] The first XR device 12.1 can include a PCF integration unit similar to the PCF integration unit 1300 of the second XR device 12.2. When the map sender 122 sends the canonical map 120 to the first XR device 12.1, the map sender 122 can send the persistent pose 1332 and the PCF 1330 associated with the canonical map 120 and originating from the second XR device 12.2. The first XR device 12.1 can store the PCF and the persistent pose in the data storage on the storage device of the first XR device 12.1. Then, the first XR device 12.1 can utilize the persistent pose and the PCF originating from the second XR device 12.2 for image display relative to the PCF. Additionally or alternatively, the first XR device 12.1 can obtain, generate, use, upload, and download PCFs and persistent poses in a manner similar to that of the second XR device 12.2 described above.

[0283] In the illustrated example, the first XR device 12.1 generates a local tracking map (hereinafter referred to as "the Figure 1 "), and the map storage routine 118 receives the Figure 1 from the first XR device 12.1. Then, the map storage routine 118 stores the Figure 1 as the canonical map 120 on the storage device of the server 20.

[0284] The second XR device 12.2 includes a map download system 126, an anchor recognition system 128, a positioning module 130, a canonical map merger 132, a local content positioning system 134, and a map publisher 136.

[0285] In use, the map transmitter 122 sends the canonical map 120 to the second XR device 12.2, and the map download system 126 downloads and stores the canonical map 120 as the canonical map 133 from the server 20.

[0286] The anchor recognition system 128 is connected to the world surface determination routine 78. The anchor recognition system 128 identifies anchors based on the objects detected by the world surface determination routine 78. The anchor recognition system 128 uses the anchors to generate a second map (the Figure 2 ). As shown in the loop 138, the anchor recognition system 128 continues to identify anchors and continues to update the Figure 2 . Based on the data provided by the world surface determination routine 78, the positions of the anchors are recorded as three-dimensional data. The world surface determination routine 78 receives images from the real object detection camera 44 and depth data from the depth sensor 135 to determine the positions of the surfaces and their relative distances from the depth sensor 135.

[0287] The positioning module 130 is connected to the canonical map 133 and the Figure 2 . The positioning module 130 repeatedly attempts to position the Figure 2 to the canonical map 133. The canonical map merger 132 is connected to the canonical map 133 and the Figure 2 . When the positioning module 130 positions the Figure 2 to the canonical map 133, the canonical map merger 132 merges the canonical map 133 into the Figure 2 anchors. Then, the missing data included in the canonical map is used to update the Figure 2 .

[0288] The local content positioning system 134 is connected to the Figure 2 . The local content positioning system 134 can be, for example, a system that allows a user to position local content at a specific location within the world coordinate framework. Then, the local content attaches itself to the Figure 2An anchor point. The local-to-world coordinate transformer 104 transforms the local coordinate frame into the world coordinate frame based on the settings of the local content positioning system 134. It has been referred to Figure 2 The functions of the rendering engine 30, the display system 42, and the data channel 62 have been described.

[0289] The map publisher 136 uploads the map Figure 2 to the server 20. The map storage routine 118 of the server 20 then stores the map Figure 2 in the storage medium of the server 20.

[0290] The map merging algorithm 124 merges the map Figure 2 with the canonical map 120. When more than two maps have been stored (e.g., three or four maps related to the same or adjacent regions of the physical world), the map merging algorithm 124 merges all the maps into the canonical map 120 to render a new canonical map 120. Then, the map sender 122 sends the new canonical map 120 to any and all devices 12.1 and 12.2 located in the region represented by the new canonical map 120. When the devices 12.1 and 12.2 align their respective maps to the canonical map 120, the canonical map 120 becomes the upgraded map.

[0291] Fig.17 Illustrates examples of generating key frames for a map of a scene according to some embodiments. In the example shown, a first key frame KF1 is generated for the door on the left wall of the room. A second key frame KF2 is generated for the corner area where the floor, left wall, and right wall of the room intersect. A third key frame KF3 is generated for the window area on the right wall of the room. On the floor of the wall, a fourth key frame KF4 is generated in the distal area of the carpet. A fifth key frame KF5 is generated for the area of the carpet closest to the user.

[0292] Fig.18 Shows examples of generating persistent poses for the map of Fig.17 according to some embodiments. In some embodiments, a new persistent pose is created when the device measures a threshold distance traveled, and / or when the application requests a new persistent pose (PP). In some embodiments, the threshold distance can be 3 meters, 5 meters, 20 meters, or any other suitable distance. Selecting a smaller threshold distance (e.g., 1m) can result in an increased computational load because a larger number of PPs can be created and managed compared to a larger threshold distance. Selecting a larger threshold distance (e.g., 40m) can result in an increased virtual content placement error because a smaller number of PPs will be created, which will result in a smaller number of PCFs being created, meaning that the virtual content attached to the PCF may be a relatively large distance (e.g., 30m) away from the PCF, and the error increases with the increasing distance from the PCF to the virtual content.

[0293] In some embodiments, a PP can be created at the start of a new session. This initial PP can be considered as zero and can be visualized as the center of a circle with a radius equal to the threshold distance. When the device reaches the circumference of the circle and, in some embodiments, the application requests a new PP, the new PP can be placed at the current position of the device (at the threshold distance). In some embodiments, if the device can find an existing PP within the threshold distance from the new position of the device, a new PP will not be created at the threshold distance. In some embodiments, when a new PP is created (e.g., Fig.14 PP 1150 in ), the device attaches one or more of the closest keyframes to the PP. In some embodiments, the position of the PP relative to the keyframes can be based on the position of the device when the PP was created. In some embodiments, a PP will not be created when the device travels the threshold distance, unless the application requests a PP.

[0294] In some embodiments, when an application has virtual content to display to the user, the application can request a PCF from the device. The PCF request from the application can trigger a PP request, and a new PP will be created after the device travels the threshold distance. Fig.18 Shows a first permanent pose PP1, which can have the closest keyframes (e.g., KF1, KF2, and KF3) attached by calculating the relative pose between the keyframes and the persistent pose. Fig.18 Also shows a second permanent pose PP2, which can have the closest keyframes (e.g., KF4 and KF5) attached.

[0295] Fig.19 Shows an example of generating a PCF for a Fig.17 map according to some embodiments. In the example shown, PCF1 can include PP1 and PP2. As described above, the PCF can be used to display image data related to the PCF. In some embodiments, each PCF can have coordinates and a PCF descriptor in another coordinate frame (e.g., the world coordinate frame), e.g., to uniquely identify the PCF. In some embodiments, the PCF descriptor can be calculated based on the feature descriptors of the features in the frame associated with the PCF. In some embodiments, the various constellations of the PCF can be combined to represent the real world in a persistent manner that requires less data and less data transmission.

[0296] Figures 20A to 20C is a schematic diagram showing an example of establishing and using a persistent coordinate frame. Fig. 20AShows two users 4802A, 4802B with respective local tracking maps 4804A, 4804B that are not yet located to a canonical map. The origins 4806A, 4806B of each user are depicted by a coordinate system (e.g., world coordinate system) in their respective regions. These origins of each tracking map may be local to each user as they depend on the orientation of their respective devices when tracking is initiated.

[0297] When the sensors of the user device scan the environment, the device can capture images as described above in connection with Fig.14 that may contain features representing persistent objects such that those images can be classified as key frames and persistent poses can be created based on those key frames. In this example, the tracking map 4802A includes a persistent pose (PP) 4808A; the tracking 4802B includes a PP 4808B.

[0298] Similarly as described above in connection with Fig.14 Some PPs can be classified as PCFs, which are used to determine the orientation of virtual content for rendering it to the user. Fig. 20B Shows that the XR devices worn by the respective users 4802A, 4802B can create local PCFs 4810A, 4810B based on the PPs 4808A, 4808B. Fig. 20C Shows that persistent content 4812A, 4812B (e.g., virtual content) can be attached to the PCFs 4810A, 4810B by the respective XR devices.

[0299] In this example, the virtual content can have a virtual content coordinate frame that can be used by the application generating the virtual content regardless of how the virtual content should be displayed. For example, the virtual content can be specified as a surface at a specific position and angle relative to the virtual content coordinate frame, such as a triangle of a mesh. To render this virtual content to the user, the positions of those surfaces can be determined relative to the user who is to perceive the virtual content.

[0300] Attaching the virtual content to the PCF can simplify the calculations involved in determining the position of the virtual content relative to the user. The position of the virtual content relative to the user can be determined by applying a series of transformations. Some of these transformations may change and may be updated frequently. Other of these transformations may be stable and may be updated frequently or not at all. In any case, the transformations can be applied with a relatively low computational burden such that the position of the virtual content can be updated frequently relative to the user, thereby providing a realistic appearance for the rendered virtual content.

[0301] In FIG. 20A to FIG. 20CIn the example, the device of User 1 has a coordinate system that is related to the coordinate system defining the map origin through the transformation rig1_T_w1. The device of User 2 has a similar transformation rig2_T_w2. These transformations can be represented as 6 degrees of transformation, specifying translation and rotation to align the device coordinate system with the map coordinate system. In some embodiments, the transformation can be represented as two separate transformations, one specifying translation and the other specifying rotation. Therefore, it should be understood that the transformation can be expressed in a form that simplifies calculations or provides other advantages.

[0302] The transformation tracking the origin of the map and the PCF identified by the corresponding user device is represented as pcf1_T_w1 and pcf2_T_w2. In this example, the PCF and the PP are the same, so the same transformation also characterizes the PP.

[0303] Therefore, the position of the user device relative to the PCF can be calculated through the serial application of these transformations, such as rig1_T_pcf1 = (rig1_T_w1) * (pcf1_T_w1).

[0304] As Fig. 20C shown, through the transformation of obj1_T_pcf1, the virtual content is positioned relative to the PCF. This transformation can be set by the application generating the virtual content, which can receive information describing the physical object relative to the PCF from the world reconstruction system. To render the virtual content to the user, the transformation to the coordinate system of the user device is calculated, which can be calculated by associating the virtual content coordinate frame to the origin of the tracking map through the transformation obj1_t_w1 = (obj1_T_pcf1) * (pcf1_T_w1). Then, this transformation can be related to the user's device through the further transformation rig1_T_w1.

[0305] Based on the output from the application generating the virtual content, the position of the virtual content can change. When it changes, the end-to-end transformation from the source coordinate system to the destination coordinate system can be recalculated. Additionally, the position and / or head pose of the user can change as the user moves. As a result, the transformation rig1_T_w1 can change, and any end-to-end transformation depending on the user's position or head pose can also change.

[0306] The transformation rig1_T_w1 can be updated as the user moves based on tracking the user's position relative to a stationary object in the physical world. Such tracking can be performed by the headset positioning component or other components of the system that processes the image sequence as described above. Such an update can be made by determining the user's pose relative to a fixed reference frame (such as the PP).

[0307] In some embodiments, since the PP is used as the PCF, the position and orientation of the user device can be determined relative to the most recent persistent pose or, in this example, the PCF. Such determination can be made by identifying feature points characterizing the PP in the current image captured by sensors on the device. Using image processing techniques such as stereo image analysis, the position of the device relative to those feature points can be determined. Based on this data, the system can calculate the change in the transformation associated with the user movement based on the relationship rig1_T_pcf1 = (rig1_T_w1)*(pcf1_T_w1).

[0308] The system can determine and apply the transformation in a computationally efficient order. For example, by tracking the user's pose and defining the position of the virtual content relative to the PP or PCF built based on the persistent pose, the need to calculate rig1_T_w1 from the measurements that yield rig1_T_pcf1 can be avoided. Thus, the transformation from the source coordinate system of the virtual content to the destination coordinate system of the user device can be based on the measured transformation according to the expression (rig1_T_pcf1)*(obj1_t_pcf1), where the first transformation is measured by the system and the latter transformation is provided by the application that specifies the rendering of the virtual content. In embodiments where the virtual content is positioned relative to the origin of the map, the end-to-end transformation can relate the virtual object coordinate system to the PCF coordinate system based on a further transformation between the map coordinates and the PCF coordinates. In embodiments where the virtual content is positioned relative to a PP or PCF different from the PP or PCF for which the user's position is tracked, a transformation can be made between the two. Such a transformation can be fixed and can be determined, for example, from a map where both appear.

[0309] For example, a transformation-based method can be implemented in a device having components that process sensor data to build a tracking map. As part of this process, these components can identify feature points that can be used as persistent poses, which in turn can become the PCF. These components can limit the number of persistent poses generated for the map to provide an appropriate spacing between the persistent poses while allowing the user to be close enough to the persistent pose positions regardless of their location in the physical environment to accurately calculate the user's pose, as shown above in conjunction with Figures 17 to 19 As shown. As the closest persistent pose to the user is updated due to user movement, the refinement of the tracking map or otherwise allows any transformation for calculating the position of the virtual content relative to the user that depends on the PP (or PCF, if being used) to be updated and stored for use, at least until the user leaves that persistent pose. Nevertheless, by calculating and storing the transformation, the computational burden each time the position of the virtual content is updated can be relatively low, enabling it to be performed with a relatively low latency.

[0310] FIG. 20A to FIG. 20C It shows positioning relative to a tracking map, and each device has its own tracking map. However, a transformation can be generated relative to any map coordinate system. Content persistence between user sessions of an XR system can be achieved by using a persistent map. A shared experience for users can also be achieved by using a map to which multiple user devices can be oriented.

[0311] In some embodiments described in more detail below, the position of virtual content can be specified relative to coordinates in a canonical map, the format of the canonical map being set such that any one of a plurality of devices can use the map. Each device may maintain a tracking map and can determine changes in the user's pose relative to the tracking map. In this example, the transformation between the tracking map and the canonical map can be determined by a "localization" process, which can be performed by matching structures in the tracking map (such as one or more persistent poses) to one or more structures in the canonical map (such as one or more PCFs).

[0312] Techniques for creating and using a canonical map in this way are described in more detail below.

[0313] Depth Keyframes

[0314] The techniques described herein rely on the comparison of image frames. For example, to establish the position of a device relative to a tracking map, new images can be captured using sensors worn by the user, and the XR system can search for images in the image set used to create the tracking map that share at least a predetermined number of points of interest with the new images. As an example of another scenario involving image frame comparison, the tracking map can be localized to the canonical map by first looking for an image frame in the tracking map associated with a persistent pose that is similar to an image frame in the canonical map associated with a PCF. Alternatively, the transformation between two canonical maps can be calculated by first looking for similar image frames in the two maps.

[0315] Depth keyframes provide a way to reduce the processing amount required to identify similar image frames. For example, in some embodiments, the comparison can be between image features (e.g., "2D features") in a new 2D image and 3D features in a map. This comparison can be performed in any suitable manner, such as by projecting a 3D image onto a 2D plane. Conventional methods such as Bag of Words (BoW) search for the 2D features of a new image in a database that includes all 2D features in the map, which may require a large amount of computing resources, especially when the map represents a large area. Then, the conventional method locates images that share at least one 2D feature with the new image, and these images may include images that are not useful for locating meaningful 3D features in the map. Then, the conventional method locates 3D features that are not meaningful relative to the 2D features in the new image.

[0316] The inventors have recognized and understood techniques for retrieving images in a map using fewer memory resources (e.g., one quarter of the memory resources used by BoW), higher efficiency (e.g., 2.5 ms of processing time per keyframe, 100 μs for comparing 500 keyframes), and higher accuracy (e.g., for a 1024-dimensional model, the retrieval recall rate is 20% better than BoW, and for a 256-dimensional model, the retrieval recall rate is 5% better than BoW).

[0317] To reduce computation, a descriptor can be calculated for an image frame, and this descriptor can be used to compare the image frame with other image frames. The descriptor can be stored instead of or in addition to the image frame and feature points. In a map in which persistent poses and / or PCFs can be generated based on image frames, the descriptors of one or more image frames based on which each persistent pose or PCF is generated can be stored as part of the persistent pose and / or PCF.

[0318] In some embodiments, the descriptor can be calculated based on feature points in the image frame. In some embodiments, a neural network is configured to calculate a unique frame descriptor representing the image. The image can have a resolution higher than 1 megabyte, so as to capture sufficient details of the 3D environment within the field of view of the device worn by the user in the image. The frame descriptor can be much shorter, such as a digital string, e.g., in the range of 128 bytes to 512 bytes or any number of digital strings in between.

[0319] In some embodiments, the neural network is trained such that the computed frame descriptors indicate similarity between images. An image in the map can be located by identifying, in a database including images for generating the map, the nearest image that can have a frame descriptor within a predetermined distance of the frame descriptor of the new image. In some embodiments, the distance between images can be represented by the difference between the frame descriptors of the two images.

[0320] Fig.21 is a block diagram showing a system for generating descriptors for individual images according to some embodiments. In the example shown, a frame embedding generator 308 is shown. In some embodiments, the frame embedding generator 308 can be used within the server 20, but can alternatively or additionally be executed in whole or in part in one of the XR devices 12.1 and 12.2 or any other device that processes images for comparison with other images.

[0321] In some embodiments, the frame embedding generator can be configured to generate a reduced data representation of an image from an initial size (e.g., 76,800 bytes) to a final size (e.g., 256 bytes), which reduced data representation still indicates the content in the image despite the reduced size. In some embodiments, the frame embedding generator can be used to generate a data representation of an image, which image can be a key frame or otherwise a frame used. In some embodiments, the frame embedding generator 308 can be configured to convert an image located at a particular position and orientation into a unique digital string (e.g., 256 bytes). In the example shown, an image 320 captured by the XR device can be processed by a feature extractor 324 to detect a point of interest 322 in the image 320. The point of interest may or may not be derived from the feature points identified as described above for feature 1120 ( Fig.14 ) or as otherwise described herein. In some embodiments, the point of interest can be represented by descriptors as described above for descriptor 1130 ( Fig.14 ), and these points of interest can be generated using a depth sparse feature method. In some embodiments, each point of interest 322 can be represented by a digital string (e.g., 32 bytes). For example, there can be n features (e.g., 100), and each feature is represented by a 32-byte string.

[0322] In some embodiments, the frame embedding generator 308 may include a neural network 326. The neural network 326 may include a multi-layer perceptron unit 312 and a max pooling unit 314. In some embodiments, the multi-layer perceptron (MLP) unit 312 may include a multi-layer perceptron that can be trained. In some embodiments, the multi-layer perceptron 312 may reduce the points of interest 322 (e.g., descriptors for the points of interest), and output a weighted combination 310 of the descriptors. For example, the MLP may reduce n features to m features less than n features.

[0323] In some embodiments, the MLP unit 312 may be configured to perform matrix multiplication. The multi-layer perceptron unit 312 receives multiple points of interest 322 of the image 320 and converts each point of interest into a corresponding string of numbers (e.g., 256). For example, there may be 100 features, and each feature may be represented by a string of 256 numbers. In this example, a matrix with 100 horizontal rows and 256 vertical columns may be created. Each row may have a series of 256 numbers that vary in magnitude, some smaller and some larger. In some embodiments, the output of the MLP may be an n×256 matrix, where n represents the number of points of interest extracted from the image. In some embodiments, the output of the MLP may be an m×256 matrix, where m is the number of points of interest reduced from n.

[0324] In some embodiments, the MLP 312 may have a training phase and a usage phase, during which the model parameters for the MLP are determined. In some embodiments, the MLP may be trained as shown in Fig.25 The input training data may include triplets of data, where a triplet includes 1) a query image, 2) a positive sample, and 3) a negative sample. The query image may be considered a reference image.

[0325] In some embodiments, the positive sample may include an image similar to the query image. For example, in some embodiments, similar may mean having the same object in the query image and the positive sample image, but viewed from different angles. In some embodiments, similar may mean having the same object in the query image and the positive sample image, but the object is shifted relative to the other image (e.g., left, right, up, down).

[0326] In some embodiments, negative samples can include images that are not similar to the query image. For example, in some embodiments, a dissimilar image may not include any objects that are prominent in the query image, or may include only a small portion (e.g., <10%, 1%) of the prominent objects in the query image. In contrast, for example, a similar image may have a majority (e.g., >50% or >75%) of the objects in the query image.

[0327] In some embodiments, points of interest can be extracted from the images in the input training data and the points of interest can be converted into feature descriptors. These descriptors can be calculated for both the training images as shown in Fig.25 and the features extracted from the operation of the frame embedding generator 308 for Fig.21 . In some embodiments, as described in U.S. Patent Application 16 / 190,948, deep sparse feature (DSF) processing can be used to generate the descriptors (e.g., DSF descriptors). In some embodiments, the DSF descriptor is n×32-dimensional. The descriptors can then be passed through a model / MLP to create a 256-byte output. In some embodiments, the model / MLP can have the same structure as the MLP 312 such that once the model parameters are set through training, the resulting trained MLP can be used as the MLP 312.

[0328] In some embodiments, the feature descriptors (e.g., the 256 bytes output from the MLP model) can then be sent to a triplet margin loss module (which can be used only during the training phase and not during the usage phase of the MLP neural network). In some embodiments, the triplet margin loss module can be configured to select the parameters of the model to reduce the difference between the 256-byte output from the query image and the 256-byte output from the positive sample, and to increase the difference between the 256-byte output from the query image and the 256-byte output from the negative sample. In some embodiments, the training phase can include feeding multiple triplet input images into the learning process to determine the model parameters. The training process can continue, for example, until the difference for the positive images is minimized and the difference for the negative images is maximized, or until other appropriate exit criteria are met.

[0329] Referring again to Fig.21, the frame embedding generator 308 may include a pooling layer, shown here as a max pooling unit 314. The max pooling unit 314 may analyze each column to determine the largest number in the corresponding column. The max pooling unit 314 may combine the maximum values of the numbers in each column of the output matrix of the MLP 312 into a global feature string 316 of, for example, 256 numbers. It should be understood that the images processed in the XR system may be expected to have high-resolution frames, potentially having millions of pixels. The global feature string 316 is a relatively small number, which occupies relatively little memory and is easy to search compared to the image (e.g., having a resolution higher than 1 megabyte). Thus, the image can be searched without analyzing each original frame from the camera, and it is also cheaper to store 256 bytes instead of the full frame.

[0330] Fig. 22 is a flowchart showing a method 2200 for calculating an image descriptor according to some embodiments. The method 2200 may start with receiving (act 2202) a plurality of images captured by an XR device worn by a user. In some embodiments, the method 2200 may include determining (act 2204) one or more key frames from the plurality of images. In some embodiments, act 2204 may be skipped and / or may instead occur after step 2210.

[0331] The method 2200 may include: identifying (act 2206) one or more points of interest in the plurality of images using an artificial neural network; and calculating (act 2208) a feature descriptor for each of the points of interest. The method may include calculating (act 2210) a frame descriptor for each image, thereby representing the image based at least in part on the feature descriptors calculated for the points of interest identified in the image using the artificial neural network.

[0332] Fig.23FIG. 2300 is a flow chart showing a method 2300 for localization using image descriptors according to some embodiments. In this example, a new image frame depicting the current location of an XR device can be compared to image frames stored in association with points in a map (e.g., persistent poses or PCFs as described above). Method 2300 can begin with receiving (act 2302) a new image captured by an XR device worn by a user. Method 2300 can include identifying (act 2304) one or more recent key frames in a database that includes key frames for generating one or more maps. In some embodiments, the recent key frames can be identified based on rough spatial information and / or previously determined spatial information. For example, the rough spatial information can indicate that the XR device is located in a geographic area represented by a 50m×50m region of the map. Image matching can be performed only on points within this region. As another example, based on tracking, the XR system can know that the XR device was previously near a first persistent pose in the map and was then moving in the direction of a second persistent pose in the map. This second persistent pose can be considered the most recent persistent pose, and the key frame stored with it can be considered the most recent key frame. Alternatively or additionally, other metadata such as GPS data or WiFi fingerprints can be used to select the most recent key frame or a set of the most recent key frames.

[0333] Regardless of how the most recent key frames are selected, frame descriptors can be used to determine whether the new image matches any of the frames selected as being associated with nearby persistent poses. This determination can be made by comparing the frame descriptor of the new image to the frame descriptors of the most recent key frames or a subset of key frames in the database selected in any other suitable manner, and selecting the key frames having frame descriptors within a predetermined distance of the frame descriptor of the new image. In some embodiments, the distance between two frame descriptors can be calculated by obtaining the difference between two numerical strings that can represent the two frame descriptors. In embodiments where the strings are processed as vectors of multiple strings, the difference can be calculated as a vector difference.

[0334] Once a matching image frame is identified, the orientation of the XR device relative to that image frame can be determined. Method 2300 can include performing (act 2306) feature matching on 3D features in the map corresponding to the identified most recent key frame, and calculating (act 2308) the pose of the device worn by the user based on the feature matching results. In this way, computationally intensive matching of feature points in two images can be performed for as few as one image that has been determined to be a possible match for the new image.

[0335] Fig.24is a flowchart showing a method 2400 for training a neural network according to some embodiments. Method 2400 may start by generating (action 2402) a data set including a plurality of image sets. Each of the plurality of image sets may include a query image, a positive sample image, and a negative sample image. In some embodiments, the plurality of image sets may include synthetic record pairs configured to teach the neural network basic information (such as shape), for example. In some embodiments, the plurality of image sets may include real record pairs that may be recorded according to the physical world.

[0336] In some embodiments, inliers may be calculated by fitting an essential matrix between two images. In some embodiments, sparse overlap may be calculated as the intersection over union (IoU) of the points of interest seen in two images. In some embodiments, a positive sample may include at least twenty points of interest that are the same as those in the query image as inliers. A negative sample may include fewer than ten inliers. The negative sample may have fewer than half of the sparse points that overlap with the sparse points of the query image.

[0337] Method 2400 may include calculating (action 2404) a loss for each image set by comparing the query image with the positive sample image and the negative sample image. Method 2400 may include modifying (action 2406) the artificial neural network based on the calculated loss such that the distance between the frame descriptor generated by the artificial neural network for the query image and the frame descriptor for the positive sample image is less than the distance between the frame descriptor for the query image and the frame descriptor for the negative sample image.

[0338] It should be understood that although the methods and apparatuses for generating global descriptors for individual images are described above, the methods and apparatuses may be configured to generate descriptors for individual maps. For example, a map may include a plurality of key frames, and each key frame may have a frame descriptor as described above. A max pooling unit may analyze the frame descriptors of the key frames of the map and combine the frame descriptors into a unique map descriptor for the map.

[0339] In addition, it should be understood that other architectures may be used for the processing described above. For example, a separate neural network for generating DSF descriptors and frame descriptors is described. This method is computationally efficient. However, in some embodiments, frame descriptors may be generated based on the selected feature points without first generating DSF descriptors.

[0340] Ranking and merging maps

[0341] Described herein are methods and apparatus for ranking and merging multiple environmental maps in an X Reality (XR) system. Map merging can enable maps representing overlapping portions of the physical world to be combined to represent a larger area. Ranking the maps can enable efficient execution of the techniques described herein, including map merging, which involves selecting maps from a set of maps based on similarity. In some embodiments, for example, the system can maintain a set of canonical maps that are formatted in a manner accessible to any of a number of XR devices. These canonical maps can be formed by merging selected tracked maps from those devices with other tracked maps or previously stored canonical maps. The canonical maps can be ranked, for example, for selecting one or more canonical maps to merge with a new tracked map and / or for selecting one or more canonical maps from the set for use in a device.

[0342] To provide a realistic XR experience to a user, an XR system must understand the user's physical environment in order to correctly associate the position of virtual objects relative to real objects. Information about the user's actual environment can be obtained from environmental maps of the user's location.

[0343] The inventors have recognized and understood that an XR system can provide an enhanced XR experience to multiple users sharing the same world including real and / or virtual content by effectively sharing environmental maps of the real / physical world collected by multiple users, whether those users are present in the world at the same time or at different times. However, there are significant challenges in providing such a system. Such a system can store multiple maps generated by multiple users and / or the system can store multiple maps generated at different times. For operations that may be performed using previously generated maps (such as, for example, localization as described above), a large amount of processing may be required to identify relevant environmental maps of the same world (e.g., the same real-world location) from all of the environmental maps collected in the XR system. In some embodiments, there may be only a small number of environmental maps accessible to the device for, e.g., localization. In some embodiments, there may be a large number of environmental maps accessible to the device. The inventors have recognized and understood techniques for quickly and accurately ranking the relevance of environmental maps from all possible environmental maps (such as, for example, Fig.28 the universe of all canonical maps 120 in). High-ranked maps can then be selected for further processing, such as rendering virtual objects on a user display to interact realistically with the physical world around the user, or merging the map data collected by that user with stored maps to create a larger or more accurate map.

[0344] In some embodiments, a stored map related to a user's task at a location in the physical world can be identified by filtering the stored maps based on multiple criteria. These criteria can indicate a comparison of a tracking map generated by the user's wearable device in the location with a candidate environment map stored in a database. The comparison can be performed based on metadata associated with the map, such as a Wi-Fi fingerprint detected by the device generating the map and / or a set of BSSIDs to which the device is connected while forming the map. The comparison can also be performed based on compressed or uncompressed content of the map. Comparisons based on compressed representations can be performed by comparing vectors calculated from the map content. For example, a comparison based on an uncompressed map can be performed by locating a tracking map within a stored map, or vice versa. Multiple comparisons can be performed in sequence based on the computation time required to reduce the number of candidate maps to be considered, where comparisons involving less computation will be performed earlier in the sequence than other comparisons that require more computation.

[0345] Fig.26 An AR system 800 configured to rank and merge one or more environment maps according to some embodiments is depicted. The AR system may include a traversable world model 802 of the AR device. Information to populate the traversable world model 802 may come from sensors on the AR device, which may include data stored in a processor 804 (e.g., Figure 4 The processor may perform some or all of the processing to convert the sensor data into a map. Such a map may be a tracking map, because the AR device may build a tracking map while collecting sensor data as it operates in an area. Figure 1 In addition, regional attributes may be provided to indicate the area represented by the tracking map. These regional attributes may be geographic location identifiers, such as coordinates expressed as latitude and longitude, or IDs used by the AR system to represent locations. Alternatively or in addition, regional attributes may be measured characteristics that have a high probability of being unique to the area. Regional attributes may, for example, be derived from parameters of wireless networks detected in the area. In some embodiments, regional attributes may be associated with a unique address of an access point that the AR system is nearby and / or connected to. For example, regional attributes may be associated with a MAC address or basic service set identifier (BSSID) of a 5G base station / router, a Wi-Fi router, or the like.

[0346] exist Fig.26 In the example of FIG. 8 , the tracking map can be merged with other maps of the environment. The map ranking portion 806 receives the tracking map from the device PW 802 and communicates with the map database 808 to select and rank the environment map from the map database 808. The selected map with a higher ranking is sent to the map merging portion 810.

[0347] The map merging section 810 can perform a merging process on the maps sent from the map ranking section 806. The merging process may involve merging the tracking map with some or all of the ranked maps and sending the new merged map to the traversable world model 812. The map merging section can merge maps by identifying maps that depict overlapping portions of the physical world. Those overlapping portions can be aligned so that the information in the two maps can be aggregated into a final map. Canonical maps can be merged with other canonical maps and / or tracking maps.

[0348] Aggregation may involve extending one map with information from another map. Alternatively or additionally, aggregation may involve adjusting the representation of the physical world in one map based on information in another map. For example, a later map may reveal an object that has caused a feature point to move, so the map can be updated based on the later information. Alternatively, two maps may characterize the same region with different feature points, and aggregation may involve selecting a set of feature points from both maps to better represent the region. Regardless of the specific processing that occurs during the merging process, in some embodiments, the PCFs from all of the maps being merged can be retained so that applications that localize content relative to them can continue to do so. In some embodiments, the merging of maps can result in redundant persistent poses, and some of the persistent poses can be deleted. When a PCF is associated with a persistent pose to be deleted, the merged map may need to modify the PCF to be associated with the persistent poses remaining in the map after merging.

[0349] In some embodiments, as maps are extended and / or updated, they can be refined. Refinement may involve performing calculations to reduce internal inconsistencies between feature points that may represent the same object in the physical world. Such inconsistencies may arise from inaccurate poses associated with keyframes that provide feature points representing the same object in the physical world. For example, such inconsistencies may arise when an XR device calculates a pose relative to a tracking map that is itself based on an estimated pose, so errors in the pose estimation accumulate, resulting in a "drift" in pose accuracy over time. The map can be refined by performing bundle adjustment or other operations to reduce the inconsistencies of feature points from multiple keyframes.

[0350] During refinement, the position of the persistent point relative to the map origin can change. Accordingly, the transforms associated with that persistent point, such as the persistent pose or PCF, may change. In some embodiments, an XR system that incorporates map refinement (whether as part of a merge operation or for other reasons) can recalculate the transforms associated with any persistent points that have changed. These transforms may be pushed from the component that calculates the transform to the component that uses the transform so that any use of the transform can be based on the updated position of the persistent point.

[0351] The traversable world model 812 can be a cloud model that can be shared by multiple AR devices. The traversable world model 812 can store or otherwise access the environmental map in the map database 808. In some embodiments, when updating a previously computed environmental map, the previous version of the map can be deleted to remove the out-of-date map from the database. In some embodiments, when updating a previously computed environmental map, the previous version of the map can be archived, enabling access / viewing of the previous version of the environment. In some embodiments, permissions can be set such that only AR systems with certain read / write access rights can trigger the deletion / archiving of the previous version of the map.

[0352] These environmental maps created from the tracking maps provided by one or more AR devices / systems can be accessed by the AR devices in the AR system. The map ranking section 806 can also be used to provide environmental maps to the AR devices. The AR devices can send messages requesting the environmental map of their current location, and the map ranking section 806 can be used to select and rank the environmental maps relevant to the requesting device.

[0353] In some embodiments, the AR system 800 can include a downsampling section 814 configured to receive a merged map from the cloud PW 812. The merged map received from the cloud PW 812 can be in a storage format for the cloud, which can include high-resolution information such as a large number of PCFs per square meter or multiple image frames or a large set of feature points associated with the PCFs. The downsampling section 814 can be configured to downsample the cloud-format map into a format suitable for storage on the AR device. The device-format map can contain less data, such as fewer PCFs or less data stored for each PCF, to accommodate the limited local computing power and storage space of the AR device.

[0354] Fig. 27is a simplified block diagram showing a plurality of canonical maps 120 that can be stored in a remote storage medium such as the cloud. Each canonical map 120 can include a plurality of canonical map identifiers that indicate the location of the canonical map within physical space, such as somewhere on Earth. These canonical map identifiers can include one or more of the following identifiers: a region identifier represented by a longitude and latitude range, a frame descriptor (e.g., Fig.21 the global feature string 316 in Fig.21 the feature descriptor 310 in

[0355] and a device identifier indicating one or more devices that contributed to the map.

[0356] Fig.28 is a schematic diagram showing a method of selecting a canonical map according to some embodiments, which can be used to locate a new tracking map to one or more canonical maps. The method can start by accessing (action 120) the world of canonical maps 120, which, as an example, can be stored in a database of traversable worlds (e.g., traversable world module 538). The world of canonical maps can include canonical maps from all previously visited locations. The XR system can filter the world of all canonical maps into a small subset or just one map. It should be understood that in some embodiments, due to bandwidth limitations, it is not possible to send all canonical maps to the viewing device. Sending a selected subset that is chosen as a candidate for matching the tracking map to the device can reduce the bandwidth and latency associated with accessing the remote database of maps.

[0357] The method can include filtering (action 300) the world of canonical maps based on a region having a predetermined size and shape. In Fig. 27In the example, each square can represent an area. Each square can cover 50m × 50m. Each square can have six adjacent areas. In some embodiments, action 300 can select at least one canonical map 120 that covers the longitude and latitude, where the longitude and latitude include the longitude and latitude of the location identifier received from the XR device, as long as there is at least one map at that longitude and latitude. In some embodiments, action 300 can select at least one adjacent canonical map that covers the longitude and latitude adjacent to the matching canonical map. In some embodiments, action 300 can select multiple matching canonical maps and multiple adjacent canonical maps. Action 300 can, for example, reduce the number of canonical maps by approximately ten times, for example, from thousands to hundreds, to form a first filtered selection. Alternatively or additionally, criteria other than latitude and longitude can be used to identify adjacent maps. For example, the XR device may have previously been located using the canonical maps in the set as part of the same session. The cloud service can retain information about the XR device, including the maps previously located to. In this example, the maps selected at action 300 can include those that cover areas adjacent to the maps to which the XR device was located.

[0358] The method can include filtering (action 302) the first filtered selection of canonical maps based on Wi-Fi fingerprints. Action 302 can determine the latitude and longitude based on the Wi-Fi fingerprints received from the XR device as part of the location identifier. Action 302 can compare the latitude and longitude from the Wi-Fi fingerprints with the latitude and longitude of the canonical map 120 to determine one or more canonical maps that form a second filtered selection. Action 302 can reduce the number of canonical maps by approximately ten times, for example, from hundreds of canonical maps to dozens (e.g., 50) of canonical maps that form the second selection. For example, the first filtered selection can include 130 canonical maps, the second filtered selection can include 50 of the 130 canonical maps, and can exclude the other 80 of the 130 canonical maps.

[0359] The method can include filtering (action 304) the second filtered selection of canonical maps based on key frames. Action 304 can compare the data representing the images captured by the XR device with the data representing the canonical map 120. In some embodiments, the data representing the images and / or maps can include feature descriptors (e.g., Fig.25 the DSF descriptor in Fig.21in 316). Action 304 can provide a third filtering option for the canonical maps. In some embodiments, for example, the output of action 304 can be only five canonical maps out of the 50 canonical maps identified after the second filtering option. Then, the map transmitter 122 sends one or more canonical maps based on the third filtering option to the viewing device. Action 304 can reduce the number of canonical maps by approximately tenfold, for example, from dozens of canonical maps to a single-digit number of canonical maps (e.g., 5) that form the third option. In some embodiments, the XR device can receive the canonical maps in the third filtering option and attempt to locate to the received canonical maps.

[0360] For example, action 304 can filter the canonical maps 120 based on the global feature string 316 of the canonical map 120 and the global feature string 316 based on an image captured by the viewing device (e.g., an image that can be part of the user's local tracking map). Thus, Fig. 27 each canonical map 120 in has one or more global feature strings 316 associated therewith. In some embodiments, the global feature string 316 can be obtained when the XR device submits image or feature details to the cloud and the cloud processes these image or feature details to generate the global feature string 316 for the canonical map 120.

[0361] In some embodiments, the cloud can receive the feature details of the live / new / current image captured by the viewing device, and the cloud can generate the global feature string 316 of the live image. Then, the cloud can filter the canonical maps 120 based on the live global feature string 316. In some embodiments, the global feature string can be generated locally on the viewing device. In some embodiments, the global feature string can be generated remotely, e.g., in the cloud. In some embodiments, the cloud can send the filtered canonical maps together with the global feature string 316 associated with the filtered canonical maps to the XR device. In some embodiments, when the viewing device locates its tracking map to the canonical map, it can do so by matching the global feature string 316 of the local tracking map with the global feature string of the canonical map.

[0362] It should be understood that the operations of the XR device may not perform all actions (300, 302, 304). For example, if the world of the canonical map is relatively small (e.g., 500 maps), the XR device attempting to localize can filter the world of the canonical map based on Wi-Fi fingerprints (e.g., action 302) and keyframes (e.g., action 304), but omits region-based filtering (e.g., action 300). Also, it is not necessary to compare the entire map. For example, in some embodiments, the comparison of two maps can result in the identification of common persistent points, such as persistent poses or PCFs that appear in both the new map and a map selected from the map world. In that case, descriptors can be associated with the persistent points, and those descriptors can be compared.

[0363] Fig.29 is a flowchart showing a method 900 for selecting one or more ranked environmental maps according to some embodiments. In the illustrated embodiment, ranking is performed on the AR device of the user who is creating the tracking map. Thus, the tracking map can be used to rank the environmental maps. In embodiments where the tracking map is not available, some or all of the selection and ranking of the environmental map that does not explicitly depend on the tracking map can be used.

[0364] Method 900 can start at action 902, where a set of maps in a database of environmental maps (which can be formatted as canonical maps) located near the location where the tracking map is being formed can be accessed and then filtered for ranking. Additionally, at action 902, at least one regional attribute of the region in which the user's AR device is operating is determined. In a scenario where the user's AR device is constructing a tracking map, the regional attribute can correspond to the region on which the tracking map is being created. As a specific example, the regional attribute can be calculated based on the received signal from an access point to a computer network while the AR device is computing the tracking map.

[0365] Fig.30 depicts an exemplary map ranking portion 806 of an AR system 800 according to some embodiments. The map ranking portion 806 can be executed in a cloud computing environment as it can include a portion executed on the AR device and a portion executed on a remote computing system such as the cloud. The map ranking portion 806 can be configured to execute at least a portion of method 900.

[0366] Fig.31ADepicts an example of the regional attributes AA1 - AA8 of the tracking map (TM) 1102 and the environmental maps CM1 - CM4 in a database according to some embodiments. As shown, the environmental maps can be associated with multiple regional attributes. The regional attributes AA1 - AA8 can include parameters of the wireless network detected by the AR device calculating the tracking map 1102. For example, the basic service set identifier (BSSID) of the network to which the AR device is connected and / or the strength of the received signal from the access point of the wireless network through, for example, the network tower 1104. The parameters of the wireless network can conform to protocols including Wi-Fi and 5G NR. In Fig.32 the example shown, the regional attribute is the fingerprint of the area where the user's AR device collects sensor data to form the tracking map.

[0367] Fig.31B Depicts an example of the determined geographical location 1106 of the tracking map 1102 according to some embodiments. In the example shown, the determined geographical location 1106 includes the centroid point 1110 and the area 1108 surrounding the centroid point. It should be understood that the determination of the geographical location in this application is not limited to the format shown. The determined geographical location can have any suitable format, including, for example, different area shapes. In this example, using a database that associates regional attributes with geographical locations, the geographical location is determined from the regional attributes. The database is commercially available. For example, a database that associates Wi-Fi fingerprints with locations expressed as latitude and longitude and can be used for this operation.

[0368] In Fig.29 the embodiment of, the map database containing the environmental maps can also include the location data of those maps, including the latitude and longitude covered by the maps. The processing at action 902 may require selecting a set of environmental maps from this database that cover the same latitude and longitude determined for the regional attributes of the tracking map.

[0369] Action 904 is the first filtering of the set of environmental maps accessed in action 902. In action 902, the environmental maps are retained in the set based on their proximity to the geographical location of the tracking map. This filtering step can be performed by comparing the latitudes and longitudes associated with the tracking map and the environmental maps in the set.

[0370] Fig.32Depicts an example of operation 904 according to some embodiments. Each region attribute may have a corresponding geographical location 1202. The set of environmental maps may include an environmental map having at least one region attribute having a geographical location that overlaps the determined geographical location of the tracking map. In the example shown, the set of identified environmental maps includes environmental maps CM1, CM2, and CM4, each having at least one region attribute having a geographical location that overlaps the determined geographical location of the tracking map 1102. CM3 associated with region attribute AA6 is not included in the set because it is outside the determined geographical location of the tracking map.

[0371] Other filtering steps may also be performed on the set of environmental maps to reduce / rank the number of environmental maps in the set that are ultimately processed (such as for map merging or providing traversable world information to a user device). Method 900 may include filtering (operation 906) the set of environmental maps based on the similarity of one or more identifiers of network access points associated with the tracking map and the environmental maps of the set. During the formation of a map, a device that collects sensor data to generate the map may be connected to a network through a network access point (such as via Wi-Fi or a similar wireless communication protocol). The access point may be identified by a BSSID. When a user device moves through the area where data is collected to form a map, the user device may connect to multiple different access points. Similarly, when multiple devices provide information to form a map, the devices may have been connected through different access points, and for this reason, multiple access points may also be used when forming the map. Thus, there may be multiple access points associated with a map, and the set of access points may be an indication of the map location. The signal strength from an access point may be reflected as an RSSI value, which may provide further geographical information. In some embodiments, a list of BSSID and RSSI values may form a region attribute for the map.

[0372] In some embodiments, filtering the set of environmental maps based on the similarity of one or more identifiers of network access points may include: retaining in the set of environmental maps the environmental map having the highest Jaccard similarity with at least one region attribute of the tracking map based on the one or more identifiers of the network access points. Fig.33 Depicts an example of operation 906 according to some embodiments. In the example shown, the network identifier associated with region attribute AA7 may be determined as the identifier of the tracking map 1102. The set of environmental maps after operation 906 includes: environmental map CM2, which may have a region attribute within a higher Jaccard similarity with AA7; and environmental map CM4, which also includes region attribute AA7. Environmental map CM1 is not included in the set because it has the lowest Jaccard similarity with AA7.

[0373] The processing of actions 902 - 906 can be performed based on metadata associated with the map without actually accessing the content of the map stored in the map database. Other processing may involve accessing the content of the map. Action 908 indicates accessing the environmental map remaining in the subset after filtering based on the metadata. It should be understood that if subsequent operations can be performed on the accessed content, this action can be performed earlier or later in the process.

[0374] Method 900 may include filtering (action 910) a set of environmental maps based on the similarity of a metric representing the content of the environmental maps of a tracking map and a set of environmental maps. The metric representing the content of the tracking map and the environmental maps may include a vector of values calculated from the content of the map. For example, as described above, the depth keyframe descriptor calculated for one or more keyframes used to form the map can provide a metric for comparing the map or parts of the map. The metric can be calculated from the map obtained at action 908 or can be pre - calculated and stored as metadata associated with those maps. In some embodiments, filtering a set of environmental maps based on the similarity of a metric representing the content of the environmental maps of a tracking map and a set of environmental maps may include: retaining in the set of environmental maps the environmental map having the minimum vector distance between the feature vector of the tracking map and the vector representing the environmental map in the set of environmental maps.

[0375] Method 900 may include further filtering (action 912) a set of environmental maps based on the degree of match between a portion of the tracking map and portions of the environmental maps in the set of environmental maps. The degree of match can be determined as part of the localization process. As a non - limiting example, localization can be performed by identifying critical points in the tracking map and the environmental maps that are similar enough to represent the same part of the physical world. In some embodiments, the critical points can be features, feature descriptors, keyframes, key assemblies, persistent poses, and / or PCFs. Then, a set of critical points in the tracking map may be aligned to produce an optimal fit with the set of critical points in the environmental map. The mean square distance between the corresponding critical points may be calculated and, if below a threshold for a particular region of the tracking map, used as an indication that the tracking map and the environmental map represent the same region of the physical world.

[0376] In some embodiments, filtering a set of environmental maps based on the degree of match between a portion of the tracking map and portions of the environmental maps in the set of environmental maps may include: calculating the volume of the physical world represented by the tracking map, which is also represented in the environmental maps of the set of environmental maps; and retaining in the set of environmental maps the environmental maps having a larger calculated volume than the environmental maps filtered out from the set. Fig.34Depicts an example of action 912 according to some embodiments. In the example shown, a set of environmental maps after action 912 includes environmental map CM4, which has an area 1402 that matches an area of tracking map 1102. Environmental map CM1 is not included in the set because it does not have an area that matches an area of tracking map 1102.

[0377] In some embodiments, the set of environmental maps can be filtered in the order of action 906, action 910, and action 912. In some embodiments, the set of environmental maps can be filtered based on action 906, action 910, and action 912, and action 906, action 910, and action 912 can be performed in the order from lowest to highest based on the processing required to perform the filtering. Method 900 can include loading (action 914) the set of environmental maps and data.

[0378] In the example shown, the user database stores a region identifier indicating the region where the AR device is used. The region identifier can be a region attribute, which can include parameters of a wireless network detected by the AR device during use. The map database can store a plurality of environmental maps constructed from data provided by the AR device and associated metadata. The associated metadata can include a region identifier derived from the region identifier of the AR device that provided the data, and the environmental map is constructed from this data. The AR device can send a message to the PW module indicating that a new tracking map is being created or has been created. The PW module can calculate a region identifier for the AR device and update the user database based on the received parameters and / or the calculated region identifier. The PW module can also determine the region identifier associated with the AR device that requests the environmental map, identify the set of environmental maps from the map database based on the region identifier, filter the set of environmental maps, and send the filtered set of environmental maps to the AR device. In some embodiments, the PW module can filter the set of environmental maps based on one or more criteria, which can include, for example, the geographical location of the tracking map, the similarity of one or more identifiers of network access points associated with the tracking map and the environmental maps of the set, the similarity of a measure representing the content of the tracking map and the environmental maps of the set, and the degree of match between a part of the tracking map and a part of the environmental maps of the set.

[0379] Several aspects of some embodiments have been described so far. It should be understood that various changes, modifications, and improvements will readily occur to those skilled in the art. As an example, the embodiments are described in the context of an augmented (AR) environment. It should be understood that some or all of the techniques described herein can be applied in an MR environment or more generally in other XR environments and VR environments.

[0380] As another example, embodiments are described in the context of a device such as a wearable device. It should be understood that some or all of the techniques described herein may be implemented via a network (such as the cloud), discrete applications, and / or devices, any suitable combination of a network and discrete applications.

[0381] In addition, Fig.29 examples of criteria that can be used to filter candidate maps to produce a set of high-ranked maps are provided. Instead of or in addition to the described criteria, other criteria may be used. For example, if multiple candidate maps have similar values of a metric for filtering out less desirable maps, the characteristics of the candidate maps can be used to determine which maps are retained as candidates or filtered out. For example, larger or denser candidate maps may be prioritized over smaller candidate maps. In some embodiments, Figure 27-28 it may be described Figure 29-34 all or part of the systems and methods described in

[0382] Fig.35 and 36 is a schematic diagram showing an XR system configured to rank and merge multiple environmental maps according to some embodiments. In some embodiments, a traversable world (PW) may determine when to trigger ranking and / or merging of maps. In some embodiments, determining which maps to use may be at least partially based on the depth keyframes described above with respect to Figure 21 to Figure 25 described.

[0383] Fig.37 is a block diagram showing a method 3700 of creating an environmental map of a physical world according to some embodiments. Method 3700 may localize a tracking map (act 3702) captured by an XR device worn by a user to a group of canonical maps (e.g., canonical maps selected by the method of Fig.28 and / or the method 900 of FIG. 900). Act 3702 may include localizing key assemblies of the tracking map into the group of canonical maps. The localization result of each key assembly may include the localized pose of the key assembly and a set of 2D to 3D feature correspondences.

[0384] In some embodiments, method 3700 may include splitting (act 3704) the tracking map into connected parts, which may robustly merge the maps by merging connected segments. Each connected part may include key assemblies within a predetermined distance. Method 3700 may include: merging (act 3706) connected parts larger than a predetermined threshold into one or more canonical maps; and removing the merged connected parts from the tracking map.

[0385] In some embodiments, method 3700 may include merging (action 3708) canonical maps in a group that are merged with the same connected portions of the tracking map. In some embodiments, method 3700 may include promoting (action 3710) the remaining connected portions of the tracking map that have not been merged with any canonical maps to canonical maps. In some embodiments, method 3700 may include merging (action 3712) the persistent poses and / or PCFs of the tracking map and the canonical maps, where the canonical maps are merged with at least one connected portion of the tracking map. In some embodiments, method 3700 may include finalizing (action 3714) the canonical maps, for example, by fusing map points and pruning redundant key assemblies.

[0386] Fig.38A and 38B illustrates an environment map 3800 created by updating a canonical map 700 according to some embodiments, where the canonical map 700 can be upgraded with a new tracking map from a tracking map 700( Figure 7 ). As illustrated and described with respect to Figure 7 , the canonical map 700 may provide a floor plan 706 of reconstructed physical objects represented by points 702 in the corresponding physical world. In some embodiments, the map points 702 may represent features of a physical object, which may include multiple features. A new tracking map of the physical world can be captured and uploaded to the cloud for merging with map 700. The new tracking map may include map points 3802 and key assemblies 3804, 3806. In the example shown, the key assembly 3804 represents a key assembly that has been successfully located in the canonical map by, for example, establishing a correspondence with the key assembly 704 of map 700 (as Fig.38B shown). On the other hand, the key assembly 3806 represents a key assembly that has not been located in map 700. In some embodiments, the key assembly 3806 can be promoted to a separate canonical map.

[0387] Figures 39A to 39F is a schematic diagram showing an example of a cloud-based persistent coordinate system that provides a shared experience for users in the same physical space. Fig.39A illustrates that a canonical map 4814 from the cloud, for example, is received by XR devices worn by users 4802A and 4802B of FIG. 20A to FIG. 20C . The canonical map 4814 may have a canonical coordinate frame 4806C. The canonical map 4814 may have a PCF 4810C with multiple associated PPs (e.g., 4818A, 4818B in Fig.39C ).

[0388] Fig.39BShows the relationship established by the XR device between its respective world coordinate systems 4806A, 4806B and the canonical coordinate frame 4806C. For example, this can be done by locating the canonical map 4814 on the respective device. For each device, aligning the tracking map to the canonical map can result in a transformation between its local world coordinate system and the coordinate system of the canonical map for each device.

[0389] Fig.39C Shows that the transformation (e.g., transformation 4816A, transformation 4816B) between the local PCF (e.g., PCF 4810A, PCF 4810B) on the respective device and the corresponding persistent pose (e.g., PP 4818A, PP 4818B) on the canonical map can be calculated as a result of the alignment. Using these transformations, each device can use its local PCF to determine where to display virtual content attached to PP 4818A, PP 4818B or other persistent points on the canonical map relative to the local device, where the local PCF can be detected locally on the device by processing the images detected by the sensors on the device. Such a method can accurately position virtual content relative to each user and can enable each user to have the same experience of virtual content in the physical space.

[0390] Fig.39D Shows a snapshot of the persistent pose from the canonical map to the local tracking map. It can be seen that the local tracking maps are interconnected by persistent poses. Fig.39E Shows that the PCF 4810A on the device worn by user 4802A can be accessed through PP 4818A in the device worn by user 4802B. Fig.39F Shows that the tracking maps 4804A, 4804B and the canonical map 4814 can be merged. In some embodiments, some PCFs can be removed due to the merger. In the example shown, the merged map includes the PCF 4810C of the canonical map 4814, but does not include the PCF 4810A, PCF 4810B of the tracking maps 4804A, 4804B. After the map merger, the PPs previously associated with PCF 4810A, PCF 4810B can be associated with PCF 4810C.

[0391] Example

[0392] Fig.40 And Fig.41 Shows an example of generating a tracking map by Fig. 9 the first XR device 12.1. Fig.40 Is a two-dimensional representation of a three-dimensional first local tracking map (map Figure 1 ) which can be generated by Fig. 9 the first XR device. Fig.41 is a block diagram showing an upload from a first XR device according to some embodiments to a server Fig. 9 of Figure 1 .

[0393] Fig.40 shows the ground Figure 1 and virtual content (Content 123 and Content 456) on the first XR device 12.1. The ground Figure 1 has an origin (Origin 1). The ground Figure 1 includes a number of PCFs (PCF a to PCF d). From the perspective of the first XR device 12.1, for example, PCF a is located at the origin of the ground Figure 1 and has X, Y, and Z coordinates of (0, 0, 0), and PCF b has X, Y, and Z coordinates of (-1, 0, 0). Content 123 is associated with PCF a. In this example, Content 123 has X, Y, and Z relationships relative to PCF a of (1, 0, 0). Content 456 has a relationship relative to PCF b. In this example, Content 456 has X, Y, and Z relationships relative to PCF b of (1, 0, 0).

[0394] In Fig.41 , the first XR device 12.1 uploads the ground Figure 1 to the server 20. The server 20 now has a canonical map based on the ground Figure 1 . The first XR device 12.1 has a canonical map that is empty at this stage. For the purpose of discussion, and in some embodiments, the server 20 does not include other maps other than the ground Figure 1 . There is no map stored on the second XR device 12.2.

[0395] The first XR device 12.1 also sends its Wi-Fi signature data to the server 20. The server 20 can use the Wi-Fi signature data to determine the approximate location of the first XR device 12.1 based on intelligence collected from other devices that have connected to the server 20 or other servers in the past with the GPS locations of such other recorded devices. The first XR device 12.1 can now end the first session (see Figure 8 ), and can disconnect from the server 20.

[0396] Fig.42 is a schematic diagram of an XR system according to some embodiments Fig.16 showing that after the first user 14.1 terminates the first session, the second user 14.2 has initiated a second session using the second XR device of the XR system. Fig.43ABlock diagram showing a second user 14.2 initiating a second session. The first user 14.1 is shown in dashed lines because the first session of the first user 14.1 has ended. The second XR device 12.2 starts recording an object. The server 20 can use various systems with different granularities to determine that the second session of the second XR device 12.2 is in the same vicinity as the first session of the first XR device 12.1. For example, the first XR device 12.1 and the second XR device 12.2 can include Wi-Fi signature data, Global Positioning System (GPS) positioning data, GPS data based on Wi-Fi signature data, or any other data indicating location to record their positions. Alternatively, the PCF identified by the second XR device 12.2 can display a similarity with the Figure 1 PCF of the ground.

[0397] As Fig.43B shown, the second XR device starts and begins to collect data, such as images 1110 from one or more cameras 44, 46. As Fig.14 shown, in some embodiments, an XR device (e.g., the second XR device 12.2) can collect one or more images 1110 and perform image processing to extract one or more features / areas of interest 1120. Each feature can be converted into a descriptor 1130. In some embodiments, the descriptor 1130 can be used to describe a keyframe 1140, which can have the position and orientation of additional associated images. One or more keyframes 1140 can correspond to a single persistent pose 1150, which can be automatically generated after a threshold distance (e.g., 3 meters) from a previous persistent pose 1150. One or more persistent poses 1150 can correspond to a single PCF 1160, which can be automatically generated after a predetermined distance (e.g., every 5 meters). Over time, as the user continues to move around the user's environment and the XR device continues to collect more data (such as images 1110), additional PCFs (e.g., PCF 3 and PCF 4, 5) may be created. One or two applications 1180 can run on the XR device and provide virtual content 1170 to the XR device for presentation to the user. The virtual content can have an associated content coordinate frame, which can be placed relative to one or more PCFs. As Fig.43B shown, the second XR device 12.2 creates three PCFs. In some embodiments, the second XR device 12.2 can attempt to locate one or more canonical maps stored on the server 20.

[0398] In some embodiments, as Fig.43C shown, the second XR device 12.2 can download the canonical map 120 from the server 20. The ground on the second XR device 12.2 Figure 1 It includes PCFs a to d and the origin 1. In some embodiments, the server 20 may have multiple canonical maps for respective locations and may determine that during a first session, the second XR device 12.2 is located in the same vicinity as the first XR device 12.1 and send the canonical map of that vicinity to the second XR device 12.2.

[0399] Fig.44 It shows that the second XR device 12.2 starts to identify PCFs for generating the Figure 2 . The second XR device 12.2 only identifies a single PCF, i.e., PCF 1,2. The X, Y, and Z coordinates of PCF 1,2 of the second XR device 12.2 may be (1, 1, 1). The Figure 2 has its own origin (origin 2), which may be based on the head pose of device 2 during the current head pose session at the start of the device. In some embodiments, the second XR device 12.2 may immediately attempt to localize the Figure 2 to the canonical map. In some embodiments, since the system cannot identify any or sufficient overlap between the two maps, the Figure 2 may not be able to be localized to the canonical map (the Figure 1 ), i.e., the localization may fail. In some embodiments, the system may localize based on the PCF comparison between the local map and the canonical map. In some embodiments, the system may localize based on the persistent pose comparison between the local map and the canonical map. In some embodiments, the system may localize based on the key frame comparison between the local map and the canonical map.

[0400] Fig.45 It shows the Figure 2 after the second XR device 12.2 identifies other PCFs (PCF 1,2, PCF 3, PCF 4,5) of the Figure 2 . The second XR device 12.2 attempts to localize the Figure 2 to the canonical map again. Since the Figure 2 has been expanded to overlap with at least a part of the canonical map, the localization attempt will succeed. In some embodiments, the overlap between the local tracking map, the Figure 2 and the canonical map may be represented by PCFs, persistent poses, key frames, or any other suitable intermediate or derived constructs.

[0401] In addition, the second XR device 12.2 has associated content 123 and content 456 with PCF 1,2 and PCF 3 of the Figure 2 . Content 123 has X, Y, and Z coordinates (1, 0, 0) relative to PCF 1,2. Similarly, relative to the Figure 2In the PCF 3, the X, Y, and Z coordinates of content 456 are (1, 0, 0).

[0402] Fig.46A and Fig.46B shown Figure 2 Successful positioning to the canonical map. The overlapping area / volume / cross-section of map 1410 is shown Figure 1 and the common part with the canonical map. Since Figure 2 PCF 3 and 4, 5 were created before positioning, while the canonical map was created before Figure 2 creating PCF a and c, different PCFs were created to represent the same volume in the actual space (e.g., in different maps).

[0403] As Fig.47 shown, the second XR device 12.2 extends Figure 2 , to include PCF a-d from the canonical map. Including PCF a-d represents Figure 2 positioning to the canonical map. In some embodiments, the XR system can perform an optimization step to remove duplicate PCFs from the overlapping area, such as PCF in 1410, PCF 3, and PCF 4, 5. After Figure 2 positioning, the placement of virtual content (such as content 456 and content 123) will be updated to be associated with the closest updated PCF. The virtual content appears in the same real-world position relative to the user, despite the change in the PCF attachment of the content and despite the update of the Figure 2 PCF of the Figure 2 .

[0404] As Fig.48 shown, the second XR device 12.2 continues to extend Figure 2 , for example, when the user walks around the real world, the second XR device 12.2 will identify other PCFs (PCF e, f, g, and h). It should also be noted that the Figure 1 does not extend in Fig.47 and Fig.48 .

[0405] Referring Fig.49 , the second XR device 12.2 uploads the Figure 2 to the server 20. The server 20 stores the Figure 2 together with the canonical Figure 1 . In some embodiments, when the session for the second XR device 12.2 ends, the Figure 2 can be uploaded to the server 20.

[0406] The canonical map within the server 20 now includes PCF i, which is not included in the Figure 1 In. When a third XR device (not shown) uploads a map to the server 20 and the map includes PCF i, the canonical map on the server 20 may be extended to include PCF i.

[0407] In Fig.50 , the server 20 merges the Figure 2 with the canonical map to form a new canonical map. The server 20 determines that PCFs a to d are common to the canonical map and the Figure 2 . The server extends the canonical map to include PCFs e to h and PCFs 1, 2 from the Figure 2 to form a new canonical map. The canonical maps on the first XR device 12.1 and the second XR device 12.2 are based on the Figure 1 and are out of date.

[0408] In Fig.51 , the server 20 sends the new canonical map to the first XR device 12.1 and the second XR device 12.2. In some embodiments, this may occur when the first XR device 12.1 and the second device 12.2 attempt to perform positioning during a different or new or subsequent session. The first XR device 12.1 and the second XR device 12.2 proceed as described above to align their respective local maps ( Figure 1 and Figure 2 ) to the new canonical map.

[0409] As Fig.52 shows, the head coordinate frame 96 or "head pose" is related to the PCF in the Figure 2 . In some embodiments, the origin of the map, origin 2, is based on the head pose of the second XR device 12.2 at the start of the session. When creating a PCF during the session, the PCF is placed relative to the world coordinate frame origin 2. The PCF of the Figure 2 is used as a persistent coordinate frame relative to the canonical coordinate frame, where the world coordinate frame can be the world coordinate frame of the previous session (e.g., Fig.40 the origin 1 of the Figure 1 in Fig. 9 ). The transformation from the world coordinate frame to the head coordinate frame 96 has been discussed previously with reference to Fig.52 . The head coordinate frame 96 shown in Figure 2 has only two orthogonal axes that are in specific coordinate positions relative to the PCF of the Figure 2 and at a specific angle relative to the Figure 2 . However, it should be understood that the head coordinate frame 96 is located in a three-dimensional position relative to the PCF of the

[0410] InFigure 53 In this case, the head coordinate frame 96 has been moved relative to the Figure 2 PCF of the ground. Since the second user 14.2 has moved their head, the head coordinate frame 96 has moved. The user can move their head in six degrees of freedom (6dof). The head coordinate frame 96 can thus move in 6dof (i.e., in three dimensions from its previous position in Figure 52 and relative to the Figure 2 PCF of the ground about three orthogonal axes). When Figure 9 the real object detection camera 44 and the inertial measurement unit 48 in respectively detect the real object and the movement of the head unit 22, the head coordinate frame 96 is adjusted. More information about head pose tracking is disclosed in U.S. Patent Application Serial No. 16 / 221,065, titled "Enhanced Pose Determination for Display Device", and is hereby incorporated by reference in its entirety.

[0411] Figure 54 It is shown that sounds can be associated with one or more PCFs. The user can, for example, wear a headset or earphones with stereo. The sound position through the earphones can be simulated using conventional techniques. The position of the sound can be located at a fixed position such that when the user rotates their head to the left, the position of the sound rotates to the right, so that the user perceives the sound from the same position in the real world. In this example, the positions of the sounds are represented by sound 123 and sound 456. For ease of discussion, Figure 54 in terms of analysis is similar to Figure 48 . When the first user 14.1 and the second user 14.2 are in the same room at the same or different times, they perceive the sounds 123 and 456 as coming from the same position in the real world.

[0412] Figure 55 and Figure 56 show another implementation of the above technology. As described with reference to Figure 8 , the first user 14.1 has initiated a first session. As shown in Figure 55 , the first user 14.1 has terminated the first session, as shown by the dashed line. At the end of the first session, the first XR device 12.1 uploads the Figure 1 to the server 20. The first user 14.1 has now initiated a second session at a time later than the first session. Since the Figure 1 has been stored on the first XR device 12.1, the first XR device 12.1 does not download the Figure 1 from the server 20. If the Figure 1 is lost, the first XR device 12.1 downloads the Figure 1。Then, the first XR device 12.1 continues to build the Figure 2 PCF, locates to the Figure 1 , and further develops the canonical map as described above. Then, as described above, the Figure 2 of the first XR device 12.1 is used to associate local content, head coordinate frames, local sounds, etc.

[0413] Reference Figure 57 and Figure 58 , it is also possible that more than one user interacts with the server in the same session. In this example, the first user 14.1 and the second user 14.2 are combined by the third user 14.3 with the third XR device 12.3. Each of the XR devices 12.1, 12.2, and 12.3 starts to generate its own map, namely the Figure 1 , the Figure 2 and the Figure 3 . When the XR devices 12.1, 12.2, and 12.3 continue to develop the Figure 1 , 2 and 3, the maps are incrementally uploaded to the server 20. The server 20 merges the Figure 1 , 2 and 3 to form the canonical map. Then the canonical map is sent from the server 20 to each of the XR devices 12.1, 12.2, and 12.3.

[0414] Figure 59 Illustrates aspects of a viewing method for restoring and / or resetting a head pose according to some embodiments. In the example shown, at action 1400, the viewing device is powered on. At action 1410, in response to being powered on, a new session is initiated. In some embodiments, the new session may include establishing a head pose. By first capturing an image of the environment and then determining the surface from the image, one or more capture devices on a head-mounted frame fixed to the user's head capture the surface of the environment. In some embodiments, the surface data can be combined with data from a gravity sensor to establish the head pose. Other suitable methods for establishing the head pose can be used.

[0415] At action 1420, the processor of the viewing device inputs a routine for tracking the head pose. When the user moves their head to determine the orientation of the head-mounted frame relative to the surface, the capture devices continue to capture the surface of the environment.

[0416] At operation 1430, the processor determines whether the head pose has been lost. The head pose may be lost due to "edge" cases such as excessive reflective surfaces, low light, blank walls, outdoors, etc. that can result in low feature acquisition; or due to dynamic situations such as movement and crowds forming part of the map. The routine at 1430 allows a certain amount of time, e.g., 10 seconds, to allow sufficient time to determine whether the head pose has been lost. If the head pose has not been lost, the processor returns to 1420 and resumes tracking the head pose again.

[0417] If the head pose has been lost at operation 1430, the processor enters a routine at 1440 to recover the head pose. If the head pose is lost due to low light, a message such as the following will be displayed to the user via the display of the viewing device:

[0418] The system is detecting low light conditions. Please move to a more well-lit area.

[0419] The system will continue to monitor whether sufficient light is available and whether the head pose can be recovered. The system may alternatively determine that the low texture of the surface is causing the head pose to be lost, in which case the following prompt will be given to the user in the display as a suggestion to improve surface capture:

[0420] The system cannot detect sufficient surfaces with fine texture. Please move to an area with less rough and finer surface texture.

[0421] At operation 1450, the processor enters a routine to determine whether head pose recovery has failed. If head pose recovery has not failed (i.e., head pose recovery has been successful), the processor returns to operation 1420 by entering the tracking of the head pose again. If head pose recovery has failed, the processor returns to operation 1410 to establish a new session. As part of the new session, all cached data is invalidated and the head pose is re-established thereafter. Any suitable head tracking method can be used in combination with Figure 59 the process described in. U.S. Patent Application No. 16 / 221,065 describes head tracking and is hereby incorporated by reference in its entirety.

[0422] Figure 60FIG. shows a schematic diagram of a machine in an exemplary form of a computer system 1900 within which a set of instructions can be executed to cause the machine to perform any one or more of the methods discussed herein. In alternative embodiments, the machine operates as a stand-alone device or can be connected (e.g., networked) to other machines. Further, although only a single machine is shown, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methods discussed herein.

[0423] The exemplary computer system 1900 includes a processor 1902 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), or both), a main memory 1904 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM)), and a static memory 1906 (e.g., flash memory, static random access memory (SRAM), etc.) that communicate with each other via a bus 1908.

[0424] The computer system 1900 may further include a disk drive unit 1916 and a network interface device 1920.

[0425] The disk drive unit 1916 includes a machine-readable medium 1922 on which is stored a set or multiple sets of instructions 1924 (e.g., software) embodying any one or more of the methods or functions described herein. During execution of software by the computer system 1900, the software can also reside, in whole or at least in part, within the main memory 1904 and / or within the processor 1902, which also constitutes machine-readable media.

[0426] The software can also be sent or received over the network 18 via the network interface device 1920.

[0427] The computer system 1900 includes a driver chip 1950 for driving a projector to generate light. The driver chip 1950 includes its own data storage 1960 and its own processor 1962.

[0428] Although the machine-readable medium 1922 is shown as a single medium in the exemplary embodiment, the term "machine-readable medium" should be understood to include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) that store a set of one or more instructions. The term "machine-readable medium" should also be considered to include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by a machine and that causes the machine to perform any one or more of the methods of the present invention. Thus, the term "machine-readable medium" should be considered to include, but not be limited to, solid-state memories, optical and magnetic media, and carrier signals.

[0429] Some aspects of several embodiments have been described above. It should be understood that various changes, modifications, and improvements will occur readily to those skilled in the art.

[0430] As an example, embodiments are described in the context of an augmented reality (AR) environment. It should be understood that some or all of the techniques described herein can be applied in an MR environment or more generally in other XR environments and VR environments.

[0431] As another example, embodiments are described in the context of a device such as a wearable device. It should be understood that some or all of the techniques described herein can be implemented via a network (such as the cloud), discrete applications, and / or devices, or any suitable combination of a network and discrete applications.

[0432] In addition, Figure 29 Examples of criteria that can be used to filter candidate maps to produce a set of highly ranked maps are provided. Instead of or in addition to the criteria described, other criteria can be used. For example, if multiple candidate maps have similar values for a metric that is used to filter out less desirable maps, the characteristics of the candidate maps can be used to determine which maps are retained as candidates or filtered out. For example, larger or denser candidate maps can be preferred over smaller candidate maps.

[0433] Such changes, modifications, and improvements are intended to be part of the present disclosure and are intended to be within the spirit and scope of the present disclosure. In addition, although the advantages of the present disclosure are indicated, it should be understood that not every embodiment of the present disclosure will include every described advantage. In some cases, some embodiments may not implement any of the features described as advantageous herein. Therefore, the foregoing description and drawings are for example only.

[0434] Some embodiments relate to a portable electronic system that includes a sensor configured to capture information about a three-dimensional (3D) environment and output an image, where each image includes a plurality of pixels; and at least one processor configured to execute computer-executable instructions to process the images output by the sensor. The computer-executable instructions include instructions for processing as follows: receiving a plurality of images captured by the sensor; for at least one subset of the plurality of images: for each image in the subset of images, identifying one or more features among the plurality of pixels, where each feature corresponds to one or more pixels; calculating a feature descriptor for each feature among the one or more features; and for each image in the subset of images, calculating a frame descriptor representing the image, at least in part based on the calculated feature descriptors in the image.

[0435] In some embodiments, the sensor includes at least one million pixel circuits. The frame descriptor for each image in the plurality of images includes 512 or fewer numbers.

[0436] In some embodiments, the computer-executable instructions include further instructions for processing as follows: constructing a map of at least a portion of the 3D environment; and associating the feature descriptors for the respective frames with portions of the map generated at least in part from the respective frames.

[0437] In some embodiments, the computer-executable instructions include instructions for processing as follows: selecting one or more key frames from the plurality of images as a subset of the plurality of images, at least in part based on the position of the image relative to the 3D environment and the positions of the plurality of pixels in the plurality of images.

[0438] In some embodiments, the computer-executable instructions include instructions for processing as follows: for a key frame among the one or more key frames, identifying one or more frames associated with the map of the 3D environment that have frame descriptors that are less than a threshold distance from the frame descriptor of the key frame.

[0439] In some embodiments, the computer-executable instructions for calculating the frame descriptor include an artificial neural network.

[0440] In some embodiments, the artificial neural network includes: a multi-layer perceptron unit that is trained based on similar and dissimilar images and is configured to receive a plurality of values representing features in the image as inputs and provide a weighted combination of the plurality of values representing the features as outputs; and a max pooling unit that is configured to select a subset of the outputs of the multi-layer perceptron unit as the frame descriptor.

[0441] Some embodiments relate to a method of operating a computing system to generate at least a portion of a map of a three-dimensional (3D) environment based on sensor data collected by a device worn by a user. The method includes: receiving a plurality of images captured by a device worn by the user; determining one or more key frames from the plurality of images; identifying one or more points of interest in the one or more key frames using a first artificial neural network; calculating a feature descriptor for each point of interest using the first artificial neural network; and for each key frame of the one or more key frames, calculating a frame descriptor representing the key frame using a second artificial neural network based at least in part on the feature descriptors calculated for the points of interest identified in the key frame.

[0442] In some embodiments, the first artificial neural network and the second artificial neural network are subnetworks of an artificial neural network.

[0443] In some embodiments, the frame descriptor is unique for each key frame.

[0444] In some embodiments, each key frame of the one or more key frames has a resolution higher than 1 megabyte. The frame descriptor for each key frame of the one or more key frames is a string of fewer than 512 numbers.

[0445] In some embodiments, each feature descriptor is a 32-byte string.

[0446] In some embodiments, the frame descriptor is generated by max-pooling the feature descriptors.

[0447] In some embodiments, the method includes: receiving a new image captured by a device worn by the user; and identifying one or more nearest key frames in a database including the key frames used to generate the map, the one or more nearest key frames having frame descriptors within a predetermined distance of the frame descriptor of the new image.

[0448] In some embodiments, the method includes: performing feature matching on 3D map points of the map corresponding to the identified one or more nearest key frames; and calculating the pose of the device worn by the user based on the feature matching results.

[0449] In some embodiments, determining one or more key frames from the plurality of images includes comparing the pixels of a first image with the pixels of a second image taken immediately after the first image, and identifying the second image as a key frame when the difference between the pixels of the first image and the pixels of the second image is above or below a threshold.

[0450] In some embodiments, the method includes training a second artificial neural network by: generating a data set including a plurality of image sets, where each of the plurality of image sets includes a query image, a positive sample image, and a negative sample image; for each of the plurality of image sets in the data set, calculating a loss by comparing the query image with the positive sample image and the negative sample image; and modifying the second artificial neural network based on the calculated loss such that the distance between the frame descriptor generated by the second artificial neural network for the query image and the frame descriptor for the positive sample image is greater than the distance between the frame descriptor generated by the second artificial neural network for the query image and the frame descriptor for the negative sample image.

[0451] Some embodiments relate to a computing environment for a cross-reality system. The computing environment includes a database storing a plurality of maps. Each map contains information representing a region of a 3D environment. The information representing each region includes a frame descriptor representing an image of the region; and a non-transitory computer storage medium storing computer-executable instructions that, when executed by at least one processor in the computing environment: process an image captured by a portable device by identifying a plurality of features in the image; calculate a feature descriptor for each of the plurality of features; calculate a frame descriptor representing the image based at least in part on the feature descriptors calculated for one or more identified points of interest in the image; and select a map in the database based on a comparison between the calculated frame descriptor and the frame descriptors stored in the map database.

[0452] In some embodiments, the frame descriptor is unique for the frames stored in the database.

[0453] In some embodiments, the image has a resolution higher than 1 megabyte. The calculated frame descriptor representing the image is a string of fewer than 512 numbers.

[0454] In some embodiments, the computer-executable instructions include an artificial neural network trained by: processing a data set including a plurality of image sets, where each of the plurality of image sets includes a query image, a positive sample image, and a negative sample image; calculating a loss for the image sets in the data set by comparing the query image with the positive sample image and the negative sample image; and modifying the artificial neural network based on the calculated loss such that the distance between the frame descriptor generated by the artificial neural network for the query image and the frame descriptor for the positive sample image is less than the distance between the frame descriptor generated by the artificial neural network for the query image and the frame descriptor for the negative sample image.

[0455] In some embodiments, modifying the artificial neural network includes modifying a copy of the artificial neural network on a portable device in the computing environment.

[0456] In some embodiments, the computing environment includes a cloud platform and a plurality of portable devices in communication with the cloud platform. The cloud platform includes a database and computer-executable instructions for selecting a map. Computer-executable instructions for processing images captured by the portable devices are stored on the portable devices.

[0457] Some embodiments relate to an XR system including a first XR device that includes a first processor, a first computer-readable medium connected to the first processor, a first origin coordinate frame stored on the first computer-readable medium, a first destination coordinate frame stored on the computer-readable medium, a first data channel for receiving data representing local content, a first coordinate frame transformer executable by the first processor to transform the localization of the local content from the first origin coordinate frame to the first destination coordinate frame, and a first display system adapted to display the local content to a first user after transforming the localization of the local content from the first origin coordinate frame to the first destination coordinate frame.

[0458] Some embodiments relate to a viewing method that includes: storing a first origin coordinate frame; storing a first destination coordinate frame; receiving data representing local content; transforming the localization of the local content from the first origin coordinate frame to the first destination coordinate frame; and displaying the local content to a first user after transforming the localization of the local content from the first origin coordinate frame to the first destination coordinate frame.

[0459] Some embodiments relate to an XR system that includes: a map storage routine that stores a first map, the first map being a canonical map having a plurality of persistent coordinate frames (PCFs), each PCF of the first map having a set of coordinates; a real object detection device positioned to detect the location of a real object; a PCF identification system connected to the real object detection device to detect a PCF of a second map based on the location of the real object, each PCF of the second map having a set of coordinates; and a localization module connected to the canonical map and the second map and executable to localize the second map to the canonical map by matching a first PCF of the second map with a first PCF of the canonical map and matching a second PCF of the second map with a second PCF of the canonical map.

[0460] In some embodiments, the real object detection device is a real object detection camera.

[0461] In some embodiments, the XR system further includes: a canonical map merger connected to the canonical map and the second map and executable to merge a third PCF of the canonical map into the second map.

[0462] In some embodiments, the XR system further includes: an XR device, which includes: a head unit, which includes: a head-mounted frame, wherein the real object detection device is mounted to the head-mounted frame; a data channel that receives image data of local content; a local content positioning system that is connected to the data channel and is executable to associate the local content with an anchor point of the canonical map; and a display system that is connected to the local content positioning system to display the local content.

[0463] In some embodiments, the XR system further includes: a local-to-world coordinate transformer that transforms the local coordinate frame of the local content to the world coordinate frame of the second map.

[0464] In some embodiments, the XR system further includes: a first world frame determination routine that calculates a first world coordinate frame based on the PCF of the second map; a first world frame storage instruction that stores the world coordinate frame; a head frame determination routine that calculates a head coordinate frame that changes as the head-mounted frame moves; a head frame storage instruction that stores the first head coordinate frame; and a world-to-head coordinate transformer that transforms the world coordinate frame to the head coordinate frame.

[0465] In some embodiments, when the head-mounted frame moves, the head coordinate frame changes relative to the world coordinate frame.

[0466] In some embodiments, the XR system further includes: at least one sound element associated with at least one PCF of the second map.

[0467] In some embodiments, the first map and the second map are created by the XR device.

[0468] In some embodiments, the XR system further includes: a first XR device and a second XR device. Each XR device includes: a head unit, which includes: a head-mounted frame, wherein the real object detection device is mounted to the head-mounted frame; a data channel that receives image data of local content; a local content positioning system that is connected to the data channel and is executable to associate the local content with a PCF of the canonical map; and a display system that is connected to the local content positioning system to display the local content.

[0469] In some embodiments, the first XR device creates the PCF of the first map, and the second XR device creates the PCF of the second map, and the positioning module forms a part of the second XR device.

[0470] In some embodiments, the first map and the second map are created in a first session and a second session, respectively.

[0471] In some embodiments, the XR system further includes: a server; and a map download system forming part of the XR device, which downloads the first map from the server via a network.

[0472] In some embodiments, the positioning module repeatedly attempts to position the second map to the canonical map.

[0473] In some embodiments, the XR system further includes: a map publisher, which uploads the second map to the server via the network.

[0474] Some embodiments relate to a viewing method, including: storing a first map, the first map being a canonical map having a plurality of PCFs, each PCF of the canonical map having a set of coordinates; detecting the position of a real object; detecting a PCF of a second map based on the position of the real object, each PCF of the second map having a set of coordinates; and positioning the second map to the canonical map by matching a first PCF of the second map with a first PCF of the first map and matching a second PCF of the second map with a second PCF of the canonical map.

[0475] Some embodiments relate to an XR system, including: a server, which may have: a processor; a computer-readable medium connected to the processor; a plurality of canonical maps on the computer-readable medium; corresponding canonical map identifiers on the computer-readable medium associated with each respective canonical map, the canonical map identifiers being different from each other to uniquely identify the canonical maps; a position detector located on the computer-readable medium and executable by the processor to receive and store position identifiers from an XR device; a first filter located on the computer-readable medium and executable by the processor to compare the position identifiers with the canonical map identifiers to determine one or more canonical maps forming a first filtered selection; and a map transmitter located on the computer-readable medium and executable by the processor to transmit one or more of the canonical maps to the XR device based on the first filtered selection.

[0476] In some embodiments, each of the canonical map identifiers includes longitude and latitude, and the position identifier includes longitude and latitude.

[0477] In some embodiments, the first filter is an adjacent area filter that selects at least one matching canonical map covering the longitude and latitude that includes the location identifier and at least one adjacent map covering the longitude and latitude adjacent to the first matching canonical map.

[0478] In some embodiments, the location identifier includes a WiFi fingerprint. The XR system further includes: a second filter as a WiFi fingerprint filter located on the computer-readable medium and executable by the processor to: determine the latitude and longitude based on the WiFi fingerprint; compare the latitude and longitude from the WiFi fingerprint filter with the latitude and longitude of the canonical map to determine one or more canonical maps forming a second filtered selection within the first filtered selection, and the map transmitter transmits one or more canonical maps based on the second selection rather than the canonical maps of the first selection outside the second selection.

[0479] In some embodiments, the first filter is a WiFi fingerprint filter located on the computer-readable medium and executable by the processor to: determine the latitude and longitude based on the WiFi fingerprint; compare the latitude and longitude from the WiFi fingerprint filter with the latitude and longitude of the canonical map to determine one or more canonical maps forming the first filtered selection.

[0480] In some embodiments, the XR system further includes: a multi-layer perception unit on the computer-readable medium and executable by the processor that receives multiple features of an image and converts each feature into a corresponding digital string; a max pooling unit on the computer-readable medium and executable by the processor that combines the maximum values of each digital string into a global feature string representing the image, wherein each canonical map has at least one of the global feature strings, and the location identifier received from the XR device includes the features of the image captured by the XR device, and the features are processed by the multi-layer perception unit and the max pooling unit to determine the global feature string of the image; and a keyframe filter that compares the global feature string of the image with the global feature string of the canonical map to determine one or more canonical maps forming a third filtered selection within the second filtered selection, and the map transmitter transmits one or more canonical maps based on the third selection rather than the canonical maps of the second selection outside the third selection.

[0481] In some embodiments, the XR system further includes: a multi-layer perception unit on the computer-readable medium and executable by the processor, which receives multiple features of an image and converts each feature into a corresponding digital string; a max pooling unit on the computer-readable medium and executable by the processor, which combines the maximum values of each digital string into a global feature string representing the image, wherein each canonical map has at least one of the global feature strings, and the location identifier received from the XR device includes features of an image captured by the XR device, and the features are processed by the multi-layer perception unit and the max pooling unit to determine the global feature string of the image; and wherein the first filter is a keyframe filter, and the keyframe filter compares the global feature string of the image with the global feature string of the canonical map to determine one or more canonical maps.

[0482] In some embodiments, the XR system further includes: an XR device, which includes: a head unit, which includes: a head-mounted frame, wherein the real object detection device is mounted to the head-mounted frame; a data channel, which receives image data of local content; a local content positioning system, which is connected to the data channel and executable to associate the local content with a PCF of the canonical map; and a display system, which is connected to the local content positioning system to display the local content.

[0483] In some embodiments, the XR device includes: a map storage routine, which stores a first map, the first map being a canonical map having multiple PCFs, and each PCF of the first map having a set of coordinates; a real object detection device, which is positioned to detect the position of a real object; a PCF identification system, which is connected to the real object detection device to detect a PCF of a second map based on the position of the real object, and each PCF of the second map having a set of coordinates; and a positioning module, which is connected to the canonical map and the second map, and executable to position the second map to the canonical map by matching a first PCF of the second map with a first PCF of the canonical map and matching a second PCF of the second map with a second PCF of the canonical map.

[0484] In some embodiments, the real object detection device is a real object detection camera.

[0485] In some embodiments, the XR system further includes: a canonical map merger, which is connected to the canonical map and the second map and executable to merge a third PCF of the canonical map into the second map.

[0486] Some embodiments relate to a viewing method, including: storing a plurality of canonical maps on a computer-readable medium, each canonical map having a corresponding canonical map identifier associated with the corresponding canonical map, the canonical map identifiers being different from each other to uniquely identify the canonical maps; using a processor connected to the computer-readable medium to receive and store a location identifier from an XR device; using the processor to compare the location identifier with the canonical map identifiers to determine one or more canonical maps forming a first filtered selection; and using the processor to send a plurality of the canonical maps to the XR device based on the first filtered selection.

[0487] Some embodiments relate to an XR system, including: a processor; a computer-readable medium connected to the processor; a multi-layer perceptron unit on the computer-readable medium and executable by the processor, which receives a plurality of features of an image and converts each feature into a corresponding digital string; and a max pooling unit on the computer-readable medium and executable by the processor, which combines the maximum values of each digital string into a global feature string representing the image.

[0488] In some embodiments, the XR system includes: a plurality of canonical maps on the computer-readable medium, each canonical map having at least one of the global feature strings associated therewith; a location detector on the computer-readable medium and executable by the processor, which receives features of an image captured by the XR device, the features being processed by the multi-layer perceptron unit and the max pooling unit to determine the global feature string of the image; a keyframe filter, which compares the global feature string of the image with the global feature strings of the canonical maps to determine one or more canonical maps forming part of a filtered selection; and a map transmitter located on the computer-readable medium and executable by the processor to send one or more of the canonical maps to the XR device based on the filtered selection.

[0489] In some embodiments, the XR system includes: an XR device, which includes: a head unit, which includes: a head-mounted frame, wherein the real object detection device is mounted to the head-mounted frame; a data channel, which receives image data of local content; a local content positioning system, which is connected to the data channel and executable to associate the local content with a PCF of the canonical maps; and a display system, which is connected to the local content positioning system to display the local content.

[0490] In some embodiments, the XR system includes: an XR device, which includes: a head unit, which includes: a head-mounted frame, wherein the real object detection device is mounted to the head-mounted frame; a data channel that receives image data of local content; a local content positioning system that is connected to the data channel and is executable to associate the local content with a PCF of the canonical map; and a display system that is connected to the local content positioning system to display the local content, wherein the matching is performed by matching the global feature string of the second map with the global feature string of the canonical map.

[0491] Some embodiments relate to a viewing method, including: using a processor to receive multiple features of an image; using the processor to convert each feature into a corresponding digital string; and using the processor to combine the maximum values of each digital string into a global feature string representing the image.

[0492] Some embodiments relate to a method of operating a computing system for identifying one or more environmental maps stored in a database to merge with a tracking map calculated based on sensor data collected by a device worn by a user, wherein the device receives a signal of an access point of a computer network while calculating the tracking map, and the method includes: determining at least one regional attribute of the tracking map based on communication characteristics with the access point; determining a geographical location of the tracking map based on the at least one regional attribute; identifying a set of environmental maps stored in the database corresponding to the determined geographical location; filtering the set of environmental maps based on similarity of one or more identifiers of network access points associated with the tracking map and the environmental maps in the set of environmental maps; filtering the set of environmental maps based on similarity of a metric representing the content of the tracking map and the environmental maps in the set of environmental maps; and filtering the set of environmental maps based on a degree of matching between a portion of the tracking map and a portion of the environmental maps in the set of environmental maps.

[0493] In some embodiments, filtering the set of environmental maps based on the similarity of the one or more identifiers of the network access point includes: retaining, in the set of environmental maps, the environmental map having the highest Jaccard similarity with the at least one regional attribute of the tracking map based on the one or more identifiers of the network access point.

[0494] In some embodiments, filtering the set of environmental maps based on a similarity of a metric representing the content of the tracking map and the environmental maps in the set of environmental maps includes: retaining in the set of environmental maps the environmental map having the smallest vector distance between the feature vector of the tracking map and the vector representing the environmental map in the set of environmental maps.

[0495] In some embodiments, the metric representing the content of the tracking map and the environmental map includes a vector of values calculated from the content of the map.

[0496] In some embodiments, filtering the set of environmental maps based on a degree of match between a portion of the tracking map and a portion of the environmental maps in the set of environmental maps includes: calculating a volume of the physical world represented by the tracking map, which is also represented in the environmental maps in the set of environmental maps; and retaining in the set of environmental maps the environmental maps having a larger calculated volume than the environmental maps filtered out from the set of environmental maps.

[0497] In some embodiments, the set of environmental maps is filtered by: first, based on the similarity of the one or more identifiers; subsequently, based on the similarity of the metric representing the content; and subsequently, based on the degree of match between a portion of the tracking map and a portion of the environmental map.

[0498] In some embodiments, filtering the set of environmental maps based on the similarity of the one or more identifiers includes: filtering the set of environmental maps based on the similarity of the metric representing the content; and performing the degree of match between a portion of the tracking map and a portion of the environmental map in an order based on the processing required to perform the filtering.

[0499] In some embodiments, an environmental map is selected based on the filtering of the set of environmental maps based on the following conditions: the similarity of the metric representing the content; performing the degree of match between a portion of the tracking map and a portion of the environmental map in an order based on the processing required to perform the filtering, and information is loaded from the selected map onto the user device.

[0500] In some embodiments, an environmental map is selected based on the filtering of the set of environmental maps based on the following conditions: the similarity of the one or more identifiers; the similarity of the metric representing the content; and the degree of match between a portion of the tracking map and a portion of the environmental map, and the tracking map is merged with the selected environmental map.

[0501] Some embodiments relate to a cloud computing environment for an augmented reality system configured to communicate with a plurality of user devices including sensors. The cloud computing environment includes: a user database storing area identifiers indicating areas in which the plurality of user devices are used, the area identifiers including parameters of a wireless network detected by the user devices during use; a map database storing a plurality of environmental maps constructed from data and associated metadata provided by the plurality of user devices, the associated metadata including area identifiers derived from the area identifiers of the plurality of user devices that provided the data from which the maps were constructed, the area identifiers including parameters of a wireless network detected by the user devices that provided the data from which the maps were constructed; a non-transitory computer storage medium storing computer-executable instructions that, when executed by at least one processor in the cloud computing environment: receive messages from the plurality of user devices including parameters of a wireless network detected by the user devices, calculate area identifiers for the user devices, and update the user database based on the received parameters and / or the calculated area identifiers; and receive requests for environmental maps from the plurality of user devices, determine area identifiers associated with the user devices requesting the environmental maps, identify multiple sets of environmental maps from the map database at least in part based on the area identifiers, filter the multiple sets of environmental maps, and send the filtered multiple sets of environmental maps to the user devices, wherein filtering a set of environmental maps is based on the similarity between the parameters of the wireless network detected by the user device from which the request for the environmental map originated and the parameters of the wireless network in the map database for the environmental map in the set of environmental maps.

[0502] In some embodiments, the computer-executable instructions are further configured to, when executed by at least one processor in the cloud computing environment, receive a tracking map from a user device requesting an environmental map; and filtering a set of environmental maps is further based on the similarity of a metric representing the content of the tracking map and the environmental map in the set of environmental maps.

[0503] In some embodiments, the computer-executable instructions are further configured to, when executed by at least one processor in the cloud computing environment, receive a tracking map from a user device requesting an environmental map; and filtering a set of environmental maps is further based on the degree of match between a portion of the tracking map and a portion of the environmental map in the set of environmental maps.

[0504] In some embodiments, the parameters of the wireless network include the basic service set identifier (BSSID) of the network to which the user device is connected.

[0505] In some embodiments, filtering the set of environmental maps based on similarities of parameters of a wireless network includes: calculating a similarity between a plurality of BSSIDs stored in the user database associated with the user equipment requesting the environmental map and BSSIDs stored in the map database associated with the environmental map in the set of environmental maps.

[0506] In some embodiments, the area identifier indicates a geographical location by longitude and latitude.

[0507] In some embodiments, determining the area identifier includes accessing the area identifier from the user database.

[0508] In some embodiments, determining the area identifier includes receiving the area identifier in a message received from the plurality of user devices.

[0509] In some embodiments, the parameters of the wireless network comply with protocols including Wi-Fi and 5G NR.

[0510] In some embodiments, the computer-executable instructions are further configured to, when executed by at least one processor in the cloud computing environment, receive a tracking map from a user device; and filter the set of environmental maps further based on a degree of match between a portion of the tracking map and a portion of the environmental maps in the set of environmental maps.

[0511] In some embodiments, the computer-executable instructions are further configured to, when executed by at least one processor in the cloud computing environment: receive a tracking map from a user device and determine an area identifier associated with the tracking map based on the user device providing the tracking map; select a second set of environmental maps from the map database at least in part based on the area identifier associated with the tracking map; and update the map database based on the received tracking map, wherein the update includes merging the received tracking map with one or more environmental maps in the second set of environmental maps.

[0512] In some embodiments, the computer-executable instructions are further configured to, when executed by at least one processor in the cloud computing environment, filter the second set of environmental maps based on a degree of match between a portion of the received tracking map and a portion of the environmental maps in the second set of environmental maps; and merging the tracking map with one or more environmental maps in the second set of environmental maps includes: merging the tracking map with one or more environmental maps in the filtered second set of environmental maps.

[0513] Some embodiments relate to an XR system, including: a real object detection device that detects multiple surfaces of real-world objects; a PCF recognition system connected to the real object detection device to generate a map based on the real-world objects; a persistent coordinate frame (PCF) generation system that generates a first PCF based on the map and associates the first PCF with the map; a first storage medium and a second storage medium on a first XR device and a second XR device respectively; and at least a first processor and a second processor of the first XR device and the second XR device that store the first PCF in the first storage medium of the first XR device and the second storage medium of the second XR device respectively.

[0514] In some embodiments, a keyframe generator that can be executed by the at least one processor to transform multiple camera images into multiple corresponding keyframes; a persistent pose calculator that can be executed by the at least one processor to generate a persistent pose by averaging the multiple keyframes; a tracking map and persistent pose transformer that can be executed by the at least one processor to transform a tracking map to the persistent pose to determine the persistent pose at the origin of the tracking map; a persistent pose and PCF transformer that can be executed by the at least one processor to transform the persistent pose to the first PCF to determine the first PCF relative to the persistent pose; a PCF and image data transformer that can be executed by the at least one processor to transform the first PCF into image data; and a display device that displays the image data related to the first PCF to the user.

[0515] In some embodiments, the detection device is the detection device of the first XR device connected to the processor of the first XR device.

[0516] In some embodiments, the map is a first map on the first XR device, and the processor that generates the first map is the first XR device processor of the first XR device.

[0517] In some embodiments, the processor that generates the first PCF is the first XR device processor of the first XR device.

[0518] In some embodiments, the processor that associates the first PCF with the first map is the first XR device processor of the first XR device.

[0519] In some embodiments, the XR system includes: an application executable by the first XR device processor; a first PCF tracker executable by the first XR device processor and including an open prompt from the application to open the first PCF tracker, wherein the first PCF tracker generates the first PCF only when the first PCF tracker is opened.

[0520] In some embodiments, the first PCF tracker has a close prompt to close the first PCF tracker from the application, wherein when the first PCF tracker is closed, the first PCF tracker terminates first PCF generation.

[0521] In some embodiments, the XR system includes: a map publisher executable by the first XR device processor to send the first PCF to a server; a map storage routine executable by the server processor of the server to store the first PCF on a storage device of the server; and the server processor of the server to send the first PCF to the second XR device; and a map download system executable by the second XR device processor of the second XR device to download the first PCF from the server.

[0522] In some embodiments, the XR system includes: an application executable by the second XR device processor; and a second PCF tracker executable by the second XR device processor and including an open prompt from the application to open the second PCF tracker, wherein the second PCF tracker generates a second PCF only when the second PCF tracker is opened.

[0523] In some embodiments, the second PCF tracker has a close prompt to close the second PCF tracker from the application, wherein when the second PCF tracker is closed, the second PCF tracker terminates second PCF generation.

[0524] In some embodiments, the XR system includes: a map publisher executable by the second XR device processor to send the second PCF to the server.

[0525] In some embodiments, the XR system includes: a persistent pose acquirer, which can be executed by the first XR device processor to download a persistent pose from the server; a PCF checker, which can be executed by the first XR device processor to obtain a PCF from a first storage device of the first XR device based on the persistent pose; and a coordinate frame calculator, which can be executed by the first XR device processor to calculate a coordinate frame based on the PCF obtained from the first storage device.

[0526] Some embodiments relate to a viewing method, including: using at least one detection device to detect multiple surfaces of a real-world object; using at least one processor to generate a map based on the real-world object; using at least one processor to generate a first PCF based on the map; using the at least one processor to associate the first PCF with the map; and storing the first PCF in a first storage medium of the first XR device and a second storage medium of the second XR device respectively using at least a first processor of the first XR device and a second processor of the second XR device.

[0527] In some embodiments, the viewing method includes: using the at least one processor to transform multiple camera images into multiple corresponding key frames; using the at least one processor to generate a persistent pose by averaging the multiple key frames; using the at least one processor to transform a tracking map into the persistent pose to determine the persistent pose at an origin of the tracking map; using the at least one processor to transform the persistent pose into the first PCF to determine the first PCF relative to the persistent pose; using the at least one processor to transform the first PCF into image data; and using a display device to display the image data related to the first PCF to the user.

[0528] In some embodiments, the detection device is a detection device of the first XR device connected to the first XR device processor.

[0529] In some embodiments, the map is a first map on the first XR device, and the processor that generates the first map is the first XR device processor of the first XR device.

[0530] In some embodiments, the processor that generates the first PCF is the first XR device processor of the first XR device.

[0531] In some embodiments, the processor that associates the first PCF with the first map is the first XR device processor of the first XR device.

[0532] In some embodiments, the viewing method includes: executing an application using the first XR device processor; and using the first XR device processor to open a first PCF tracker with an open prompt from the application, wherein the first PCF tracker generates the first PCF only when the first PCF tracker is open.

[0533] In some embodiments, the viewing method includes: using the first XR device processor to close the first PCF tracker according to a close prompt from the application, wherein the first PCF tracker terminates first PCF generation only when the first PCF tracker is closed.

[0534] In some embodiments, the viewing method includes: using the first XR device processor to send the first PCF to a server; using the server processor of the server to store the first PCF on a storage device of the server; and using the server processor of the server to send the first PCF to the second XR device; and using the second XR device processor of the second XR device to receive the first PCF from the server.

[0535] In some embodiments, the viewing method includes: executing an application using the second XR device processor; and using the second XR device processor to open a second PCF tracker with an open prompt from the application, wherein the second PCF tracker generates a second PCF only when the second PCF tracker is open.

[0536] In some embodiments, the viewing method includes: using the first XR device processor to close the second PCF tracker with a close prompt of the application, wherein the second PCF tracker terminates second PCF generation only when the second PCF tracker is closed.

[0537] In some embodiments, the viewing method includes: using the second XR device processor to upload the second PCF to the server.

[0538] In some embodiments, the viewing method includes: using the first XR device processor to determine a persistent pose from the server; using the first XR device processor to obtain a PCF based on the persistent pose from a first storage device of the first XR device; and using the first XR device processor to calculate a coordinate frame based on the PCF obtained from the first storage device.

[0539] Some embodiments relate to an XR system, including: a first XR device, which may include: a first XR device processor; a first XR device storage device connected to the first XR device processor; a set of instructions on the first XR device processor, including: a download system capable of being executed by the first XR device processor to download a persistent pose from a server; a PCF acquirer capable of being executed by the first XR device processor to acquire a PCF based on the persistent pose from the first storage device of the first XR device; and a coordinate frame calculator capable of being executed by the first XR device processor to calculate a coordinate frame based on the PCF acquired from the first storage device.

[0540] Some embodiments relate to a viewing method, including: using a first XR device processor of a first XR device to download a persistent pose from a server; using the first XR device processor to acquire a PCF based on the persistent pose from the first storage device of the first XR device; and using the first XR device processor to calculate a coordinate frame based on the PCF acquired from the first storage device.

[0541] Some embodiments relate to an XR device, including: a server, which may include: a server processor; a server storage device connected to the server processor; a map storage routine capable of being executed by the server processor of the server to store a first PCF associated with a map on the server storage device of the server; and a map transmitter capable of being executed by the server processor to send the map and the first PCF to a first XR device.

[0542] Some embodiments relate to a viewing method, including: using a server processor of a server to store a first PCF associated with a map on the server storage device of the server; and using the server processor of the server to send the map and the first PCF to a first XR device.

[0543] Some embodiments relate to a viewing method, including: entering tracking of a head pose by a processor of an XR device by capturing an environment with a capture device fixed to a head-mounted frame fixed to a user's head and determining an orientation of the head-mounted frame; determining by the processor whether the head pose is lost due to an inability to determine the orientation of the head-mounted frame; and if the head pose is lost, entering, by the processor, a pose recovery mode to establish the head pose by determining the orientation of the head-mounted frame.

[0544] In some embodiments, if the head pose is not lost, the processor enters tracking of the head pose.

[0545] In some embodiments, pose recovery includes: the processor displaying a message with suggestions to the user to improve the capture of the environment.

[0546] In some embodiments, the suggestion is at least one of increasing light and refining texture.

[0547] In some embodiments, the viewing method i...

Claims

1. An electronic system, comprising: one or more sensors configured to capture information about a three-dimensional (3D) environment, the captured information including a plurality of images; and at least one processor configured to execute computer-executable instructions to generate a map of at least a portion of the 3D environment based on the plurality of images, wherein the computer-executable instructions further include instructions for: identifying a plurality of features in the plurality of images at least in part based on criteria indicating persistence associated with the identified features; selecting a plurality of key frames from the plurality of images, the selection being at least in part based on the plurality of features of the selected key frames; generating one or more coordinate frames at least in part based on the identified features of the selected key frames; and storing the one or more coordinate frames as one or more persistent coordinate frames in association with the map of the 3D environment.

2. The electronic system according to claim 1, wherein: the one or more sensors include a plurality of pixel circuits arranged in a two-dimensional array such that each image of the plurality of images includes a plurality of pixels; and each feature corresponds to a plurality of pixels.

3. The electronic system according to claim 1, wherein storing the one or more coordinate frames includes storing for each of the one or more coordinate frames: a descriptor representing at least a subset of the features in the selected key frame from which the coordinate frame is generated.

4. The electronic system according to claim 1, wherein storing the one or more coordinate frames includes storing for each of the one or more coordinate frames: at least a subset of the features in the selected key frame from which the coordinate frame is generated.

5. The electronic system according to claim 1, wherein storing the one or more coordinate frames includes storing for each of the one or more coordinate frames: a transformation between the coordinate frame of the map of the 3D environment and the persistent coordinate frame; and geographic information indicating the location of the selected key frame within the 3D environment from which the coordinate frame is generated.

6. The electronic system according to claim 5, wherein the geographic information includes a WiFi fingerprint of the location.

7. The electronic system according to claim 1, wherein the computer-executable instructions include instructions for calculating a feature descriptor for each feature using an artificial neural network.

8. The electronic system according to claim 1, wherein: the computer-executable instructions further include: an application programming interface configured to provide information characterizing a persistent coordinate frame among the one or more persistent coordinate frames to an application executing on a portable electronic system; instructions for refining the map of the 3D environment based on a second plurality of images; adjusting one or more of the persistent coordinate frames at least in part based on the second plurality of images; and instructions for providing a notification of the adjusted persistent coordinate frame through the application programming interface.

9. The electronic system according to claim 8, wherein: Adjusting the one or more persistent coordinate frames includes: adjusting the translation and rotation of the one or more persistent coordinate frames relative to the origin of the map of the 3D environment.

10. The electronic system according to claim 9, wherein: The electronic system includes a wearable device, and the one or more sensors are mounted on the wearable device; The map is a tracking map calculated on the wearable device, and The origin of the map is determined based on the position where the wearable device is powered on.

11. An electronic system, comprising: One or more sensors configured to capture information about a three-dimensional (3D) environment, the captured information including a plurality of images; and At least one processor configured to execute computer-executable instructions to generate a map of at least a portion of the 3D environment based on the plurality of images, wherein the computer-executable instructions further include instructions for: Identifying a plurality of features in the plurality of images; Selecting a plurality of key frames from the plurality of images, the selection being at least partially based on the plurality of features of the selected key frames; Generating one or more coordinate frames at least partially based on the identified features of the selected key frames; and Storing the one or more coordinate frames as one or more persistent coordinate frames in association with the map of the 3D environment, wherein: The computer-executable instructions include instructions for calculating a feature descriptor for each feature using a first artificial neural network; and The computer-executable instructions include instructions for implementing a second artificial neural network configured to calculate a frame descriptor representing a key frame at least partially based on the feature descriptors calculated for the identified features in the key frames.

12. An electronic system, comprising: One or more sensors configured to capture information about a three-dimensional (3D) environment, the captured information including a plurality of images; and At least one processor configured to execute computer-executable instructions to generate a map of at least a portion of the 3D environment based on the plurality of images, wherein the computer-executable instructions further include instructions for: Identifying a plurality of features in the plurality of images; Selecting a plurality of key frames from the plurality of images, the selection being at least partially based on the plurality of features of the selected key frames; Generating one or more coordinate frames at least partially based on the identified features of the selected key frames; and Storing the one or more coordinate frames as one or more persistent coordinate frames in association with the map of the 3D environment, wherein: Identifying the plurality of features in the plurality of images includes: selecting a number of pixel groups less than a predetermined maximum value as the identified features based on a measure of similarity to a group of pixels depicting a persistent object part.

13. The electronic system according to claim 12, wherein: The one or more sensors include a plurality of pixel circuits arranged in a two-dimensional array such that each of the plurality of images includes a plurality of pixels; and Each feature corresponds to a plurality of pixels.

14. The electronic system according to claim 12,[[]] wherein,[[]] Storing the one or more coordinate frames includes storing for each of the one or more coordinate frames: A descriptor that represents at least a subset of the features in the selected key frame from which the coordinate frame is generated.

15. The electronic system according to claim 12,[[]] wherein,[[]] Storing the one or more coordinate frames includes storing for each of the one or more coordinate frames: At least a subset of the features in the selected key frame from which the coordinate frame is generated.

16. The electronic system according to claim 12,[[]] wherein,[[]] Storing the one or more coordinate frames includes storing for each of the one or more coordinate frames: The transformation between the coordinate frame of the map of the 3D environment and the persistent coordinate frame; and Geographic information that indicates the position of the selected key frame within the 3D environment from which the coordinate frame is generated.

17. The electronic system according to claim 12,[[]] wherein,[[]] The computer-executable instructions include instructions for calculating feature descriptors for individual features using an artificial neural network.

18. The electronic system according to claim 17,[[]] wherein: The artificial neural network is a first artificial neural network; and The computer-executable instructions include instructions for implementing a second artificial neural network configured to calculate a frame descriptor representing a key frame based at least in part on the feature descriptors calculated for the identified features in the key frame.

19. The electronic system according to claim 12,[[]] wherein: The computer-executable instructions further include: An application programming interface configured to provide information characterizing a persistent coordinate frame among the one or more persistent coordinate frames to an application executing on a portable electronic system; Instructions for refining the map of the 3D environment based on a second plurality of images; Adjusting one or more of the persistent coordinate frames at least in part based on the second plurality of images; Instructions for providing notification of the adjusted persistent coordinate frame through the application programming interface.

20. The electronic system according to claim 19,[[]] wherein: Adjusting the one or more persistent coordinate frames includes adjusting the translation and rotation of the one or more persistent coordinate frames relative to the origin of the map of the 3D environment.

21. The electronic system according to claim 20,[[]] wherein: The electronic system includes a wearable device, and the one or more sensors are mounted on the wearable device; The map is a tracking map calculated on the wearable device, and The origin of the map is determined based on the position where the wearable device is powered on.

22. The electronic system according to claim 12,[[]] wherein: The electronic system includes a wearable device, and the one or more sensors are mounted on the wearable device; The computer-executable instructions further include instructions for the following operations: Tracking the movement of a portable device; And Based on the tracked movement indicating that the movement of the wearable device exceeds a threshold distance, controlling the execution timing of instructions for generating one or more coordinate frames and / or instructions for storing one or more persistent coordinate frames, wherein the threshold distance is between two meters and twenty meters.

23. An electronic system, Comprising: One or more sensors configured to capture information about a three-dimensional (3D) environment, the captured information including a plurality of images; And At least one processor configured to execute computer-executable instructions to generate a map of at least a portion of the 3D environment based on the plurality of images, wherein the computer-executable instructions further include instructions for the following operations: Identifying a plurality of features in the plurality of images; Selecting a plurality of key frames from the plurality of images, the selection being at least partially based on the plurality of features of the selected key frames; Generating one or more coordinate frames at least partially based on the identified features of the selected key frames; and Storing the one or more coordinate frames as one or more persistent coordinate frames in association with the map of the 3D environment, wherein: The electronic system includes a wearable device, and the one or more sensors are mounted on the wearable device; The computer-executable instructions further include instructions for the following operations: Tracking the movement of a portable device; and Based on the tracked movement indicating that the movement of the wearable device exceeds a threshold distance, controlling the execution timing of instructions for generating one or more coordinate frames and / or instructions for storing one or more persistent coordinate frames, wherein the threshold distance is between two meters and twenty meters.

24. The electronic system according to claim 23, Wherein, The computer-executable instructions include instructions for calculating a feature descriptor for each feature using an artificial neural network.

25. The electronic system according to claim 24, Wherein: The artificial neural network is a first artificial neural network; and The computer-executable instructions include instructions for implementing a second artificial neural network configured to calculate a frame descriptor representing a key frame at least partially based on the feature descriptors calculated for the identified features in the key frames.

26. The electronic system according to claim 23, Wherein: The computer-executable instructions further include: An application programming interface configured to provide information characterizing a persistent coordinate frame in the one or more persistent coordinate frames to an application executing on a portable electronic system; Instructions for refining the map of the 3D environment based on a second plurality of images; Adjusting one or more of the persistent coordinate frames in the persistent coordinate frames at least partially based on the second plurality of images; Instructions for providing a notification of the adjusted persistent coordinate frame through the application programming interface.

27. The electronic system according to claim 26, wherein: adjusting the one or more persistent coordinate frames includes: adjusting the translation and rotation of the one or more persistent coordinate frames relative to the origin of the map of the 3D environment.

28. The electronic system according to claim 27, wherein: the electronic system includes a wearable device, and the one or more sensors are mounted on the wearable device; the map is a tracking map calculated on the wearable device, and the origin of the map is determined based on the location where the wearable device is powered on.

Images