Cross reality system

Abstract

To provide an ideal cross reality system.SOLUTION: An augmented reality viewing system is described. A local coordinate frame of local content is transformed to a world coordinate frame. A further transformation is made to a head coordinate frame and a further transformation is made to a camera coordinate frame including all pupil positions of an eye. One or more users may interact with a viewing system in separate sessions. If a canonical map is available, an earlier map is downloaded onto a viewing device of a user. The viewing device then generates another map and identifies the position of a subsequent map to the canonical map.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to and benefit of U.S. Provisional Patent Application No. 62 / 718,357, filed Aug. 13, 2018, and entitled "VIEWING DEVICE OR VIEWING DEVICES HAVING ONE OR MORE COORDINATE FRAME TRANSFORMERS," which is incorporated herein by reference in its entirety. This patent application also claims priority to and benefit of U.S. Provisional Patent Application No. 62 / 742,237, filed Oct. 5, 2018, and entitled "COORDINATE FRAME PROCESSING AUGMENTED REALITY," which is incorporated herein by reference in its entirety. This patent application also claims priority to and benefit of U.S. Provisional Patent Application No. 62 / 812,935, filed March 1, 2019, entitled "MERGING A PLURALITY OF INDIVIDUALLY MAPPED ENVIRONMENTS," which is incorporated herein by reference in its entirety, and U.S. Provisional Patent Application No. 62 / 815,955, filed March 8, 2019, entitled "VIEWING DEVICE OR VIEWING DEVICES HAVING ONE OR MORE COORDINATE FRAME TRANSFORMERS," which is incorporated herein by reference in its entirety. This patent application also claims priority to and benefit of U.S. Provisional Patent Application No. 62 / 868,786, filed June 28, 2019, entitled "RANKING AND MERGING A PLURALITY OF ENVIRONMENT MAPS," which is incorporated herein by reference in its entirety. This patent application also claims priority to and benefit of U.S. Provisional Patent Application No. 62 / 870,954, filed July 5, 2019, and entitled “RANKING AND MERGING A PLURALITY OF ENVIRONMENT MAPS,” which is incorporated by reference in its entirety herein.This patent application also claims priority to and benefit of U.S. Provisional Patent Application No. 62 / 884,109, filed August 8, 2019, and entitled "A VIEWING SYSTEM," which is incorporated by reference in its entirety herein.

[0002] This application relates generally to cross reality systems. [Background technology]

[0003] A computer may control a human user interface and create an X-reality (XR or cross-reality) environment in which some or all of the XR environment is generated by the computer as it is perceived by the user. These XR environments may be virtual reality (VR), augmented reality (AR), and mixed reality (MR) environments in which some or all of the XR environment may be generated by the computer, in part, using data that describes the environment. This data may describe virtual objects, for example, that may be rendered so that the user can sense or perceive and interact with the virtual objects as part of the physical world. The user may experience these virtual objects as a result of data being rendered and presented through a user interface device, such as a head-mounted display device. The data may control audio that may be displayed to the user for viewing or played audibly to the user, or may control a tactile (or haptic) interface, allowing the user to experience touch sensations that the user senses or perceives as they feel the virtual objects.

[0004] XR systems can be useful for many applications ranging from scientific visualization, medical training, engineering design and prototyping, remote manipulation and telepresence, and personal entertainment. AR and MR, in contrast to VR, involve one or more objects in association with real objects in the physical world. The experience of a virtual object interacting with a real object generally enhances the user's enjoyment when using an XR system and also opens up possibilities for a variety of applications that present realistic and easily understandable information about how the physical world can be altered.

[0005] An XR system may represent the physical world around a user of the system as a "mesh." The mesh may be represented by a number of interconnected triangles. Each triangle has edges that join points on the surface of an object in the physical world such that each triangle represents a portion of the surface. Information about portions of the surface, such as color, texture, or other properties, may be associated and stored within the triangles. In operation, the XR system may process image information and detect points and surfaces to create or update a mesh. Summary of the Invention [Means for solving the problem]

[0006] Aspects of the present application relate to methods and apparatus for providing X-reality (cross-reality or XR) scenes. The techniques described herein may be used together, separately, or in any suitable combination.

[0007] Some embodiments relate to an XR system including a first XR device including a first processor, a first computer-readable medium connected to the first processor, a first origin coordinate frame stored on the first computer-readable medium, a first destination coordinate frame stored on the computer-readable medium, a first data channel receiving data representing local content, a first coordinate frame converter executable by the first processor to convert a positioning of the local content from the first origin coordinate frame to the first destination coordinate frame, and a first display system adapted to display the local content to a first user after converting the positioning of the local content from the first origin coordinate frame to the first destination coordinate frame.

[0008] Some embodiments relate to a viewing method that includes storing a first origin coordinate frame, storing a first destination coordinate frame, receiving data representing local content, transforming a positioning of the local content from the first origin coordinate frame to the first destination coordinate frame, and displaying the local content to a first user after transforming the positioning of the local content from the first origin coordinate frame to the first destination coordinate frame.

[0009] Some embodiments relate to an XR system including: a map storage routine that stores a first map, the first map being a reference map having a plurality of anchors, each anchor of the first map having a set of coordinates; a real object detection device positioned to detect locations of real objects; an anchor identification system connected to the real object detection device and that detects anchors of a second map based on locations of the real objects, each anchor of the second map having a set of coordinates; and a localization module connected to the reference map and the second map and executable to localize the second map relative to the reference map by matching a first anchor of the second map to a first anchor of the reference map and matching a second anchor of the second map to a second anchor of the reference map.

[0010] Some embodiments relate to a viewing method that includes storing a first map, the first map being a reference map having a plurality of anchors, each anchor of the reference map having a set of coordinates; detecting locations of real objects; detecting anchors of a second map based on the locations of the real objects, each anchor of the second map having a set of coordinates; and locating the second map relative to the reference map by matching the first anchor of the second map to the first anchor of the first map and matching the second anchor of the second map to the second anchor of the reference map.

[0011] Some embodiments relate to an XR system including a server that may have a processor, a computer readable medium connected to the processor, a plurality of reference maps on the computer readable medium, individual reference map identifiers on the computer readable medium associated with each individual reference map, the reference map identifiers being different from one another and uniquely identifying the reference map, a position detector on the computer readable medium and executable by the processor to receive and store the position identifier from the XR device, a first filter on the computer readable medium and executable by the processor to compare the position identifier and the reference map identifier and determine one or more reference maps forming a first filtered selection, and a map transmitter on the computer readable medium and executable by the processor to transmit one or more of the reference maps to the XR device based on the first filtered selection.

[0012] Some embodiments relate to a viewing method including storing a plurality of reference maps on a computer readable medium, each reference map having a respective reference map identifier associated therewith, the reference map identifiers being different from one another and uniquely identifying the reference map; receiving and storing, with a processor coupled to the computer readable medium, a location identifier from the XR device; comparing, with the processor, the location identifier and the reference map identifier to determine one or more reference maps forming a first filtered selection; and transmitting, with the processor, the plurality of reference maps to the XR device based on the first filtered selection.

[0013] Some embodiments relate to an XR system including a processor; a computer-readable medium coupled to the processor; a multi-layer perception unit on the computer-readable medium and executable by the processor to receive a plurality of features of the image and convert each feature into a separate number sequence; and a max pooling unit on the computer-readable medium and executable by the processor to combine a maximum value of each number sequence into a global feature sequence representing the image.

[0014] Some embodiments relate to a method of viewing that includes receiving, with a processor, a plurality of features of an image; converting, with the processor, each feature into a separate number sequence; and combining, with the processor, the maximum value of each sequence into a global feature sequence that represents the image.

[0015] Some embodiments relate to a method of operating a computing system to identify one or more environmental maps stored in a database and merge with a tracking map calculated based on sensor data collected by a device worn by a user, the device receiving a signal of an access point to a computer network while calculating the tracking map, the method including the steps of determining at least one area attribute of the tracking map based on characteristics of communication with the access point, determining a geographic location of the tracking map based on the at least one area attribute, identifying a set of environmental maps stored in the database corresponding to the determined geographic location, filtering the set of environmental maps based on similarity of one or more identifiers of network access points associated with the tracking map and an environmental map of the set of environmental maps, filtering the set of environmental maps based on similarity of a metric representative of content of the environmental map of the tracking map and the set of environmental maps, and filtering the set of environmental maps based on a degree of matching between a portion of the tracking map and a portion of an environmental map of the set of environmental maps.

[0016] Some embodiments include a user database that stores area identifications indicative of areas in which a plurality of user devices are used, the area identifications comprising parameters of wireless networks detected by the user devices when in use; and a map database that stores a plurality of environmental maps constructed from data provided by the plurality of user devices and associated metadata, the associated metadata comprising area identifications derived from area identifications of the plurality of user devices that provided the data from which the map was constructed, the area identifications comprising parameters of wireless networks detected by the user devices that provided the data from which the map was constructed; and a map database that, when executed by at least one processor in a cloud computing environment, receives messages from a plurality of user devices comprising parameters of wireless networks detected by the user devices, and stores area identifiers for the user devices. and updating a user database based on the received parameters and / or the calculated area identifier, receiving requests for the environmental maps from a plurality of user devices, determining an area identifier associated with the user devices requesting the environmental maps, identifying a set of environmental maps from the map database based at least in part on the area identifier, filtering the set of environmental maps, and transmitting the filtered set of environmental maps to the user devices, wherein filtering the set of environmental maps is based on similarity between parameters of a wireless network detected by the user device from which the request for the environmental map originated and parameters of a wireless network in the map database for the environmental maps in the set of environmental maps.

[0017] Some embodiments relate to an XR system including a real object detection device that detects multiple surfaces of a real-world object; an anchor identification system connected to the real object detection device and that generates a map based on the real-world object; a persistent coordinate frame (PCF) generation system that generates a first PCF based on the map and associates the first PCF with the map; first and second storage media on the first and second XR devices, respectively; and at least first and second processors of the first and second XR devices that store the first PCF in the first and second storage media of the first and second XR devices, respectively.

[0018] Some embodiments relate to a viewing method including: detecting a plurality of surfaces of a real-world object with at least one detection device; generating a map based on the real-world object with at least one processor; generating a first PCF based on the map with the at least one processor; associating the first PCF with the map with the at least one processor; and storing the first PCF in first and second storage media of the first and second XR devices, respectively, with at least first and second processors of the first and second XR devices.

[0019] Some embodiments relate to an XR system including a first XR device, which may include a first XR device processor, a first XR device storage device connected to the first XR device processor, a download system executable by the first XR device processor to download a sustained posture from a server, a PCF reader executable by the first XR device processor to read a PCF from the first storage device of the first XR device based on the sustained posture, and a coordinate frame calculator executable by the first XR device processor to calculate a coordinate frame based on the PCF read from the first storage device.

[0020] Some embodiments relate to a viewing method including the steps of: downloading a sustained pose from a server using a first XR device processor of a first XR device; reading a PCF from a first storage device of the first XR device based on the sustained pose using the first XR device processor; and calculating a coordinate frame based on the PCF read from the first storage device using the first XR device processor.

[0021] Some embodiments relate to a viewing method including a server, which may include a server processor, a server storage device connected to the server processor, a map storage routine executable by the server processor of the server to store a first PCF associated with the map on the server storage device of the server, and a map transmitter executable by the server processor to transmit the map and the first PCF to a first XR device. The present invention also provides a viewing method including the steps of: storing, by the server processor of the server, a first PCF associated with the map on the server storage device of the server; and transmitting, by the server processor of the server, the map and the first PCF to a first XR device.

[0022] Some embodiments relate to a viewing method that includes entering head pose tracking by a processor of the XR device using a capture device on a head-mounted frame affixed to the user's head to capture a surface of the environment and determine an orientation of the head-mounted frame relative to the surface, determining by the processor whether the head pose has been lost due to an inability to determine an orientation of the head-mounted frame relative to the surface, and if the head pose has been lost, entering by the processor into a pose recovery mode to establish the head pose by determining an orientation of the head-mounted frame relative to the surface. The present invention provides, for example, the following: (Item 1) Coordinate frame transformation An XR system, A first XR device, the first XR device comprising: A first processor; a first computer readable medium coupled to the first processor; a first origin coordinate frame stored on the first computer readable medium; and a first destination coordinate frame stored on the computer readable medium; a first data channel for receiving data representing local content; a first coordinate frame converter executable by the first processor to convert a positioning of the local content from the first origin coordinate frame to the first destination coordinate frame; a first display system adapted to display local content to a first user after transforming a positioning of the local content from the first origin coordinate frame to the first destination coordinate frame; The first XR device, including An XR system equipped with (Item 2) The first XR device further comprises: a first real object detection device for detecting a position of a real object; a first world surface determination routine, the first world surface determination routine being executable by the first processor to identify a location of at least one point on a surface of the real object; and a first world frame determination routine, the first world frame determination routine being executable by the first processor to calculate a first world coordinate frame based on the at least one point, and one of the first origin and a first destination coordinate frame being the first world coordinate frame; first world frame storage instructions, the first world frame storage instructions executable by the first processor to store the world coordinate frame on the computer readable medium; 2. The XR system according to item 1, comprising: (Item 3) 3. The XR system of item 2, wherein the first real object detection device is a camera. (Item 4) 3. The XR system of item 2, wherein the first real object detection device detects the positioning of multiple real objects. (Item 5) 3. The XR system of item 2, wherein the first world surface determination routine identifies a positioning of a plurality of points on a surface of the real object. (Item 6) 6. The XR system of item 5, wherein the first world frame determination routine calculates the first world coordinate frame based on the plurality of points. (Item 7) The first XR device further comprises: a first head mountable frame; a first head frame determination routine, the first head frame determination routine being executable by the first processor to calculate a first head coordinate frame that changes in response to movement of the first head mountable frame, and one of the first origin and first destination coordinate frames being the first head coordinate frame; a first head frame storage instruction, the first head frame storage instruction being executable by the first processor to store the first head coordinate frame on the first computer readable medium; 3. The XR system according to item 2, comprising: (Item 8) 8. The XR system of claim 7, wherein the first coordinate frame converter is a first world / head coordinate converter that converts the first world coordinate frame into the first head coordinate frame. (Item 9) The first XR device further comprises: 8. The XR system of item 7, further comprising a first inertial measurement unit attached to the first head-mountable frame and configured to detect movement of the first head-mountable frame, the first head frame determination routine calculating the first head coordinate frame based on measurements of the first inertial measurement unit. (Item 10) The first device further comprises: 8. The XR system of item 7, further comprising a first moving tracking camera affixed to the first head-mountable frame for detecting movement of the first head-mountable frame, and the first head frame determination routine calculates the first head coordinate frame based on images captured by the first moving tracking camera. (Item 11) The first XR device further comprises: a first local frame determination routine, the first local frame determination routine being executable by the first processor to calculate a first local coordinate frame of the local content, one of the first origin and a first destination coordinate frame being the local coordinate frame; a first local frame store instruction, the first local frame store instruction being executable by the first processor to store the local coordinate frame on the computer readable medium; 3. The XR system according to item 2, comprising: (Item 12) 12. The XR system of claim 11, wherein the first coordinate frame converter is a first local / world coordinate converter that converts the first local coordinate frame to the first world coordinate frame. (Item 13) The first XR device further comprises: 2. The XR system of claim 1, further comprising: a first camera frame stored on the first computer-readable medium, the first camera frame including a plurality of eye positions for an eye moving relative to the first head-mountable frame; and the first coordinate frame converter is a first head / camera coordinate converter that converts the first head coordinate frame to the first camera coordinate frame. (Item 14) A visual inspection method, comprising: storing a first origin coordinate frame; storing a first destination coordinate frame; Receiving data representing local content; Transforming positioning of local content from the first origin coordinate frame to the first destination coordinate frame; displaying the local content to a first user after transforming the positioning of the local content from the first origin coordinate frame to the first destination coordinate frame; A method comprising: Standard Map (Item 15) An XR system, a map storage routine for storing a first map, the first map being a reference map having a plurality of anchors, each anchor of the first map having a set of coordinates; a real object detection device positioned to detect the location of a real object; an anchor identification system coupled to the real object detection device and configured to detect anchors of a second map based on locations of the real objects, each anchor of the second map having a set of coordinates; a location determination module coupled to the reference map and the second map and operable to locate the second map relative to the reference map by matching a first anchor of the second map to a first anchor of the reference map and matching a second anchor of the second map to a second anchor of the reference map; An XR system equipped with (Item 16) Item 16. The XR system of item 15, wherein the real object detection device is a real object detection camera. (Item 17) Item 16. The XR system of item 15, further comprising a reference map embedder connected to the reference map and the second map and executable to embed a third anchor of the reference map into the second map. (Item 18) An XR device, comprising: A head unit, a head-mountable frame, the real object detection device being mounted on the head-mountable frame; a data channel for receiving image data of a local content; a local content positioning system coupled to said data channel and operable to associate said local content with an anchor of said reference map; a display system coupled to the local content location system for displaying the local content; A head unit comprising: Including XR devices Item 16. The XR system of item 15, further comprising: (Item 19) 20. The XR system of claim 18, further comprising a local / world coordinate converter that converts a local coordinate frame of the local content into a world coordinate frame of the second map. (Item 20) a first world frame determination routine that calculates a first world coordinate frame based on anchors of the second map; a first world frame storage instruction for storing the world coordinate frame; a head frame determination routine for calculating a head coordinate frame that changes in response to movement of the head mountable frame; a head frame storage instruction for storing the first head coordinate frame; a world / head coordinate converter for converting the world coordinate frame into the head coordinate frame; 19. The XR system of item 18, further comprising: (Item 21) 21. The XR system of claim 20, wherein the head coordinate frame changes relative to the world coordinate frame as the head-mountable frame moves. (Item 22) 20. The XR system of item 18, further comprising at least one sound element associated with at least one anchor of the second map. (Item 23) 20. The XR system of item 18, wherein the first and second maps are created by the XR device. (Item 24) a first and a second XR device, each XR device comprising: A head unit, a head-mountable frame, the real object detection device being mounted on the head-mountable frame; a data channel for receiving image data of a local content; a local content positioning system coupled to said data channel and operable to associate said local content with an anchor of said reference map; a display system coupled to the local content location system for displaying the local content; A head unit comprising: The first and second XR devices, including Item 16. The XR system of item 15, further comprising: (Item 25) 25. The XR system of claim 24, wherein the first XR device creates an anchor for the first map and the second XR device creates an anchor for the second map, and the location determination module forms part of the second XR device. (Item 26) 26. The XR system of item 25, wherein the first and second maps are created in a first and second session, respectively. (Item 27) 26. The XR system of item 25, wherein the first and second maps are created in the same session. (Item 28) A server; a map download system forming part of the XR device and configured to download the first map from a server via a network; 19. The XR system of item 18, further comprising: (Item 29) 16. The XR system of claim 15, wherein the localization module repeatedly attempts to localize the second map against the reference map. (Item 30) Item 16. The XR system of item 15, further comprising a map publisher that uploads the second map to the server via the network. (Item 31) The reference map is stored on the server; a map merging algorithm on the server that merges the second map with the reference map; a map transmitter for transmitting the reference map after the second map is merged with the reference map; Item 16. The XR system of item 15, further comprising: (Item 32) A visual inspection method, comprising: storing a first map, the first map being a reference map having a plurality of anchors, each anchor of the reference map having a set of coordinates; Detecting a location of a real object; detecting anchors in a second map based on the locations of the real objects, each anchor in the second map having a set of coordinates; locating the second map relative to the reference map by matching a first anchor of the second map to a first anchor of the first map and matching a second anchor of the second map to a second anchor of the reference map; A method comprising: Criterion Map Filtering (Item 33) An XR system, A server, A processor; a computer readable medium coupled to the processor; a plurality of reference maps on the computer readable medium; a respective reference map identifier on the computer readable medium associated with each respective reference map, the reference map identifiers being different from one another and uniquely identifying the reference map; a position detector, the position detector being on the computer readable medium and executable by the processor to receive and store a position identifier from an XR device; a first filter, the first filter being on the computer readable medium and executable by the processor to compare the location identifier to the criteria map identifier and determine one or more criteria maps forming a first filtered selection; a map transmitter on the computer-readable medium and executable by the processor to transmit one or more of the reference maps to the XR device based on the first filtered selection; A server having An XR system equipped with (Item 34) 34. The XR system of item 33, wherein each of the reference map identifiers includes a longitude and a latitude, and each of the location identifiers includes a longitude and a latitude. (Item 35) Item 35. The XR system of item 34, wherein the first filter is a neighborhood area filter that selects at least one matching criterion map that covers longitudes and latitudes that include the longitude and latitude of the location identifier and at least one neighborhood map that covers longitudes and latitudes adjacent to the first matching criterion map. (Item 36) the location identifier comprises a WiFi fingerprint; a second filter, the second filter being a WiFi fingerprint filter, on the computer readable medium and configured to be generated by the processor; determining a latitude and longitude based on the WiFi fingerprint; comparing the latitudes and longitudes from the WiFi fingerprint filter with the latitudes and longitudes of the criteria map to determine one or more criteria maps forming a second filtered selection within the first filtered selection, the map transmitter transmitting one or more criteria maps based on the second selection and not transmitting any criteria maps based on the first selection outside the second selection; a second filter operable to Item 36. The XR system of item 35, further comprising: (Item 37) The first filter is a WiFi fingerprint filter, the WiFi fingerprint filter being on the computer readable medium, and the WiFi fingerprint filter is generated by the processor. determining a latitude and longitude based on the WiFi fingerprint; comparing the latitudes and longitudes from the WiFi fingerprint filter with the latitudes and longitudes of the reference map to determine one or more reference maps forming the first filtered selection; Item 34. The XR system according to item 33, which is executable to perform the steps: (Item 38) a multi-layer perception unit, the multi-layer perception unit being on the computer readable medium and executable by the processor, the multi-layer perception unit receiving a plurality of features of an image and converting each feature into a respective sequence of numbers; a max pooling unit, the max pooling unit being on the computer readable medium and executable by the processor, combining a maximum value of each sequence into a global feature sequence representing the image, each reference map having at least one of the global feature sequences, the location identifier received from the XR device including features of an image captured by the XR device that are developed by the multi-layer perception unit and the max pooling unit to determine a global feature sequence of the image; a keyframe filter, the keyframe filter comparing a global feature sequence of the image with a global feature sequence of the criterion map to determine one or more criterion maps forming a third filtered selection within the second filtered selection, the map transmitter transmitting one or more criterion maps based on the third selection and not transmitting a criterion map based on the second selection outside the third selection; 37. The XR system of item 36, further comprising: (Item 39) a multi-layer perception unit, the multi-layer perception unit being on the computer readable medium and executable by the processor, the multi-layer perception unit receiving a plurality of features of an image and converting each feature into a respective sequence of numbers; a max pooling unit, the max pooling unit being on the computer readable medium and executable by the processor, combining a maximum value of each sequence into a global feature sequence representing the image, each reference map having at least one of the global feature sequences, the location identifier received from the XR device including features of an image captured by the XR device that are developed by the multi-layer perception unit and the max pooling unit to determine a global feature sequence of the image; Further equipped with Item 34. The XR system of item 33, wherein the first filter is a keyframe filter that compares a global feature sequence of the image with a global feature sequence of the reference map and determines one or more reference maps. (Item 40) An XR device, comprising: A head unit, a head-mountable frame, the real object detection device being mounted on the head-mountable frame; a data channel for receiving image data of a local content; a local content positioning system coupled to said data channel and operable to associate said local content with an anchor of said reference map; a display system coupled to the local content location system and displaying the local content; A head unit comprising: Including XR devices 34. The XR system of item 33, further comprising: (Item 41) The XR device is a map storage routine for storing a first map, the first map being a reference map having a plurality of anchors, each anchor of the first map having a set of coordinates; a real object detection device positioned to detect the location of a real object; an anchor identification system coupled to the real object detection device and configured to detect anchors of a second map based on locations of the real objects, each anchor of the second map having a set of coordinates; a location determination module coupled to the reference map and the second map and operable to locate the second map relative to the reference map by matching a first anchor of the second map to a first anchor of the reference map and matching a second anchor of the second map to a second anchor of the reference map; Item 41. The XR system according to item 40, comprising: (Item 42) 42. The XR system of item 41, wherein the real object detection device is a real object detection camera. (Item 43) Item 42. The XR system of item 41, further comprising a reference map embedder connected to the reference map and the second map and executable to embed a third anchor of the reference map into the second map. (Item 44) A visual inspection method, comprising: storing a plurality of criteria maps on a computer readable medium, each having a respective criteria map identifier associated with said respective criteria map identifier, said criteria map identifiers being different from one another and uniquely identifying said criteria maps; receiving and storing, with a processor coupled to the computer readable medium, a location identifier from the XR device; using the processor to compare the location identifiers to the criteria map identifiers to determine one or more criteria maps forming a first filtered selection; transmitting, with the processor, a plurality of the reference maps to the XR device based on the first filtered selection; A method comprising: Keyframes (Item 45) An XR system, A processor; a computer readable medium coupled to the processor; a multi-layer perception unit, the multi-layer perception unit being on the computer readable medium and executable by the processor, the multi-layer perception unit receiving a plurality of features of an image and converting each feature into a respective sequence of numbers; a max pooling unit, the max pooling unit being on the computer readable medium and executable by the processor, for combining a maximum value of each sequence into a global feature sequence representing the image; An XR system equipped with (Item 46) a plurality of criteria maps on the computer readable medium, each criteria map having at least one of the global feature columns associated therewith; a position detector, the position detector being on the computer readable medium and executable by the processor to receive from the XR device features of an image captured by the XR device, which are processed by the multi-layer perception unit and the max pooling unit to determine a global feature sequence of the image; a keyframe filter that compares global feature sequences of the image with global feature sequences of the criteria maps to determine one or more criteria maps that form part of a filtered selection; a map transmitter, the map transmitter being on the computer readable medium and executable by the processor to transmit one or more of the reference maps to the XR device based on the filtered selection; Item 46. The XR system of item 45, further comprising: (Item 47) An XR device, comprising: A head unit, a head-mountable frame, the real object detection device being mounted on the head-mountable frame; a data channel for receiving image data of a local content; a local content positioning system coupled to said data channel and operable to associate said local content with an anchor of said reference map; a display system coupled to the local content location system and displaying the local content; A head unit comprising: Including XR devices Item 46. The XR system of item 45, further comprising: (Item 48) An XR device, comprising: A head unit, a head-mountable frame, the real object detection device being mounted on the head-mountable frame; a data channel for receiving image data of a local content; a local content positioning system coupled to said data channel and operable to associate said local content with an anchor of said reference map; a display system coupled to the local content location system and configured to display the local content, the matching being performed by matching a global feature sequence of the second map to a global feature sequence of the reference map; and A head unit comprising: Including XR devices Item 48. The XR system of item 47, further comprising: (Item 49) A visual inspection method, comprising: receiving, with a processor, a plurality of features of the image; converting each feature into a separate sequence of numbers using said processor; combining, with said processor, the maximum value of each sequence into a global feature sequence representative of said image; A method comprising: Ranking and merging maps (from issue '823) (Item 50) 1. A method of operating a computing system to identify and merge one or more environmental maps stored in a database with a tracking map calculated based on sensor data collected by a device worn by a user, the device receiving signals of access points to a computer network while calculating the tracking map, the method comprising: determining at least one area attribute of the tracking map based on characteristics of communications with the access points; determining a geographic location of the tracking map based on the at least one area attribute; identifying a set of environment maps stored in the database that correspond to the determined geographic location; filtering the set of environmental maps based on a similarity of one or more identifiers of network access points associated with the environmental maps of the set of tracking maps and environmental maps; filtering the set of environment maps based on a similarity of a metric representative of the content of the tracking map and the set of environment maps; filtering the set of environment maps based on a degree of matching between the portion of the tracking map and a portion of the environment map of the set of environment maps; A method comprising: (Item 51) Item 51. The method of item 50, wherein filtering the set of environmental maps based on similarity of one or more identifiers of the network access points includes retaining within the set of environmental maps the environmental map with the highest Jaccard similarity to at least one area attribute of the tracking map based on the one or more identifiers of the network access points. (Item 52) Item 51. The method of item 50, wherein filtering the set of environment maps based on a similarity of a metric representing the content of the environment maps of the tracking map and the set of environment maps includes retaining in the set of environment maps an environment map with a minimum vector distance between a vector of characteristics of the tracking map and a vector representing an environment map in the set of environment maps. (Item 53) 51. The method of claim 50, wherein the metric representative of the contents of the tracking map and the environment map comprises a vector of values ​​calculated from the contents of the maps. (Item 54) Filtering the set of environment maps based on a degree of matching between the portion of the tracking map and a portion of the environment map of the set of environment maps includes: calculating a volume of the physical world represented by said tracking map that is also represented in an environment map of a set of environment maps; retaining in the set of environment maps those environment maps with calculated volumes larger than the filtered out environment maps from said set; 51. The method according to item 50, comprising: (Item 55) A set of environment maps is First, based on similarity of the one or more identifiers, Then, based on the similarity of the metric representing the content, Then, based on a degree of matching between the portion of the tracking map and the portion of the environment map, 51. The method according to item 50, wherein the information is filtered. (Item 56) the similarity of said one or more identifiers; the similarity of said metric representing the content; a degree of matching between the portion of the tracking map and the portion of the environment map; filtering the set of environment maps based on The method according to item 50. (Item 57) The environment map is the similarity of said one or more identifiers; the similarity of said metric representing the content; a degree of matching between the portion of the tracking map and the portion of the environment map; The set of environment maps is selected based on filtering the set of environment maps based on Information is loaded onto the user device from the selected environment map. The method according to item 50. (Item 58) The environment map is the similarity of said one or more identifiers; the similarity of said metric representing the content; a degree of matching between a portion of the tracking map and a portion of the environment map; The set of environment maps is selected based on filtering the set of environment maps based on The tracking map is merged with the selected environment map. The method according to item 50. (Item 59) 1. A cloud computing environment for an augmented reality system configured for communication with a plurality of user devices equipped with sensors, comprising: a user database storing area identifications indicative of areas in which the plurality of user devices are used, the area identifications comprising parameters of wireless networks detected by the user devices when in use; a map database storing a plurality of environmental maps constructed from data and associated metadata provided by the plurality of user devices, the associated metadata comprising area identifications derived from area identifications of the plurality of user devices that provided data from which the map was constructed, the area identifications comprising parameters of wireless networks detected by the user devices that provided data from which the map was constructed; A non-transitory computer storage medium storing computer-executable instructions that, when executed by at least one processor in the cloud computing environment, receiving messages from the plurality of user devices comprising parameters of wireless networks detected by the user devices, calculating area identifiers for the user devices, and updating the user database based on the received parameters and / or the calculated area identifier; receiving requests for environmental maps from the plurality of user devices, determining an area identifier associated with a user device requesting the environmental maps, identifying a set of environmental maps from the map database based at least in part on the area identifier, filtering the set of environmental maps, and transmitting the filtered set of environmental maps to the user device; Do the following: filtering the set of environmental maps is based on similarities between parameters of wireless networks detected by the user device from which the request for the environmental map originated and parameters of wireless networks in the map database for the environmental maps in the set of environmental maps; Non-transitory computer storage media and A cloud computing environment. (Item 60) The computer-executable instructions, when executed by at least one processor in the cloud computing environment, are further configured to receive a tracking map from a user device that requests an environment map; filtering the set of environment maps further comprises filtering the set of environment maps based on a similarity of a metric representative of the content of the tracking map and the set of environment maps. Item 59. A cloud computing environment as described in item 59. (Item 61) The computer-executable instructions, when executed by at least one processor in the cloud computing environment, are further configured to receive a tracking map from a user device that requests an environment map; Filtering the set of environment maps further includes filtering the set of environment maps based on a degree of matching between the portion of the tracking map and the portion of the environment map of the set of environment maps. Item 59. A cloud computing environment as described in item 59. (Item 62) 60. The cloud computing environment of claim 59, wherein the wireless network parameters include a Basic Service Set Identifier (BSSID) of a network to which the user device is connected. (Item 63) Item 63. The cloud computing environment of item 62, wherein filtering the set of environmental maps based on similarity of wireless network parameters includes calculating a similarity between a plurality of BSSIDs stored in a user database associated with a user device requesting the environmental map and a BSSID stored in a map database associated with an environmental map of the set of environmental maps. (Item 64) 60. The cloud computing environment of claim 59, wherein the area identifier indicates a geographic location by longitude and latitude. (Item 65) 60. The cloud computing environment of claim 59, wherein determining an area identifier includes accessing an area identifier from the user database. (Item 66) 60. The cloud computing environment of claim 59, wherein determining the area identifier includes receiving an area identifier in messages received from the plurality of user devices. (Item 67) 60. The cloud computing environment of item 59, wherein the parameters of the wireless network comply with protocols including Wi-Fi and 5GNR. (Item 68) The computer-executable instructions, when executed by at least one processor in the cloud computing environment, are further configured to receive a tracking map from a user device; Filtering the set of environment maps further includes filtering the set of environment maps based on a degree of matching between the portion of the tracking map and the portion of the environment map of the set of environment maps. Item 59. A cloud computing environment as described in item 59. (Item 69) The computer-executable instructions further, when executed by at least one processor in the cloud computing environment, receiving a tracking map from a user device and determining an area identifier associated with the tracking map based on the user device providing the tracking map; selecting a second set of environment maps from the map database based at least in part on an area identifier associated with the tracking map; updating the map database based on the received tracking map, the updating including merging the received tracking map with one or more environment maps in the second set of environment maps; 60. The cloud computing environment of item 59, configured to: (Item 70) The computer-executable instructions, when executed by at least one processor in the cloud computing environment, are further configured to filter the second set of environment maps based on a degree of matching between a portion of the received tracking map and a portion of an environment map of the second set of environment maps; merging the tracking map with one or more environment maps in the second set of environment maps includes merging the tracking map with one or more environment maps in the filtered second set of environment maps. Item 69. The cloud computing environment according to item 69. PCF sharing (Item 71) An XR system, a real object detection device for detecting multiple surfaces of a real world object; an anchor identification system coupled to the real object detection device and configured to generate a map based on the real world objects; a PCF generation system that generates a first PCF based on the map and associates the first PCF with the map; a first and a second storage medium, the first and the second storage medium being on a first and a second XR device, respectively; at least a first and a second processor of the first and second XR devices, the first and second processors storing the first PCF in first and second storage media of the first and second XR devices, respectively; An XR system equipped with (Item 72) a keyframe generator, the keyframe generator being executable by the at least one processor to convert a plurality of camera images into a plurality of distinct keyframes; a sustained pose calculator, the sustained pose calculator being executable by the at least one processor to generate a sustained pose by averaging the plurality of keyframes; and a tracking map and sustained attitude converter, the tracking map and sustained attitude converter being executable by the at least one processor to convert a tracking map to the sustained attitude and to determine the sustained attitude at an origin for the tracking map; a sustained attitude and PCF converter executable by the at least one processor to convert the sustained attitude to the first PCF and to determine the first PCF for the sustained attitude; a PCF and image data converter executable by the at least one processor to convert the first PCF into image data; and a display device for displaying the image data for the first PCF to the user; 72. The XR system of item 71, further comprising: (Item 73) 72. The XR system of claim 71, wherein the detection device is a detection device of the first XR device connected to the first XR device processor. (Item 74) 73. The XR system of item 72, wherein the map is a first map on the first XR device, and a processor that generates the first map is a first XR device processor of the first XR device. (Item 75) Item 75. The XR system of item 74, wherein the processor that generates the first PCF is a first XR device processor of the first XR device. (Item 76) Item 76. The XR system of item 75, wherein the processor that associates the first PCF and the first map is a first XR device processor of the first XR device. (Item 77) an application executable by the first XR device processor; a first PCF tracker, the first PCF tracker being executable by the first XR device processor and including an on prompt for switching on the first PCF tracker from the application, the first PCF tracker generating the first PCF only when the first PCF tracker is switched on; 77. The XR system of item 76, further comprising: (Item 78) The first PCF tracker has an off prompt for switching off the first PCF tracker from the application, and the first PCF tracker terminates the first PCF generation when the first PCF tracker is switched off. 80. The XR system of item 77, further comprising: (Item 79) a map issuer, the map issuer being executable by the first XR device processor to transmit the first PCF to a server; a map storage routine, the map storage routine being executable by a server processor of the server to store the first PCF on a storage device of the server; transmitting, using a server processor of the server, the first PCF to the second XR device; a map download system, the map download system being executable by a second XR device processor of the second XR device to download the first PCF from the server; 77. The XR system of item 76, further comprising: (Item 80) an application executable by the second XR device processor; and a second PCF tracker, the second PCF tracker being executable by the second XR device processor and including an on prompt for switching on the second PCF tracker from the application, the second PCF tracker generating a second PCF only when the second PCF tracker is switched on; 80. The XR system of item 79, further comprising: (Item 81) The second PCF tracker has an off prompt for switching off the second PCF tracker from the application, and the second PCF tracker terminates the second PCF generation when the second PCF tracker is switched off. 81. The XR system of item 80, further comprising: (Item 82) 80. The XR system of item 79, further comprising a map issuer, the map issuer being executable by the second XR device processor to transmit the second PCF to the server. (Item 83) a persistent attitude obtainer, the persistent attitude obtainer being executable by the first XR device processor to download persistent attitudes from the server; a PCF validator executable by the first XR device processor to read a PCF from a first storage device of the first XR device based on the sustained attitude; a coordinate frame calculator executable by the first XR device processor to calculate a coordinate frame based on the PCF read from the first storage device; 77. The XR system of item 76, further comprising: (Item 84) A visual inspection method, comprising: detecting a plurality of surfaces of a real-world object with at least one sensing device; generating, with at least one processor, a map based on the real world objects; generating, with at least one processor, a first PCF based on the map; using the at least one processor to associate the first PCF with the map; storing, using at least first and second processors of the first and second XR devices, the first PCF in first and second storage media of the first and second XR devices, respectively; A method comprising: (Item 85) converting, with the at least one processor, a plurality of camera images into a plurality of distinct key frames; generating, with the at least one processor, a sustained pose by averaging the plurality of keyframes; and using the at least one processor to transform a tracking map to the sustained pose and determine the sustained pose at an origin for the tracking map; converting, by the at least one processor, the sustained attitude to the first PCF and determining the first PCF for the sustained attitude; converting the first PCF into image data with the at least one processor; displaying the image data for the first PCF to the user using a display device; Item 85. The viewing method according to item 84, further comprising: (Item 86) 85. The viewing method of claim 84, wherein the detection device is a detection device of the first XR device connected to the first XR device processor. (Item 87) Item 86. The viewing method of item 85, wherein the map is a first map on the first XR device, and the processor that generates the first map is a first XR device processor of the first XR device. (Item 88) Item 88. The viewing method of item 87, wherein the processor that generates the first PCF is a first XR device processor of the first XR device. (Item 89) Item 89. The viewing method of item 88, wherein the processor that associates the first PCF and the first map is a first XR device processor of the first XR device. (Item 90) executing an application using the first XR device processor; switching on a first PCF tracker with the first XR device processor upon prompting from the application, the first PCF tracker generating the first PCF only when the first PCF tracker is switched on; 90. The viewing method according to item 89, further comprising: (Item 91) and switching off the first PCF tracker with the first XR device processor upon an off prompt from the application, the first PCF tracker terminating first PCF generation when the first PCF tracker is switched off. Item 91. The viewing method according to Item 90, further comprising: (Item 92) transmitting, using the first XR device processor, the first PCF to a server; storing, using a server processor of the server, the first PCF on a storage device of the server; transmitting, using a server processor of the server, the first PCF to the second XR device; receiving the first PCF from the server using a second XR device processor of the second XR device; 90. The viewing method according to item 89, further comprising: (Item 93) executing an application using the second XR device processor; switching on a second PCF tracker with the second XR device processor upon prompting from the application, the second PCF tracker generating a second PCF only if the second PCF tracker is switched on; Item 93. The viewing method according to Item 92, further comprising: (Item 94) and switching off the second PCF tracker with the first XR device processor upon an off prompt from the application, the second PCF tracker terminating second PCF generation when the second PCF tracker is switched off. Item 94. The viewing method according to Item 93, further comprising: (Item 95) uploading the second PCF to the server using the second XR device processor; Item 93. The viewing method according to Item 92, further comprising: (Item 96) determining a sustained pose from the server using the first XR device processor; reading, with the first XR device processor, a PCF from a first storage device of the first XR device based on the sustained attitude; calculating, with the first XR device processor, a coordinate frame based on the PCF read from the first storage device; 90. The viewing method according to item 89, further comprising: PCF Download (Item 97) An XR system, A first XR device, a first XR device processor; a first XR device storage device coupled to the first XR device processor; A set of instructions on the first XR device processor, comprising: a download system executable by the first XR device processor to download a sustained pose from a server; and a PCF reader, the PCF reader executable by the first XR device processor to read a PCF from a first storage device of the first XR device based on the sustained attitude; a coordinate frame calculator executable by the first XR device processor to calculate a coordinate frame based on the PCF read from the first storage device; A set of instructions, including The first XR device, including An XR system equipped with (Item 98) A visual inspection method, comprising: downloading, using a first XR device processor of the first XR device, a persistent attitude from a server; reading, with the first XR device processor, a PCF from a first storage device of the first XR device based on the sustained attitude; calculating, with the first XR device processor, a coordinate frame based on the PCF read from the first storage device; A method comprising: PCF Server (Item 99) A visual inspection method, comprising: A server, A server processor; a server storage device connected to the server processor; a map storage routine, the map storage routine executable by a server processor of the server to store the first PCF associated with a map on a server storage device of the server; a map transmitter, the map transmitter being executable by a server processor to transmit, by the server processor, the map and the first PCF to a first XR device; Including, Server A method comprising: (Item 100) A visual inspection method, comprising: storing, using a server processor of the server, the first PCF associated with a map on a server storage device of the server; transmitting, using a server processor of the server, the map and the first PCF to a first XR device; A method comprising: Head Pose Recovery and Reset (Item 101) A visual inspection method, comprising: entering head pose tracking by a processor of the XR device, capturing a surface of the environment with a capture device on a head-mounted frame affixed to the user's head and determining an orientation of the head-mounted frame relative to the surface; determining, by the processor, whether head pose has been lost due to an inability to determine an orientation of the head mounted frame relative to the surface; if head pose is lost, entering, by the processor, a pose recovery mode to establish the head pose by determining an orientation of the head mounted frame relative to the surface; A method comprising: (Item 102) Item 102. The method of claim 101, wherein if the head pose is not lost, the processor enters head pose tracking. (Item 103) Posture restoration is displaying, by the processor, a message to the user with suggestions for improving capture of the surface. Item 102. The viewing method according to item 101. (Item 104) Item 104. The viewing method according to item 103, wherein the suggestion is at least one of increasing light and refining texture. (Item 105) determining, by said processor, whether the restoration has failed; if the restoration fails, initiating a new session including establishing a head pose by said processor; Item 102. The viewing method according to Item 101, further comprising: (Item 106) Item 106. The viewing method of item 105, further comprising displaying, by the processor, a message to the user that a new session will be started. (Item 107) Item 102. The viewing method of item 101, further comprising, if head pose is not lost, entering head pose tracking, by the processor. (Item 108) 1. An electronic system portable by a user, comprising: one or more sensors configured to capture images of one or more physical objects in a scene, the images being in a first coordinate frame; an application configured to execute computer-executable instructions and render virtual content within the scene, the application displaying the virtual content in a second coordinate frame different from the first coordinate frame such that the display of the virtual content is independent of eye rotations within the user's head and / or deformations of the one or more sensors; An electronic system comprising: (Item 109) the first coordinate frame is a world coordinate frame having a world origin; the world origin is a first pose of the electronic system when the electronic system is powered on to capture the image; Item 109. The electronic system of item 108. (Item 110) the second coordinate frame is a camera coordinate frame having a camera origin; the camera origin is determined based, at least in part, on dimensions of the electronic system and one or more orientations of one or more sensors of the electronic system when capturing the image; Item 109. The electronic system of item 108. (Item 111) At least one processor configured to execute computer-executable instructions to provide the virtual content to the application, the computer-executable instructions comprising: determining a local coordinate frame for the virtual content based, at least in part, on one or more physical objects in the scene; and transforming image data for the virtual content in the local coordinate frame into image data for the virtual content in the second coordinate frame; at least one processor having instructions for performing Item 109. The electronic system of item 108, comprising: (Item 112) Item 109. The electronic system of item 108, wherein the local coordinate frame is determined, at least in part, based on one or more nodes on an outer surface of a prism or bounding box that encloses the virtual content. (Item 113) Transforming image data for virtual content in the local coordinate frame into image data for virtual content in the second coordinate frame includes: transforming image data for the virtual content in the local coordinate frame into image data for the virtual content in the first coordinate frame; Transforming image data for the virtual content in the first coordinate frame into image data for the virtual content in the second coordinate frame; Item 112. The electronic system of item 111, comprising: (Item 114) Transforming image data for virtual content in the first coordinate frame into image data for virtual content in the second coordinate frame includes: Transforming image data for the virtual content in the first coordinate frame into image data for the virtual content in a head coordinate frame; transforming image data for the virtual content in the head coordinate frame into image data for the virtual content in the second coordinate frame; Including, the head coordinate frame having a head origin determined at least in part based on one or more poses of one or more sensors of the electronic system when capturing the image; Item 112. The electronic system of item 111. (Item 115) The computer executable instructions include: instructions for determining a persistent coordinate frame (PCF) based, at least in part, on images captured centered on the one or more physical objects in the scene, such that the PCF is distinct from, but related to, the world coordinate frame; Item 112. The electronic system of item 111, comprising: (Item 116) 1. A method of operating a computing system to render a virtual object within a scene comprising one or more physical objects, the method comprising: capturing a plurality of images centered on the scene from one or more sensors of a first device worn by a user; calculating one or more sustained poses based at least in part on the plurality of images; and generating a persistent coordinate frame based, at least in part, on the calculated one or more persistent poses such that information of the plurality of images may be accessed at different times by one or more applications running on the first device and / or the second device via the persistent coordinate frame; A method comprising: (Item 117) Calculating the one or more sustained poses based at least in part on the plurality of images includes: extracting one or more features from each of the plurality of images; generating a descriptor for each of said one or more features; generating a key frame for each of the plurality of images based at least in part on the descriptors; generating the one or more sustained poses based at least in part on the one or more key frames; and Item 117. The method of item 116, comprising: (Item 118) Generating the persistent coordinate frame based at least in part on the calculated one or more persistent poses includes: generating said persistent coordinate frame when said first device has traveled a predetermined distance from where said plurality of images were captured. Item 117. The method of item 116, comprising: (Item 119) Item 119. The method according to item 118, wherein the predetermined distance is between 2 and 20 meters such that both the consumption of computing resources of the device and the placement error of the virtual object are controlled to generate the one or more sustained postures. (Item 120) generating an initial sustained attitude when the first device is powered on; generating a first sustained pose at the current location of the first device when the first device reaches a circumference of a circle with the initial sustained pose as the center of the circle and a radius equal to a threshold distance; Item 118. The method according to item 117, comprising: (Item 121) the circle is a first circle, The method further includes generating a second sustained pose at the current location of the first device when the device reaches a second circular circumference with the first sustained pose as a center of the circle and a radius equal to twice the threshold distance. The method according to item 120. (Item 122) Item 121. The method of item 120, wherein the first persistent pose is not generated when the first device finds an existing persistent pose within the threshold distance from the current position of the first device. (Item 123) Item 121. The method of item 120, wherein the first device associates with the first sustained pose one or more of the plurality of key frames that are within a predetermined distance to the first sustained pose. (Item 124) Item 121. The method of item 120, wherein the first persistent attitude is not generated when an application running on the first device does not request a persistent attitude. (Item 125) 1. An electronic system portable by a user, comprising: one or more sensors configured to capture images of one or more physical objects in a scene; an application configured to execute computer-executable instructions to render virtual content within the scene; at least one processor configured to execute computer-executable instructions to provide image data for the virtual content to the application, the computer-executable instructions comprising: and instructions for generating a persistent coordinate frame based, at least in part, on the captured image. At least one processor and An electronic system comprising: [Brief description of the drawings]

[0023] The accompanying drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component that is illustrated in various figures is represented by a like numeral. For purposes of clarity, not every component is labeled in every figure.

[0024] [Figure 1] FIG. 1 is a schematic diagram illustrating a cross-reality (XR) system, according to some embodiments.

[0025] [Diagram 2] FIG. 2 is a block diagram of a first XR device of the XR system of FIG. 1 according to some embodiments.

[0026] [Diagram 3] FIG. 3 is a schematic diagram illustrating the transformation of an origin coordinate frame into a destination coordinate frame, e.g., to correctly render local XR content, according to some embodiments.

[0027] [Figure 4] FIG. 4 is a top plan view illustrating a pupil-based coordinate frame, according to some embodiments.

[0028] [Diagram 5] FIG. 5 is a top plan view illustrating a camera coordinate frame including all pupil positions according to some embodiments.

[0029] [Figure 6] FIG. 6 is a block diagram of the XR system of FIG. 1 including an additional second XR device, according to some embodiments.

[0030] [Figure 7] FIG. 7 is a schematic diagram of the display system of FIG. 2 according to some embodiments.

[0031] [Figure 8]FIG. 8 is a two-dimensional representation of a three-dimensional first local tracking map (Map 1) that may be generated by the first XR device of FIG. 2 according to some embodiments.

[0032] [Figure 9] FIG. 9 is a block diagram illustrating the steps of uploading map 1 from a first XR device to the server of FIG. 6 according to some embodiments.

[0033] [Figure 10] FIG. 10 is a schematic diagram illustrating the XR system of FIG. 6 in accordance with some embodiments, showing that after a first user ends a first session, a second user starts a second session using a second XR device of the XR system.

[0034] [Figure 11A] FIG. 11A is a block diagram illustrating a new session for the second XR device of FIG. 10 according to some embodiments.

[0035] [Figure 11B] FIG. 11B is a block diagram illustrating the creation of a tracking map for the second XR device of FIG. 10 according to some embodiments.

[0036] [Figure 11C] FIG. 11C is a block diagram illustrating downloading a reference map from a server to the second XR device of FIG. 10 according to some embodiments.

[0037] [Figure 11D] FIG. 11D is a block diagram illustrating the creation of a persistent coordinate frame (PCF) and XR content bound to the PCF, according to some embodiments.

[0038] [Figure 12]FIG. 12 is a schematic diagram illustrating a localization attempt to locate a second tracking map (Map 2) relative to a reference map, which may be generated by the second XR device of FIG. 10 in accordance with some embodiments.

[0039] [Figure 13] FIG. 13 is a schematic diagram illustrating a location attempt to locate the second tracking map (Map 2) of FIG. 12 relative to a reference map, which may be further deployed with XR content associated with an anchor in Map 2, in accordance with some embodiments.

[0040] [Figure 14A] 14A-14B are schematic diagrams illustrating successful localization of map 2 of FIG. 13 relative to a reference map, according to some embodiments. [Figure 14B] 14A-14B are schematic diagrams illustrating successful localization of map 2 of FIG. 13 relative to a reference map, according to some embodiments.

[0041] [Figure 15] FIG. 15 is a schematic diagram illustrating a reference map generated by including an anchor or anchors from the reference map of FIG. 14A into map 2 of FIG. 13, according to some embodiments.

[0042] [Figure 16] FIG. 16 is a schematic diagram illustrating the reference map of FIG. 15 with a further extension of map 2 on a second XR device, according to some embodiments.

[0043] [Figure 17] FIG. 17 is a block diagram illustrating the steps of uploading map 2 from a second XR device to a server according to some embodiments.

[0044] [Figure 18]FIG. 18 is a block diagram illustrating merging Map2 with a reference map according to some embodiments.

[0045] [Figure 19] FIG. 19 is a block diagram illustrating the transmission of a new reference map from a server to a first and second XR device according to some embodiments.

[0046] [Figure 20] FIG. 20 is a block diagram illustrating a two-dimensional representation of map 2 and the head coordinate frame of a second XR device referenced to map 2, according to some embodiments.

[0047] [Figure 21] FIG. 21 is a block diagram illustrating adjustments of the head coordinate frame that can occur in six degrees of freedom in two dimensions according to some embodiments.

[0048] [Figure 22] FIG. 22 is a block diagram illustrating a reference map on a second XR device where sounds are localized relative to anchors of map 2, according to some embodiments.

[0049] [Figure 23] 23 and 24 are perspective and block diagrams illustrating use of the XR system when a first user has finished a first session and the first user has started a second session using the XR system, according to some embodiments. [Figure 24] 23 and 24 are perspective and block diagrams illustrating use of the XR system when a first user has finished a first session and the first user has started a second session using the XR system, according to some embodiments.

[0050] [Diagram 25]25 and 26 are perspective and block diagrams illustrating the use of an XR system when three users use the XR system simultaneously within the same session, according to some embodiments. [Figure 26] 25 and 26 are perspective and block diagrams illustrating the use of an XR system when three users use the XR system simultaneously within the same session, according to some embodiments.

[0051] [Figure 27] FIG. 27 is a block diagram illustrating a two-dimensional diagram representing latitude and longitude of a reference map stored on a server, according to some embodiments.

[0052] [Figure 28] FIG. 28 is a schematic diagram illustrating a series of filters for selecting criteria maps for transmission by a server to an XR device, according to some embodiments.

[0053] [Figure 29A] 29A-29C are block diagrams illustrating a system for generating global feature sequences of images and / or maps, according to some embodiments. [Figure 29B] 29A-29C are block diagrams illustrating a system for generating global feature sequences of images and / or maps, according to some embodiments. [Figure 29C] 29A-29C are block diagrams illustrating a system for generating global feature sequences of images and / or maps, according to some embodiments.

[0054] [Figure 29D] FIG. 29D is a block diagram illustrating a method of training a neural network, according to some embodiments.

[0055] [Diagram 30]FIG. 30 is a sketch illustrating an example of a simplified augmented reality (AR) scene, according to some embodiments.

[0056] [Diagram 31] FIG. 31 is a sketch of an example simplified AR scene illustrating an example world reconstruction use case including visual occlusion, physics-based interaction, and environmental inference, according to some embodiments.

[0057] [Diagram 32] FIG. 32 is a schematic diagram illustrating data flow within an AR system configured to provide an experience of AR content that interacts with the physical world, according to some embodiments.

[0058] [Diagram 33] FIG. 33 is a schematic diagram illustrating an example of an AR display system, according to some embodiments.

[0059] [Figure 34A] FIG. 34A is a schematic diagram illustrating a user wearing an AR display system that renders AR content as the user moves through a physical world environment, according to some embodiments.

[0060] [Figure 34B] FIG. 34B is a schematic diagram illustrating a viewing optics assembly and associated components, according to some embodiments.

[0061] [Figure 35A] FIG. 35A is a schematic diagram illustrating an AR system using a world reconstruction system, according to some embodiments.

[0062] [Figure 35B] FIG. 35B is a schematic diagram illustrating a passable world, according to some embodiments.

[0063] [Diagram 36] FIG. 36 is an example environment map, according to some embodiments.

[0064] [Figure 37] FIG. 37 is a schematic diagram illustrating an AR system configured to rank and merge multiple environment maps, according to some embodiments.

[0065] [Figure 38] FIG. 38 is a flowchart illustrating a method for selecting a plurality of ranked environment maps according to some embodiments.

[0066] [Figure 39] FIG. 39 is a schematic diagram illustrating an example map ranking portion of the AR system of FIG. 37, according to some embodiments.

[0067] [Figure 40A] FIG. 40A is a schematic diagram illustrating an example of area attributes of a tracking map (TM) and an environment map in a database, according to some embodiments.

[0068] [Figure 40B] FIG. 40B is a schematic diagram illustrating an example of determining the geographic location of a tracking map (TM) for the geographic location filtering of FIG. 38, according to some embodiments.

[0069] [Diagram 41] FIG. 41 is a schematic diagram illustrating an example of the geographic location filtering of FIG. 38, according to some embodiments.

[0070] [Diagram 42] FIG. 42 is a schematic diagram illustrating an example of the Wi-Fi BSSID filtering of FIG. 38, according to some embodiments.

[0071] [Diagram 43] FIG. 43 is a schematic diagram illustrating an example of localization of FIG. 38, according to some embodiments.

[0072] [Diagram 44] FIG. 44 is a schematic diagram illustrating a room and key frames established for various areas within the room, according to some embodiments.

[0073] [Diagram 45] FIG. 45 is a schematic diagram illustrating establishment of sustained poses based on key frames, according to some embodiments.

[0074] [Figure 46] FIG. 46 is a schematic diagram illustrating establishment of a persistent coordinate frame (PCF) based on a persistent attitude, according to some embodiments.

[0075] [Figure 47] FIG. 47 is a flowchart illustrating a method of establishing and using a PCF, according to some embodiments.

[0076] [Figure 48A] 48A-48I are schematic diagrams illustrating an example of establishing and using the PCF of FIG. 47. [Figure 48B] 48A-48I are schematic diagrams illustrating an example of establishing and using the PCF of FIG. 47. [Figure 48C] 48A-48I are schematic diagrams illustrating an example of establishing and using the PCF of FIG. 47. [Figure 48D] 48A-48I are schematic diagrams illustrating an example of establishing and using the PCF of FIG. 47. [Figure 48E] 48A-48I are schematic diagrams illustrating an example of establishing and using the PCF of FIG. 47. [Fig.48F] 48A-48I are schematic diagrams illustrating an example of establishing and using the PCF of FIG. 47. [Figure 48G] 48A-48I are schematic diagrams illustrating an example of establishing and using the PCF of FIG. 47. [Fig. 48H] 48A-48I are schematic diagrams illustrating an example of establishing and using the PCF of FIG. 47. [Fig.48I] 48A-48I are schematic diagrams illustrating an example of establishing and using the PCF of FIG. 47.

[0077] [Figure 49] FIG. 49 is a flowchart illustrating a method for restoring and resetting head pose, according to some embodiments.

[0078] [Figure 50] FIG. 50 is a block diagram of a machine, in the form of a computer, that may find application within the system of the present invention, according to some embodiments.

[0079] [Figure 51] 51-53 are block diagrams illustrating examples of determining which map to use based, at least in part, on deep keyframe (DKF) analysis, according to some embodiments. [Figure 52] 51-53 are block diagrams illustrating examples of determining which map to use based, at least in part, on deep keyframe (DKF) analysis, according to some embodiments. [Figure 53] 51-53 are block diagrams illustrating examples of determining which map to use based, at least in part, on deep keyframe (DKF) analysis, according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0080] Described herein are methods and apparatus for providing an X-reality (XR or cross-reality) scene. To provide a realistic XR experience to multiple users, the XR system must understand the user's physical surroundings in order to properly correlate the location of virtual objects in relation to real objects. The XR system may build an environmental map of the scene, which may be created from image and depth information collected with sensors that are part of the XR device worn by the user of the XR system.

[0081] The inventors have realized and appreciated techniques for operating an XR system and providing an XR scene for a more immersive user experience by allowing a persistent environment map to be created, stored, and retrieved by any of multiple users of the XR system. Such a persistent environment map may enable one or more features that enhance the XR experience. For example, a wearable device worn by a user may retrieve a previously created and stored appropriate environment map from a persistent storage device, such as a cloud storage device, after being turned on. The previously stored map may be based on data about the environment collected with sensors on the user's wearable device at a previous time or times the user was present. Retrieving the persistent map may enable use of the wearable device without scanning the physical world with sensors on the wearable device. Alternatively or additionally, a user may similarly retrieve an appropriate environment map in response to entering a new area of ​​the physical world.

[0082] In a multi-user XR system, a persistent environment map accessed by one user may have been created and stored by another user, and / or may have been constructed by aggregating data about the physical world collected by sensors on wearable devices worn by multiple users that previously existed within at least a portion of the physical world represented by the persistent environment map.

[0083] Furthermore, sharing data about the physical world between multiple users may enable a shared user experience of virtual content. Two XR devices with access to the same persistent environment map may both be localized, for example, relative to the persistent environment map. Once localized, the user devices may render virtual content, having a location defined by a fiducial, into the persistent environment map by translating the location to a frame or fiducial maintained by the user device. The user device may use this local frame of reference to control the user device's display to render the virtual content within the defined location.

[0084] To support these and other possible functions, an XR system may include components that develop, maintain, and use one or more persistent environmental maps based on data about the physical world collected using sensors on the user device. These components may be distributed across the XR system, with some running, for example, on the head-mounted portion of the user device. Other components may run on a computer associated with the user that is coupled to the head-mounted portion via a local or personal area network. Still others may run at remote locations, such as one or more servers accessible via a wide area network.

[0085] These components may include, for example, components that may identify, from information about the physical world collected by one or more user devices, information that is of sufficient quality to be stored as a persistent environment map. An example of such a component, described in more detail below, is the PCF integration unit 1300 in FIG. 6.

[0086] As another example, these components may include components that may help determine an appropriate persistent environment map that may be retrieved and used by a user device. An example of such a component, described in more detail below, is a map ranking component. Such a component may, for example, receive input from a user device and identify one or more persistent environment maps that are likely to represent an area of ​​the physical world in which the device is operating. The map ranking component may, for example, help select a persistent environment map to be used by the local device when rendering virtual content, gathering data about the environment, or performing other actions. The map ranking component may alternatively or additionally help identify a persistent environment map to be updated as additional information about the physical world is gathered by one or more user devices.

[0087] Another component may be an application programming interface (API). The platform that builds and maintains the persistent environmental map may provide one or more APIs to enable separately written applications to obtain information about the physical world.

[0088] Still other components may be transforms that convert information captured or described relative to one frame of reference to be translated into another frame of reference. For example, a sensor may be attached to a head mounted display such that data read from the sensor indicates the location of an object in the physical world relative to the wearer's head pose. One or more transforms may be applied to relate that location information to a coordinate frame associated with the persistent environment map. Similarly, data that indicates where a virtual object should be rendered when represented in the coordinate frame of the persistent environment map may undergo one or more transforms to be in the frame of reference of the display on the user's head. There may be multiple such transforms, as described in more detail below. These transforms may be partitioned across components of the XR system such that they may be efficiently updated and / or applied in a distributed system.

[0089] In some embodiments, map information may be represented so that it may be easily shared among users and distributed components, including applications. Information about the physical world may be represented, for example, as a coordinate frame with one or more anchors that represent recognized features within the physical world. Such maps may be sparse and provide less than all of the available information about the physical world so that they may be efficiently processed and transferred.

[0090] In some embodiments, a persistent environment map may be constructed from information collected by multiple user devices. The XR devices may construct separate tracking maps (TMs) using information collected by sensors in the XR devices worn by users of the system at various locations and times. Each TM may contain points that may be associated with real objects in the environment. Potentially, in addition to providing input and creating and maintaining the persistent environment map, the TMs may be used to track the user's movements within the scene, allowing the XR system to estimate the head pose of individual users based on the TMs.

[0091] This codependency between creating an environment map and estimating head pose constitutes a significant challenge. Substantial processing may be required to simultaneously create an environment map and estimate head pose. Because latency would make the XR experience less immersive for the user, processing must be performed quickly as objects move in the scene (e.g., moving a cup on a table) and as the user moves in the scene. On the other hand, the weight of the XR device should be light for the user to wear comfortably, so the XR device may provide limited computational resources. Thus, the lack of computational resources cannot be compensated for with more sensors. Furthermore, more sensors and more computational resources lead to heat, which may cause deformation of the XR device.

[0092] The inventors have realized and appreciated techniques for operating XR systems and providing XR scenes for a more immersive user experience, such as head pose estimation at a frequency of 1 kHz, low usage of computational resources such as four video graphic array (VGA) cameras operating at 30 Hz, one inertial measurement unit (IMU) operating at 1 kHz, the computational power of a single advanced RISC machine (ARM) core, less than 1 GB of memory, and less than 100 Mbps of network bandwidth. These techniques involve generating and maintaining environment maps, reducing the processing required to estimate head pose, and providing and consuming data with low computational overhead.

[0093] These techniques may include hybrid tracking, such that the XR system may exploit both (1) patch-based tracking of distinguishable points between successive images of the environment (frame / frame tracking) and (2) matching of a descriptor-based map of points of interest in a current image with the corresponding points of interest's known real-world locations (map / frame tracking). In frame / frame tracking, the XR system may track a particular point of interest (salient point), such as a corner, between captured images of the real-world environment. For example, the display system may identify the location of a visual point of interest in a current image that was contained (e.g., located) in a previous image. This identification may be accomplished, for example, using a photometric error minimization process. In map / frame tracking, the XR system may access map information, which indicates the real-world locations (e.g., three-dimensional coordinates) of the points of interest, and match points of interest contained in the current image to points of interest indicated in the map information. Information about the points of interest is stored as a descriptor in the map database. The XR system may then calculate its pose based on the matched visual features. US Patent Application No. 16 / 221,065 describes hybrid tracking and is incorporated herein by reference in its entirety.

[0094] These techniques may include reducing the amount of data processed when constructing the map, such as by constructing a sparse map using a set of mapped points and keyframes, and / or by dividing the map into blocks and enabling updates by blocks. The mapped points may be associated with a three-dimensional (3D) world reconstruction of real objects in the environment. The keyframes may include information selected from camera-captured data. U.S. Patent Application No. 16 / 520,582 describes determining and / or evaluating a localization map and is incorporated herein by reference in its entirety.

[0095] These techniques may include creating dynamic maps based on one or more coordinate systems in real space across one or more sessions, generating persistent coordinate frames (PCFs) over the sparse maps that can be exposed to XR applications, for example via an application programming interface (API), and in some embodiments using deep keyframe analysis to rank and map multiple maps created by one or more XR devices, and restoring and resetting head pose.

[0096] The techniques described herein may be used together or separately with many types of devices and for many types of scenes, including wearable or portable devices with limited computing resources, that provide augmented reality scenes.

[0097] Coordinate Frame

[0098] Described herein are methods and apparatus for providing virtual content using an XR system independent of the location of the eye that views the virtual content. Conventionally, virtual content is re-rendered in response to any movement of the display system. For example, when a user wearing a display system views a virtual representation of a three-dimensional (3D) object on the display and walks around the area in which the 3D object appears, the 3D object should be re-rendered for each viewpoint so that the user has the perception that he or she is walking around the object that occupies real space. However, re-rendering consumes significant computational resources of the system and introduces artifacts due to latency.

[0099] The inventors have recognized and appreciated that head pose (e.g., location and orientation of a user wearing an XR system) may be used to render virtual content independent of eye rotation in the user's head. In some embodiments, a dynamic map of a scene may be generated based on multiple coordinate frames in real space across one or more sessions such that virtual content interacting with the dynamic map may be robustly rendered independent of eye rotation in the user's head and / or independent of sensor deformations caused by, for example, heat generated during fast computationally intensive operations. In some embodiments, the configuration of multiple coordinate frames may enable a first XR device worn by a first user and a second XR device worn by a second user to recognize a common location in a scene. In some embodiments, the configuration of multiple coordinate frames may enable a first and second user wearing the first and second XR devices to view virtual content in the same location in a scene.

[0100] In some embodiments, the tracking map may be constructed in a world coordinate frame, which may have a world origin. The world origin may be the first pose of the XR device when the XR device is powered on. The world origin may be aligned to gravity so that the developer of the XR application can obtain gravity alignment without extra work. Different tracking maps may be constructed in different world coordinate frames because the tracking maps may be captured by the same XR device in different sessions and / or by different XR devices worn by different users. In some embodiments, a session of the XR device may start when the device is powered on and continue until it is powered off. In some embodiments, the XR device may have a head coordinate frame, which may have a head origin. The head origin may be the current pose of the XR device when the image is taken. The difference between the head pose of the world coordinate frame and the head coordinate frame may be used to estimate the tracking route.

[0101] In some embodiments, the XR device may have a camera coordinate frame, which may have a camera origin. The camera origin may be the current pose of one or more sensors of the XR device. The inventors have recognized and appreciated that configuration of the camera coordinate frame enables robust display of virtual content independent of eye rotations in the user's head. This configuration also enables robust display of virtual content independent of sensor deformations due to, for example, heat generated during movement.

[0102] In some embodiments, the XR device may have a head unit with a head-mountable frame that a user may fasten to his / her head and that may include two waveguides, one in front of each of the user's eyes. The waveguides may be transparent so that ambient light from real-world objects may transmit through the waveguides and the user may see the real-world objects. Each waveguide may transmit projected light from a projector to a respective eye of the user. The projected light may form an image on the retina of the eye. The retina of the eye thus receives the ambient light and the projected light. The user may simultaneously see the real-world object and one or more virtual objects created by the projected light. In some embodiments, the XR device may have sensors that detect real-world objects around the user. These sensors may be, for example, cameras that capture images that may be processed to identify the location of the real-world objects.

[0103] In some embodiments, the XR system may assign a local coordinate frame to the virtual content, as opposed to tying the virtual content in a world coordinate frame. Such a configuration allows the virtual content to be tied to a more persistent frame location, such as a persistent coordinate frame (PCF). In some embodiments, when a head pose session ends, the local coordinate frame may be transformed to a world coordinate frame. In some embodiments, the world coordinate frame may be the origin of the map of the previous session, located relative to the current local coordinate frame. When the location of an object changes, the XR device may detect the change in the environment map and determine the movement of the head unit worn by the user relative to the real-world object.

[0104] 1 illustrates an XR system 10, according to some embodiments. The XR system may include a first XR device 12.1 worn by a first user 14.1, a real object in the form of a table 16, a network 18, and a server 20.

[0105] In the illustrated example, the first XR device 12.1 includes a head unit 22, a belt pack 24, and a cable connection 26. The first user 14.1 fastens the head unit 22 to his head and the belt pack 24, remote from the head unit 22, on his waist. The cable connection 26 connects the head unit 22 to the belt pack 24. The head unit 22 includes technology used to display a virtual object or objects to the first user 14.1 while still allowing the first user 14.1 to see a real object, such as a table 16. The belt pack 24 primarily includes the processing and communication capabilities of the first XR device 12.1. In some embodiments, the processing and communication capabilities may reside in whole or in part in the head unit 22, such that the belt pack 24 may be removed or located in another device, such as a backpack.

[0106] In the illustrated embodiment, the belt pack 24 is connected to the network 18 via a wireless connection. A server 20 is connected to the network 18 and holds data representing local content. The belt pack 24 downloads data representing the local content from the server 20 via the network 18. The belt pack 24 provides the data to the head unit 22 via a cable connection 26. The head unit 22 may include a display having a light source, e.g., a laser light source or a light emitting diode (LED) light source, and a waveguide for guiding the light.

[0107] In some embodiments, the first user 14.1 may carry a head unit 22 on his head and a belt pack 24 on his waist. The belt pack 24 may download image data from the server 20 via the network 18. The first user 14.1 may see the table 16 through the display of the head unit 22. A projector, forming part of the head unit 22, may receive the image data from the belt pack 24 and generate light based on the image data. The light may travel through one or more of the waveguides forming part of the display of the head unit 22. The light may then exit the waveguide and propagate onto the retina of the first user's 14.1 eye. The projector may generate light in a pattern that is replicated on the retina of the first user's 14.1 eye. The light that strikes the retina of the first user's 14.1 eye may have a selected depth of field such that the first user 14.1 perceives an image at a preselected depth behind the waveguide. In addition, both eyes of the first user 14.1 may receive slightly different images such that the brain of the first user 14.1 perceives a three-dimensional image or multiple images at a selected distance from the head unit 22. In the illustrated embodiment, the first user 14.1 perceives the virtual content 28 above the table 16. The virtual content 28 and its location and proportion of distance from the first user 14.1 are determined by data representing the virtual content 28 and the various coordinate frames used to display the virtual content 28 to the first user 14.1.

[0108] In the illustrated embodiment, the virtual content 28 is invisible from the perspective of the drawing and is visible to the first user 14.1 through the use of the first XR device 12.1. The virtual content 28 may initially reside as a data structure in the visual data and algorithms in the beltpack 24. The data structure may then be revealed as light when the projector of the head unit 22 generates light based on the data structure. It should be understood that although the virtual content 28 does not exist in a three-dimensional space in front of the first user 14.1, the virtual content 28 is still represented in FIG. 1 in a three-dimensional space for illustrative purposes. Visualization of computer data in a three-dimensional space may be used in this description to illustrate how the data structures that facilitate the rendering as perceived by one or more users are interrelated among the data structures in the beltpack 24.

[0109] 2 illustrates components of a first XR device 12.1 according to some embodiments. The first XR device 12.1 may include a head unit 22 and various components that form part of the visual data and algorithms, including, for example, a rendering engine 30, various coordinate systems 32, various origin and destination coordinate frames 34, and various origin / destination coordinate frame converters 36.

[0110] The head unit 22 may include a head-mountable frame 40 , a display system 42 , a real object detection camera 44 , a mobile tracking camera 46 , and an inertial measurement unit 48 .

[0111] The head-mountable frame 40 may have a shape that is attachable to the head of the first user 14.1 in Figure 1. The display system 42, the real object detection camera 44, the mobile tracking camera 46, and the inertial measurement unit 48 are mounted to the head-mountable frame 40 and may therefore move with the head-mountable frame 40.

[0112] The coordinate system 32 may include a local data system 52 , a world frame system 54 , a head frame system 56 , and a camera frame system 58 .

[0113] Local data system 52 may include a data channel 62, a local frame determination routine 64, and local frame storage instructions 66. Data channel 62 may be an internal software routine, a hardware component such as an external cable or radio frequency receiver, or a hybrid component such as an open port. Data channel 62 may be configured to receive image data 68 representing virtual content.

[0114] A local frame determination routine 64 may be connected to the data channel 62. The local frame determination routine 64 may be configured to determine a local coordinate frame 70. In some embodiments, the local frame determination routine may determine the local coordinate frame based on a real-world object or real-world location. In some embodiments, the local coordinate frame may be based on an origin at the time of head pose session initialization. In some embodiments, the local coordinate frame may be located where the device was booted up, or a new location if the head pose was lost during the boot session. In some embodiments, the local coordinate frame may be the origin at the start of the head pose session. In some embodiments, the virtual content may also have a content coordinate frame associated with it. In some embodiments, the content coordinate frame may be based on the top edge relative to the bottom edge of the browser window, the head or feet of a character, a node on the outer surface of a prism or bounding box that encloses the virtual content, or any other suitable location for placing a coordinate frame that defines the facing direction of the virtual content and where the virtual content should be placed (e.g., a node such as a placement node or anchor node) with which the virtual content should be placed, etc.

[0115] The local frame storage instruction 66 may be connected to the local frame determination routine 64. One skilled in the art will understand that software modules and routines are "connected" to one another through subroutines, calls, etc. The local frame storage instruction 66 may store the local coordinate frame 70 as the local coordinate frame 72 in the origin and destination coordinate frame 34. In some embodiments, the origin and destination coordinate frame 34 may be one or more coordinate frames that may be manipulated or transformed in order for the virtual content to persist between sessions. In some embodiments, a session may be a period of time between boot-up and shutdown of an XR device. The two sessions may be two startup and shutdown periods for a single XR device, or startup and shutdown for two different XR devices.

[0116] In some embodiments, the origin and destination coordinate frames 34 may be coordinate frames involved in one or more transformations required for the first user's XR device and the second user's XR device to perceive a common location. In some embodiments, the destination coordinate frame may be the output of a series of calculations and transformations applied to the target coordinate frames in order for the first and second users to view virtual content at the same location.

[0117] Rendering engine 30 may be connected to data channel 62. Rendering engine 30 may receive image data 68 from data channel 62 such that rendering engine 30 may render virtual content based, at least in part, on image data 68.

[0118] A display system 42 may be connected to the rendering engine 30. The display system 42 may include components that convert image data 68 into visible light. The visible light may form two patterns, one for each eye. The visible light may enter the eye of the first user 14.1 in FIG. 1 and may be detected on the retina of the eye of the first user 14.1.

[0119] The real object detection camera 44 may include one or more cameras that may capture images from different sides of the head mountable frame 40. The mobile tracking camera 46 may include one or more cameras that capture images on a side of the head mountable frame 40. One set of one or more cameras may be used in place of two sets of one or more cameras representing the real object detection camera 44 and the mobile tracking camera 46. In some embodiments, the cameras 44, 46 may capture images.

[0120] In some embodiments, images captured by cameras 44, 46 may be computed into one or more key frames (e.g., key frames 1, 2 in FIG. 11D). A key frame may correspond to a camera pose. A key frame may include one or more camera images captured at a camera pose. In some embodiments, the XR system may determine that a portion of a camera image captured at a camera pose is not useful and therefore not include that portion in a key frame. Thus, using key frames to match new images with earlier knowledge of the scene reduces the use of computational resources of the XR system. In some embodiments, a key frame may be an image, i.e., image data, at a location with a certain direction / angle. In some embodiments, a key frame may be a location and direction from which one or more map points may be observed. In some embodiments, a key frame may be a coordinate frame with an ID. U.S. Patent Application No. 15 / 877,359 describes key frames and is incorporated herein by reference in its entirety.

[0121] The inertial measurement unit 48 may include a number of devices used to detect movement of the head unit 22. The inertial measurement unit 48 may include a gravity sensor, one or more accelerometers, and one or more gyroscopes. The sensors of the inertial measurement unit 48 combine to track movement of the head unit 22 in at least three orthogonal directions and approximately at least three orthogonal axes.

[0122] In the illustrated embodiment, the world frame system 54 includes a world surface determination routine 78, a world frame determination routine 80, and world frame storage instructions 82. The world surface determination routine 78 is connected to the real object detection camera 44. The world surface determination routine 78 receives images and / or key frames based on images captured by the real object detection camera 44, processes the images, and identifies surfaces within the images. A depth sensor (not shown) may determine the distance to the surface. The surface is thus represented by data in three dimensions, including its size, shape, and distance from the real object detection camera.

[0123] In some embodiments, the world surface determination routine 78 may convert one or more keyframes into a persistent pose (e.g., persistent pose 1, 2 in FIG. 11D). The persistent pose may be an output of the routine 78. In some embodiments, the persistent pose may be a coordinate location and / or orientation having one or more associated keyframes. In some embodiments, the persistent pose may be created automatically after the user has traveled a certain distance, e.g., 3 meters. In some embodiments, the persistent pose may act as a reference point during localization. In some embodiments, the data determined by the world surface determination routine 78 may be stored in a passable world (e.g., a digital representation of real objects in the physical world that may be stored and updated as the real objects in the physical world change).

[0124] In some embodiments, the passable world may comprise or be created from one or more reference maps, each of which may include a corresponding world coordinate frame. The reference map may originate as a tracking map (TM) (e.g., TM 1102 in FIG. 40A), which may be converted to a reference map after a TM session is completed. In some embodiments, the TM may be a head pose sparse map of the XR device. In some embodiments, the XR device may send one or more TMs to the cloud for merging with additional TMs captured by the XR device at different times or by other XR devices.

[0125] In some embodiments, the persistent frame determination routine 80 may be connected to the world surface determination routine 78 and may determine a persistent coordinate frame (PCF) 84. A PCF may include multiple PCF anchors (e.g., PCF anchors 1-5 in FIG. 11D). A new PCF anchor may be determined based on a predefined distance allowed between different PCF anchors. The world surface determination routine 78 may store the PCF anchors, which may constitute a PCF, in a passable world. In some embodiments, the world frame determination routine 80 may convert one or more persistent poses to a PCF anchor once the user has progressed a predetermined distance, e.g., 5 meters. In some embodiments, the PCF anchor may be associated with one or more world coordinate frames 86, e.g., in a passable world. In some embodiments, the world coordinate frames 86 may be stored in the origin and destination coordinate frames 34.

[0126] In some embodiments, the content coordinate frame of the virtual content itself (i.e., a coordinate frame that is tied to the virtual content independent of its real-world location) may be associated with one or more PCF anchors. The inventors have recognized and appreciated that associating virtual content with PCF anchors may give the virtual content a more persistent frame position as opposed to being measured directly relative to the world coordinate frame. For example, if a virtual lamp is placed on a table, there may be multiple data points on the table that provide placement input for the relative positioning of the virtual lamp that do not change substantially over time. In contrast, if the world map is created as a function of some orientation and position, and the user changes position or orientation, thus forcing a new world coordinate frame, the virtual lamp may continue to utilize the same local coordinate frame rather than adjusting to the new world frame, which may introduce jitter or position shifts into the appearance of the lamp.

[0127] In some embodiments, the persistent frame storage instruction 82 may be connected to the persistent frame determination routine 80 and receive persistent coordinate frames 84 from the persistent frame determination routine 80. In some embodiments, the persistent frame storage instruction 82 may store the persistent coordinate frames 84 locally in a database and / or in the cloud (which may depend, for example, on security settings).

[0128] The head frame system 56 may include a head frame determination routine 90 and head frame storage instructions 92. The head frame determination routine 90 may be connected to the moving tracking camera 46 and the inertial measurement unit 48. The head frame determination routine 90 may use data from the moving tracking camera 46 and the inertial measurement unit 48 to calculate a head coordinate frame 94. For example, the inertial measurement unit 48 may have a gravity sensor that determines the direction of gravity relative to the head unit 22. The moving tracking camera 46 may continuously capture images that are used by the head frame determination routine 90 to refine the head coordinate frame 94. The head unit 22 moves as the first user 14.1 in FIG. 1 moves his head. The moving tracking camera 46 and the inertial measurement unit 48 may continually provide data to the head frame determination routine 90 so that the head frame determination routine 90 may update the head coordinate frame 94.

[0129] The head frame storage instruction 92 may be connected to the head frame determination routine 90 and receive a head coordinate frame 94 from the head frame determination routine 90. The head frame storage instruction 92 may store the head coordinate frame 94 as a head coordinate frame 96 in the origin and destination coordinate frame 34. The head frame storage instruction 92 may repeatedly store the updated head coordinate frame 94 as the head coordinate frame 96 when the head frame determination routine 90 recalculates the head coordinate frame 94. In some embodiments, the head coordinate frame may be the location of the wearable XR device 12.1 relative to the local coordinate frame 72.

[0130] The camera frame system 58 may include camera intrinsic properties 98. The camera intrinsic properties 98 may include the dimensions of the head unit 22, which are characteristic of its design and manufacture. The camera intrinsic properties 98 may be used to calculate a camera coordinate frame 100, which is stored in the origin and destination coordinate frames 34.

[0131] In some embodiments, the camera coordinate frame 100 may include all of the pupil positions of the left eye of the first user 14.1 in Figure 1. As the left eye moves from left to right or up and down, the pupil positions of the left eye are located within the camera coordinate frame 100. In addition, the pupil positions of the right eye are located within the camera coordinate frame 100 for the right eye. In some embodiments, the camera coordinate frame 100 may include the location of the camera relative to a local coordinate frame when an image is taken.

[0132] The origin / destination coordinate frame converter 36 may include a local / world coordinate converter 104, a world / head coordinate converter 106, and a head / camera coordinate converter 108. The local / world coordinate converter 104 may receive the local coordinate frame 72 and transform the local coordinate frame 72 to the world coordinate frame 86. The transformation of the local coordinate frame 72 to the world coordinate frame 86 may be represented as a local coordinate frame that is transformed within the world coordinate frame 86 to a world coordinate frame 110.

[0133] The world / head coordinate converter 106 may transform from the world coordinate frame 86 to the head coordinate frame 96. The world / head coordinate converter 106 may transform a local coordinate frame that is transformed to the world coordinate frame 110 into the head coordinate frame 96. The transformation may be expressed as a local coordinate frame that is transformed to the head coordinate frame 112 in the head coordinate frame 96.

[0134] The head / camera coordinate converter 108 may convert from the head coordinate frame 96 to the camera coordinate frame 100. The head / camera coordinate converter 108 may convert the local coordinate frame, which is converted to the head coordinate frame 112, to a local coordinate frame, which is converted to the camera coordinate frame 114 in the camera coordinate frame 100. The local coordinate frame, which is converted to the camera coordinate frame 114, may be brought into the rendering engine 30. The rendering engine 30 may render image data 68 representing the local content 28 based on the local coordinate frame, which is converted to the camera coordinate frame 114.

[0135] 3 is a spatial representation of various origin and destination coordinate frames 34. A local coordinate frame 72, a world coordinate frame 86, a head coordinate frame 96, and a camera coordinate frame 100 are represented in the figure. In some embodiments, the local coordinate frame associated with the XR content 28 may define the position and rotation of the virtual content (e.g., provide a nodal and facing direction), which may then be transformed into a local and / or world coordinate frame and / or PCF when the virtual content is placed in the real world and thus may be viewed by a user. Each camera may have its own camera coordinate frame 100 that encompasses all pupil positions of one eye. Reference numerals 104A and 106A represent the transformations performed by the local-to-world coordinate converter 104, the world-to-head coordinate converter 106, and the head-to-camera coordinate converter 108 in FIG. 2, respectively.

[0136] FIG. 4 depicts a camera rendering protocol for transforming from a head coordinate frame to a camera coordinate frame, according to some embodiments. In the illustrated example, the pupil for one eye moves from position A to B. A virtual object that is intended to appear stationary would be projected onto the depth plane at one of two positions A or B, depending on the position of the pupil (assuming the camera is configured to use the pupil as its coordinate frame). As a result, using a pupil coordinate frame that is transformed into the head coordinate frame would cause jitter in the stationary virtual object as the eye moves from position A to position B. This situation is referred to as view-dependent display or projection.

[0137] As depicted in Figure 5, the camera coordinate frame (e.g., CR) is positioned to encompass all pupil positions, but the object projection will now be consistent regardless of pupil positions A and B. The head coordinate frame transforms to the CR frame, which is referred to as view-independent display or projection. Image reprojection may be applied to the virtual content to account for changes in eye position, however, the rendering is still at the same position, so jitter is minimized.

[0138] As depicted in Fig. 6, the second XR device 12.2, which may be in the same scene as the first XR device 12.1, may include a persistent coordinate frame (PCF) integration unit 1300, an application 1302 that generates image data 68, and a frame embedding generator 308. In some embodiments, the map download system 126, the anchor identification system 128, the map 2, the location module 130, the reference map embedder 132, the reference map 133, and the map publisher 136 may be grouped into a passable world unit 1304. The PCF integration unit 1300 may be connected to the passable world unit 1304 and other components of the second XR device 12.2 to enable reading, creating, using, uploading, and downloading PCF anchors. The coordinate frame constructed by the PCF anchors may enable more persistence in a changing world, e.g., where people move in and out of a scene and objects such as doors move relative to a scene, require less storage space and transmission rates, and enable the use of individual PCF anchors and their relationships to each other and / or integrated constellations of PCF anchors to map a scene.

[0139] In some embodiments, the PCF integration unit 1300 may include a PCF anchor 1306 previously stored in a data storage device on a storage unit of the second XR device 12.2, a PCF tracker 1308, a persistent attitude obtainer 1310, a PCF verifier 1312, a PCF generation system 1314, a coordinate frame calculator 1316, a persistent attitude calculator 1318, and three converters including a tracking map and persistent attitude converter 1320, a persistent attitude and PCF converter 1322, and a PCF and image data converter 1324.

[0140] In some embodiments, the PCF tracker 1308 may have an on prompt and an off prompt that are selectable by the application 1302. The application 1302 may be executable by a processor of the second XR device 12.2 to display virtual content, for example. The application 1302 may have a call, via an on prompt, to switch the PCF tracker 1308 on. The PCF tracker 1308 may generate a PCF anchor when the PCF tracker 1308 is switched on. The application 1302 may have a subsequent call, via an off prompt, to switch the PCF tracker 1308 off. The PCF tracker 1308 terminates PCF generation when the PCF tracker 1308 is switched off.

[0141] In some embodiments, the server 20 may include a plurality of sustained attitudes 1332 and a plurality of PCF anchors 1330 previously stored in association with the reference map 120. The map transmitter 122 may transmit the reference map 120 to the second XR device 12.2 together with the sustained attitudes 1332 and / or the PCF anchors 1330. The sustained attitudes 1332 and the PCF anchors 1330 may be stored on the second XR device 12.2 in association with the reference map 133. Once map 2 is located relative to the reference map 133, the sustained attitudes 1332 and the PCF anchors 1330 may be stored in association with map 2.

[0142] In some embodiments, the persistent attitude obtainer 1310 may obtain a persistent attitude for map 2. The PCF verifyer 1312 may be coupled to the persistent attitude obtainer 1310. The PCF verifyer 1312 may retrieve PCF anchors from the PCF anchor 1306 based on the persistent attitude retrieved by the persistent attitude obtainer 1310. The PCF anchors retrieved by the PCF verifyer 1312 may form an initial group of PCF anchors to be used for image display based on the PCF anchors.

[0143] In some embodiments, application 1302 may request that additional PCF anchors be generated. For example, if a user moves into an area not previously mapped, application 1302 may turn on PCF tracker 1308. PCF generation system 1314 may be connected to PCF tracker 1308 and begin generating PCF anchors based on map 2 as map 2 begins to expand. The PCF anchors generated by PCF generation system 1314 may form a second group of PCF anchors that may be used for PCF-based image display.

[0144] The coordinate frame calculator 1316 may be connected to the PCF validator 1312. After the PCF validator 1312 reads the PCF anchor, the coordinate frame calculator 1316 may call the head coordinate frame 96 to determine the head pose of the second XR device 12.2. The coordinate frame calculator 1316 may also call the sustained pose calculator 1318. The sustained pose calculator 1318 may be directly or indirectly connected to the frame embedding generator 308. In some embodiments, the frame embedding generator 308 may output a global feature sequence 316, which may be used as a data representation (e.g., frame or key frame) for an image. In some embodiments, an image / frame may be designated as a key frame after a threshold distance, e.g., 3 meters, from the previous key frame has been advanced. The sustained pose calculator 1318 may generate a sustained pose based on multiple, e.g., 3 key frames. In some embodiments, the sustained pose may essentially be an average of the coordinate frames of multiple key frames.

[0145] A tracking map and sustained attitude converter 1320 may be coupled to the map 2 and sustained attitude calculator 1318. The tracking map and sustained attitude converter 1320 may convert map 2 to a sustained attitude and determine a sustained attitude at the origin for map 2.

[0146] The persistent attitude and PCF converter 1322 may be connected to the tracking map and persistent attitude converter 1320 and further to the PCF verifier 1312 and the PCF generation system 1314. The persistent attitude and PCF converter 1322 may convert the persistent attitude (for which the tracking map has been transformed) to a PCF anchor from the PCF verifier 1312 and the PCF generation system 1314 and determine a PCF for the persistent attitude.

[0147] The PCF and image data converter 1324 may be connected to the persistent attitude and PCF converter 1322 and the data channel 62. The PCF and image data converter 1324 converts the PCF into image data 68. The rendering engine 30 is connected to the PCF and image data converter 1324 and may display the image data 68 for the PCF anchor to a user.

[0148] The PCF integration unit 1300 may store additional PCF anchors generated using the PCF generation system 1314 in the PCF anchor 1306. The PCF anchor 1306 may be stored for the sustained attitude. The map publisher 136 may read the PCF anchor 1306 and the sustained attitude associated with the PCF anchor 1306 when the map publisher 136 transmits map 2 to the server 20 and the map publisher 136 also transmits the PCF and sustained attitude associated with map 2 to the server 20. When the map storage routine 118 of the server 20 stores map 2, the map storage routine 118 may also store the sustained attitude and PCF anchor generated by the second viewing device 12.2. The map merge algorithm 124 may create the reference map 120 with the sustained attitude and PCF anchor of map 2 associated with the reference map 120 and stored in the sustained attitude 1332 and the PCF anchor 1330, respectively.

[0149] The first XR device 12.1 may include a PCF integration unit similar to the PCF integration unit 1300 of the second XR device 12.2. When the map transmitter 122 transmits the reference map 120 to the first XR device 12.1, the map transmitter 122 may transmit the sustained attitude 1332 and the PCF 1330 associated with the reference map 120 and resulting from the second XR device 12.2. The first XR device 12.1 may store the PCF anchors and sustained attitudes in a data storage device on the storage device of the first XR device 12.1. The first XR device 12.1 may then utilize the sustained attitudes and PCF anchors resulting from the second XR device 12.2 for image display relative to the PCF anchors. Additionally or alternatively, the first XR device 12.1 may read, generate, utilize, upload and download PCF anchors and persistent postures in a manner similar to the second XR device 12.2, as described above.

[0150] FIG. 44 illustrates keyframes generated for a room. In the illustrated example, a first keyframe KF1 is generated for a door on the left wall of the room. A second keyframe KF2 is generated for the area in the corner where the floor, left wall, and right wall of the room meet. A third keyframe KF3 is generated for the window area on the right wall of the room. A fourth keyframe KF4 is generated for the area at the edge of a rug on the floor of the wall. A fifth keyframe KF5 is generated for the area of ​​the rug closest to the user.

[0151] FIG. 45 illustrates the persistence postures PP1 and PP2 in the user's environment. In some embodiments, a new persistence posture is created when the device measures a threshold distance traveled and / or when an application requests a new persistence posture. In some embodiments, the threshold distance may be 3 meters, 5 meters, 20 meters, or any other suitable distance. Selecting a smaller threshold distance (e.g., 1 m) may result in an increase in computational load since a larger number of PPs may be created and managed compared to a larger threshold distance. Selecting a larger threshold distance (e.g., 40 m) may result in fewer PPs being created and fewer PCFs being created, which may result in an increase in virtual content placement error since virtual content tied to a PCF anchor may be a relatively large distance (e.g., 30 m) away from the PCF anchor, meaning that the error may increase with the increase in the distance from the anchor to the virtual content.

[0152] In some embodiments, a PP may be created at the beginning of a new session. This initial PP may be considered zero and may be visualized as the center of a circle with a radius equal to the threshold distance. When the device reaches the circumference of the circle and, in some embodiments, an application requests a new PP, a new PP may be placed at the device's current location (threshold distance). In some embodiments, a new PP will not be created at the threshold distance if the device is able to find an existing PP within the threshold distance from the device's new position. In some embodiments, when a new PP is created, the device binds one or more of the closest key frames to the PP. In some embodiments, the location of the PP relative to the key frame may be based on the device's location at the time the PP is created. In some embodiments, a PP will not be created even if the device progresses the threshold distance unless an application requests a PP.

[0153] In some embodiments, an application may request a PCF from the device when the application has virtual content to display to the user. A PCF request from the application may trigger a PP request, and a new PP will be created after the device progresses a threshold distance. FIG. 45 illustrates a first persistent posture PP1 that may cause the closest key frames (e.g., KF1, KF2, and KF3) to be linked. FIG. 45 also illustrates a second persistent posture PP2 that may cause the closest key frames (e.g., KF4 and KF5) to be linked.

[0154] FIG. 46 illustrates PCF anchors, which may include PP1 and PP2. As described above, PCFs may be used to display image data for a PCF. In some embodiments, each PCF may have coordinates and descriptors in another coordinate frame (e.g., the world coordinate frame). In some embodiments, various constellations of PCF anchors may be combined to represent the real world, requiring less data and less data transmission in a persistent manner.

[0155] FIG. 47 illustrates a method 4700 of establishing and using persistent coordinate frames, according to some embodiments. Method 4700 may begin with capturing images of a scene using one or more sensors of an XR device (act 4702). Multiple cameras may be used, and one camera may generate multiple images. Method 4700 may include extracting features from the captured images (4704), generating descriptors for the extracted features (act 4706), generating key frames based on the descriptors (act 4708), and generating persistent poses based on the key frames (act 4710). In some embodiments, key frames may be generated based on each of the images, and an average of the key frames for all images of all cameras may be used to generate persistent poses. Method 4700 may include generating anchors based on the persistent poses (act 4712). The anchors may include one or more persistent poses. The method 4700 may include associating at least one of the anchors with image data of the virtual content to be displayed by the XR device (act 4714).

[0156] 48A-48I are schematic diagrams illustrating examples of establishing and using persistent coordinate frames that may enable data sharing between XR devices. FIG. 48A shows two users 4802A, 4802B with individual local tracking maps 4804A, 4804B that are not localized relative to a reference map. Origins 4806A, 4806B for each user are described by a coordinate system (e.g., a world coordinate system) within their individual areas. In some embodiments, the TM 4802A may include a persistent pose (PP) 4808A and the TM 4802B may include a PP 4808B. FIG. 48B shows that the XR devices worn by the individual users 4802A, 4802B may create local PCF anchors 4810A, 4810B based on the PP 4808A, 4908B. FIG. 48C shows that persistent content 4812 (e.g., virtual content) can now be bound to PCF anchors 4810A, 4810B. FIG. 48D shows that, for example, a reference map 1814 from the cloud is received by both devices. FIG. 48E shows that both devices locate to the reference map 4814 on their respective devices. FIG. 48F shows that each device creates a link (e.g., link 4816A, 4816B) between a local PCF anchor (e.g., PCF anchor 4810A, 4810B) on the respective device and a respective persistent posture (e.g., PP 4818A, 4818B) on the reference map. FIG. 48G shows a snapshot of the persistent posture from the reference map to the local tracking map. As can be seen, the local tracking maps are connected to each other via persistent postures. Figure 48H shows that PCF anchor 4810A on a device worn by user 4802A is accessible in a device worn by user 4802B. Figure 481 shows that the maps may all eventually merge and the PCFs may "coalesce" in a connected constellation.

[0157] Mapping and Location 6 illustrates in further detail the first XR device 12.1 and the second XR device 12.2 and the visual data and algorithms of the server 20 according to some embodiments. Although not shown, the first XR device 12.1 may be configured identically to the second XR device 12.2.

[0158] The server 20 may include a map storage routine 118 , a reference map 120 , a map transmitter 122 , and a map merging algorithm 124 .

[0159] In the illustrated embodiment, the first XR device 12.1 generates a local tracking map (hereinafter referred to as “Map 1”), and the map storage routine 118 receives Map 1 from the first XR device 12.1. The map storage routine 118 then stores Map 1 as a reference map 120 on the storage device of the server 20.

[0160] The second XR device 12.2 includes a map download system 126, an anchor identification system 128, a location determination module 130, a reference map embedder 132, a local content positioning system 134, and a map publisher 136.

[0161] In use, the map transmitter 122 transmits the reference map 120 to the second XR device 12.2, and the map download system 126 downloads and stores the reference map 120 from the server 20 as the reference map 133.

[0162] The anchor identification system 128 is connected to the world surface determination routine 78. The anchor identification system 128 identifies anchors based on objects detected by the world surface determination routine 78. The anchor identification system 128 uses the anchors to generate a second map (Map 2). As shown by cycle 138, the anchor identification system 128 continues to identify anchors and update Map 2. The locations of the anchors are recorded as three-dimensional data based on the data provided by the world surface determination routine 78. The world surface determination routine 78 receives images from the real object detection camera 44 and depth data from the depth sensor 135 and determines the location of the surface and its relative distance from the depth sensor 135.

[0163] A location module 130 is connected to the reference map 133 and map 2. The location module 130 repeatedly attempts to locate map 2 relative to the reference map 133. A reference map embedder 132 is connected to the reference map 133 and map 2. Once the location module 130 locates map 2 relative to the reference map 133, the reference map embedder 132 embeds the reference map 133 into the anchor of map 2. Map 2 is then updated with the missing data contained in the reference map.

[0164] A local content positioning system 134 is connected to map 2. The local content positioning system 134 may be, for example, a system in which a user may locate local content at a particular location in a world coordinate frame. The local content itself is then tied to an anchor in map 2. The local-to-world coordinate converter 104 converts the local coordinate frame to the world coordinate frame based on the settings of the local content positioning system 134. The functionality of the rendering engine 30, the display system 42, and the data channel 62 are described with reference to FIG. 2.

[0165] Map publisher 136 uploads Map 2 to server 20. Map storage routine 118 of server 20 then stores Map 2 in the server 20's storage medium.

[0166] A map merging algorithm 124 merges map 2 with the reference map 120. When more than two maps are stored, for example three or four maps, the map merging algorithm 124 merges all maps into the reference map 120 and renders the new reference map 120. The map transmitter 122 then transmits the new reference map 120 to every device 12.1 and 12.2 within the area represented by the new reference map 120. Once the devices 12.1 and 12.2 locate their respective maps against the reference map 120, the reference map 120 becomes the recommended map.

[0167] 7 illustrates in more detail the display system 42. The display system 42 includes a stereoscopic analyzer 144 that is connected to the rendering engine 30 and forms part of the visual data and algorithms.

[0168] The display system 42 further includes left and right projectors 166A and 166B and left and right waveguides 170A and 170B. The left and right projectors 166A and 166B are connected to a power supply. Each projector 166A and 166B has an individual input through which image data is provided to the individual projector 166A or 166B. When powered, the individual projector 166A or 166B generates light in a two-dimensional pattern and emits light therefrom. The left and right waveguides 170A and 170B are positioned to receive light from the left and right projectors 166A and 166B, respectively. The left and right waveguides 170A and 170B are transparent waveguides.

[0169] In use, a user mounts head-mountable frame 40 on their head. Components of head-mountable frame 40 may include, for example, a strap (not shown) that wraps around the back of the user's head. Left and right waveguides 170A and 170B are then positioned in front of the user's left and right eyes 220A and 220B.

[0170] The rendering engine 30 feeds the image data it receives into the stereoscopic analyzer 144. The image data is three-dimensional image data of the local content 28 in FIG. 1. The image data is projected onto multiple virtual planes. The stereoscopic analyzer 144 analyzes the image data and determines left and right image data sets based on the image data for projection onto each depth plane. The left and right image data sets are data sets representing two-dimensional images that are projected in three dimensions to give the user the perception of depth.

[0171] The stereoscopic analyzer 144 captures left and right image data sets into the left and right projectors 166A and 166B, which then create left and right light patterns. The components of the display system 42 are shown in plan view, but it should be understood that the left and right patterns are two-dimensional patterns when shown in a front elevation view. Each light pattern includes a number of pixels. For illustrative purposes, light rays 224A and 226A from two of the pixels are shown exiting the left projector 166A and entering the left waveguide 170A. The light rays 224A and 226A reflect off the side of the left waveguide 170A. Although rays 224A and 226A are shown propagating through internal reflection from left to right within left waveguide 170A, it should be understood that rays 224A and 226A may also propagate into the plane of the paper using refractive and reflective systems.

[0172] Light rays 224A and 226A exit the left optical waveguide 170A through pupil 228A and then enter the left eye 220A through the pupil 230A of the left eye 220A. Light rays 224A and 226A then impinge on the retina 232A of the left eye 220A. Thus, the left light pattern impinges on the retina 232A of the left eye 220A. The user is given the perception that the pixels formed on the retina 232A are pixels 234A and 236A, which the user perceives as being at a distance on the side of the left waveguide 170A facing the left eye 220A. Depth perception is created by manipulating the focal length of the light.

[0173] Similarly, stereoscopic analyzer 144 captures a right image data set to right projector 166B. Right projector 166B transmits a right light pattern, which is represented by pixels in the form of rays 224B and 226B. Rays 224B and 226B reflect in right waveguide 170B and exit through pupil 228B. Rays 224B and 226B then enter through pupil 230B of right eye 220B and impinge on retina 232B of right eye 220B. The pixels of rays 224B and 226B are perceived as pixels 134B and 236B behind right waveguide 170B.

[0174] The patterns created on the retinas 232A and 232B are perceived individually as left and right images, which differ slightly from each other due to the function of the stereoscopic analyzer 144. The left and right images are perceived as a three-dimensional rendering in the user's brain.

[0175] As mentioned, the left and right waveguides 170A and 170B are transparent. Light from a real object, such as a table 16, on the side of the left and right waveguides 170A and 170B facing the eyes 220A and 220B can be projected through the left and right waveguides 170A and 170B and impinge on the retinas 232A and 232B.

[0176] FIG. 8 illustrates a map 1 and local content (content 123 and content 456) on a first XR device 12.1. Map 1 includes several anchors (anchor a-anchor d). From the perspective of the first XR device 12.1, anchor a has, as an example, X, Y, and Z coordinates of (0,0,0), and anchor b has X, Y, and Z coordinates of (-1,0,0). Content 123 is associated with anchor a. In this example, content 123 has an X, Y, and Z relationship to anchor a of (1,0,0). Content 456 has a relationship to anchor b. In this example, content 456 has an X, Y, and Z relationship to anchor b of (1,0,0). Map 1 also has an origin (origin 1).

[0177] In Fig. 9, the first XR device 12.1 uploads map 1 to the server 20. The server 20 now has a reference map, based on map 1. The first XR device 12.1 has a reference map, which is empty at this stage. The server 20, for purposes of discussion, in some embodiments does not contain any other maps than map 1. No maps are stored on the second XR device 12.2.

[0178] The first XR device 12.1 also transmits its Wi-Fi signature data to the server 20. The server 20 may use the Wi-Fi signature data to determine a rough location of the first XR device 12.1 based on intelligence gathered from other devices that have been connected to the server 20 or other servers in the past, as well as recorded GPS locations of such other devices.

[0179] The first XR device 12.1 may now end the first session (see FIG. 1) and disconnect from the server 20.

[0180] 10 and 11A show the start of a second session by the second user 14.2. The first user 14.1 is shown in phantom because the first session by the first user 14.1 has ended. The second XR device 12.2 starts recording objects. Various systems with variable granularity may be used by the server 20 to determine that the second session by the second XR device 12.2 is within the same vicinity of the first session by the first XR device 12.1. For example, Wi-Fi signature data, Global Positioning System (GPS) positioning data, GPS data based on Wi-Fi signature data, or any other data indicating a location may be included in the first and second XR devices 12.1 and 12.2 to record the location. Alternatively, the anchors identified by the second XR device 12.2 may show similarity to the anchors of map 1.

[0181] As shown in FIG. 11B, the second XR device boots up and begins to collect data, such as images 1110, from one or more cameras 44, 46. As shown in FIG. 11D, in some embodiments, the XR device (e.g., the second XR device 12.2) may collect one or more images 1110, perform image processing, and extract one or more features / points of interest 1120. Each feature may be converted into a descriptor 1130. In some embodiments, the descriptor 1130 may be used to describe a key frame 1140, which may have a location and orientation of an associated image that is tied to it. The one or more key frames 1140 may correspond to a single sustained pose 1150, which may be automatically generated after a threshold distance, e.g., 3 meters, from a previous sustained pose 1150. The one or more sustained poses 1150 may correspond to a single PCF 1160, which may be automatically generated after a predetermined distance, e.g., every 5 meters. Over time, as the user continues to move around the user's environment and the XR device continues to collect more data, such as images 1110, additional PCF anchors (e.g., PCF anchors 4, 5 and PCF anchor 3) may be created. Applications, namely, two 1180, may be launched on the XR device and provide virtual content 1170 to the XR device for presentation to the user. The virtual content may have an associated content coordinate frame, which may be placed relative to one or more PCF anchors. As shown in FIG. 11B, the second XR device 12.2 creates three PCF anchors. In some embodiments, the second XR device 12.2 may attempt to locate relative to one or more reference maps stored on the server 20. In some embodiments, as shown in FIG. 11C, the second XR device 12.2 may download the reference map 120 from the server 20. Map 1 on the second XR device 12.2 includes anchor ad and origin 1. In some embodiments, the server 20 may have multiple reference maps for various locations and may determine that the second XR device 12.2 is within the same vicinity as the first XR device 12.1 during the first session and send the reference map for that vicinity to the second XR device 12.2.

[0182] FIG. 12 shows that the second XR device 12.2 begins identifying anchors for the purpose of generating map 2. The second XR device 12.2 has only identified a single anchor, namely, PCF1,2. The X, Y, and Z coordinates of PCF1,2 for the second XR device 12.2 may be (1,1,1). Map 2 has its own origin (origin 2), which may be based on the head pose of device 2 for the current head pose session. In some embodiments, the second XR device 12.2 may immediately attempt to locate map 2 against the reference map. In some embodiments, map 2 may be impossible to locate against the reference map (map 1) (i.e., the localization may fail) because the system does not recognize any or sufficient overlap between the two maps. In some embodiments, the system may locate based on an anchor / PCF comparison between the local map and the reference map. In some embodiments, the system may locate based on a persistent pose comparison between the local map and the reference map. In some embodiments, the system may locate based on keyframe comparison between the local map and the reference map.

[0183] 13 shows map 2 after the second XR device 12.2 identifies additional anchors (PCF1,2, PCF3, PCF4,5) for map 2. The second XR device 12.2 again attempts to locate map 2 against the reference map. Because map 2 has been extended to overlap with at least a portion of the reference map, the localization attempt will be successful. In some embodiments, the overlap between the local tracking map, map 2, and the reference map may be represented by PCFs / anchors, persistent poses, keyframes, or any other suitable intermediate or derived constructs.

[0184] Additionally, the second XR device 12.2 has associated content 123 and content 456 with PCF1,2 and PCF3 of map 2. Content 123 has X, Y, and Z coordinates with respect to PCF1,2 of (1,0,0).

[0185] Similarly, the X, Y, and Z coordinates of content 456 for PCF3 in map 2 are also (1,0,0).

[0186] 14A and 14B illustrate the successful localization of map 2 relative to the reference map. The overlapping area / volume / section of map 1410 represents the intersection with map 1 and the reference map. Map 2 created PCFs 3 and 4, 5 prior to localization, and the reference map created anchors a and c before map 2 was created, so different anchors / PCFs were created to represent the same volume in real space (e.g., different maps).

[0187] As shown in FIG. 15, the second XR device 12.2 extends map 2 to include the anchor ad from the reference map. The inclusion of the anchor ad represents the localization of map 2 relative to the reference map. In some embodiments, the XR system may perform an optimization step to remove duplicate anchors / PCFs from the overlapping areas of the PCFs in 1410, i.e., PCF3 and PCF4,5. After map 2 is localized, placement of virtual content, such as content 456 and content 123, will be to the nearest updated anchor / PCF in the updated map 2. The virtual content will appear in the same real-world location to the user despite the changed PCF binding for the content and despite the updated anchor for map 2.

[0188] As shown in Figure 16, the second XR device 12.2 continues to expand map 2 as additional anchors (anchors e, f, g, and h) are identified by the second XR device 12.2, e.g., as the user walks around the real world. Also, note that map 1 is not expanded in Figures 15 and 16.

[0189] 17, the second XR device 12.2 uploads map 2 to the server 20. The server 20 stores the reference map along with map 2. In some embodiments, map 2 may be uploaded to the server 20 when the session for the second XR device 12.2 ends.

[0190] The reference map in the server 20 now includes anchor i, which is not included in map 1 on the first XR device 12.1. The reference map on the server 20 may be expanded to include anchor i when a third XR device (not shown) uploads a map to the server 20 and such map includes anchor i.

[0191] In Figure 18, the server 20 merges map 2 with the reference map to form a new reference map. The server 20 determines that anchor ad is common to the reference map and map 2. The server extends the reference map to include anchors eh and PCF1,2 from map 2 to form a new reference map. The reference maps on the first and second XR devices 12.1 and 12.2 are based on map 1 and become out of date.

[0192] In Fig. 19, the server 20 transmits the new reference map to the first and second XR devices 12.1 and 12.2. In some embodiments, this may occur when the first and second XR devices 12.1 and 12.2 attempt to locate during a different or new or subsequent session. The first and second XR devices 12.1 and 12.2 proceed to locate their respective local maps (Map 1 and Map 2, respectively) against the new reference map, as described above.

[0193] As shown in FIG. 20, the head coordinate frame 96 or "head pose" is relative to the anchor in map 2. In some embodiments, the map's origin, i.e., origin 2, is based on the head pose of the second XR device 12.2 at the beginning of the session. As anchors are created during the session, they are placed relative to the local coordinate frame, i.e., origin 2. The map 2 anchor serves as a persistent coordinate frame relative to the world coordinate frame, which may be the local coordinate frame of the previous session (e.g., origin 1 of map 1 in FIG. 8). The transformation from the world coordinate frame to the head coordinate frame 96 was previously described with reference to FIG. 2. The head coordinate frame 96 shown in FIG. 20 has only two orthogonal axes that are at a specific coordinate position relative to the map 2 anchor and at a specific angle relative to map 2. However, it should be understood that the head coordinate frame 96 is in a certain three-dimensional location relative to the map 2 anchor and has three orthogonal axes in three-dimensional space.

[0194] In FIG. 21, the head coordinate frame 96 has moved relative to the anchor of map 2. The head coordinate frame 96 has moved because the second user 14.2 has moved his head. A user can move his head in six degrees of freedom (6dof). The head coordinate frame 96 can therefore move in 6dof, i.e., in three dimensions from its previous location in FIG. 20, and in about three orthogonal axes relative to the anchor of map 2. The head coordinate frame 96 is adjusted as the real object detection camera 44 and the inertial measurement unit 48 in FIG. 2 detect the movement of the real object and the head unit 22, respectively. Further information regarding head pose tracking is disclosed in U.S. Patent Application No. 16 / 221,065, entitled "Enhanced Pose Determination for Display Device," which is incorporated herein by reference in its entirety.

[0195] FIG. 22 shows that a sound may be associated with one or more anchors. A user may, for example, wear headphones or earphones with stereophonic sound. The location of the sound through the headphones can be simulated using conventional techniques. The location of the sound may be located at a stationary position such that when the user rotates his / her head to the left, the location of the sound rotates to the right, thus the user perceives the sound originating from the same location in the real world. In this example, the location of the sound is represented by sound 123 and sound 456. For purposes of discussion, FIG. 22 is similar in its analysis to FIG. 16. When a first and a second user 14.1 and 14.2 are located in the same room, at the same or different times, they perceive sound 123 and sound 456 originating from the same location in the real world.

[0196] 23 and 24 illustrate further implementations of the technique described above. The first user 14.1 started a first session as described with reference to FIG. 1. As shown in FIG. 23, the first user 14.1 ended the first session as indicated by the phantom line. At the end of the first session, the first XR device 12.1 uploaded map 1 to the server 20. The first user 14.1 now started a second session at a time after the first session. The first XR device 12.1 does not download map 1 from the server 20 because map 1 is already stored on the first XR device 12.1. If map 1 is lost, the first XR device 12.1 downloads map 1 from the server 20. The first XR device 12.1 then proceeds to build an anchor for map 2, locate relative to map 1, and further develop a reference map as described above. Map 2 is then used to correlate local content, head coordinate frames, local sounds, etc. as explained above.

[0197] 25 and 26, it may also be possible for more than one user to interact with the server in the same session. In this example, the first user 14.1 and the second user 14.2 are joined by a third user 14.3 with a third XR device 12.3. Each XR device 12.1, 12.2, and 12.3 starts to generate its own map, namely, map 1, map 2, and map 3, respectively. As the XR devices 12.1, 12.2, and 12.3 continue to develop maps 1, 2, and 3, the maps are progressively uploaded to the server 20. The server 20 merges maps 1, 2, and 3 to form a reference map. The reference map is then transmitted from the server 20 to each of the XR devices 12.1, 12.2, and 12.3.

[0198] Multiple criterion maps and keyframes for criterion map filtering and localization 27 illustrates a number of reference maps 120 stored in a data storage device on a computer-readable medium of server 20. Each reference map 120 has a number of reference map identifiers associated with it. These reference map identifiers include areas on planet Earth, represented by ranges of longitude and latitude. The reference map identifiers also include global feature columns.

[0199] The reference map 120 is illustrated geographically in a two-dimensional pattern as it may exist on the surface of the Earth. The longitude and latitude are different from each other and uniquely identify the reference map. The server 20 has a location detector that receives the location data, including the longitude and latitude, the Wi-Fi fingerprint, and the image features taken by the individual viewing device 12.1 or 12.2. The location detector then stores the location identifier. Not all viewing devices provide the longitude and latitude as location identifiers to the server 20. However, the viewing devices will typically include the Wi-Fi fingerprint and image features as location identifiers.

[0200] FIG. 28 illustrates the reference map 120 and a series of filters, including a nearby area filter 300, a Wi-Fi fingerprint filter 302, and a key frame filter 304, that the server 20 executes to determine one or more maps to be sent to the viewing device, as described above, followed by location determination 306 on the viewing device.

[0201] The reference map 120 may have many reference maps, such as thousands of reference maps. In some embodiments, due to bandwidth limitations, it may not be possible to transmit all of the reference maps to the viewing device.

[0202] In some embodiments, the neighborhood area filter 300 may select at least one matching criteria map 120 that covers the longitude and latitude of the location identifier received from the viewing device, as long as at least one map exists at that longitude and latitude. In some embodiments, the neighborhood area filter 300 further selects at least one nearby criteria map that covers the longitude and latitude adjacent to the matching criteria map. In some embodiments, the neighborhood area filter 300 may select multiple matching criteria maps and multiple nearby criteria maps. The neighborhood area filter 300 may, for example, reduce the number of criteria maps from several thousand to, for example, 130, forming a first filtered selection. In some embodiments, the neighborhood area filter 300 may be configured to reduce the number of criteria maps 120. In other words, the number of criteria maps that are populated into the neighborhood area filter 300 may exceed the number of criteria maps that are retrieved from the neighborhood area filter 300.

[0203] In some embodiments, the Wi-Fi fingerprint filter 302 forms a second filter. The Wi-Fi fingerprint filter 302 determines a latitude and longitude based on the Wi-Fi fingerprint received from the viewing device as part of the location identifier. The Wi-Fi fingerprint filter 302 then compares the latitude and longitude from the Wi-Fi fingerprint filter 302 with the latitude and longitude of the criteria map 120 to determine one or more criteria maps forming a second filtered selection within the first filtered selection established by the neighborhood area filter 300. The second filtered selection may, for example, reduce the number of criteria maps to 50 criteria maps forming a second selection that does not include any criteria maps based on the first filtered selection outside the second filtered selection. In this example, the first filtered selection thus includes 130 criteria maps and the second filtered selection includes 50 of the 130 criteria maps and does not include the remaining 80 of the 130 criteria maps.

[0204] The key frame filter 304 further reduces the number of reference maps, for example to five reference maps. The key frame filter 304 may be a computer vision filter that compares data represented by the image captured by the viewing device with data representing the reference maps 120. The key frame filter 304 thus provides as output a third filtered selection that includes only five of the 50 reference maps identified following the second filtered selection. The map transmitter 122 then transmits one or more reference maps to the viewing device based on the third filtered selection. The viewing device receives the five reference maps in this example and some embodiments and attempts to locate against all five reference maps at 306.

[0205] 29A-29C illustrate a frame embedding generator 308 that, in some embodiments, may be used within the server 20 (or one of the viewing devices 12.1 and 12.2) to generate a unique and reduced data representation of an image from an initial size (e.g., 76,800 bytes) to a final size (e.g., 256 bytes / 1 bit). In some embodiments, the frame embedding generator may be configured to generate a data representation for an image, which may be a key frame or a frame. In some embodiments, the frame embedding generator 308 may be configured to convert an image at a particular location and orientation into a unique 256-byte sequence of numbers. An image 320 taken by the viewing device may be processed by the viewing device to detect features 324 within the image 320 (e.g., using a deep sparse feature method), where each feature 322 may be represented by a sequence 326 of 32 numbers. The features 322 are shown together by block 310. For example, there may be n=100 features, where each feature is represented by a sequence of 32 numbers.

[0206] In some embodiments, the frame embedding generator 308 may include a multi-layer perceptron unit 312 and a max pooling unit 314. In some embodiments, the multi-layer perceptron (MLP) unit 312 may comprise a multi-layer perceptron, which may comprise a neural network. In some embodiments, the MLP unit 312 may be configured to perform matrix multiplication. The multi-layer perceptron unit 312 receives a plurality of features 322 of an image 320 and converts each feature into a separate sequence of 256 numbers. In this example, there may be 100 features, each represented by a byte size of 256 numbers. The matrix may be created to have, in this example, 100 horizontal rows and 256 vertical columns. Each row has a sequence of 256 numbers of varying magnitude, some smaller and some larger. In some embodiments, the output of the MLP may be an n by 256 matrix 328, where n represents the number of features extracted from the image.

[0207] In some embodiments, the MLP may comprise a neural network, which may have a training phase, for example to determine model parameters, and a use phase. In some embodiments, the MLP may be trained as illustrated in FIG. 29D. The input training data may comprise data in three sets, the three sets comprising 1) a query image, 2) positive samples, and 3) negative samples. The query image may be considered a reference image. The positive samples may comprise images that are similar to the query image. For example, in some embodiments, similar may be having the same object in both the query and positive sample images, but viewed from different angles. In some embodiments, similar may be having the same object in both the query and positive sample images, but having an object that is shifted (e.g., left, right, up, down) relative to the other image. The negative samples may comprise images that are not similar to the query image. For example, in some embodiments, different may be having an object in the query image but not in the sample image. In some embodiments, different may be having a majority (e.g., >50% or >75%) of the object in the query image, but only a small portion (e.g., <10%, 1%) of the same object in the negative sample. In some embodiments, features (e.g., sparse features) may be extracted from the images in the input training data and converted into descriptors. In some embodiments, a deep sparse feature (DSF) process may be used to generate the descriptors (e.g., DSF descriptors, feature descriptors) as described in U.S. Patent Application No. 16 / 190,948. In some embodiments, the DSF descriptors are n×32 in size. The descriptors may then be passed through a model / MLP to create a 256-byte output. In some embodiments, the model / MLP may be the MLP described in Figures 29A-29C. The 256 bytes output from the MLP model may then be sent to a triplet margin loss module (and may only be used during the training phase, not during the use phase of the MLP neural network).In some embodiments, the triplet margin loss module may be configured to minimize the difference between the 256 bytes output from the query image and the 256 bytes output from the positive samples and maximize the difference between the 256 bytes output from the query image and the 256 bytes output from the negative samples. In some embodiments, the training phase may include feeding a plurality of triplet input images into a learning process (e.g., MLP, model) and determining model parameters.

[0208] A max pooling unit 314 analyzes each string and determines the maximum number in the individual strings. The max pooling unit 314 combines the maximum value of each string of numbers into a global feature string 316 of 256 numbers. The global feature string 316 is a relatively small number that occupies relatively little memory and is easily searchable. Thus, it is possible to search images without analyzing each original frame from the camera, and it is also cheaper to store 256 bytes instead of a full frame.

[0209] In some embodiments, the key frame filter 304 in FIG. 28 may filter the reference map 120 based on the global feature sequence 316 of the reference map 120 and the global feature sequence 316 based on an image captured by the viewing device (e.g., an image that may be part of a local tracking map for the user). Each one of the reference maps 120 in FIG. 27 thus has one or more global feature sequences 316 associated with it. In some embodiments, the global feature sequence 316 may be obtained when the viewing device submits an image or feature details to the server 20, which processes the image or feature details and generates the global feature sequence 316 for the reference map 120. In some embodiments, the server 20 may receive feature details of a live / new / current image captured by the viewing device, and the server 20 may generate the global feature sequence 316 for the live image and filter the reference map 120 based on the live global feature sequence 316. In some embodiments, the global feature sequence may be generated on the local viewing device. In some embodiments, the global feature string may be generated remotely, for example on the cloud or server 20. In some embodiments, the viewing device may take one or more images and perform the process illustrated in Figure 29C to bind the global feature string 316 to the local tracking map. In some embodiments, the global feature string 316 may be a data representation of a frame or keyframe in a local tracking map or reference map on the local device or cloud.

[0210] In some embodiments, the server 20 may transmit to the viewing device, along with the filtered reference map, a global feature sequence 316 associated with the filtered reference map. In some embodiments, when the viewing device locates its tracking map relative to the reference map, it may do so by matching the global feature sequence 316 of the local tracking map with the global feature sequence of the reference map.

[0211] Ranking and merging maps Described herein are methods and apparatus for ranking and merging multiple environment maps (e.g., reference maps) within an X-Reality (XR) system. To provide a user with a realistic XR experience, the XR system must know the user's physical surroundings to properly correlate the location of virtual objects in relation to real objects. Information about the user's physical surroundings may be obtained from an environment map related to the user's location. The environment map may be created from image and depth information collected with sensors that are part of an XR device worn by a user of the XR system. The environment map may be constructed using information collected by sensors worn by users of the system at various locations and times. Each environment map may include mapped points that are associated with three-dimensional (3D) world reconstructions of real objects in the environment, respectively.

[0212] The inventors have recognized and appreciated that an XR system may provide an enhanced XR experience to multiple users sharing the same world, comprising real and / or virtual content, regardless of whether those users are present in the world at the same or different times, by enabling efficient sharing of real / physical world environmental maps collected by multiple users. However, significant challenges exist in providing such a system. Such a system may store multiple maps generated by multiple users, and / or the system may store multiple maps generated at different times. For example, as described above, with respect to operations that may be performed using previously generated maps, such as localization, substantial processing may be required to identify relevant environmental maps of the same world (e.g., the same real-world location) from all environmental maps collected within the XR system. In some embodiments, such as those described in Figures 8-26, there may be only a small number of environmental maps that a device may access, for example, for localization. In some embodiments, such as those described in connection with Figures 27-29C above, there may be a large number of environmental maps that a device may access. The inventors have recognized and appreciated techniques for quickly and accurately ranking the relevance of environmental maps from all possible environmental maps, such as, for example, the population of all reference maps 120 in Figure 28. Highly ranked maps may then be selected for further processing, such as for rendering virtual objects on a user display to interact realistically with the physical world around the user, or for merging the stored map with map data collected by the user to create a larger or more accurate map.

[0213] In some embodiments, stored maps relevant to a task for a user at a location in the physical world may be identified by filtering the stored maps based on multiple criteria. The criteria may refer to a comparison of a tracked map generated by the user's wearable device at the location with candidate environment maps stored in a database, for example, as described above with respect to Figs. 27-29C. The comparison may be performed based on metadata associated with the map, such as a Wi-Fi fingerprint detected by the device generating the map, and / or the set of BSSIDs to which the device is connected while forming the map. The comparison may also be performed based on compressed or uncompressed content of the map. A comparison based on a compressed representation may be performed by, for example, a comparison of vectors calculated from the map content, for example, as described in Figs. 29A-29C. A comparison based on an uncompressed map may be performed by, for example, locating the tracked map against a stored map, or vice versa. Multiple comparisons may be performed in an order based on the calculation time required to reduce the number of candidate maps for consideration, with comparisons involving fewer calculations being performed before other comparisons requiring more calculations.

[0214] The techniques described herein may be used together or separately with and for many types of scenes, including wearable or portable devices with limited computing resources that provide XR scenes. In some embodiments, the techniques may be implemented by one or more services that form part of the XR system.

[0215] Figures 30 and 31 illustrate such a scene. For illustrative purposes, an AR system is used as an example of an XR system. Figures 32-35 illustrate an example AR system including one or more processors, memory, sensors, and a user interface that can operate in accordance with the techniques described herein.

[0216] 30, an outdoor AR scene 354 is depicted in which a user of the AR technology sees a physical world park-like setting 356 featuring people, trees, buildings in the background, and a concrete platform 358. In addition to these items, the user of the AR technology also perceives that they "see" a robotic figure 357 standing on the physical world concrete platform 358, and a flying cartoon-like avatar character 352, thereby appearing to be an anthropomorphic bumblebee, although these elements (e.g., the avatar character 352 and the robotic figure 357) do not exist within the physical world. Due to the extreme complexity of the human visual perception and nervous system, it is difficult to produce AR technology that facilitates comfortable, natural-feeling, rich presentation of virtual image elements among other virtual or physical world image elements.

[0217] Such AR scenes can be achieved using a system that includes a world reconstruction component that can build and update a digital representation of the physical world surfaces around the user. This representation can be used to occlude rendering, place virtual objects, in physics-based interactions, and for virtual character path planning and navigation, or for other operations in which information about the physical world is used. FIG. 31 depicts another example of an indoor AR scene 400 showing an example world reconstruction use case including visual occlusion 402, physics-based interactions 404, and environmental inference 406, according to some embodiments.

[0218] The exemplary scene 400 is a living room with a wall, a bookshelf on one side of the wall, a floor lamp in the corner of the room, a floor, a sofa, and a coffee table on the floor. In addition to these physical items, the user of the AR technology also perceives virtual objects such as an image on the wall behind the sofa, a bird that has flown in through the door, a deer peeking out from the bookshelf, and a decoration in the form of a windmill placed on the coffee table. For the image on the wall, the AR technology requires information not only about the surface of the wall, but also about objects and surfaces in the room, such as the lamp shape, that occlude the image to correctly render the virtual object. For the bird that has flown in, the AR technology requires information about all objects and surfaces around the room to render the bird with realistic physics to avoid the objects and surfaces or bounce off if the bird collides. For the deer, the AR technology requires information about surfaces such as the floor or coffee table to calculate where the deer should be placed. With respect to the windmill, the system may identify it as a separate object from the table and determine that it is movable, while a corner of a shelf or a corner of a wall may be determined to be stationary. Such specificity may be used in decisions regarding the portions of the scene to be used or updated in each of the various operations.

[0219] The scene may be presented to the user via a system that includes multiple components, including a user interface that may stimulate one or more user senses, including vision, hearing, and / or touch. In addition, the system may include one or more sensors that may measure parameters of the physical portion of the scene, including the user's position and / or movement within the physical portion of the scene. Furthermore, the system may include one or more computing devices with associated computer hardware, such as memory. These components may be integrated into a single device or distributed across multiple interconnected devices. In some embodiments, some or all of these components may be integrated into a wearable device.

[0220] FIG. 32 depicts an AR system 502 configured to provide an experience of AR content that interacts with a physical world 506, according to some embodiments. The AR system 502 may include a display 508. In the illustrated embodiment, the display 508 may be worn by a user as part of a headset such that the user may wear the display over their eyes, such as a pair of goggles or glasses. At least a portion of the display may be transparent such that the user may observe a see-through reality 510. The see-through reality 510 may correspond to a portion of the physical world 506 within a current viewpoint of the AR system 502, which may correspond to a user's viewpoint when the user is wearing a headset incorporating both the display and sensors of the AR system and obtaining information about the physical world.

[0221] The AR content may also be presented on the display 508, overlaid on the see-through reality 510. To provide accurate interaction between the AR content and the see-through reality 510 on the display 508, the AR system 502 may include a sensor 522 configured to capture information about the physical world 506.

[0222] The sensor 522 may include one or more depth sensors that output a depth map 512. Each depth map 512 may have a number of pixels, each of which may represent a distance to a surface in the physical world 506 in a particular direction relative to the depth sensor. Raw depth data may come from the depth sensors to create a depth map. Such a depth map may be updated as fast as the depth sensors can form a new image, which may be hundreds or thousands of times per second. However, the data may be noisy and incomplete, and may have holes, shown as black pixels on the illustrated depth map.

[0223] The system may include other sensors, such as image sensors. The image sensors may obtain monocular or stereoscopic information that may be processed to represent the physical world in other ways. For example, images may be processed in the world reconstruction component 516 to create a mesh that represents connected portions of objects in the physical world. Metadata about such objects, including, for example, color and surface texture, may also be obtained using the sensors and stored as part of the world reconstruction.

[0224] The system may also obtain information about the user's head pose relative to the physical world. In some embodiments, the sensor 522 may include an inertial measurement unit, which may be used to calculate and / or determine the head pose 514. The head pose 514 for a depth map may indicate the current viewpoint of the sensor capturing the depth map, for example with six degrees of freedom (6DoF), but the head pose 514 may be used for other purposes, such as relating image information to a particular part of the physical world or relating the position of a display worn on the user's head to the physical world. In some embodiments, the head pose information may be derived in a manner other than an IMU, such as from analysis of objects in an image.

[0225] The world reconstruction component 516 may receive the depth map 512 and head pose 514 and any other data from the sensors and integrate the data into a reconstruction 518. The reconstruction 518 may be more complete and less noisy than the sensor data. The world reconstruction component 516 may update the reconstruction 518 using spatial and temporal averages of the sensor data from multiple viewpoints over time.

[0226] Reconstruction 518 may include a representation of the physical world in one or more data formats, including, for example, voxels, meshes, planes, etc. Different formats may represent alternative representations of the same portion of the physical world or may represent different portions of the physical world. In the illustrated example, on the left side of reconstruction 518, a portion of the physical world is presented as a global surface, and on the right side of reconstruction 518, a portion of the physical world is presented as a mesh.

[0227] The reconstruction 518 may be used as or processed into a map that provides information about the portion of the physical world represented by the reconstruction. The map may indicate the location of features and / or structures, such as edges and / or planes. In addition, the map may include information about the user's pose from which the points or structures are observable to the user, which may be referred to in some embodiments as a keyframe or keyrig. In the illustrated embodiment, the map represents a portion of the physical world in which a user of a single wearable device is present. In combination with content that characterizes that portion of the physical world, the map may include metadata. The metadata may, for example, indicate the time of capture of the sensor information used to form the map. The metadata may alternatively or additionally indicate the location of the sensor at the time of capture of the information used to form the map. The location may be represented directly, such as with information from a GPS chip, or indirectly, such as with a Wi-Fi signature indicating the strength of signals received from one or more wireless access points while the sensor data was being collected, and / or with the BSSID of the wireless access point to which the user device was connected while the sensor data was being collected.

[0228] Reconstruction 518 may be used for AR functions, such as producing a surface representation of the physical world for occlusion handling or physics-based processing. This surface representation may change as the user moves or objects in the physical world change. Aspects of reconstruction 518 may be used by component 520, for example, to produce a changing global surface representation in world coordinates that may be used by other components.

[0229] AR content may be generated, such as by an AR application 504, based on this information. The AR application 504 may be, for example, a game program that performs one or more functions based on information about the physical world, such as visual occlusion, physics-based interactions, and environmental inference. It may perform these functions by querying data in different formats from a reconstruction 518 produced by the world reconstruction component 516. In some embodiments, the component 520 may be configured to output updates as the representation in a region of interest of the physical world changes. The region of interest may be set to approximate a portion of the physical world in the vicinity of a user of the system, such as a portion in the user's field of view, or may be projected (predicted / determined) to be within the user's field of view.

[0230] The AR application 504 may use this information to generate and update AR content, the virtual portions of which may be presented on the display 508 in combination with the see-through reality 510 to create a realistic user experience.

[0231] In some embodiments, the AR experience may be provided to the user through a wearable display system. FIG. 33 illustrates an example of a wearable display system 580 (hereinafter referred to as "system 580"). System 580 includes a head-mounted display device 562 (hereinafter referred to as "display device 562") and various mechanical and electronic modules and systems that support the functionality of display device 562. Display device 562 may be coupled to a frame 564, which is wearable by a user or viewer 560 of the display system (hereinafter referred to as "user 560") and configured to position display device 562 in front of the eyes of user 560. According to various embodiments, display device 562 may be a sequential display. Display device 562 may be monocular or binocular. In some embodiments, display device 562 may be an example of display 508 in FIG. 32.

[0232] In some embodiments, a speaker 566 is coupled to the frame 564 and positioned proximate to the ear canal of the user 560. In some embodiments, another speaker, not shown, is positioned adjacent to another ear canal of the user 560 to provide stereo / adjustable sound control. The display device 562 is operatively coupled, such as by wired leads or wireless connectivity 568, to a local data processing module 570, which may be mounted in a variety of configurations, such as fixedly attached to the frame 564, fixedly attached to a helmet or hat worn by the user 560, built into headphones, or otherwise removably attached to the user 560 (e.g., in a backpack configuration, in a belt-coupled configuration).

[0233] The local data processing module 570 may include a processor and digital memory such as non-volatile memory (e.g., flash memory), both of which may be utilized to aid in processing, caching, and storing data. The data includes data a) captured from sensors (e.g., which may be operatively coupled to the frame 564 or otherwise attached to the user 560), such as image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, and / or gyroscopes, and / or b) obtained and / or processed using the remote processing module 572 and / or remote data repository 574, possibly for passing to the display device 562 after processing or retrieval.

[0234] The local data processing module 570 may be operatively coupled to a remote processing module 572 and a remote data repository 574 by communications links 576, 578, such as via wired or wireless communication links, respectively, such that the remote modules 572, 574 are operatively coupled to each other and available as resources to the local data processing module 570. In some embodiments, the world reconstruction component 516 in FIG. 32 may be implemented, at least in part, within the local data processing module 570. For example, the local data processing module 570 may be configured, at least in part, to execute computer-executable instructions to generate a physical world representation based, at least in part, on at least a portion of the data.

[0235] In some embodiments, processing may be distributed across local and remote processors. With respect to the world reconstruction component 516, for example, local processing may be used to build a map (e.g., a tracking map) on the user device based on sensor data collected with sensors on the user's device. Such a map may be used by applications on the user's device. In addition, previously created environmental maps (e.g., a world map and / or a reference map) may be stored in the remote data repository 574. If a suitable stored environmental map is available, it may be used instead of or in addition to the tracking map. The environmental map may be loaded onto the user device to enable the user device to render virtual content without delays associated with scanning a location and building a tracking map from sensor data obtained during the scan. Alternatively or in addition, the tracking map may be merged with previously stored environmental maps to augment or improve the quality of those environmental maps. Processing to determine whether a suitable previously created environment map is available and / or whether to merge the tracking map with one or more stored environment maps may occur within the local data processing module 570 or the remote processing module 572.

[0236] In some embodiments, the local data processing module 570 may include one or more processors (e.g., a graphics processing unit (GPU)) configured to analyze and process the data and / or image information. In some embodiments, the local data processing module 570 may include a single processor (e.g., a single-core or multi-core ARM processor), which would limit the computational budget of the local data processing module 570 but enable smaller devices. In some embodiments, the world reconstruction component 516 may generate the physical world representation in real time over a non-predetermined space using less of a computational budget than a single Advanced RISC Machine (ARM) core, such that the remaining computational budget of the single ARM core may be accessed for other uses, such as, for example, extracting meshes.

[0237] In some embodiments, the remote data repository 574 may include a digital data storage facility, which may be available through the Internet or other networking configurations in a "cloud" resource configuration. In some embodiments, all data is stored and all computations are performed within the local data processing module 570, allowing for fully autonomous use from the remote modules. In some embodiments, all data is stored and all or most computations are performed within the remote data repository 574, allowing for smaller devices. The world reconstruction may be stored in whole or in part within this repository 574, for example.

[0238] In embodiments in which data is stored remotely and accessible via a network, the data may be shared by multiple users of the augmented reality system. For example, a user device may upload its tracking map, which may be stored in a database of environmental maps. In some embodiments, the uploading of the tracking map occurs at the end of a session. In some embodiments, the uploading of the tracking map may occur continuously, semi-continuously, intermittently, at predefined times, after a predefined period from a previous upload, or when triggered by an event. A tracking map uploaded by any user device may be used to extend or improve a previously stored environmental map, whether based on data from that user device or any other user device. Similarly, an environmental map downloaded to a user device may be based on data from that user device or any other user device. In this way, a high-quality environmental map may be readily available to a user to improve their experience with the AR system.

[0239] In some embodiments, the local data processing module 570 is operably coupled to a battery 582. In some embodiments, the battery 582 is a removable power source, such as a commercially available battery. In other embodiments, the battery 582 is a lithium ion battery. In some embodiments, the battery 582 includes both an internal lithium ion battery that is rechargeable by the user 560 during periods of non-operation of the system 580, and a removable battery, so that the user 560 may operate the system 580 for longer periods of time without being plugged in and having to charge the lithium ion battery or shutting off the system 580 and replacing the battery.

[0240] FIG. 34A illustrates a user 530 wearing an AR display system that renders AR content as the user 530 moves through a physical world environment 532 (hereafter referred to as “environment 532”). The user 530 positions the AR display system at a location 534, and the AR display system records ambient information of the passable world (e.g., digital representations of real objects in the physical world that can be stored and updated as the real objects in the physical world change) for the location 534, such as pose relationships with mapped features or directional audio inputs. The location 534 is aggregated against a data input 536 and processed by at least a passable world module 538, which may be implemented, for example, by processing on the remote processing module 572 of FIG. 33. In some embodiments, the passable world module 538 may include a world reconstruction component 516. FIG. 35B depicts a component architecture of a PW, according to some embodiments.

[0241] The passable world module 538 determines, at least in part, where and how the AR content 540 may be placed in the physical world, as determined from the data input 536. The AR content is "placed" in the physical world by presenting both a representation of the physical world and the AR content via the user interface, where the AR content is rendered as if it were interacting with objects in the physical world, and the objects in the physical world are presented as if the AR content were obscuring the user's view of those objects, when appropriate. In some embodiments, the AR content may be placed by appropriately selecting a portion of a fixed element 542 (e.g., a table) from the reconstruction (e.g., reconstruction 518) and determining the shape and location of the AR content 540. As an example, the fixed element may be a table, and the virtual content may be positioned to appear on the table. In some embodiments, the AR content may be placed in a structure in the field of view 544, which may be the current field of view or an estimated future field of view. In some embodiments, the AR content may be persisted to a mapped (e.g., mesh) model 546 of the physical world.

[0242] As depicted, the fixed element 542 serves as a proxy (e.g., a digital copy) for any fixed element in the physical world that may be stored in the passable world module 538 such that the user 530 may perceive content on the fixed element 542 without the system having to map it to the fixed element 542 each time the user 530 sees it. The fixed element 542 may thus be a mesh model that is mapped, determined from a previous modeling session or from a separate user, but is nevertheless stored on the passable world module 538 for future reference by multiple users. The passable world module 538 may thus recognize the environment 532 from a previously mapped environment and display the AR content without the user's 530's device having to first map all or part of the environment 532, saving computational processes and cycles and avoiding latency of any rendered AR content.

[0243] A mapped mesh model 546 of the physical world may be created by the AR display system, and appropriate surfaces and metrics for interacting with and displaying the AR content 540 can be mapped and stored in the passable world module 538 for future retrieval by the user 530 or other users, without the need to remap or model, fully or partially. In some embodiments, the data inputs 536 are inputs such as geographic location, user identification, and current activity to indicate to the passable world module 538 which fixed elements 542 of one or more fixed elements are available, the AR content 540 last placed on the fixed element 542, and whether that same content should be displayed (such AR content is "persistent" content regardless of whether the user is viewing a particular passable world model).

[0244] Even in embodiments where an object is considered fixed (e.g., a kitchen table), the passable world module 538 may be updated from time to time to account for possible changes in the physical world. Models of fixed objects may be updated at a very low frequency. Other objects in the physical world may be moving or otherwise not considered fixed (e.g., a kitchen chair). To render the AR scene with a realistic feel, the AR system may update the positions of these non-fixed objects at a much higher frequency than those used to update fixed objects. To enable accurate tracking of all of the objects in the physical world, the AR system may draw information from multiple sensors, including one or more image sensors.

[0245] FIG. 34B is a schematic illustration of a viewing optics assembly 548 and associated components. In some embodiments, two eye tracking cameras 550 are directed towards the user's eye 549 to detect metrics of the user's eye 549, such as eye shape, eyelid occlusion, pupil direction, and glints on the user's eye 549. In some embodiments, one of the sensors may be a depth sensor 551, such as a time-of-flight sensor, that emits signals into the world and detects reflections of those signals from nearby objects to determine the distance to a given object. The depth sensor may, for example, quickly determine whether objects have entered the user's field of view, either as a result of the objects' motion or a change in the user's posture. However, information about the location of objects in the user's field of view may alternatively or additionally be collected using other sensors. Depth information may be obtained, for example, from a stereoscopic image sensor or a plenoptic sensor.

[0246] In some embodiments, the world camera 552 records a wider view than the surroundings, maps the environment 532, and detects inputs that may affect the AR content. In some embodiments, the world camera 552 and / or the camera 553 may be grayscale and / or color image sensors that may output grayscale and / or color image frames at fixed time intervals. The camera 553 may also capture a physical world image within the user's field of view at a specific time. The pixels of a frame-based image sensor may be repeatedly sampled even if their values ​​are immutable. The world camera 552, the camera 553, and the depth sensor 551 each have a separate field of view 554, 555, and 556, respectively, and collect and record data from a physical world scene, such as the physical world environment 532 depicted in FIG. 34A.

[0247] An inertial measurement unit 557 may determine the movement and orientation of the viewing optics assembly 548. In some embodiments, each component is operably coupled to at least one other component. For example, the depth sensor 551 is operably coupled to the eye tracking camera 550 as a check of the measured accommodation to the actual distance the user's eyes 549 are looking at.

[0248] It should be understood that the viewing optics assembly 548 may include some of the components illustrated in FIG. 34B, or may include components instead of or in addition to the components illustrated. In some embodiments, for example, the viewing optics assembly 548 may include two world cameras 552 instead of four. Alternatively, or in addition, the cameras 552 and 553 need not capture visible light images of their full field of view. The viewing optics assembly 548 may include other types of components. In some embodiments, the viewing optics assembly 548 may include one or more dynamic visual sensors (DVS), whose pixels may asynchronously respond to relative changes in light intensity exceeding a threshold.

[0249] In some embodiments, the viewing optics assembly 548 may not include a depth sensor 551 based on time-of-flight information. In some embodiments, for example, the viewing optics assembly 548 may include one or more plenoptic cameras, whose pixels may capture the light intensity and angle of incident light, from which depth information can be determined. For example, the plenoptic camera may include an image sensor overlaid with a transmissive diffractive mask (TDM). Alternatively, or in addition, the plenoptic camera may include an image sensor containing angle-sensing pixels and / or phase-detection autofocus pixels (PDAF) and / or a microlens array (MLA). Such sensors may serve as a depth information source instead of, or in addition to, the depth sensor 551.

[0250] It should also be understood that the configuration of components in FIG. 34B is illustrated as an example. Viewing optics assembly 548 may include components with any suitable configuration, which may be configured to provide the user with the maximum field of view practical for a particular set of components. For example, if viewing optics assembly 548 has one world camera 552, the world camera may be located in a central region of the viewing optics assembly instead of on the side.

[0251] Information from sensors in the viewing optics assembly 548 may be coupled to one or more of the processors in the system. The processors may generate data that may be rendered to cause the user to perceive the virtual content as interacting with objects in the physical world. The rendering may be implemented in any suitable manner, including generating image data that depicts both the physical and virtual objects. In other embodiments, the physical and virtual content may be depicted in a scene by modulating the opacity of a display device through which the user sees the physical world. The opacity may be controlled to create the appearance of virtual objects and block the user from seeing objects in the physical world that are occluded by the virtual objects. In some embodiments, the image data may include only the virtual content, which may be modified (e.g., clipping the content and taking occlusion into account) so that the virtual content is perceived by the user to interact realistically with the physical world when viewed through the user interface. Regardless of how the content is presented to the user, a model of the physical world is required so that features of the virtual objects that may be affected by the physical objects, including the shape, position, motion, and visibility of the virtual objects, may be correctly calculated. In some embodiments, the model may include a reconstruction of the physical world, such as reconstruction 518.

[0252] The model may be created from data collected from sensors on the user's wearable device, although in some embodiments the model may be created from data collected by multiple users, which may be aggregated in a computing device remote from all users (and may be "in the cloud").

[0253] The model may be created, at least in part, by a world reconstruction system, such as the world reconstruction component 516 of FIG. 32, depicted in further detail in FIG. 35A. The world reconstruction component 516 may include a perception module 660, which may generate, update, and store representations for portions of the physical world. In some embodiments, the perception module 660 may represent a portion of the physical world within the reconstruction range of a sensor as a plurality of voxels. Each voxel corresponds to a 3D cube of a predetermined volume within the physical world and may include surface information and indicate whether a surface exists within the volume represented by the voxel. A voxel may be assigned a value indicating whether its corresponding volume has been determined to include a surface of a physical object, has been determined to be empty, or has not yet been measured with a sensor, and thus its value is unknown. It should be understood that values ​​indicative of voxels determined to be empty or unknown need not be explicitly stored, and that the values ​​of voxels may be stored in computer memory in any suitable manner, including not storing information regarding voxels determined to be empty or unknown.

[0254] In addition to generating information for the persisted world representation, perception module 660 may identify and output indications of changes in the area surrounding a user of the AR system. Such indications of changes may trigger updates to volumetric data stored as part of the persisted world, or trigger other functions, such as triggering generate AR content, update AR content, component 604.

[0255] In some embodiments, the perception module 660 may identify changes based on a signed distance function (SDF) model. The perception module 660 may be configured to receive sensor data, such as a depth map 660a and a head pose 660b, and then fuse the sensor data into an SDF model 660c. The depth map 660a may provide the SDF information directly, or images may be processed to arrive at the SDF information. The SDF information represents distances from sensors used to capture that information. Since those sensors may be part of the wearable unit, the SDF information may represent the physical world from the perspective of the wearable unit, and therefore the user. The head pose 660b may enable the SDF information to be related to voxels in the physical world.

[0256] In some embodiments, the perception module 660 may generate, update, and store representations for portions of the physical world that are within a perception range. The perception range may be determined, at least in part, based on the reconstruction range of the sensor, which may be determined, at least in part, based on the limits of the observation range of the sensor. As a specific example, an active depth sensor that operates using active IR pulses may operate reliably over a range of distances, creating an observation range for the sensor that may be a few centimeters or tens of centimeters to several meters.

[0257] The world reconstruction component 516 may include additional modules that may interact with the perception module 660. In some embodiments, the persisted world module 662 may receive a representation for the physical world based on data obtained by the perception module 660. The persisted world module 662 may also include representations of the physical world in various formats. In some embodiments, the representation of the physical world may be images, features from images, feature descriptors, keyframes / keyrigs, persisted poses, PCFs, and / or global feature strings. For example, volumetric metadata 662b, such as voxels, may be stored along with the meshes 662c and planes 662d. In some embodiments, other information, such as depth maps, may also be saved.

[0258] In some embodiments, the perception module 660 may include modules that generate representations for the physical world in various formats, including, for example, mesh 660d, planar, and semantic 660e. These modules may generate the representations based on data within the perception range of one or more sensors at the time the representations are generated and data captured at previous times and information in the persisted world module 662. In some embodiments, these components may operate on depth information captured using a depth sensor. However, the AR system may also include a vision sensor and generate such representations by analyzing monocular or binocular visual information.

[0259] In some embodiments, these modules may operate on regions of the physical world. They may be triggered to update a subregion of the physical world when the perception module 660 detects a change in the physical world in that subregion. Such a change may be detected, for example, by detecting a new surface in the SDF model 660c, or by other criteria, such as a change in the values ​​of a sufficient number of voxels representing the subregion.

[0260] The world reconstruction component 516 may include components 664 that may receive a representation of the physical world from the perception module 660. Information about the physical world may be pulled by these components, for example, according to usage requests from applications. In some embodiments, information may be pushed to the usage components, such as via indications of changes in pre-identified areas or changes in the physical world representation within a perception range. The components 664 may include game programs and other components that implement processing for, for example, visual occlusion, physics-based interactions, and environmental inference.

[0261] In response to a query from component 664, perception module 660 may transmit a representation for the physical world in one or more formats. For example, when component 664 indicates the use is for visual occlusion or physics-based interaction, perception module 660 may transmit a representation of a surface. When component 664 indicates the use is for environmental inference, perception module 660 may transmit meshes, planes, and semantics of the physical world.

[0262] In some embodiments, perception module 660 may include a component that provides formatting information to component 664. An example of such a component may be a ray casting component 660f. A usage component (e.g., component 664) may, for example, query information about the physical world from a particular viewpoint. The ray casting component 660f may select from one or more representations of the physical world data within a field of view from that viewpoint.

[0263] As should be understood from the foregoing description, the perception module 660 or another component of the AR system may process data and create a 3D representation of a portion of the physical world. The data to be processed may be reduced, at least in part, by thinning out a portion of the 3D reconstruction volume based on the camera frustum and / or depth image, extracting and persisting planar data, capturing, persisting, and updating the 3D reconstruction data in a block that enables local updates while maintaining neighborhood consistency, where occlusion data is derived from a combination of one or more depth data sources, providing occlusion data to an application that generates such a scene, and / or performing multi-stage mesh simplification.

[0264] The world reconstruction system may integrate sensor data from multiple viewpoints of the physical world over time. The pose (e.g., position and orientation) of the sensor may be tracked as the device containing the sensor is moved. As the frame pose of the sensor and how it relates to other poses is understood, each of these multiple viewpoints of the physical world may be fused together into a single combined reconstruction. The reconstruction may be more complete and less noisy than the original sensor data by using spatial and temporal averaging (i.e., averaging data from multiple viewpoints over time) or any other suitable method.

[0265] The reconstruction may contain data of different levels of sophistication, including, for example, raw data such as live depth data, fused volumetric data such as voxels, and calculated data such as meshes.

[0266] The passable world (PW) (e.g., PW538) may include an environment map that is created at least in part based on data reconstructed by a world reconstruction system of an AR device worn by multiple users. During a session of an AR experience, each AR device (such as the wearable device described above in connection with FIG. 33) may create a tracking map (TM). In some embodiments, the tracking map may include a sparse map of the head pose of the AR device scanning the environment and information about objects detected within the environment. The AR device may transmit the tracking map to the cloud and merge it with an environment map, for example, selected from the environment maps stored in the cloud. In some embodiments, the selected environment map may be transmitted from the cloud to the AR device for merging.

[0267] An environment map may provide information about a portion of the physical world represented by the reconstruction data processed to create a separate map. FIG. 36 depicts an example environment map 700, according to some embodiments. The environment map 700 may include a plan view 706 of a reconstructed physical object in the corresponding physical world represented by the point 702. The environment map 700 may also include a tracking route 710 along which a device is traveled to collect data related to the point 702. The tracking route 710 may include a key rig 704, which may represent a location where the AR device uploaded the reconstruction data and associated metadata, such as head pose and / or image data. In some embodiments, a key rig may also be referred to as a key frame. In the illustrated example, the environment map includes two tracking routes 710a and 710b, which may indicate that the environment map 700 is created by merging two tracking maps, i.e., one created along the tracking route 710a and the other created along the tracking route 710b. In some embodiments, the two tracking maps may be created by two different AR devices at the same time, by two different AR devices at different times, or by the same AR device at different times. In some embodiments, the two tracking maps may be created by one AR device and uploaded to the cloud through two network access points, each of which may be accessible within a designated area range.

[0268] The environment map may have any of a number of formats, depending on, for example, the storage location of the environment map, including, for example, the local storage and the remote storage of the AR device. For example, the map in the remote storage may have a higher resolution than the map in the local storage. To transmit the higher resolution map from the remote storage to the local storage, the map may be downsampled or otherwise converted to a suitable format. In some embodiments, a slice or portion of the high resolution map from the remote storage is transmitted to the local storage, and the slice or portion is not downsampled.

[0269] The database of environment maps may be updated as new tracking maps are created. To determine which of the potentially numerous environment maps in the database should be updated, the updating step may include efficiently selecting one or more environment maps stored in the database that are associated with the new tracking map. The selected one or more environment maps may be ranked by relevance, and one or more of the highest ranked maps may be selected for processing to merge the new tracking map with the higher ranked selected environment map and create one or more updated environment maps.

[0270] FIG. 37 depicts an AR system 800 configured to rank and merge one or more environmental maps according to some embodiments. The AR system may include a passable world model 802 of the AR device. Information for populating the passable world model 802 may come from sensors on the AR device, which may include computer-executable instructions stored in the processor 804 (e.g., local data processing module 570) that may perform some or all of the processing for converting sensor data into a map. Such a map may be a tracking map, such as may be built as sensor data is collected as the AR device operates within an area. Along with the tracking map, area attributes may be provided to indicate the area that the tracking map represents. These area attributes may be coordinates presented as latitude and longitude or geographic location identifiers, such as IDs used by the AR system to represent a location. Alternatively or in addition, the area attributes may be measured characteristics that have a high likelihood of being unique for the area. The area attributes may be derived, for example, from parameters of wireless networks detected within the area. In some embodiments, the area attribute may be associated with a unique address of an access point to which the AR system is nearby and / or connected. For example, the area attribute may be associated with a MAC address or Basic Service Set Identifier (BSSID) of a 5G base station / router, Wi-Fi router, and the like.

[0271] 37, the tracking map may be merged with other maps of the environment. The map ranking portion 806 receives the tracking map from the device PW 802, communicates with the map database 808, and selects and ranks selected environment maps from the map database 808. The selected maps that are ranked higher are sent to the map merging portion 810.

[0272] The map merging portion 810 may perform a merging process on the maps sent from the map ranking portion 806. The merging process may involve merging some or all of the tracked and ranked maps and transmitting the new merged map to the passable world model 812. The passable world model 812 may be a cloud model, which may be shared by multiple AR devices. The passable world model 812 may store or otherwise have access to their environment maps in the map database 808. In some embodiments, when a previously calculated environment map is updated, the previous version of the map may be deleted, removing the outdated map from the database. In some embodiments, when a previously calculated environment map is updated, the previous version of the map may be archived, enabling reading / viewing of the previous version of the environment. In some embodiments, permissions may be set such that only AR systems with certain read / write access may trigger the deletion / archiving of the previous version of the map.

[0273] These environment maps created from the tracking maps provided by one or more AR devices / systems may be accessed by the AR devices in the AR system. The map ranking portion 806 may also be used in providing the environment maps to the AR devices. The AR device may send a message requesting an environment map for its current location, and the map ranking portion 806 may be used to select and rank the environment map associated with the requesting device.

[0274] In some embodiments, the AR system 800 may include a downsampling portion 814 configured to receive the merged map from the cloud PW 812. The merged map received from the cloud PW 812 may be in a storage format for the cloud, which may include dense information. The downsampling portion 814 may be configured to downsample the cloud format map to a format suitable for storage on the AR device. The device format map may be sparsely populated to accommodate the limited local computing power and storage space of the AR device.

[0275] U.S. Provisional Patent Application No. 62 / 812,935, entitled "Merging a Plurality of Individually Mapped Environments," describes an exemplary method and apparatus for merging environmental maps. A copy of the filing version of U.S. Application No. 62 / 812,935 is attached as an attachment and is incorporated by reference in its entirety for all purposes. Techniques such as those described in that application may be used to merge environmental maps.

[0276] 38 is a flow chart illustrating a method 900 of selecting one or more ranked environment maps according to some embodiments. In the illustrated embodiment, the ranking step is performed for the user's AR device to create a tracking map. Thus, the tracking map is available for use in ranking the environment maps. In embodiments where a tracking map is not available, some or all of the parts of the selecting and ranking steps of the environment maps that do not explicitly rely on a tracking map may be used.

[0277] The method 900 may begin at act 902, where a set of maps from a database of environmental maps (sometimes referred to as reference maps) in the vicinity of where the tracking map was formed may be accessed and then filtered for ranking. Additionally, at act 902, at least one area attribute is determined for an area in which the user's AR device is operating. In a scenario in which the user's AR device is building a tracking map, the area attribute may correspond to the area over which the tracking map is created. As a specific example, the area attribute may be calculated based on signals received from an access point to a computer network while the AR device was calculating the tracking map.

[0278] 39 depicts an example map ranking portion 806 of an AR system 800, according to some embodiments. The map ranking portion 806 may include a portion of an AR device and a portion of a remote computing system, such as a cloud, and thus may be a cloud computing environment. The map ranking portion 806 may be configured to implement at least a portion of the method 900.

[0279] FIG. 40A depicts an example of area attributes AA1-AA8 of the tracking map (TM) 1102 and the environment maps CM1-CM4 in the database, according to some embodiments. As shown, the environment map may be associated with multiple area attributes. The area attributes AA1-AA8 may include parameters of the wireless network detected by the AR device that calculates the tracking map 1102, such as the basic service set identifier (BSSID) of the network to which the AR device is connected, and / or the strength of the signal of the access point received by the wireless network, for example, through the network tower 1104. The parameters of the wireless network may conform to protocols including Wi-Fi and 5GNR. In the example illustrated in FIG. 9, the area attributes are fingerprints of the areas in which the user AR device collected sensor data and formed the tracking map.

[0280] FIG. 40B depicts an example of a determined geographic location 1106 of a tracking map 1102, according to some embodiments. In the illustrated example, the determined geographic location 1106 includes a centroid point 1110 and an area 1108 surrounding the centroid point. It should be understood that the determination of geographic locations in this application is not limited to the format shown. The determined geographic location may have any suitable format, including, for example, different area shapes. In this example, the geographic location is determined from the area attributes using a database that associates the area attributes with the geographic location. The database is commercially available, for example, a database that associates Wi-Fi fingerprints with locations, expressed as latitude and longitude, that can be used for the present operation.

[0281] 38 embodiment, the map database containing the environmental maps may also include location data for those maps, including the latitude and longitude covered by the maps. Processing in act 902 may involve selecting from that database a set of environmental maps that cover the same latitude and longitude determined for the area attribute of the tracking map.

[0282] Act 904 is a first filtering of the set of environmental maps accessed in act 902. In act 902, environmental maps are retained in the set based on their proximity to the geographic location of the tracking map. This filtering step may be performed by comparing the latitudes and longitudes associated with the tracking map and the environmental maps in the set.

[0283] FIG. 41 depicts an example of act 904, according to some embodiments. Each area attribute may have a corresponding geographic location 1202. The set of environment maps may include environment maps with at least one area attribute that has a geographic location that overlaps with the determined geographic location of the tracking map. In the illustrated example, the set of identified environment maps includes environment maps CM1, CM2, and CM4, each having at least one area attribute that has a geographic location that overlaps with the determined geographic location of the tracking map 1102. Environment map CM3, associated with area attribute AA6, is not included in the set because it is outside the determined geographic location of the tracking map.

[0284] Other filtering steps may also be performed on the set of environmental maps to reduce / rank the number of environmental maps in the set that are ultimately processed (such as for map merging or for providing passable world information to the user device). Method 900 may include filtering the set of environmental maps (act 906) based on similarity of one or more identifiers of network access points associated with the environmental maps of the set of tracking maps and environmental maps. During formation of the map, the devices that collect sensor data and generate the map may be connected to the network through network access points, such as through Wi-Fi or a similar wireless communication protocol. The access points may be identified by a BSSID. The user device may connect to multiple different access points as it moves through an area, collects data, and forms the map. Similarly, when multiple devices provide information to form a map, the devices may be connected through different access points, and thus, for similar reasons, there may be multiple access points used in forming the map. Thus, there may be multiple access points associated with the map, and the set of access points may be an indication of the location of the map.

[0285] In some embodiments, filtering the set of environment maps based on similarity of one or more identifiers of the network access points may include retaining in the set of environment maps the environment map with the highest Jaccard similarity with at least one area attribute of the tracking map based on the one or more identifiers of the network access points. FIG. 42 depicts an example of act 906, according to some embodiments. In the example shown, a network identifier associated with area attribute AA7 may be determined as the identifier for tracking map 1102. The set of environment maps after act 906 includes environment map CM2, which may have an area attribute within a higher Jaccard similarity with AA7, and environment map CM4, which also includes area attribute AA7. Environment map CM1 is not included in the set because it has the lowest Jaccard similarity with AA7.

[0286] The processing in acts 902-906 may be performed without actually accessing the contents of the map stored in the map database based on metadata associated with the map. Other processing may involve accessing the contents of the map. Act 908 illustrates accessing the environmental maps remaining in the subset after filtering based on the metadata. It should be understood that this act may be performed either earlier or later in the process, where subsequent operations may be performed using the accessed content.

[0287] The method 900 may include filtering the set of environment maps based on similarity of a metric representative of the content of the tracking and environment maps of the set (act 910). The metric representative of the content of the tracking and environment maps may include a vector of values ​​calculated from the content of the maps. The metric may be calculated from the maps retrieved in act 908 or may be pre-calculated and stored as metadata associated with the maps. In some embodiments, filtering the set of environment maps based on similarity of a metric representative of the content of the tracking and environment maps of the set of environment maps may include retaining in the set of environment maps an environment map with a minimum vector distance between a vector of characteristics of the tracking map and a vector representing an environment map in the set of environment maps.

[0288] The method 900 may further include filtering the set of environment maps (act 912) based on the degree of matching between the portion of the tracking map and the portion of the environment map of the set of environment maps. The degree of matching may be determined as part of the localization process. As a non-limiting example, localization may be performed by identifying interest points in the tracking map and the environment map that are similar enough that they may represent the same portion of the physical world. In some embodiments, the interest points may be features, feature descriptors, keyframes / keyrigs, sustained poses, and / or PCFs. The set of interest points in the tracking map may then be matched to produce a best match with the set of interest points in the environment map. The mean squared distance between corresponding interest points may be calculated and, if it is below a threshold for a particular region of the tracking map, is used as an indication that the tracking map and the environment map represent the same region of the physical world.

[0289] In some embodiments, filtering the set of environment maps based on the degree of matching between the portion of the tracking map and the portion of the environment map of the set of environment maps may include calculating a volume of the physical world represented by the tracking map that is also represented in the environment map of the set of environment maps, and retaining in the set of environment maps an environment map with a calculated volume that is greater than the filtered out environment map of the set. FIG. 43 depicts an example of act 912, according to some embodiments. In the example shown, the set of environment maps after act 912 includes environment map CM4, which has an area 1402 that matches an area of ​​tracking map 1102. Environment map CM1 is not included in the set because it does not have an area that matches an area of ​​tracking map 1102.

[0290] In some embodiments, the set of environment maps may be filtered in the order of acts 906, acts 910, and acts 912. In some embodiments, the set of environment maps may be filtered based on acts 906, acts 910, and acts 912, which may be performed in an order based on the processing required to perform the filtering from lowest to highest.

[0291] The method 900 may include loading a set of environment maps and data (act 914).

[0292] In the illustrated embodiment, the user database stores an area identification indicating an area in which the AR device was used. The area identification may be an area attribute, which may include parameters of a wireless network detected by the AR device when in use. The map database stores a plurality of environment maps constructed from data provided by the AR device and associated metadata. The associated metadata may include an area identification derived from an area identification of the AR device that provided the data from which the environment map was constructed. The AR device may send a message to the PW module to indicate that a new tracking map is created or is being created. The PW module calculates an area identifier for the AR device and updates the user database based on the received parameters and / or the calculated area identifier. The PW module may also determine an area identifier associated with the AR device requesting the environment map, identify a set of environment maps from the map database based on the area identifier, filter the set of environment maps, and transmit the filtered set of environment maps to the AR device. In some embodiments, the PW module may filter the set of environmental maps based on one or more criteria including, for example, the geographic location of the tracking map, the similarity of one or more identifiers of network access points associated with the tracking map and the environmental maps of the set of environmental maps, the similarity of metrics representing the content of the tracking map and the environmental maps of the set of environmental maps, and the degree of matching between a portion of the tracking map and a portion of the environmental map of the set of environmental maps.

[0293] While several aspects of several embodiments have been described above, it will be appreciated that various alterations, modifications, and improvements will readily occur to those skilled in the art.

[0294] As an example, the embodiments are described in relation to an augmented (AR) environment, it being understood that some or all of the techniques described herein may be applied in MR environments, and more generally in other XR and VR environments.

[0295] As another example, embodiments are described in connection with devices, such as wearable devices. It should be understood that some or all of the techniques described herein may be implemented via a network (such as the cloud), a discrete application, and / or any suitable combination of devices, networks, and discrete applications.

[0296] Further, FIG. 38 provides examples of criteria that may be used to filter the candidate maps and result in a set of highly ranked maps. Other criteria may be used instead of or in addition to the criteria described. For example, if multiple candidate maps have similar values ​​of a metric used to filter out less desirable maps, characteristics of the candidate maps may be used to determine which maps are retained as candidate maps or filtered out. For example, larger or denser candidate maps may be prioritized over smaller candidate maps. In some embodiments, FIGS. 27-28 may describe all or part of the systems and methods described in FIGS. 37-39.

[0297] 51-53 illustrate an example of determining a map to be used based, at least in part, on deep keyframe (DKF) analysis, according to some embodiments. The DKF analysis may be part of a process for determining an environment map to be used by an XR device based on a TM generated by the XR device and / or for determining a current head pose with respect to a user (e.g., the position and / or orientation of the user's head). In some embodiments, the DKF analysis employs suitable deep learning techniques to predict or otherwise determine a suitable environment map to be used by providing keyframes of the TM as input to a deep learning architecture, such as a neural network. In some embodiments, the DKF analysis is used as part of a process for ranking the environment maps, also referred to as criteria maps (CMs). For example, the CMs can be ranked based on the similarity of DKF descriptors between the TM and each CM being analyzed as a candidate. As shown in the example of FIG. 53, the DKF analysis can be part of a hierarchical (e.g., multi-level) analysis for selecting CMs, where geographic location filtering is performed as a (e.g., first) top-level analysis, Wi-Fi BSSID filtering is performed as the next level, and the DKF analysis is performed as a third (e.g., final) level for ranking the CMs. The highest ranked CM or set of CMs may be provided to the XR device and used and / or further analyzed. The ranking analysis may be performed in the cloud (e.g., using a SPIFF analysis module, as shown in the example of FIGS. 51-53). At each level, the set of candidate CMs may be filtered (e.g., reduced) until a single map (or a subset of maps) remains after the bottom-level analysis based on the DKF. The map ranking module shown in FIG. 51 may perform the ranking and pass the results to a map merging module, described above.

[0298] U.S. Patent Application No. 16 / 520,582 and U.S. Provisional Patent Application No. 62 / 702,829 are incorporated herein by reference and describe further details of map sharing based on map quality, simultaneous visibility graphical map sanitization, and creation of locatable maps from camera imagery. Head Pose Tracking Recovery and Reset

[0299] FIG. 49 illustrates aspects of a viewing method for restoring and / or resetting head pose, according to some embodiments. In the illustrated example, in act 1400, the viewing device is powered on. In act 1410, in response to being powered on, a new session is started. In some embodiments, the new session may include establishing a head pose. One or more capture devices on a head-mounted frame affixed to the user's head capture the surfaces of the environment by first capturing images of the environment and then determining the surfaces from the images. The surface data is also combined with data from a gravity sensor to establish the head pose.

[0300] In act 1420, the viewing device's processor enters a routine for head pose tracking. The capture device continues to capture surfaces of the environment and determine the orientation of the head-mounted frame relative to the surfaces as the user moves their head.

[0301] In act 1430, the processor determines whether the head pose has been lost. The head pose may be lost due to "edge" cases such as too many reflective surfaces, low light, bare walls, outdoors, etc., which may result in low feature acquisition, or due to dynamic cases such as a crowd that moves and forms part of the map. The routine at 1430 allows a certain amount of time to pass, for example 10 seconds, to allow sufficient time to determine whether the head pose has been lost. If the head pose has not been lost, the processor returns to 1420 and again goes into tracking the head pose.

[0302] If the head pose is lost in act 1430, the processor enters a routine to restore the head pose in 1440. If the head pose is lost due to low light, a message such as the following message is displayed to the user through the display of the viewing device:

[0303] The system is detecting low light conditions. Move to an area with more light.

[0304] The system will continue to monitor whether sufficient light is available and whether the head pose can be restored. The system may alternatively determine that low texture on the surface is causing the head pose to be lost, in which case the user is given the following prompt in the display as a suggestion to improve the capture of the surface:

[0305] The system is unable to detect surfaces with sufficient fine texture. Move to an area where the surface has less coarse texture and more fine texture.

[0306] In act 1450, the processor enters a routine to determine whether head pose reconstruction has failed. If head pose reconstruction has not failed (i.e., head pose reconstruction was successful), the processor returns to act 1420 by entering head pose tracking again. If head pose reconstruction has failed, the processor returns to act 1410 and establishes a new session. As part of the new session, all cached data is invalidated and henceforth the head pose is newly established.

[0307] 50 shows a schematic representation of a machine in an exemplary form of a computer system 1900 on which a set of instructions for causing the machine to perform any one or more of the methodologies discussed herein may be executed according to some embodiments. In alternative embodiments, the machine may operate as a stand-alone device or may be connected (e.g., networked) to other machines. Furthermore, although only a single machine is illustrated, the term "machine" shall also be taken to include any collection of machines that, individually or together, execute a set (or sets) of instructions to perform any one or more of the methodologies discussed herein.

[0308] The exemplary computer system 1900 includes a processor 1902 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), or both), a main memory 1904 (e.g., read only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM), etc.), and a static memory 1906 (e.g., flash memory, static random access memory (SRAM), etc.), which communicate with each other via a bus 1908.

[0309] The computer system 1900 may further include a disk drive unit 1916 and a network interface device 1920 .

[0310] The disk drive unit 1916 includes a machine-readable medium 1922 on which is stored one or more sets of instructions 1924 (e.g., software) that embody any one or more of the methodologies or functions described herein. Software may also reside, completely or at least partially, within the main memory 1904 and / or within the processor 1902 during its execution by the computer system 1900, the main memory 1904, and the processor 1902, and may also constitute a machine-readable medium.

[0311] The software may also be transmitted or received over the network 18 via the network interface device 1920 .

[0312] The computer system 1900 includes a driver chip 1950 that is used to drive a projector and generate light. The driver chip 1950 includes its own data storage device 1960 and its own processor 1962.

[0313] While the machine-readable medium 1922 is shown in the exemplary embodiment to be a single medium, the term "machine-readable medium" should be taken to include a single medium or multiple media (e.g., centralized or distributed databases and / or associated caches and servers) that store one or more sets of instructions. The term "machine-readable medium" should also be taken to include any medium capable of storing, encoding, or carrying a set of instructions for execution by a machine, causing the machine to perform any one or more of the methodologies of the present invention. The term "machine-readable medium" should thus be taken to include, but is not limited to, solid-state memories, optical and magnetic media, and carrier wave signals.

[0314] While several aspects of several embodiments have been described above, it will be appreciated that various alterations, modifications, and improvements will readily occur to those skilled in the art.

[0315] As an example, the embodiments are described in relation to an augmented (AR) environment, it being understood that some or all of the techniques described herein may be applied in MR environments, or more generally, in other XR and VR environments.

[0316] As another example, embodiments are described in connection with devices, such as wearable devices. It should be understood that some or all of the techniques described herein may be implemented via a network (such as the cloud), a discrete application, and / or any suitable combination of devices, networks, and discrete applications.

[0317] Further, Figure 38 provides examples of criteria that may be used to filter the candidate maps and result in a set of highly ranked maps. Other criteria may be used instead of or in addition to the criteria described. For example, if multiple candidate maps have similar values ​​of a metric used to filter out less desirable maps, characteristics of the candidate maps may be used to determine which maps are retained as candidate maps or filtered out. For example, larger or denser candidate maps may be prioritized over smaller candidate maps.

[0318] Such alterations, modifications, and improvements are intended to be part of this disclosure and are intended to be within the spirit and scope of this disclosure. Additionally, advantages of the disclosure are set forth, but it should be understood that not all embodiments of the disclosure include all described advantages. Some embodiments may not implement any feature described herein as advantageous, and in some instances. Thus, the foregoing description and drawings are by way of example only.

[0319] The foregoing embodiments of the present disclosure can be implemented in any of a number of ways. For example, the embodiments may be implemented using hardware, software, or a combination thereof. When implemented in software, the software code can be executed on any suitable processor or collection of processors, whether provided in a single computer or distributed among multiple computers. Such a processor may be implemented as an integrated circuit, with one or more processors in an integrated circuit component, including commercially available integrated circuit components known in the art, such as a CPU chip, a GPU chip, a microprocessor, a microcontroller, or a coprocessor, to name a few. In some embodiments, the processor may be implemented in a custom circuit, such as an ASIC, or in a semi-custom circuit resulting from configuring a programmable logic device. As a further alternative, the processor may be part of a larger circuit or semiconductor device, whether commercially available, semi-custom, or custom. As a specific example, some commercially available microprocessors have multiple cores, such that one or a subset of those cores may constitute a processor. However, the processor may be implemented using a circuit in any suitable format.

[0320] Further, it should be understood that a computer may be embodied in any of a number of forms, such as a rack-mounted computer, a desktop computer, a laptop computer, or a tablet computer, etc. In addition, a computer may be embodied in devices not generally considered computers but with suitable processing capabilities, including a personal digital assistant (PDA), a smart phone, or any suitable portable or fixed electronic device.

[0321] A computer may also have one or more input and output devices. These devices can be used, among other things, to present a user interface. Examples of output devices that may be used to provide a user interface include a printer or display screen for visual presentation of output, or a speaker or other sound generating device for audible presentation of output. Examples of input devices that may be used for a user interface include keyboards and pointing devices such as mice, touchpads, and digitizing tablets. As another example, a computer may receive input information through speech recognition or in other audible formats. In the illustrated embodiment, the input / output devices are illustrated as physically separate from the computing device. However, in some embodiments, the input and / or output devices may be physically integrated into the same unit as the processor or other elements of the computing device. For example, the keyboard may be implemented as a soft keyboard on a touch screen. In some embodiments, the input / output devices may be completely disconnected from the computing device and functionally integrated through a wireless connection.

[0322] Such computers may be interconnected by one or more networks in any suitable form, including as a local area network or a wide area network, such as an enterprise network or the Internet. Such networks may be based on any suitable technology and operate according to any suitable protocol, and may include wireless networks, wired networks, or fiber optic networks.

[0323] Also, the various methods and processes outlined herein may be coded as software that is executable on one or more processors employing any one of a variety of operating systems or platforms. In addition, such software may be written using any of a number of suitable programming languages ​​and / or programming or scripting tools, and may be compiled as executable machine language code or intermediate code that runs on a framework or virtual machine.

[0324] In this aspect, the present disclosure may be embodied as a computer-readable storage medium (or multiple computer-readable media) (e.g., a computer memory, one or more floppy disks, compact disks (CDs), optical disks, digital video disks (DVDs), magnetic tapes, flash memories, circuitry in field programmable gate arrays or other semiconductor devices, or other tangible computer storage media) encoded with one or more programs that, when executed on one or more computers or other processors, perform methods implementing the various embodiments discussed above. As is evident from the foregoing examples, a computer-readable storage medium may retain information for a sufficient time to provide computer-executable instructions in a non-transitory form. Such a computer-readable storage medium or media may be transportable, as described above, such that one or more programs stored thereon may be loaded onto one or more different computers or other processors to implement various aspects of the present disclosure. As used herein, the term "computer-readable storage medium" encompasses only computer-readable media that may be considered a manufacture (i.e., an article of manufacture) or machine. In some embodiments, the present disclosure may be embodied as a computer-readable medium other than a computer-readable storage medium, such as a propagated signal.

[0325] The term "program" or "software" is used herein in a general sense to refer to any type of computer code or set of computer executable instructions that can be employed to program a computer or other processor to implement various aspects of the present disclosure, as described above. In addition, according to one aspect of the present embodiment, it should be understood that one or more computer programs that, when executed, perform the methods of the present disclosure need not reside on a single computer or processor, but can be distributed in a modular manner among several different computers or processors to implement various aspects of the present disclosure.

[0326] Computer-executable instructions may be in many forms, such as program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Typically the functionality of the program modules may be combined or distributed as desired in various embodiments.

[0327] Also, the data structures may be stored in the computer-readable medium in any suitable form. For ease of illustration, the data structures may be shown to have fields that are related through locations within the data structure. Such relationships may be achieved by allocating storage for the fields with locations in the computer-readable medium that similarly convey the relationship between the fields. However, any suitable mechanism may be used to establish relationships between information in the fields of the data structures, including through the use of pointers, tags, or other mechanisms that establish relationships between data elements.

[0328] Various aspects of the disclosure may be used alone, in combination, or in various arrangements not specifically discussed in the foregoing embodiments, and therefore, its application is not limited to the details and arrangements of components set forth in the foregoing description or illustrated in the drawings. For example, aspects described in one embodiment may be combined in any manner with aspects described in other embodiments.

[0329] The present disclosure may also be embodied as a method, examples of which are provided. The acts performed as part of the method may be ordered in any suitable manner. Thus, while the illustrative embodiments are shown as sequential acts, embodiments may be constructed in which acts are performed in an order different from that shown, which may include performing some acts simultaneously.

[0330] The use of ordinal terms such as "first," "second," "third," etc. in the claims to modify claim elements does not, by itself, imply any priority, precedence, or ordering of one claim element relative to another element, or the temporal order in which acts of a method are performed, but rather the ordinal terms are used solely as markers to distinguish one claim element having a certain name from another element having the same name (due to the use of the ordinal terms) to distinguish between claim elements.

[0331] Also, the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of "including," "comprising," "having," "containing," "with," and variations thereof herein is meant to encompass the items listed thereafter and equivalents and additional items.

Claims

1. An electronic system portable by a user, comprising: one or more sensors configured to capture images of one or more physical objects in a scene, the images being in a first coordinate frame; an application configured to execute computer-executable instructions for rendering virtual content within the scene, the application displaying the virtual content in a second coordinate frame different from the first coordinate frame such that the display of the virtual content is independent of eye rotation within the user's head and / or deformations of the portable electronic system that change the relative positions of the one or more sensors; At least one processor configured to execute computer-executable instructions for providing the virtual content to the application, the computer-executable instructions comprising: determining a local coordinate frame for the virtual content based at least in part on the one or more physical objects in the scene; transforming image data of the virtual content in the local coordinate frame into image data of the virtual content in the second coordinate frame; At least one processor having instructions for performing An electronic system comprising:

2. The first coordinate frame is a world coordinate frame; The electronic system of claim 1 , wherein the world coordinate frame is a first pose of the electronic system when the electronic system is powered on to capture the image.

3. The electronic system of claim 2, wherein the world coordinate frame is aligned with gravity.

4. The electronic system of claim 2, wherein the computer executable instructions include instructions for a world frame determination routine configured to determine the world coordinate frame.

5. The computer-executable instructions include instructions for a persistent frame determination routine configured to determine a persistent coordinate frame (PCF); The electronic system of claim 4 , wherein the world frame determination routine is configured to transform one or more sustained poses into a PCF that is associated with, but distinct from, the world coordinate frame.

6. The computer-executable instructions include instructions for a world surface determination routine configured to output a sustained attitude; The electronic system of claim 5 , wherein the persistent frame determination routine is configured to determine each PCF based on one or more persistent attitudes from the world surface determination routine.

7. The second coordinate frame is a camera coordinate frame; The electronic system of claim 1 , wherein the camera coordinate frame is determined based at least in part on dimensions of the electronic system and one or more orientations of the one or more sensors of the electronic system when capturing the image.

8. The electronic system of claim 1, wherein the local coordinate frame is determined based at least in part on one or more nodes on an outer surface of a prism or bounding box that encloses the virtual content.

9. The method of claim 8, further comprising: converting the image data of the virtual content in the local coordinate frame to the image data of the virtual content in the second coordinate frame. transforming the image data of the virtual content in the local coordinate frame to image data of the virtual content in the first coordinate frame; Transforming the image data of the virtual content in the first coordinate frame into the image data of the virtual content in the second coordinate frame; The electronic system according to any one of claims 1 to 8, comprising:

10. The method of claim 1, further comprising: converting the image data of the virtual content in the first coordinate frame to the image data of the virtual content in the second coordinate frame, the method comprising: Transforming the image data of the virtual content in the first coordinate frame into the image data of the virtual content in a head coordinate frame; Transforming the image data of the virtual content in the head coordinate frame to the image data of the virtual content in the second coordinate frame; Including, The electronic system of claim 9 , wherein the head coordinate frame is determined based at least in part on one or more poses of the one or more sensors of the electronic system when capturing the image.

11. The computer executable instructions:

3. The electronic system of claim 2, comprising instructions for determining a persistent coordinate frame (PCF) based at least in part on the captured images of the one or more physical objects in the scene, such that the PCF is different from but associated with the world coordinate frame.

Images