Cross reality system with fast localization

Abstract

A cross-reality system enables any of multiple devices to efficiently and accurately access previously persisted maps, which are uniform maps of very large environments, and render virtual content defined in relation to those maps. The cross-reality system can rapidly process batches of images acquired using portable devices and determine whether sufficient consistency exists across the batch in the calculated localizations. Processing on at least one image from the batch can determine a coarse localization of the device relative to the map. This coarse localization result can be used in a refined localization to process the image from which it was generated. The coarse localization result can also be selectively propagated to the refined localization process for other images in the batch, allowing the coarse localization process to be skipped for other images.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Provisional Patent Application No. 62 / 981,939, filed February 26, 2020, and entitled “CROSS REALITY SYSTEM WITH FAST LOCALIZATION,” which is incorporated by reference in its entirety under 35 U.S.C. §119(e).

[0002] This application relates generally to cross reality systems. [Background technology]

[0003] A computer may control a human user interface and create a cross-reality (XR) environment in which some or all of the XR environment is generated by the computer as it is perceived by the user. These XR environments may be virtual reality (VR), augmented reality (AR), and mixed reality (MR) environments in which some or all of the XR environment may be generated by the computer, in part, using data that describes the environment. This data may describe virtual objects, for example, that may be rendered so that a user can sense or perceive and interact with the virtual objects as part of the physical world. The user may experience these virtual objects as a result of data being rendered and presented through a user interface device, such as a head-mounted display device. The data may control audio that may be displayed to the user for viewing or played audibly to the user, or may control a tactile (or haptic) interface, allowing the user to experience touch sensations that the user senses or perceives as they feel the virtual objects.

[0004] XR systems can be useful for many applications ranging from scientific visualization, medical training, engineering design and prototyping, remote manipulation and telepresence, and personal entertainment. AR and MR, in contrast to VR, involve one or more objects in association with real objects in the physical world. The experience of virtual objects interacting with real objects greatly enhances the user's enjoyment when using XR systems and also opens up possibilities for a variety of applications that present realistic and easily understandable information about how the physical world can be altered.

[0005] To realistically render virtual content, an XR system may build a representation of the physical world around a user of the system. This representation may be built, for example, by processing images acquired using sensors on a wearable device that forms part of the XR system. In such a system, a user may perform an initialization routine by looking around a room or other physical environment in which the user intends to use the XR system until the system acquires sufficient information for the system to build a representation of that environment. As the system operates and the user moves around the environment or into other environments, sensors on the wearable device may acquire additional information and extend or update the representation of the physical world. Summary of the Invention [Means for solving the problem]

[0006] Aspects of the present application relate to methods and apparatus for providing cross-reality (XR) scenes. The techniques described herein may be used together, separately, or in any suitable combination.

[0007] In some aspects, a system is provided that supports specification of a location of virtual content relative to one or more persistent maps in a database of persistent maps. The system includes a location service comprising a communication component configured to receive information from a portable electronic device about a plurality of sets of features within an image of a three-dimensional (3D) environment of the portable electronic device, and at least one processor configured to execute computer-executable instructions. The computer-executable instructions implement a location component, coupled to the communication component, configured to calculate a plurality of candidate location determinations for the plurality of sets of features by performing a coarse location determination of the first set of features relative to the one or more persistent maps in the database of persistent maps with respect to a first set of features of the plurality of sets of features, the performing the coarse location determination including calculating a first coarse transformation of the portable electronic device relative to the persistent map in the database of persistent maps. For the first set of features, a refined localization of the first set of features relative to the persistent map is performed, where the step of performing the refined localization of the first set of features includes calculating a first candidate localization of the portable electronic device based on the first coarse transformation. For a second set of features of the plurality of sets of features, a refined localization of the second set of features relative to the persistent map is performed, where the step of performing the refined localization of the second set of features includes calculating a second candidate localization of the portable electronic device based on the first coarse transformation. A shared term based localization among the plurality of candidate localizations, including the first candidate localization and the second candidate localization, is selected.

[0008] In some embodiments, the step of performing the coarse localization further includes identifying a first set of features in the first collection of features and identifying a second set of features in the persistent map, where features of the first set of features correspond to individual features of the second set of features based on similarity of at least one characteristic.

[0009] In some embodiments, at least one characteristic of the feature comprises a feature descriptor calculated from pixels in the image that depict the feature.

[0010] In some embodiments, the feature descriptor comprises deep sparse features (DSF).

[0011] In some embodiments, the step of performing coarse localization further includes the steps of selecting a first subset of the first set of features and a second subset of the second set of features such that multiple candidate coarse transformations and associated measures of error are calculated for multiple iterations; determining a candidate coarse transformation between the first subset of features and the second subset of features based on a measure of alignment error between the first subset and the second subset transformed with the candidate transformations; selecting a selected candidate coarse transformation from among the multiple candidate coarse transformations based on the associated measure of alignment error; and providing a first coarse transformation based on the selected candidate coarse transformation.

[0012] In some embodiments, the selected localization comprises a localization transformation between a local coordinate frame of the portable electronic device and a coordinate frame of the persistent map, and the localization component is further configured to transmit the localization transformation to the portable electronic device.

[0013] In some embodiments, each of the multiple sets of features comprises a posed feature rig.

[0014] In some embodiments, the pose-equipped feature rig comprises a plurality of feature descriptors and a pose associated with each of the plurality of feature descriptors.

[0015] In some embodiments, the computer-executable instructions implementing the localization component further comprise instructions for determining whether to perform refined localization of the second set of features based on satisfaction of one or more criteria.

[0016] In some embodiments, the computer-executable instructions implementing the localization component further comprise instructions for performing a refined localization of the second set of features based on a second coarse transformation calculated from the second set of features when the one or more criteria are not satisfied.

[0017] In some embodiments, the one or more criteria comprise a disparity above a threshold amount between the image from which the first set of features was extracted and the image from which the second set of features was extracted.

[0018] In some embodiments, the one or more criteria includes calculating a confidence in the first candidate location location above a first threshold.

[0019] In some embodiments, calculating the first candidate location determination includes rejecting a location determination with a confidence below a second threshold as the first candidate location determination.

[0020] In some embodiments, the computer-executable instructions for selecting a location location based on commonalities among a plurality of candidate location locations calculated for a plurality of sets of features include determining whether there are commonalities among a plurality of candidate location locations calculated for a plurality of sets of features, including a second candidate location location, that are above a first threshold, and recalculating a candidate location location for a set of features of the plurality of sets of features, where the candidate location location was calculated based on a rough transformation from a different set of features, based on the determined commonalities that are below the first threshold.

[0021] In some embodiments, the computer-executable instructions for selecting a location location based on commonalities among a plurality of candidate location locations calculated for a plurality of sets of features determine whether there are commonalities among the plurality of candidate location locations calculated for a plurality of sets of features that exceed a second threshold, the plurality of candidate location locations comprising recalculated candidate location locations, the first threshold being higher than the second threshold.

[0022] In some embodiments, determining whether there are any common terms above a first threshold comprises comparing a most frequent value of the plurality of candidate location locations to the first threshold.

[0023] In some embodiments, the step of performing a coarse localization for the first set of features further includes selecting a sub-region of the persistent map based on location metadata associated with the first set of features and the selected sub-region of the persistent map.

[0024] In some embodiments, a portable electronic device configured to operate within a cross reality system in combination with the portable electronic device having a device coordinate frame, the portable electronic device comprising: one or more sensors configured to capture information about the 3D environment, the captured information comprising a plurality of images; and at least one processor configured to execute second computer-executable instructions, the second computer-executable instructions comprising instructions for: extracting, for each image of the plurality of images, a plurality of features and associating each of the plurality of features with pose information in the device coordinate frame to generate information about a plurality of sets of features, including transmitting the information about the plurality of extracted features and the associated pose information to a location service over a network; and receiving from the location service a transformation relating the device coordinate frame to the second coordinate frame.

[0025] In some embodiments, the portable electronic device further comprises a buffer, and the computer executable instructions on the portable electronic device further comprise instructions for storing information about the set of extracted features and associated pose information for each of the plurality of images in the buffer, and transmitting the information about the plurality of sets of features over the network, including transmitting the contents of the buffer.

[0026] In some embodiments, the portable electronic device comprises a display, and the computer-executable instructions on the portable electronic device further comprise instructions for rendering virtual content having a location defined in a second coordinate frame on the display at a position calculated based at least in part on the received transformation.

[0027] In some embodiments, the information about the multiple extracted features comprises a descriptor for each feature.

[0028] In some aspects, a method of calculating a pose between a first component and a map is implemented in which the pose of the first component is represented by a plurality of sets of features oriented relative to a coordinate frame of the first component. The method includes calculating a plurality of candidate localizations based on a respective set of features of the plurality of sets of features, and determining a localization based on the plurality of candidate localizations. The calculating a plurality of candidate localizations includes calculating a first coarse localization for a set of features in the first portion of the plurality of sets of features for a first portion of the plurality of sets of features, and calculating a first refined localization based on the first coarse localization as the first candidate localization of the plurality of candidate localizations. For a second portion of the plurality of sets of features, one or more criteria are evaluated to determine whether to calculate a second refined localization based on the first coarse localization as a second candidate localization of the plurality of candidate localizations.

[0029] In some embodiments, the method further includes calculating a second refined localization based on the first coarse localization as a second candidate localization of the plurality of candidate localizations for a set of features in a second portion of the plurality of sets of features for which it has been determined to calculate a second refined localization based on the first coarse localization.

[0030] In some embodiments, the method further includes calculating a second coarse localization for a set of features in a second portion of the plurality of sets of features for which it was determined not to calculate a second refined localization based on the first coarse localization, and calculating a second refined localization based on the second coarse localization as a second candidate localization of the plurality of candidate localizations.

[0031] In some embodiments, evaluating the one or more criteria to determine whether to calculate a second refined localization based on the first coarse localization includes determining whether to calculate a second refined localization based on the first coarse localization based on a disparity between an image from which the set of features in the first portion was extracted and an image from which the set of features in the second portion was extracted exceeding a threshold.

[0032] In some embodiments, evaluating the one or more criteria to determine whether to calculate a second refined localization based on the first coarse localization includes determining whether to calculate a second refined localization based on the first coarse localization based on a measure of error for the first refined localization for a set of features in the first portion being below a threshold.

[0033] In some embodiments, determining a location based on the plurality of candidate location determinations includes evaluating the plurality of candidate location determinations for shared terms based on a shared term criterion depending on whether a second candidate location determination for the set of features in the second portion was calculated as a second refined location determination based on the first rough location determination.

[0034] In some embodiments, the multiple sets of features comprise a batch of posed feature rigs received from a portable electronic device.

[0035] In some embodiments, the plurality of sets of features comprises a posed feature rig within a tracking map formed by the portable electronic device.

[0036] Some embodiments include selecting a map from a database of stored maps based on location metadata included with the batch's pose-feature rig.

[0037] In some embodiments, calculating a first coarse localization for the set of features in the first portion includes identifying a set of features in the set of features in the first portion that correspond to features in the map, iteratively forming subsets of the corresponding set of features and determining a candidate coarse transformation and associated accuracy for each subset, and selecting a coarse transformation from among the iteratively formed candidate coarse transformations based on the determined accuracy of the selected coarse transformation.

[0038] In some embodiments, calculating the first refined localization based on the first coarse localization includes searching for a transformation that matches a set of features identified in the collection of features in the first portion with corresponding features in the map, the searching being constrained around the coarse transformation calculated for the first coarse localization.

[0039] In some aspects, a non-transitory computer-readable medium is provided that comprises computer-executable instructions that, when executed by at least one processor, are configured to implement a method for calculating an pose between a first component and a map. The pose of the first component is represented by a plurality of sets of features oriented relative to a coordinate frame of the first component. The method includes calculating a plurality of candidate localizations based on individual sets of features of the plurality of sets of features, and determining a localization based on the plurality of candidate localizations. The calculating a plurality of candidate localizations includes calculating a first coarse localization for a set of features in the first portion of the plurality of sets of features for a first portion of the plurality of sets of features, and calculating a first refined localization based on the first coarse localization as the first candidate localization of the plurality of candidate localizations. For a second portion of the plurality of sets of features, one or more criteria are evaluated to determine whether to calculate a second refined localization based on the first coarse localization as a second candidate localization of the plurality of candidate localizations.

[0040] The above description is provided by way of illustration and is not intended to be limiting. The present invention provides, for example, the following: (Item 1) 1. A system that supports specification of a location of virtual content relative to one or more persistent maps in a database of persistent maps, the system comprising a location service, the location service comprising: a communication component configured to receive, from a portable electronic device, information about a plurality of sets of features within an image of a three-dimensional (3D) environment of the portable electronic device; at least one processor configured to execute computer-executable instructions to implement a location determination component coupled to the communication component, the location determination component comprising: With respect to a first set of features of the plurality of sets of features, performing a coarse location of the first set of features relative to one or more persistent maps in the database of persistent maps, where performing the coarse location includes calculating a first coarse transformation of the portable electronic device relative to a persistent map in the database of persistent maps; performing a refined localization of the first set of features relative to the persistent map, where performing a refined localization of the first set of features includes calculating a first candidate localization of the portable electronic device based on the first coarse transformation; With respect to a second set of features of the plurality of sets of features, performing a refined localization of the second set of features relative to the persistent map, the performing a refined localization of the second set of features including calculating a second candidate localization of the portable electronic device based on the first coarse transformation; calculating a plurality of candidate localizations for the plurality of sets of features by selecting a location location based on a commonality among the plurality of candidate location locations, including the first candidate location location and the second candidate location location; at least one processor configured to A system comprising: (Item 2) Performing a rough localization further comprises: identifying a first set of features within the first collection of features; identifying a second set of features in the persistence map, wherein features of the first set of features correspond to individual features of the second set of features based on similarity of at least one characteristic; 2. The system according to item 1, comprising: (Item 3) 3. The system of claim 2, wherein at least one characteristic of the feature comprises a feature descriptor calculated from pixels in an image depicting the feature. (Item 4) 4. The system of claim 3, wherein the feature descriptor comprises deep sparse features (DSF). (Item 5) Performing a rough localization further comprises: For multiple iterations, multiple candidate rough transformations and associated measures of error are calculated. selecting a first subset of the first set of features and a second subset of the second set of features; determining a candidate coarse transformation between the first subset of features and the second subset of features based on a measure of a matching error between the first subset transformed using the candidate transformation and the second subset; selecting a selected candidate coarse transformation from among the plurality of candidate coarse transformations based on an associated measure of the alignment error; providing the first coarse transformation based on the selected candidate coarse transformations; and 3. The system according to item 2, comprising: (Item 6) the selected localization comprises a localization transformation between a local coordinate frame of the portable electronic device and a coordinate frame of the persistent map; 2. The system of claim 1, wherein the localization component is further configured to transmit the localization transform to the portable electronic device. (Item 7) 2. The system of claim 1, wherein each of the plurality of sets of features comprises a pose feature rig. (Item 8) 8. The system of claim 7, wherein the pose-attached feature rig comprises a plurality of feature descriptors and a pose associated with each of the plurality of feature descriptors. (Item 9) The computer-executable instructions implementing the location component further include: determining whether to perform a refined localization of the second set of features based on satisfaction of one or more criteria; 2. The system of claim 1, comprising instructions for: (Item 10) The computer-executable instructions implementing the location component further include: performing a refined localization of the second set of features based on a second coarse transformation calculated from the second set of features when the one or more criteria are not satisfied. 10. The system of claim 9, comprising instructions for: (Item 11) Item 11. The system of item 10, wherein the one or more criteria comprises a disparity greater than a threshold amount between an image from which the first set of features was extracted and an image from which the second set of features was extracted. (Item 12) Said one or more criteria may be: Item 11. The system of item 10, further comprising calculating a confidence in the first candidate location location that is above a first threshold. (Item 13) 13. The system of claim 12, wherein calculating the first candidate location determination includes rejecting a location determination with a confidence level less than a second threshold as the first candidate location determination, the first threshold being higher than the second threshold. (Item 14) The computer executable instructions for selecting a location location based on commonalities among the candidate location locations calculated for the plurality of sets of features include: determining whether there are any shared terms above a first threshold among the plurality of candidate localizations calculated for the plurality of sets of features including the second candidate localization; recalculating candidate localizations for a set of features of the plurality of sets of features where the candidate localizations were calculated based on rough transformations from different sets of features based on the determined commonality terms being below the first threshold. Item 11. The system according to item 10. (Item 15) The computer executable instructions for selecting a location location based on commonalities among the candidate location locations calculated for the plurality of sets of features include: 15. The system of claim 14, further comprising: determining whether there are shared terms among the plurality of candidate localizations calculated for the plurality of sets of features that exceed a second threshold, the plurality of candidate localizations comprising the recalculated candidate localizations, and the first threshold being higher than the second threshold. (Item 16) 15. The system of claim 14, wherein determining whether there are any common terms that exceed the first threshold comprises comparing a most frequent value of the plurality of candidate location determinations to the first threshold. (Item 17) Performing a rough location determination for the first set of features further includes: selecting a sub-region of the persistent map based on location metadata associated with the first set of features and the selected sub-region of the persistent map. 2. The system according to item 1, comprising: (Item 18) In combination with a portable electronic device configured to operate within a cross reality system, the portable electronic device having a device coordinate frame, the portable electronic device comprising: one or more sensors configured to capture information about the 3D environment, the captured information comprising a plurality of images; At least one second processor configured to execute second computer-executable instructions, the second computer-executable instructions comprising: extracting a plurality of features for each of the plurality of images; Associating pose information in the device coordinate frame with each of the plurality of features; generating information about the plurality of sets of features by transmitting information about the plurality of sets of features over a network to a location service, including transmitting information about the plurality of extracted features and the associated pose information; receiving from the location service a transformation relating the device coordinate frame to a second coordinate frame; at least one second processor having instructions for performing Item 2. The system of item 1, comprising: (Item 19) The portable electronic device further comprises a buffer; The computer executable instructions for the portable electronic device further include: storing, for each of the plurality of images in the buffer, information about the set of extracted features and the associated pose information; transmitting information about the plurality of sets of characteristics over the network, including transmitting the contents of the buffer; and 20. The system of claim 18, comprising instructions for performing the steps of: (Item 20) the portable electronic device comprises a display; 20. The system of claim 18, wherein the computer-executable instructions of the portable electronic device further comprise instructions for rendering virtual content having a location defined within the second coordinate frame on the display at a position calculated based at least in part on the received transformation. (Item 21) 20. The system of claim 18, wherein the information about the plurality of extracted features comprises a descriptor for each feature. (Item 22) 1. A method of computing a pose between a first component and a map, the pose of the first component being represented by a plurality of sets of features oriented relative to a coordinate frame of the first component, the method comprising: calculating a plurality of candidate localizations based on respective sets of features of the plurality of sets of features; determining a location location based on the plurality of candidate location locations; and Including, Calculating the plurality of candidate localizations includes: With respect to a first portion of the plurality of sets of features, calculating a first coarse localization for a set of features within a first portion of the plurality of sets of features; calculating a first refined localization based on the first coarse localization as a first candidate localization of the plurality of candidate localizations; With respect to a second portion of the plurality of sets of features, evaluating one or more criteria to determine whether to calculate a second refined location determination based on the first coarse location determination as a second candidate location determination of the plurality of candidate location determinations; A method comprising: (Item 23) for a set of features in a second portion of the plurality of sets of features for which it is determined to calculate the second refined localization based on the first coarse localization, calculating the second refined location determination based on the first coarse location determination as a second candidate location determination of the plurality of candidate location determinations; 23. The method of claim 22, further comprising: (Item 24) for a set of features in a second portion of the plurality of sets of features for which it is determined not to calculate the second refined localization based on the first rough localization, calculating a second coarse localization for a set of features in a second portion of the plurality of sets of features for which it has been determined not to calculate the second refined localization based on the first coarse localization; calculating the second refined location determination based on the second coarse location determination as a second candidate location determination of the plurality of candidate location determinations; 24. The method of claim 23, further comprising: (Item 25) Evaluating the one or more criteria to determine whether to calculate the second refined location based on the first coarse location includes: determining whether to calculate the second refined localization based on the first coarse localization based on a disparity between an image from which the set of features in the first portion was extracted and an image from which the set of features in the second portion was extracted exceeding a threshold; 24. The method according to item 23, comprising: (Item 26) Evaluating the one or more criteria to determine whether to calculate the second refined location based on the first coarse location includes: determining whether to calculate the second refined location based on the first coarse location based on a measure of error for the first refined location for a set of features within the first portion being below a threshold; 24. The method according to item 23, comprising: (Item 27) Determining the location location based on the plurality of candidate location locations includes: evaluating the plurality of candidate localizations for common terms based on a common term criterion depending on whether the second candidate localization for a set of features within the second portion was calculated as the second refined localization based on the first rough localization. 23. The method according to item 22, comprising: (Item 28) 23. The method of claim 22, wherein the plurality of sets of features comprises a batch of posed feature rigs received from a portable electronic device. (Item 29) 23. The method of claim 22, wherein the plurality of sets of features comprises a posed feature rig within a tracking map formed by the portable electronic device. (Item 30) 30. The method of claim 28, further comprising selecting the map from a database of stored maps based on location metadata included with the batch of pose-equipped feature rigs. (Item 31) Calculating the first coarse localization with respect to a set of features within the first portion includes: identifying a set of features in the collection of features in the first portion that correspond to features in the map; iteratively forming subsets of the set of corresponding features and determining a candidate rough transformation and associated accuracy for each subset; selecting a coarse transformation from among said iteratively formed candidate coarse transformations based on the determined accuracy of the selected coarse transformation; 23. The method according to item 22, comprising: (Item 32) Calculating the first refined localization based on the first coarse localization includes: searching for a transformation that matches a set of features identified in a feature collection in the first portion with corresponding features in the map, the searching being constrained around the coarse transformation calculated for the first coarse localization; 32. The method according to claim 31, comprising: (Item 33) 1. A non-transitory computer readable medium comprising computer executable instructions that, when executed by at least one processor, are configured to perform a method for calculating a pose between a first component and a map, the pose of the first component being represented by a plurality of sets of features oriented relative to a coordinate frame of the first component, the method comprising: calculating a plurality of candidate localizations based on respective sets of features of the plurality of sets of features; determining a location location based on the plurality of candidate location locations; and Including, Calculating the plurality of candidate localizations includes: With respect to a first portion of the plurality of sets of features, calculating a first coarse localization for a set of features within a first portion of the plurality of sets of features; calculating a first refined localization based on the first coarse localization as a first candidate localization of the plurality of candidate localizations; With respect to a second portion of the plurality of sets of features, evaluating one or more criteria to determine whether to calculate a second refined location determination based on the first coarse location determination as a second candidate location determination of the plurality of candidate location determinations; 16. A non-transitory computer readable medium comprising: [Brief description of the drawings]

[0041] The accompanying drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component that is illustrated in various figures is represented by a like numeral. For purposes of clarity, not every component is labeled in every figure.

[0042] [Figure 1] FIG. 1 is a sketch illustrating an example of a simplified augmented reality (AR) scene, according to some embodiments.

[0043] [Diagram 2] FIG. 2 is a sketch of an exemplary simplified AR scene illustrating an exemplary use case of an XR system, according to some embodiments.

[0044] [Diagram 3] FIG. 3 is a schematic diagram illustrating data flow for a single user in an AR system configured to provide the user with an experience of AR content that interacts with the physical world, according to some embodiments.

[0045] [Figure 4] FIG. 4 is a schematic diagram illustrating an example AR display system displaying virtual content for a single user, according to some embodiments.

[0046] [Figure 5A] FIG. 5A is a schematic diagram illustrating a user wearing an AR display system that renders AR content as the user moves through a physical world environment, according to some embodiments.

[0047] [Figure 5B] FIG. 5B is a schematic diagram illustrating a viewing optics assembly and associated components, according to some embodiments.

[0048] [Figure 6A] FIG. 6A is a schematic diagram illustrating an AR system using a world reconstruction system, according to some embodiments.

[0049] [Figure 6B] FIG. 6B is a schematic diagram illustrating components of an AR system that maintains a model of a passable world, according to some embodiments.

[0050] [Figure 7] FIG. 7 is a schematic diagram of a tracking map formed by a device traversing a path through the physical world.

[0051] [Figure 8] FIG. 8 is a schematic diagram illustrating a user of a cross-reality (XR) system perceiving virtual content, according to some embodiments.

[0052] [Figure 9] FIG. 9 is a block diagram of components of a first XR device of the XR system of FIG. 8 that convert between coordinate systems, according to some embodiments.

[0053] [Figure 10] FIG. 10 is a schematic diagram illustrating an example transformation of an origin coordinate frame into a destination coordinate frame for correctly rendering local XR content, according to some embodiments.

[0054] [Figure 11] FIG. 11 is a top plan view illustrating a pupil-based coordinate frame, according to some embodiments.

[0055] [Figure 12] FIG. 12 is a top plan view illustrating a camera coordinate frame including all pupil positions according to some embodiments.

[0056] [Figure 13] FIG. 13 is a schematic diagram of the display system of FIG. 9, according to some embodiments.

[0057] [Figure 14] FIG. 14 is a block diagram illustrating the creation of a persistent coordinate frame (PCF) and binding of XR content to the PCF according to some embodiments.

[0058] [Figure 15] FIG. 15 is a flowchart illustrating a method of establishing and using a PCF, according to some embodiments.

[0059] [Figure 16] FIG. 16 is a block diagram of the XR system of FIG. 8 including a second XR device, according to some embodiments.

[0060] [Figure 17] FIG. 17 is a schematic diagram illustrating a room and key frames established for various areas within the room, according to some embodiments.

[0061] [Figure 18] FIG. 18 is a schematic diagram illustrating establishment of sustained poses based on key frames, according to some embodiments.

[0062] [Figure 19] FIG. 19 is a schematic diagram illustrating establishment of a persistent coordinate frame (PCF) based on a persistent attitude, according to some embodiments.

[0063] [Figure 20] 20A-20C are schematic diagrams illustrating examples of creating a PCF, according to some embodiments.

[0064] [Figure 21]FIG. 21 is a block diagram illustrating a system for generating global descriptors for individual images and / or maps, according to some embodiments.

[0065] [Figure 22] FIG. 22 is a flowchart illustrating a method for calculating image descriptors according to some embodiments.

[0066] [Diagram 23] FIG. 23 is a flowchart illustrating a method for localization using image descriptors, according to some embodiments.

[0067] [Figure 24] FIG. 24 is a flowchart illustrating a method of training a neural network, according to some embodiments.

[0068] [Diagram 25] FIG. 25 is a block diagram illustrating a method for training a neural network, according to some embodiments.

[0069] [Figure 26] FIG. 26 is a schematic diagram illustrating an AR system configured to rank and merge multiple environment maps, according to some embodiments.

[0070] [Figure 27] FIG. 27 is a simplified block diagram illustrating multiple criteria maps stored on a remote storage medium, according to some embodiments.

[0071] [Figure 28] FIG. 28 is a schematic diagram illustrating a method of selecting a reference map, e.g., locating a new tracking map within one or more reference maps, and / or obtaining a PCF from a reference map, in accordance with some embodiments.

[0072] [Figure 29] FIG. 29 is a flowchart illustrating a method for selecting a plurality of ranked environment maps according to some embodiments.

[0073] [Diagram 30] FIG. 30 is a schematic diagram illustrating an example map ranking portion of the AR system of FIG. 26, in accordance with some embodiments.

[0074] [Figure 31A] FIG. 31A is a schematic diagram illustrating an example of area attributes of a tracking map (TM) and an environment map in a database, according to some embodiments.

[0075] [Figure 31B] FIG. 31B is a schematic diagram illustrating an example of determining the geographic location of a tracking map (TM) for the geographic location filtering of FIG. 29, according to some embodiments.

[0076] [Diagram 32] FIG. 32 is a schematic diagram illustrating an example of the geographic location filtering of FIG. 29, according to some embodiments.

[0077] [Diagram 33] FIG. 33 is a schematic diagram illustrating an example of the Wi-Fi BSSID filtering of FIG. 29, according to some embodiments.

[0078] [Diagram 34] FIG. 34 is a schematic diagram illustrating an example of the use of the location determination of FIG. 29, according to some embodiments.

[0079] [Diagram 35] 35 and 36 are block diagrams of an XR system configured to rank and merge multiple environment maps, according to some embodiments. [Diagram 36]35 and 36 are block diagrams of an XR system configured to rank and merge multiple environment maps, according to some embodiments.

[0080] [Figure 37] FIG. 37 is a block diagram illustrating a method of creating an environment map of the physical world in a canonical form according to some embodiments.

[0081] [Figure 38A] 38A and 38B are schematic diagrams illustrating an environment map created in a reference configuration by updating the tracking map of FIG. 7 with a new tracking map, according to some embodiments. [Figure 38B] 38A and 38B are schematic diagrams illustrating an environment map created in a reference configuration by updating the tracking map of FIG. 7 with a new tracking map, according to some embodiments.

[0082] [Figure 39-1] 39A-39F are schematic diagrams illustrating examples of merging maps, according to some embodiments. [Figure 39-2] 39A-39F are schematic diagrams illustrating examples of merging maps, according to some embodiments.

[0083] [Diagram 40] FIG. 40 is a two-dimensional representation of a three-dimensional first local tracking map (Map 1), according to some embodiments, which may be generated by the first XR device of FIG.

[0084] [Diagram 41] FIG. 41 is a block diagram illustrating uploading map 1 from a first XR device to the server of FIG. 9 according to some embodiments.

[0085] [Diagram 42]FIG. 42 is a schematic diagram illustrating the XR system of FIG. 16 in accordance with some embodiments, showing a second user starting a second session using a second XR device of the XR system after the first user has ended the first session.

[0086] [Figure 43A] FIG. 43A is a block diagram illustrating a new session for the second XR device of FIG. 42 according to some embodiments.

[0087] [Figure 43B] FIG. 43B is a block diagram illustrating the creation of a tracking map for the second XR device of FIG. 42 according to some embodiments.

[0088] [Figure 43C] FIG. 43C is a block diagram illustrating downloading a reference map from a server to the second XR device of FIG. 42 according to some embodiments.

[0089] [Diagram 44] FIG. 44 is a schematic diagram illustrating localization attempting to locate a second tracking map (Map 2), which may be generated by the second XR device of FIG. 42, relative to a reference map, according to some embodiments.

[0090] [Diagram 45] FIG. 45 is a schematic diagram illustrating localization attempting to locate the second tracking map (Map 2) of FIG. 44 with XR content associated with the PCF of Map 2, which may be further deployed against a reference map, in accordance with some embodiments.

[0091] [Figure 46A] 46A-46B are schematic diagrams illustrating successful location of map 2 of FIG. 45 relative to a reference map, according to some embodiments. [Figure 46B]46A-46B are schematic diagrams illustrating successful location of map 2 of FIG. 45 relative to a reference map, according to some embodiments.

[0092] [Figure 47] FIG. 47 is a schematic diagram illustrating a reference map generated by including one or more PCFs from the reference map of FIG. 46A into Map 2 of FIG. 45, in accordance with some embodiments.

[0093] [Figure 48] FIG. 48 is a schematic diagram illustrating the reference map of FIG. 47 with a further extension of map 2 on a second XR device, according to some embodiments.

[0094] [Figure 49] FIG. 49 is a block diagram illustrating uploading map 2 from a second XR device to a server, according to some embodiments.

[0095] [Figure 50] FIG. 50 is a block diagram illustrating merging Map 2 and a reference map, according to some embodiments.

[0096] [Figure 51] FIG. 51 is a block diagram illustrating the transmission of a new reference map from a server to a first and second XR device according to some embodiments.

[0097] [Figure 52] FIG. 52 is a block diagram illustrating a two-dimensional representation of map 2 and the head coordinate frame of a second XR device referenced to map 2, according to some embodiments.

[0098] [Figure 53] FIG. 53 is a block diagram illustrating adjustments of the head coordinate frame that can occur in six degrees of freedom in two dimensions according to some embodiments.

[0099] [Figure 54] FIG. 54 is a block diagram illustrating a reference map on a second XR device where sounds are localized relative to the PCFs of map 2, according to some embodiments.

[0100] [Figure 55] 55 and 56 are perspective and block diagrams illustrating use of the XR system when a first user has finished a first session and the first user has started a second session using the XR system, according to some embodiments. [Figure 56] 55 and 56 are perspective and block diagrams illustrating use of the XR system when a first user has finished a first session and the first user has started a second session using the XR system, according to some embodiments.

[0101] [Figure 57] 57 and 58 are perspective and block diagrams illustrating the use of an XR system when three users use the XR system simultaneously within the same session, according to some embodiments. [Figure 58] 57 and 58 are perspective and block diagrams illustrating the use of an XR system when three users use the XR system simultaneously within the same session, according to some embodiments.

[0102] [Figure 59] FIG. 59 is a flowchart illustrating a method for restoring and resetting head pose, according to some embodiments.

[0103] [Figure 60] FIG. 60 is a block diagram of a machine, in the form of a computer, that may find application within the system of the present invention, according to some embodiments.

[0104] [Figure 61] FIG. 61 is a schematic diagram of an example XR system in which any of a number of devices may access location services, according to some embodiments.

[0105] [Figure 62] FIG. 62 is an exemplary process flow for operation of a portable device as part of an XR system that provides cloud-based localization, according to some embodiments.

[0106] [Figure 63A] 63A, B, and C are exemplary process flows for cloud-based location determination according to some embodiments. [Figure 63B] 63A, B, and C are exemplary process flows for cloud-based location determination according to some embodiments. [Figure 63C] 63A, B, and C are exemplary process flows for cloud-based location determination according to some embodiments.

[0107] [Figure 64] FIG. 64 is a block diagram of a localization component of an XR system, according to some embodiments.

[0108] [Figure 65] FIG. 65 is an alternative embodiment of a localization component of an XR system, according to some alternative embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0109] Detailed Description Described herein are methods and apparatus for providing XR scenes. To provide a realistic XR experience to multiple users, the XR system must know the location of the user in the physical world in order to properly correlate the location of virtual objects to real objects. The inventors have recognized and appreciated methods and apparatus that are computationally efficient and rapid in locating XR devices even within large and very large environments (e.g., neighborhoods, cities, countries, the world).

[0110] The XR system may build a map of the environment in which the user device operates. The environmental map may be created from image information collected with sensors that are part of the XR device worn by the user of the XR system. Each XR device may develop a local map of its physical environment by integrating information from one or more images collected as the device operates. In some embodiments, the coordinate system of the local map is tied to the device's position and / or orientation when the device first begins scanning the physical world (e.g., starting a new session). That position and / or orientation of the device may change from session to session as the user interacts with the XR system, whether the different sessions are associated with different users, each with their own wearable device with sensors scanning the environment, or the same user using the same device at different times.

[0111] An XR system may implement one or more techniques to enable persistent actions across sessions based on persistent spatial information. The techniques may provide XR scenes for more computationally efficient and immersive experiences for single or multiple users, for example, by allowing persistent spatial information to be created, stored, and retrieved by any of multiple users of the XR system. When shared by multiple users, persistent spatial information provides a more immersive experience because it allows multiple users to experience virtual content at the same location relative to the physical world. Even when used by a single user, persistent spatial information may enable head poses on an XR device to be quickly restored and reset in a computationally efficient manner.

[0112] The persistent spatial information may be represented by a persistent map. The persistent map may be stored in a remote storage medium (e.g., the cloud). After being turned on, the wearable device worn by the user may retrieve an appropriate map that was previously created and stored from the persistent storage device. The previously stored map may be based on data about the environment collected using sensors on the user's wearable device during a previous session. Retrieving the stored map may enable use of the wearable device without completing a scan of the physical world using sensors on the wearable device. Alternatively or additionally, the device may similarly retrieve an appropriate stored map in response to entering a new area of ​​the physical world.

[0113] The stored map may be represented in a canonical form to which the local reference frame on each XR device may be related. In a multi-device XR system, a stored map accessed by one device may have been created and stored by another device and / or may have been constructed by aggregating data about the physical world collected by sensors on multiple wearable devices that pre-exists within at least a portion of the physical world represented by the stored map.

[0114] In some embodiments, persistent spatial information may be represented in a way that can be easily shared among users and among distributed components, including applications. The reference map may provide information about the physical world that may be formatted, for example, as a persistent coordinate frame (PCF). The PCF may be defined based on a set of features recognized in the physical world. The features may be selected such that they are likely to be the same for each user session of the XR system. The PCFs may be sparse, providing less than all of the available information about the physical world, so that they can be efficiently processed and transferred.

[0115] Techniques for processing persistent spatial information may also include creating dynamic maps based on the local coordinate systems of one or more devices. These maps may be sparse maps representing the physical world with feature points detected in the images used in forming the maps. A reference map may be formed by merging multiple maps created by one or more XR devices.

[0116] The relationship between the reference map and the local map for each device may be determined through a localization process that may be performed on each XR device based on a set of reference maps that are selected and transmitted to the device. Alternatively or additionally, the localization service may be provided on a remote processor, such as may be implemented in the cloud.

[0117] Two XR devices with access to the same stored map may both be localized, for example, relative to the stored map. Once localized, the user devices may render virtual content, having a location defined by a fiducial, onto the stored map by translating the location into a frame of reference maintained by the user devices. The user devices may use this local frame of reference to control their displays to render the virtual content within the defined location.

[0118] XR systems may be configured to create, share, and use persistent spatial information with low usage of computational resources and / or low latency to provide a more immersive user experience. To support these operations, the system may use techniques for efficient comparison of spatial information. Such comparison may occur, for example, as part of localization, in which a set of features from a local device are matched to a set of features in a reference map. Similarly, a map merging process may attempt to match one or more sets of features in a tracking map from a device to corresponding features in a reference map.

[0119] Map selection may aid in efficient comparison of spatial information. By selecting a set of candidate maps from an unlimited number of stored maps for comparison with spatial information provided from a user device, the number of computationally intensive comparison operations performed to identify the best match may be reduced. In some embodiments, location metadata may be associated with the maps and / or portions of the maps that may be compared. Candidate maps may be quickly selected based on matching location metadata. Computationally intensive feature matching may then be limited to only the candidate maps, which requires fewer computations than a comparison required across an unlimited number of stored maps.

[0120] In some embodiments, comparison of sets of feature points may also be streamlined, such as through the use of feature descriptors. Descriptors may have numerical values ​​that enable comparison of features, assigned by a trained neural network. Features that are likely to represent the same portion of an object in the physical world are assigned feature descriptors with similar values, such that matching features can be quickly identified based on the descriptors with similarity values.

[0121] In scenarios in which the set of features is derived from the image frames, further processing simplifications can be made by providing a simple mechanism for finding similar frames. The comparison of the feature sets can be limited simply to the set of features derived from the similar frames. The step of finding similar image frames may also be simplified by representing the image frames as a whole by a numerical descriptor. The descriptor may be computed via a transformation that maps the set of features identified in the image to the descriptor.

[0122] Using these and possibly other techniques to reduce the number of comparisons to determine whether two sets of features match, at least some computationally intensive comparisons of the sets of features may be performed. For example, two frames with matching frame descriptors are determined to match only after finding a correspondence with sufficiently low error between sets of features in those frames. Finding a correspondence between sets of features with low error may serve as confirmation that the set of features received from the local device matches the stored map. Alternatively, or in addition, the correspondence may be based on matching sets of feature points such that a transformation between coordinate frames for those sets of feature points may be determined.

[0123] To reduce the processing time spent searching for a match between the sets of features, a rough localization between the two sets may be performed. The rough localization may be used to bound the search space for the refined localization. In this way, both the rough localization and the refined localization may be computationally simple relative to a complete search for a transformation that matches the complete set of features with a matching set of features.

[0124] The inventors have recognized and appreciated techniques for further reduction in the process of matching spatial information as it occurs during the localization process. To ensure sufficient accuracy, localization may be performed based on batches of image frames, each represented by a set of features. Localization results may be determined based on shared terms among the localization results for the batch.

[0125] A reduction in processing time, accompanied by a suitable accuracy in the localization results, may be achieved by selectively performing refined localization for some sets in a batch based on coarse localization for other sets in the batch. Propagating the coarse localization result of a first set and calculating a refined localization for a second set of features may avoid the processing required to calculate a coarse localization for the second set.

[0126] The inventors have further recognized and appreciated criteria that predict favorable accuracy of the localization results when processing is simplified by propagating rough localization results calculated for some sets of features and calculating refined localization results for other sets of features, and when those criteria are not met, localization may be performed without propagating the rough localization for one set in the refined localization for another set.

[0127] The techniques described herein may be used together or separately with and for many types of devices, including wearable or portable devices with limited computing resources, that provide augmented or mixed reality scenes. In some embodiments, the techniques may be implemented by one or more services that form part of an XR system.

[0128] AR System Overview

[0129] 1 and 2 illustrate a scene with virtual content displayed in conjunction with a portion of the physical world. For illustrative purposes, an AR system is used as an example of an XR system. Figures 3-6B illustrate an example AR system including one or more processors, memory, sensors, and a user interface that can operate in accordance with the techniques described herein.

[0130] 1, an outdoor AR scene 354 is depicted in which a user of the AR technology sees a physical world park-like setting 356 featuring people, trees, buildings in the background, and a concrete platform 358. In addition to these items, the user of the AR technology also perceives that they "see" a robotic figure 357 standing on the physical world concrete platform 358, and a flying cartoon-like avatar character 352, which thereby appears to be an anthropomorphic bumblebee, although these elements (e.g., the avatar character 352 and the robotic figure 357) do not exist within the physical world. Due to the extreme complexity of the human visual perception and nervous system, it is difficult to produce AR technology that facilitates a comfortable, natural-feeling, and rich presentation of virtual image elements among other virtual or physical world image elements.

[0131] Such AR scenes may be achieved using a system that allows a user to place AR content in a physical world, determines the location in the physical world map where the AR content was placed, saves the AR scene so that the placed AR content can be reloaded for display in the physical world, for example, during different AR experience sessions, and builds a map of the physical world based on the tracking information, allowing multiple users to share the AR experience. The system may build and update a digital representation of the physical world surfaces around the user. The representation may be used to render the virtual content to appear occluded by the physical object between the user and the rendered location of the virtual content, fully or partially, for placing virtual objects, in physics-based interactions, and for virtual character path planning and navigation, or for other operations in which information about the physical world is used.

[0132] 2 depicts another example of an indoor AR scene 400, according to some embodiments, illustrating an example use case of an XR system. The example scene 400 is a living room with a wall, a bookshelf on one side of the wall, a floor lamp in the corner of the room, a floor, a sofa, and a coffee table on the floor. In addition to these physical items, a user of the AR technology also perceives virtual objects such as an image on the wall behind the sofa, a bird flying through the door, a deer peeking out from the bookshelf, and a decoration in the form of a windmill placed on the coffee table.

[0133] For an image on a wall, the AR technique requires information not only about the surface of the wall, but also about objects and surfaces in the room, such as lamp shapes, that occlude the image to render the virtual object correctly. For a flying bird, the AR technique requires information about all objects and surfaces around the room to render the bird with realistic physics to avoid objects and surfaces or bounce off if the bird collides. For a deer, the AR technique requires information about surfaces, such as the floor or coffee table, to calculate where the deer should be placed. For a windmill, the system may identify it as a separate object from the table and determine that it is movable, while the corners of the shelf or the corners of the wall may be determined to be stationary. Such specificity may be used in the decision regarding the parts of the scene to be used or updated in each of the various operations.

[0134] The virtual object may be placed in a previous AR experience session. When a new AR experience session is started in the living room, the AR technology requires that the virtual object be displayed exactly where it was placed before and be realistically visible from a different perspective. For example, a windmill should be displayed as standing on a book, not drifting above the table, even in a different location without the book. Such drifting may occur if the user's location for the new AR experience session is not accurately located in the living room. As another example, if the user is viewing a windmill from a different perspective than the perspective when the windmill was placed, the AR technology requires the corresponding side of the windmill to be displayed.

[0135] The scene may be presented to the user via a system that includes multiple components, including a user interface that may stimulate one or more user senses, such as vision, hearing, and / or touch. In addition, the system may include one or more sensors that may measure parameters of the physical portion of the scene, including the user's position and / or movement within the physical portion of the scene. Furthermore, the system may include one or more computing devices with associated computer hardware, such as memory. These components may be integrated into a single device or distributed across multiple interconnected devices. In some embodiments, some or all of these components may be integrated into a wearable device.

[0136] 3 depicts an AR system 502 configured to provide an experience of AR content that interacts with a physical world 506, according to some embodiments. The AR system 502 may include a display 508. In the illustrated embodiment, the display 508 may be worn by a user as part of a headset such that the user may wear the display over their eyes, such as a pair of goggles or glasses. At least a portion of the display may be transparent such that the user may observe a see-through reality 510. The see-through reality 510 may correspond to a portion of the physical world 506 within a current viewpoint of the AR system 502, which may correspond to a user's viewpoint when the user is wearing a headset incorporating both the display and sensors of the AR system and obtaining information about the physical world.

[0137] The AR content may also be presented on the display 508, overlaid on the see-through reality 510. To provide accurate interaction between the AR content and the see-through reality 510 on the display 508, the AR system 502 may include a sensor 522 configured to capture information about the physical world 506.

[0138] The sensor 522 may include one or more depth sensors that output depth maps 512. Each depth map 512 may have multiple pixels, each of which may represent a distance to a surface in the physical world 506 in a particular direction relative to the depth sensor. Raw depth data may come from the depth sensors to create the depth maps. Such depth maps may be updated as fast as the depth sensors can form new images, which may be hundreds or thousands of times per second. However, the data may be noisy and incomplete, and may have holes, shown as black pixels on the illustrated depth map.

[0139] The system may include other sensors, such as image sensors. The image sensors may obtain monocular or stereoscopic information that may be processed to represent the physical world in other ways. For example, images may be processed in the world reconstruction component 516 to create a mesh that represents connected portions of objects in the physical world. Metadata about such objects, including, for example, color and surface texture, may also be obtained using the sensors and stored as part of the world reconstruction.

[0140] The system may also obtain information about the user's head pose (or "pose") relative to the physical world. In some embodiments, a head pose tracking component of the system may be used to calculate the head pose in real time. The head pose tracking component may represent the user's head pose in a coordinate frame with six degrees of freedom, including, for example, translation in three perpendicular axes (e.g., forward / back, up / down, left / right) and rotation about three perpendicular axes (e.g., pitch, yaw, and roll). In some embodiments, the sensor 522 may include an inertial measurement unit ("IMU"), which may be used to calculate and / or determine the head pose 514. The head pose 514 for a depth map may indicate the current viewpoint of the sensor capturing the depth map, for example, with six degrees of freedom, although the head pose 514 may be used for other purposes, such as relating image information to a particular portion of the physical world or relating the position of a display worn on the user's head to the physical world.

[0141] In some embodiments, the head pose information may be derived in a manner other than an IMU, such as from an analysis of an object in an image. For example, the head pose tracking component may calculate a relative position and orientation of the AR device with respect to a physical object based on visual information captured by a camera and inertial information captured by an IMU. The head pose tracking component may then calculate the head pose of the AR device, for example, by comparing the calculated relative position and orientation of the AR device with features of the physical object. In some embodiments, the comparison may be made by identifying features in images captured with one or more of sensors 522 that are stable over time, such that changes in the position of these features in images captured over time can be associated with changes in the user's head pose.

[0142] The inventors have realized and appreciated techniques for operating an XR system to provide an XR scene for a more immersive user experience, such as estimating head pose at a frequency of 1 kHz with low usage of computational resources connecting the XR device, which may be configured with four video graphic array (VGA) cameras operating at 30 Hz, one inertial measurement unit (IMU) operating at 1 kHz, the computing power of a single advanced RISC machine (ARM) core, less than 1 GB of memory, and a network with less than 100 Mbps bandwidth. These techniques involve reducing the processing required to generate and maintain maps and estimate head pose, and providing and consuming data with low computational overhead. The XR system may calculate its pose based on matched visual features. U.S. Patent Application No. 16 / 221,065 describes hybrid tracking and is incorporated herein by reference in its entirety.

[0143] In some embodiments, the AR device may build a map from feature points recognized in successive images in a series of image frames captured as the user moves through the physical world with the AR device. Although each image frame may be obtained from a different pose as the user moves, the system may adjust the orientation of features in each successive image frame by matching features of the successive image frames with previously captured image frames to match the orientation of the initial image frame. Translation of the successive image frames may be used to align each successive image frame and match the orientation of the previously processed image frame such that points representing the same feature will match corresponding feature points from the previously collected image frame. The frames in the resulting map may have a common orientation that is established when the first image frame is added to the map. This map may be used to determine the user's pose in the physical world by matching features from the current image frame to the map along with a set of feature points in a common reference frame. In some embodiments, this map may be referred to as a tracking map.

[0144] In addition to enabling tracking of the user's pose within the environment, this map may enable other components of the system, such as a world reconstruction component 516, to determine the location of physical objects relative to the user. The world reconstruction component 516 may receive the depth map 512 and head pose 514 and any other data from the sensors and integrate the data into a reconstruction 518. The reconstruction 518 may be more complete and less noisy than the sensor data. The world reconstruction component 516 may update the reconstruction 518 using spatial and temporal averages of the sensor data from multiple viewpoints over time.

[0145] Reconstruction 518 may include a representation of the physical world in one or more data formats, including, for example, voxels, meshes, planes, etc. Different formats may represent alternative representations of the same portion of the physical world or may represent different portions of the physical world. In the illustrated example, on the left side of reconstruction 518, a portion of the physical world is presented as a global surface, and on the right side of reconstruction 518, a portion of the physical world is presented as a mesh.

[0146] In some embodiments, the map maintained by the head pose component 514 may be sparse with respect to other maps that may be maintained of the physical world. Rather than providing information about the location and possibly other properties of a surface, the sparse map may indicate the location of points of interest and / or structures, such as corners or edges. In some embodiments, the map may include image frames as captured by the sensor 522. These frames may be reduced to features that may represent points of interest and / or structures. Along with each frame, information about the pose of the user from which the frame was obtained may also be stored as part of the map. In some embodiments, all images obtained by the sensor may or may not be stored. In some embodiments, the system may process images as they are collected by the sensor and select a subset of image frames for further calculation. The selection may be based on one or more criteria that limit the addition of information but ensure that the map contains useful information. The system may add new image frames to the map, for example, based on overlap with previous image frames already added to the map, or based on image frames containing a sufficient number of features determined to be likely to represent stationary objects. In some embodiments, a selected image frame or a group of features from a selected image frame may serve as a keyframe for a map, which is used to provide spatial information.

[0147] In some embodiments, the amount of data processed when constructing the map may be reduced by constructing a sparse map with a set of mapped points and key frames, and / or by dividing the map into blocks and enabling block-wise updates, etc. The mapped points may be associated with points of interest in the environment. The key frames may include information selected from the camera capture data. U.S. Patent Application Serial No. 16 / 520,582 describes steps for determining and / or evaluating a localization map and is incorporated herein by reference in its entirety.

[0148] The AR system 502 may integrate sensor data from multiple viewpoints of the physical world over time. The pose (e.g., position and orientation) of the sensor may be tracked as the device containing the sensor is moved. As the frame pose of the sensor and how it relates to other poses is understood, each of these multiple viewpoints of the physical world may be fused together into a single combined reconstruction of the physical world, which may serve as an abstraction layer for the map and provide spatial information. The reconstruction may be more complete and less noisy than the original sensor data by using spatial and temporal averaging (i.e., averaging data from multiple viewpoints over time) or any other suitable method.

[0149] In the embodiment illustrated in Figure 3, the map represents a portion of the physical world in which a user of a single wearable device is present. In that scenario, the head poses associated with frames in the map may be represented as local head poses that indicate an orientation relative to an initial orientation for the single device at the start of a session. For example, the head poses may be tracked relative to an initial head pose when the device is turned on or otherwise operated to scan the environment and build a representation of that environment.

[0150] In combination with content that characterizes that portion of the physical world, the map may include metadata. The metadata may, for example, indicate the capture time of the sensor information used to form the map. The metadata may alternatively or additionally indicate the location of the sensor at the capture time of the information used to form the map. The location may be represented directly, such as with information from a GPS chip, or indirectly, such as with a wireless (e.g., Wi-Fi) signature indicating the strength of signals received from one or more wireless access points while the sensor data was being collected, and / or with an identifier such as the BSSID of a wireless access point to which the user device was connected while the sensor data was collected.

[0151] Reconstruction 518 may be used for AR functions, such as producing a surface representation of the physical world for occlusion handling or physics-based processing. This surface representation may change as the user moves or objects in the physical world change. Aspects of reconstruction 518 may be used by component 520, for example, to produce a changing global surface representation in world coordinates that may be used by other components.

[0152] AR content may be generated, such as by an AR application 504, based on this information. The AR application 504 may be, for example, a game program that performs one or more functions based on information about the physical world, such as visual occlusion, physics-based interactions, and environmental inference. It may perform these functions by querying data in different formats from a reconstruction 518 produced by a world reconstruction component 516. In some embodiments, the component 520 may be configured to output updates as the representation in a region of interest of the physical world changes. The region of interest may be set to approximate a portion of the physical world in the vicinity of a user of the system, such as a portion in the user's field of view, or projected (predicted / determined) to be within the user's field of view.

[0153] The AR application 504 may use this information to generate and update AR content. The virtual portions of the AR content may be presented on the display 508 in combination with the see-through reality 510 to create a realistic user experience.

[0154] In some embodiments, the AR experience may be provided to the user through an XR device, which may be part of a system that may include remote processing and / or remote data storage, and / or may be other wearable display devices worn by other users in some embodiments. FIG. 4 illustrates an example of a system 580 (hereinafter referred to as "system 580") that includes a single wearable device for ease of illustration. The system 580 includes a head-mounted display device 562 (hereinafter referred to as "display device 562") and various mechanical and electronic modules and systems that support the functionality of the display device 562. The display device 562 may be coupled to a frame 564, which is wearable by a user or viewer 560 of the display system (hereinafter referred to as "user 560") and configured to position the display device 562 in front of the eyes of the user 560. According to various embodiments, the display device 562 may be a sequential display. The display device 562 may be monocular or binocular. In some embodiments, display device 562 may be an example of display 508 in FIG.

[0155] In some embodiments, a speaker 566 is coupled to the frame 564 and positioned proximate to the ear canal of the user 560. In some embodiments, another speaker, not shown, is positioned adjacent to another ear canal of the user 560 to provide stereo / adjustable sound control. The display device 562 is operatively coupled, such as by wired leads or wireless connectivity 568, to a local data processing module 570, which may be mounted in a variety of configurations, such as fixedly attached to the frame 564, fixedly attached to a helmet or hat worn by the user 560, built into headphones, or otherwise removably attached to the user 560 (e.g., in a backpack configuration, in a belt-coupled configuration).

[0156] The local data processing module 570 may include a processor and digital memory, such as non-volatile memory (e.g., flash memory), both of which may be utilized to aid in processing, caching, and storing data. The data includes a) data captured from sensors (e.g., which may be operatively coupled to the frame 564 or otherwise attached to the user 560), such as image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, and / or gyroscopes, and / or b) data obtained and / or processed using the remote processing module 572 and / or remote data repository 574, possibly for passing to the display device 562 after processing or retrieval.

[0157] In some embodiments, the wearable device may communicate with remote components. The local data processing module 570 may be operatively coupled to a remote processing module 572 and a remote data repository 574 by communication links 576, 578, such as via wired or wireless communication links, respectively, such that these remote modules 572, 574 are operatively coupled to each other and available as resources to the local data processing module 570. In further embodiments, in addition to or as an alternative to the remote data repository 574, the wearable device may access a cloud-based remote data repository and / or service. In some embodiments, the head pose tracking components described above may be implemented at least in part within the local data processing module 570. In some embodiments, the world reconstruction component 516 in FIG. 3 may be implemented at least in part within the local data processing module 570. For example, the local data processing module 570 may be configured to execute computer-executable instructions to generate a map and / or a physical world representation based, at least in part, on at least a portion of the data.

[0158] In some embodiments, processing may be distributed across local and remote processors. For example, local processing may be used to build a map (e.g., a tracking map) on the user device based on sensor data collected with sensors on the user's device. Such a map may be used by applications on the user's device. In addition, a previously created map (e.g., a reference map) may be stored in the remote data repository 574. If a suitable stored or persistent map is available, it may be used instead of or in addition to a tracking map created locally on the device. In some embodiments, the tracking map may be located relative to the stored map such that a correspondence is established between the tracking map, which may be oriented relative to the position of the wearable device at the time the user turned on the system, and the reference map, which may be oriented relative to one or more persistent features. In some embodiments, the persistent map may be loaded onto the user device and enable the user device to render virtual content without the delays associated with scanning a location to build a tracking map of the user's complete environment from sensor data obtained during the scan. In some embodiments, a user device may access a remote persistent map (eg, stored on the cloud) without the need to download the persistent map onto the user device.

[0159] In some embodiments, spatial information may be communicated from the wearable device to a remote service, such as a cloud service, configured to locate the device and store it in a map maintained on the cloud service. According to one embodiment, the localization process may occur in the cloud, matching the device location to an existing map, such as a reference map, and returning a transformation that links virtual content to the wearable device location. In such an embodiment, the system may avoid communicating a map from a remote resource to the wearable device. Other embodiments are configured for both device-based and cloud-based localization, and may enable functionality, for example, when network connectivity is unavailable or the user chooses not to enable cloud-based localization.

[0160] Alternatively, or in addition, the tracking map may be merged with previously stored maps to enhance or improve the quality of those maps. Processing to determine whether suitable previously created environmental maps are available and / or whether to merge the tracking map with one or more stored environmental maps may occur within the local data processing module 570 or the remote processing module 572.

[0161] In some embodiments, the local data processing module 570 may include one or more processors (e.g., a graphics processing unit (GPU)) configured to analyze and process the data and / or image information. In some embodiments, the local data processing module 570 may include a single processor (e.g., a single-core or multi-core ARM processor), which would limit the computational budget of the local data processing module 570 but enable smaller devices. In some embodiments, the world reconstruction component 516 may generate the physical world representation in real time over a non-predetermined space using less of a computational budget than a single Advanced RISC Machine (ARM) core, such that the remaining computational budget of the single ARM core may be accessed for other uses, such as, for example, extracting meshes.

[0162] In some embodiments, the remote data repository 574 may include a digital data storage facility, which may be available through the Internet or other networking configurations in a "cloud" resource configuration. In some embodiments, all data is stored and all computations are performed in the local data processing module 570, allowing for fully autonomous use from the remote modules. In some embodiments, all data is stored and all or most computations are performed in the remote data repository 574, allowing for smaller devices. The world reconstruction may be stored in whole or in part in this repository 574, for example.

[0163] In embodiments in which data is stored remotely and accessible via a network, the data may be shared by multiple users of the augmented reality system. For example, a user device may upload its tracking map and augment it into a database of environmental maps. In some embodiments, uploading of the tracking map occurs at the end of a user session with the wearable device. In some embodiments, uploading of the tracking map may occur continuously, semi-continuously, intermittently, at predefined times, after a predefined period from a previous upload, or when triggered by an event. A tracking map uploaded by any user device may be used to augment or improve a previously stored map, whether based on data from that user device or any other user device. Similarly, a persistent map downloaded to a user device may be based on data from that user device or any other user device. In this way, a high-quality environmental map may be readily available to a user to improve their experience with the AR system.

[0164] In further embodiments, download of the persistent map may be limited and / or avoided based on localization performed on a remote resource (e.g., in the cloud). In such a configuration, the wearable device or other XR device communicates feature information (e.g., positioning information about the device at the time the feature represented in the feature information is sensed) combined with pose information to the cloud service. One or more components of the cloud service may match the feature information with an individual stored map (e.g., a reference map) and generate a transformation between the tracking map maintained by the XR device and the coordinate system of the reference map. Each XR device, having its tracking map localized relative to the reference map, may render the virtual content precisely at a location defined relative to the reference map based on its own tracking.

[0165] In some embodiments, the local data processing module 570 is operably coupled to a battery 582. In some embodiments, the battery 582 is a removable power source, such as a commercially available battery. In other embodiments, the battery 582 is a lithium ion battery. In some embodiments, the battery 582 includes both an internal lithium ion battery that is rechargeable by the user 560 during periods of non-operation of the system 580, and a removable battery, so that the user 560 may operate the system 580 for longer periods of time without being plugged in and having to charge the lithium ion battery or shutting off the system 580 and replacing the battery.

[0166] FIG. 5A illustrates a user 530 wearing an AR display system that renders AR content as the user 530 moves through a physical world environment 532 (hereafter referred to as “environment 532”). Information captured by the AR system along the user's path of movement may be processed into one or more tracking maps. The user 530 positions the AR display system at a location 534, and the AR display system records ambient information of the passable world (e.g., digital representations of real objects in the physical world that may be stored and updated as the real objects in the physical world change) relative to the location 534. That information may be stored as a pose in combination with images, features, directional audio input, or other desired data. The location 534 is aggregated against data input 536, e.g., as part of a tracking map, and processed by at least a passable world module 538, which may be implemented, e.g., by processing on the remote processing module 572 of FIG. 4. In some embodiments, the passable world module 538 may include a head pose component 514 and a world reconstruction component 516 such that the processed information, in combination with other information about the physical objects used in the rendered virtual content, may indicate the location of the objects in the physical world.

[0167] The passable world module 538 determines, at least in part, where and how the AR content 540 may be placed in the physical world, as determined from the data input 536. The AR content is "placed" in the physical world by presenting both a representation of the physical world and the AR content through the user interface, where the AR content is rendered as if it were interacting with objects in the physical world, and the objects in the physical world are presented as if the AR content were obscuring the user's view of those objects, when appropriate. In some embodiments, the AR content may be placed by appropriately selecting a portion of a fixed element 542 (e.g., a table) from the reconstruction (e.g., reconstruction 518) and determining the shape and location of the AR content 540. As an example, the fixed element may be a table, and the virtual content may be positioned to appear on the table. In some embodiments, the AR content may be placed in a structure in the field of view 544, which may be the current field of view or an estimated future field of view. In some embodiments, the AR content may be persisted to a model 546 (e.g., a mesh) of the physical world.

[0168] As depicted, the fixed element 542 serves as a proxy (e.g., a digital copy) for any fixed element in the physical world that may be stored in the passable world module 538 such that the user 530 may perceive content on the fixed element 542 without the system having to map it to the fixed element 542 each time the user 530 sees it. The fixed element 542 may thus be a mesh model determined from a previous modeling session or from a separate user, but stored by the passable world module 538 for future reference by multiple users. The passable world module 538 may thus recognize the environment 532 from a previously mapped environment and display the AR content without the user's 530's device having to first map all or part of the environment 532, saving computational processes and cycles and avoiding latency of any rendered AR content.

[0169] A mesh model 546 of the physical world may be created by the AR display system, and appropriate surfaces and metrics for interacting with and displaying the AR content 540 can be stored by the passable world module 538 for future retrieval by the user 530 or other users, without the need to recreate the model, either fully or partially. In some embodiments, the data inputs 536 are inputs such as geographic location, user identification, and current activity to indicate to the passable world module 538 which of the one or more fixed elements 542 is available, the AR content 540 last placed on the fixed element 542, and whether that same content should be displayed (such AR content is "persistent" content regardless of whether the user is viewing a particular passable world model).

[0170] Even in embodiments where objects are considered fixed (e.g., a kitchen table), the passable world module 538 may update those objects in the model of the physical world from time to time to account for possible changes in the physical world. The models of fixed objects may be updated very infrequently. Other objects in the physical world may be moving or otherwise not considered fixed (e.g., a kitchen chair). To render the AR scene with a sense of realism, the AR system may update the positions of these non-fixed objects much more frequently than those used to update fixed objects. To enable accurate tracking of all of the objects in the physical world, the AR system may draw information from multiple sensors, including one or more image sensors.

[0171] 5B is a schematic illustration of a viewing optics assembly 548 and associated components. In some embodiments, two eye tracking cameras 550 are pointed towards the user's eye 549 to detect metrics of the user's eye 549, such as eye shape, eyelid occlusion, pupil direction, and glint on the user's eye 549.

[0172] In some embodiments, one of the sensors may be a depth sensor 551, such as a time-of-flight sensor, that emits signals into the world and detects reflections of those signals from nearby objects to determine the distance to a given object. The depth sensor may, for example, quickly determine whether objects have entered the user's field of view, either as a result of the objects' motion or a change in the user's posture. However, information about the location of objects in the user's field of view may alternatively or additionally be collected using other sensors. Depth information may be obtained, for example, from a stereoscopic image sensor or a plenoptic sensor.

[0173] In some embodiments, the world camera 552 records, maps, and / or otherwise models the environment 532 and detects inputs that may affect the AR content. In some embodiments, the world camera 552 and / or the camera 553 may be grayscale and / or color image sensors that may output grayscale and / or color image frames at fixed time intervals. The camera 553 may also capture a physical world image within the user's field of view at a specific time. The pixels of a frame-based image sensor may be repeatedly sampled even if their values ​​are immutable. The world camera 552, the camera 553, and the depth sensor 551 each have a separate field of view 554, 555, and 556, and collect and record data from a physical world scene, such as the physical world environment 532 depicted in FIG. 34A.

[0174] The inertial measurement unit 557 may determine the movement and orientation of the viewing optics assembly 548. In some embodiments, the inertial measurement unit 557 provides an output indicative of the direction of gravity. In some embodiments, each component is operably coupled to at least one other component. For example, the depth sensor 551 is operably coupled to the eye tracking camera 550 as a check of the measured accommodation to the actual distance the user's eye 549 is looking at.

[0175] It should be understood that the viewing optics assembly 548 may include some of the components illustrated in FIG. 34B, or may include components instead of or in addition to the components illustrated. In some embodiments, for example, the viewing optics assembly 548 may include two world cameras 552 instead of four. Alternatively, or in addition, the cameras 552 and 553 need not capture visible light images of their full field of view. The viewing optics assembly 548 may include other types of components. In some embodiments, the viewing optics assembly 548 may include one or more dynamic visual sensors (DVS), whose pixels may asynchronously respond to relative changes in light intensity exceeding a threshold.

[0176] In some embodiments, the viewing optics assembly 548 may not include a depth sensor 551 based on time-of-flight information. In some embodiments, for example, the viewing optics assembly 548 may include one or more plenoptic cameras, whose pixels may capture the light intensity and angle of incident light, from which depth information can be determined. For example, the plenoptic camera may include an image sensor overlaid with a transmissive diffractive mask (TDM). Alternatively, or in addition, the plenoptic camera may include an image sensor containing angle-sensing pixels and / or phase-detection autofocus pixels (PDAF) and / or a microlens array (MLA). Such sensors may serve as a depth information source instead of, or in addition to, the depth sensor 551.

[0177] It should also be understood that the configuration of components in FIG. 5B is provided as an example. Viewing optics assembly 548 may include components with any suitable configuration, which may be configured to provide the user with the maximum field of view practical for a particular set of components. For example, if viewing optics assembly 548 has one world camera 552, the world camera may be located in a central region of the viewing optics assembly instead of on the side.

[0178] Information from sensors in the viewing optics assembly 548 may be coupled to one or more of the processors in the system. The processor may generate data that may be rendered to cause the user to perceive the virtual content as interacting with objects in the physical world. The rendering may be implemented in any suitable manner, including generating image data that depicts both the physical and virtual objects. In other embodiments, the physical and virtual content may be depicted in a scene by modulating the opacity of a display device through which the user sees through the physical world. The opacity may be controlled to create the appearance of the virtual object and block the user from seeing objects in the physical world that are occluded by the virtual object. In some embodiments, the image data may include only the virtual content, which may be modified (e.g., clipping the content and taking into account occlusion) to be perceived by the user as realistically interacting with the physical world when the virtual content is viewed through the user interface.

[0179] The locations on the viewing optics assembly 548 where content may be displayed to create the impression of an object at a particular location may depend on the physics of the viewing optics assembly. Additionally, the pose of the user's head relative to the physical world and the direction the user's eyes are looking will affect the location within the physical world content that is displayed at a particular location on the viewing optics assembly where the content will appear. Sensors as described above may provide information from which this information can be collected and / or calculated such that a processor receiving the sensor input can calculate where objects should be rendered on the viewing optics assembly 548 to create a desired appearance for the user.

[0180] Regardless of how content is presented to a user, a model of the physical world may be used so that characteristics of virtual objects that may be affected by physical objects, including the shape, position, motion, and visibility of the virtual objects, may be correctly calculated. In some embodiments, the model may include a reconstruction of the physical world, such as reconstruction 518.

[0181] The model may be created from data collected from sensors on the user's wearable device, although in some embodiments the model may be created from data collected by multiple users, which may be aggregated in a computing device remote from all users (and may be "in the cloud").

[0182] The model may be created, at least in part, by a world reconstruction system, such as, for example, the world reconstruction component 516 of FIG. 3, which is depicted in further detail in FIG. 6A. The world reconstruction component 516 may include a perception module 660, which may generate, update, and store representations for portions of the physical world. In some embodiments, the perception module 660 may represent a portion of the physical world within the reconstruction range of a sensor as a plurality of voxels. Each voxel corresponds to a 3D cube of a predetermined volume within the physical world and may include surface information and indicate whether a surface exists within the volume represented by the voxel. A voxel may be assigned a value indicating whether its corresponding volume has been determined to include a surface of a physical object, has been determined to be empty, or has not yet been measured with a sensor, and therefore its value is unknown. It should be understood that values ​​indicative of voxels determined to be empty or unknown need not be explicitly stored, and that the values ​​of voxels may be stored in computer memory in any suitable manner, including not storing information regarding voxels determined to be empty or unknown.

[0183] In addition to generating information for the persisted world representation, perception module 660 may identify and output indications of changes in the area surrounding a user of the AR system. Such indications of changes may trigger updates to volumetric data stored as part of the persisted world, or trigger other functions, such as triggering generate AR content, update AR content, component 604.

[0184] In some embodiments, the perception module 660 may identify changes based on a signed distance function (SDF) model. The perception module 660 may be configured to receive sensor data, such as a depth map 660a and a head pose 660b, and then fuse the sensor data into an SDF model 660c. The depth map 660a may provide the SDF information directly, or images may be processed to arrive at the SDF information. The SDF information represents distances from sensors used to capture that information. Since those sensors may be part of the wearable unit, the SDF information may represent the physical world from the perspective of the wearable unit, and therefore the user. The head pose 660b may enable the SDF information to be related to voxels in the physical world.

[0185] In some embodiments, the perception module 660 may generate, update, and store representations for portions of the physical world that are within a perception range. The perception range may be determined, at least in part, based on the reconstruction range of the sensor, which may be determined, at least in part, based on the limits of the observation range of the sensor. As a specific example, an active depth sensor that operates using active IR pulses may operate reliably over a range of distances, creating an observation range for the sensor that may be a few centimeters or tens of centimeters to several meters.

[0186] The world reconstruction component 516 may include additional modules that may interact with the perception module 660. In some embodiments, the persisted world module 662 may receive a representation for the physical world based on data obtained by the perception module 660. The persisted world module 662 may also include representations of the physical world in various formats. For example, volumetric metadata 662b, such as voxels, may be stored along with the meshes 662c and planes 662d. In some embodiments, other information, such as depth maps, may also be saved.

[0187] In some embodiments, a representation of the physical world such as that illustrated in FIG. 6A may provide relatively dense information about the physical world compared to a sparse map such as a tracking map based on feature points, as described above.

[0188] In some embodiments, the perception module 660 may include modules that generate representations for the physical world in various formats, including, for example, meshes 660d, planes, and semantics 660e. The representations for the physical world may be stored across local and remote storage media. The representations for the physical world may be described in different coordinate frames, for example, depending on the location of the storage media. For example, a representation for the physical world stored in a device may be described in a coordinate frame local to the device. The representation for the physical world may have a counterpart stored in the cloud. The counterpart in the cloud may be described in a coordinate frame shared by all devices in the XR system.

[0189] In some embodiments, these modules may generate the representation based on data within the perceptual range of one or more sensors at the time the representation is generated and data captured at earlier times and information in the persisted world module 662. In some embodiments, these components may operate on depth information captured using a depth sensor. However, the AR system may also include a vision sensor and generate such representations by analyzing monocular or binocular visual information.

[0190] In some embodiments, these modules may operate on regions of the physical world. They may be triggered to update a subregion of the physical world when the perception module 660 detects a change in the physical world in that subregion. Such a change may be detected, for example, by detecting a new surface in the SDF model 660c, or by other criteria, such as a change in the values ​​of a sufficient number of voxels representing the subregion.

[0191] The world reconstruction component 516 may include components 664 that may receive a representation of the physical world from the perception module 660. Information about the physical world may be pulled by these components, for example, according to usage requests from applications. In some embodiments, information may be pushed to the usage components, such as via indications of changes in pre-identified areas or changes in the physical world representation within a perception range. The components 664 may include game programs and other components that implement processing for, for example, visual occlusion, physics-based interactions, and environmental inference.

[0192] In response to a query from component 664, perception module 660 may transmit a representation for the physical world in one or more formats. For example, when component 664 indicates the use is for visual occlusion or physics-based interaction, perception module 660 may transmit a representation of a surface. When component 664 indicates the use is for environmental inference, perception module 660 may transmit meshes, planes, and semantics of the physical world.

[0193] In some embodiments, perception module 660 may include a component that provides formatting information to component 664. An example of such a component may be a ray casting component 660f. A usage component (e.g., component 664) may, for example, query information about the physical world from a particular viewpoint. The ray casting component 660f may select from one or more representations of the physical world data within a field of view from that viewpoint.

[0194] As should be understood from the foregoing description, the perception module 660 or another component of the AR system may process data and create a 3D representation of a portion of the physical world. The data to be processed may be reduced, at least in part, by thinning a portion of the 3D reconstruction volume based on the camera frustum and / or depth image, extracting and persisting planar data, capturing, persisting, and updating the 3D reconstruction data in a block that allows local updates while maintaining neighborhood consistency, occlusion data is derived from a combination of one or more depth data sources, providing occlusion data to an application that generates such a scene, and / or performing multi-stage mesh simplification. The reconstruction may contain data of different levels of sophistication, including, for example, raw data such as live depth data, fused volumetric data such as voxels, and calculated data such as meshes.

[0195] In some embodiments, components of the passable world model may be distributed, with some parts running locally on the XR device and some parts running remotely, such as on a network connected to a server or otherwise in the cloud. The distribution of information processing and storage between the local XR device and the cloud may affect the functionality and user experience of the XR system. For example, reducing processing on the local device by distributing processing to the cloud may enable longer battery life and reduce heat generated on the local device. However, distributing much more processing to the cloud may create undesirable latency that causes an unacceptable user experience.

[0196] FIG. 6B depicts a distributed component architecture 600 configured for spatial computing, according to some embodiments. The distributed component architecture 600 may include a passable world component 602 (e.g., PW 538 in FIG. 5A), a Lumin OS 604, an API 606, an SDK 608, and applications 610. The Lumin OS 604 may include a Linux-based kernel with custom drivers compatible with XR devices. The API 606 may include an application programming interface that gives XR applications (e.g., applications 610) access to the spatial computing features of the XR device. The SDK 608 may include a software development kit that enables the creation of XR applications.

[0197] One or more components in the architecture 600 may create and maintain a model of the passable world. In this example, sensor data is collected on the local device. Processing of the sensor data may be performed partially locally on the XR device and partially in the cloud. The PW 538 may include an environment map that is created at least in part based on data captured by AR devices worn by multiple users. During a session of an AR experience, individual AR devices (such as the wearable devices described above in connection with FIG. 4) may create a tracking map, which is one type of map.

[0198] In some embodiments, the device may include components that build both sparse and dense maps. The tracking map may serve as a sparse map and may include information about the head poses of the AR device scanning the environment and the objects detected in the environment at each head pose. The head poses may be maintained locally per device. For example, the head poses on each device may be relative to the initial head pose when the device is turned on for the session. As a result, each tracking map may be local to the device that creates it and may have its own frame of reference defined by its own local coordinate system. However, in some embodiments, the tracking map on each device may be formed such that one coordinate of its local coordinate system is aligned with the direction of gravity as measured by its sensors, such as the inertial measurement unit 557.

[0199] In some embodiments, a device may include components that build both sparse and dense maps. The tracking map may serve as a sparse map and may include information about the head pose of the AR device scanning the environment and the objects detected in the environment at each head pose. Those head poses may be maintained locally per device. For example, the head poses on each device may be relative to the initial head pose when the device is turned on for the session. As a result, each tracking map may be local to the device that creates it. The dense map may include surface information, which may be represented by meshes or depth information. Alternatively or in addition, the dense map may include higher level information derived from surface or depth information, such as the location and / or properties of planes and / or other objects.

[0200] The creation of the dense map may be independent from the creation of the sparse map in some embodiments. The creation of the dense map and the sparse map may be performed, for example, in separate processing pipelines in the AR system. Separating the processing may, for example, allow the generation or processing of different types of maps to be performed at different rates. The sparse map may, for example, be refreshed at a faster rate than the dense map. However, in some embodiments, the processing of the dense and sparse maps may be related even if performed in different pipelines. Changes in the physical world exposed in the sparse map may, for example, trigger an update of the dense map, or vice versa. Furthermore, even if created independently, the maps may be used together. For example, a coordinate system derived from the sparse map may be used to define the position and / or orientation of objects in the dense map.

[0201] The sparse and / or dense maps may be persisted for reuse by the same device and / or for sharing with other devices. Such persistence may be achieved by storing the information in a cloud. The AR device may send the tracking map to the cloud and merge it with an environment map selected from the persisted maps previously stored in the cloud, for example. In some embodiments, the selected persisted map may be sent from the cloud to the AR device for merging. In some embodiments, the persisted map may be oriented with respect to one or more persistent coordinate frames. Such maps may serve as reference maps since they may be used by any of multiple devices. In some embodiments, the model of the passable world may include or be created with one or more reference maps. Although the device performs some operations based on a coordinate frame local to the device, it may use the reference map by determining a transformation between its coordinate frame local to the device and the reference map.

[0202] The reference map may originate as a tracking map (TM) (e.g., TM 1102 in FIG. 31A), which may be promoted to the reference map. The reference map may be persisted such that a device accessing the reference map, once it has determined the transformation between its local coordinate system and the coordinate system of the reference map, may use the information in the reference map to determine the location of objects represented in the reference map in the physical world around the device. In some embodiments, the TM may be a head pose sparse map created by the XR device. In some embodiments, the reference map may be created when the XR device sends one or more TMs to a cloud server for merging with additional TMs captured by the XR device at different times or by other XR devices.

[0203] In embodiments where a tracking map is formed on a local device with one coordinate of a local coordinate frame aligned with gravity, this orientation with respect to gravity may be preserved upon creation of a reference map. For example, when a tracking map submitted for merging does not overlap with any previously stored map, it may be promoted to the reference map. Other tracking maps, which may also have an orientation with respect to gravity, may then be merged with the reference map. Merging may be performed to ensure that the resulting reference map retains its orientation with respect to gravity. Two maps may not be merged, regardless of the correspondence of feature points in the maps, for example, if the coordinates of each map aligned with gravity do not match each other with a sufficiently close tolerance.

[0204] A reference map or other map may provide information about a portion of the physical world represented by the processed data to create a separate map. FIG. 7 depicts an example tracking map 700, according to some embodiments. The tracking map 700 may provide a top view 706 of a physical object in a corresponding physical world represented by points 702. In some embodiments, the map points 702 may represent features of the physical object, which may include multiple features. For example, each corner of a table may be a feature represented by a point on the map. The features may be derived from processed images, such as may be obtained using a sensor of a wearable device in an augmented reality system. The features may be derived, for example, by processing image frames output by the sensor and identifying features based on large gradients in the images or other suitable criteria. Further processing may limit the number of features in each frame. For example, processing may select features that are likely to represent persistent objects. One or more heuristics may be applied for this selection.

[0205] The tracking map 700 may include data about points 702 collected by the device. For each image frame with data points included in the tracking map, a pose may be stored. The pose may represent the orientation from which the image frame was captured so that feature points in each image frame may be spatially correlated. The pose may be determined by positioning information, such as may be derived from a sensor, such as an IMU sensor, on the wearable device. Alternatively, or in addition, the pose may be determined from matching an image frame with another image frame that depicts an overlapping portion of the physical world. By finding such position correlation, which may be accomplished by matching a subset of feature points in the two frames, a relative pose between the two frames may be calculated. The relative pose may be appropriate for the tracking map because the map may be relative to a coordinate system local to the device, which is established based on the initial pose of the device when construction of the tracking map began.

[0206] Because much of the information collected with the sensor is likely to be redundant, not all of the feature points and image frames collected by the device may be retained as part of the tracking map. Rather, only certain frames may be added to the map. These frames may be selected based on one or more criteria, such as the degree of overlap with image frames already in the map, the number of new features they contain, or a quality metric for the features in the frame. Image frames that are not added to the tracking map may be discarded or used to revise the location of features. As a further alternative, all or most of the image frames, represented as a set of features, may be retained, but a subset of those frames may be designated as key frames, which are used for further processing.

[0207] The keyframes may be processed to produce a keyrig 704. The keyframes may be processed to produce a three-dimensional set of feature points and stored as a keyrig 704. Such processing may involve, for example, comparing image frames derived simultaneously from two cameras and stereoscopically determining the 3D positions of the feature points. Metadata such as pose may be associated with these keyframes and / or keyrigs.

[0208] The environment map may have any of a number of formats, depending on, for example, the storage location of the environment map, including, for example, the local storage and the remote storage of the AR device. For example, the map in the remote storage may have a higher resolution than the map in the local storage on the wearable device when memory is limited. To transmit the higher resolution map from the remote storage to the local storage, the map may be downsampled or otherwise converted to a suitable format, such as by reducing the number of poses per area of ​​the physical world stored in the map and / or the number of feature points stored per pose. In some embodiments, a slice or portion of the high resolution map from the remote storage may be transmitted to the local storage, and the slice or portion is not downsampled.

[0209] The database of environmental maps may be updated as new tracking maps are created. To determine which of the potentially numerous environmental maps in the database should be updated, the updating may include efficiently selecting one or more environmental maps to be stored in the database associated with the new tracking map. The selected one or more environmental maps may be ranked by relevance, and one or more of the highest ranked maps may be selected for processing to merge the new tracking map with the higher ranked selected environmental map and create one or more updated environmental maps. When the new tracking map represents a portion of the physical world over which there is no existing environmental map to update, the tracking map may be stored in the database as a new environmental map.

[0210] View Independent Display

[0211] Described herein are methods and apparatus for providing virtual content using an XR system independent of the location of the eye that views the virtual content. Conventionally, virtual content is re-rendered in response to any movement of the display system. For example, when a user wearing a display system views a virtual representation of a three-dimensional (3D) object on the display and walks around the area in which the 3D object appears, the 3D object should be re-rendered for each viewpoint so that the user has the perception that he or she is walking around the object that occupies real space. However, re-rendering consumes significant computational resources of the system and introduces artifacts due to latency.

[0212] The inventors have recognized and appreciated that head pose (e.g., location and orientation of a user wearing an XR system) may be used to render virtual content independent of eye rotations at the user's head. In some embodiments, a dynamic map of a scene may be generated based on multiple coordinate frames in real space across one or more sessions such that virtual content interacting with the dynamic map may be robustly rendered independent of eye rotations at the user's head and / or independent of sensor deformations caused by heat generated during high speed computationally intensive operations. In some embodiments, the configuration of multiple coordinate frames may enable a first XR device worn by a first user and a second XR device worn by a second user to recognize a common location in a scene. In some embodiments, the configuration of multiple coordinate frames may enable users wearing the XR devices to view virtual content in the same location of the scene.

[0213] In some embodiments, the tracking map may be constructed in a world coordinate frame, which may have a world origin. The world origin may be the first pose of the XR device when the XR device is powered on. The world origin may be aligned to gravity so that the developer of the XR application can obtain gravity alignment without extra work. Different tracking maps may be constructed in different world coordinate frames because the tracking map may be captured by the same XR device in different sessions and / or by different XR devices worn by different users. In some embodiments, a session of an XR device may start when the device is powered on and continue until it is powered off. In some embodiments, the XR device may have a head coordinate frame, which may have a head origin. The head origin may be the current pose of the XR device when the image is taken. The difference between the head pose of the world coordinate frame and the head coordinate frame may be used to estimate the tracking route.

[0214] In some embodiments, the XR device may have a camera coordinate frame, which may have a camera origin. The camera origin may be the current pose of one or more sensors of the XR device. The inventors have recognized and appreciated that the configuration of the camera coordinate frame enables a robust display of virtual content independent of eye rotations in the user's head. This configuration also enables a robust display of virtual content independent of sensor deformations due to, for example, heat generated during movement.

[0215] In some embodiments, the XR device may have a head unit with a head-mountable frame that a user may fasten to his / her head and that may include two waveguides, one in front of each of the user's eyes. The waveguides may be transparent so that ambient light from real-world objects may transmit through the waveguides and the user may see the real-world objects. Each waveguide may transmit projected light from a projector to a respective eye of the user. The projected light may form an image on the retina of the eye. The retina of the eye thus receives the ambient light and the projected light. The user may simultaneously see the real-world object and one or more virtual objects created by the projected light. In some embodiments, the XR device may have sensors that detect real-world objects around the user. These sensors may be, for example, cameras that capture images that may be processed to identify the location of the real-world objects.

[0216] In some embodiments, the XR system may assign coordinate frames to the virtual content as opposed to tying the virtual content to a world coordinate frame. Such an arrangement allows the virtual content to be described regardless of where it is rendered for the user, but may be tied to a more persistent frame location and rendered at a defined location, such as, for example, a persistent coordinate frame (PCF) as described in connection with FIG. 14-20C. As the location of the object changes, the XR device may detect the change in the environment map and determine the movement of the head unit worn by the user relative to the real-world object.

[0217] 8 illustrates a user experiencing virtual content as rendered in a physical environment by an XR system 10, according to some embodiments. The XR system may include a first XR device 12.1 worn by a first user 14.1, a network 18, and a server 20. The user 14.1 is present in the physical environment with a real object in the form of a table 16.

[0218] In the illustrated example, the first XR device 12.1 includes a head unit 22, a belt pack 24, and a cable connection 26. The first user 14.1 fastens the head unit 22 to his head and the belt pack 24, remote from the head unit 22, on his waist. The cable connection 26 connects the head unit 22 to the belt pack 24. The head unit 22 includes technology used to display a virtual object or objects to the first user 14.1 while still allowing the first user 14.1 to see a real object, such as a table 16. The belt pack 24 primarily includes the processing and communication capabilities of the first XR device 12.1. In some embodiments, the processing and communication capabilities may reside in whole or in part in the head unit 22, such that the belt pack 24 may be removed or located in another device, such as a backpack.

[0219] In the illustrated embodiment, the belt pack 24 is connected to the network 18 via a wireless connection. A server 20 is connected to the network 18 and holds data representing local content. The belt pack 24 downloads data representing the local content from the server 20 via the network 18. The belt pack 24 provides the data to the head unit 22 via a cable connection 26. The head unit 22 may include a display having a light source, e.g., a laser light source or a light emitting diode (LED) light source, and a waveguide for guiding the light.

[0220] In some embodiments, the first user 14.1 may carry a head unit 22 on his head and a belt pack 24 on his waist. The belt pack 24 may download image data representing virtual content from a server 20 via a network 18. The first user 14.1 may see a table 16 through a display of the head unit 22. A projector, forming part of the head unit 22, may receive the image data from the belt pack 24 and generate light based on the image data. The light may travel through one or more of the waveguides forming part of the display of the head unit 22. The light may then exit the waveguide and propagate onto the retina of the first user's 14.1 eye. The projector may generate light in a pattern that is replicated on the retina of the first user's 14.1 eye. The light that strikes the retina of the first user's 14.1 eye may have a selected depth of field such that the first user 14.1 perceives an image at a preselected depth behind the waveguide. In addition, both eyes of the first user 14.1 may receive slightly different images such that the brain of the first user 14.1 perceives a three-dimensional image or multiple images at a selected distance from the head unit 22. In the illustrated embodiment, the first user 14.1 perceives the virtual content 28 above the table 16. The virtual content 28 and its location and proportion of distance from the first user 14.1 are determined by data representing the virtual content 28 and the various coordinate frames used to display the virtual content 28 to the first user 14.1.

[0221] In the illustrated embodiment, the virtual content 28 is invisible from the perspective of the drawing and is visible to the first user 14.1 through the use of the first XR device 12.1. The virtual content 28 may initially reside as a data structure in the visual data and algorithms in the beltpack 24. The data structure may then be revealed as light when the projector of the head unit 22 generates light based on the data structure. It should be understood that although the virtual content 28 does not exist in a three-dimensional space in front of the first user 14.1, the virtual content 28 is still represented in FIG. 1 in a three-dimensional space for illustration of what the wearer of the head unit 22 would perceive. Visualization of computer data in a three-dimensional space may be used in this description to illustrate how the data structures that facilitate the rendering as perceived by one or more users are interrelated among the data structures in the beltpack 24.

[0222] 9 illustrates components of a first XR device 12.1 according to some embodiments. The first XR device 12.1 may include a head unit 22 and various components that form part of the visual data and algorithms, including, for example, a rendering engine 30, various coordinate systems 32, various origin and destination coordinate frames 34, and various origin / destination coordinate frame converters 36. The various coordinate systems may be based on the intrinsic properties of the XR device or may be determined by reference to other information, such as a persistent pose or persistent coordinate system as described herein.

[0223] The head unit 22 may include a head-mountable frame 40 , a display system 42 , a real object detection camera 44 , a mobile tracking camera 46 , and an inertial measurement unit 48 .

[0224] The head-mountable frame 40 may have a shape that is attachable to the head of a first user 14.1 in Figure 8. The display system 42, the real object detection camera 44, the mobile tracking camera 46, and the inertial measurement unit 48 are mounted to the head-mountable frame 40 and may therefore move with the head-mountable frame 40.

[0225] The coordinate system 32 may include a local data system 52 , a world frame system 54 , a head frame system 56 , and a camera frame system 58 .

[0226] Local data system 52 may include a data channel 62, a local frame determination routine 64, and local frame storage instructions 66. Data channel 62 may be an internal software routine, a hardware component such as an external cable or radio frequency receiver, or a hybrid component such as an open port. Data channel 62 may be configured to receive image data 68 representing virtual content.

[0227] A local frame determination routine 64 may be connected to the data channel 62. The local frame determination routine 64 may be configured to determine a local coordinate frame 70. In some embodiments, the local frame determination routine may determine the local coordinate frame based on a real-world object or real-world location. In some embodiments, the local coordinate frame may be based on the top edge relative to the bottom edge of the browser window, the head or feet of a character, a node on the outer surface of a prism or bounding box that encloses the virtual content, or any other suitable location for placing a coordinate frame that defines the facing direction of the virtual content and where the virtual content should be placed (e.g., a node such as a placement node or PCF node) with which the virtual content should be placed, etc.

[0228] The local frame storage instruction 66 may be connected to the local frame determination routine 64. One skilled in the art will understand that software modules and routines are "connected" to one another through subroutines, calls, etc. The local frame storage instruction 66 may store the local coordinate frame 70 as the local coordinate frame 72 in the origin and destination coordinate frame 34. In some embodiments, the origin and destination coordinate frame 34 may be one or more coordinate frames that may be manipulated or transformed in order for the virtual content to persist between sessions. In some embodiments, a session may be a period of time between boot-up and shutdown of an XR device. The two sessions may be two startup and shutdown periods for a single XR device, or startup and shutdown for two different XR devices.

[0229] In some embodiments, the origin and destination coordinate frames 34 may be coordinate frames involved in one or more transformations required for the first user's XR device and the second user's XR device to perceive a common location. In some embodiments, the destination coordinate frame may be the output of a series of calculations and transformations applied to the target coordinate frames in order for the first and second users to view virtual content at the same location.

[0230] Rendering engine 30 may be connected to data channel 62. Rendering engine 30 may receive image data 68 from data channel 62 such that rendering engine 30 may render virtual content based, at least in part, on image data 68.

[0231] A display system 42 may be connected to the rendering engine 30. The display system 42 may include components that convert image data 68 into visible light. The visible light may form two patterns, one for each eye. The visible light may enter the eye of the first user 14.1 in FIG. 8 and may be detected on the retina of the eye of the first user 14.1.

[0232] The real object detection camera 44 may include one or more cameras that may capture images from different sides of the head mountable frame 40. The mobile tracking camera 46 may include one or more cameras that capture images on a side of the head mountable frame 40. One set of one or more cameras may be used instead of two sets of one or more cameras representing the real object detection camera 44 and the mobile tracking camera 46. In some embodiments, the cameras 44, 46 may capture images. As explained above, these cameras may collect data that is used to build a tracking map.

[0233] The inertial measurement unit 48 may include several devices used to detect movement of the head unit 22. The inertial measurement unit 48 may include a gravity sensor, one or more accelerometers, and one or more gyroscopes. The sensors of the inertial measurement unit 48 combine to track movement of the head unit 22 in at least three orthogonal directions and approximately at least three orthogonal axes.

[0234] In the illustrated embodiment, the world frame system 54 includes a world surface determination routine 78, a world frame determination routine 80, and world frame storage instructions 82. The world surface determination routine 78 is connected to the real object detection camera 44. The world surface determination routine 78 receives images and / or key frames based on images captured by the real object detection camera 44, processes the images, and identifies surfaces within the images. A depth sensor (not shown) may determine the distance to the surface. The surface is thus represented by data in three dimensions, including its size, shape, and distance from the real object detection camera.

[0235] In some embodiments, the world coordinate frame 84 may be based on the origin at the initialization of the head pose session. In some embodiments, the world coordinate frame may be located where the device was booted up, or may be some new location if the head pose is lost during the boot session. In some embodiments, the world coordinate frame may be the origin at the start of the head pose session.

[0236] In the illustrated embodiment, the world frame determination routine 80 is coupled to the world surface determination routine 78 and determines a world coordinate frame 84 based on the location of the surface as determined by the world surface determination routine 78. The world frame storage instruction 82 is coupled to the world frame determination routine 80 and receives the world coordinate frame 84 from the world frame determination routine 80. The world frame storage instruction 82 stores the world coordinate frame 84 as a world coordinate frame 86 in the origin and destination coordinate frame 34.

[0237] The head frame system 56 may include a head frame determination routine 90 and head frame storage instructions 92. The head frame determination routine 90 may be connected to the moving tracking camera 46 and the inertial measurement unit 48. The head frame determination routine 90 may use data from the moving tracking camera 46 and the inertial measurement unit 48 to calculate a head coordinate frame 94. For example, the inertial measurement unit 48 may have a gravity sensor that determines the direction of gravity relative to the head unit 22. The moving tracking camera 46 may continuously capture images that are used by the head frame determination routine 90 to refine the head coordinate frame 94. The head unit 22 moves as the first user 14.1 in FIG. 8 moves his head. The moving tracking camera 46 and the inertial measurement unit 48 may continually provide data to the head frame determination routine 90 so that the head frame determination routine 90 may update the head coordinate frame 94.

[0238] The head frame storage instruction 92 may be connected to the head frame determination routine 90 and receive a head coordinate frame 94 from the head frame determination routine 90. The head frame storage instruction 92 may store the head coordinate frame 94 as a head coordinate frame 96 between the origin and destination coordinate frames 34. The head frame storage instruction 92 may repeatedly store the updated head coordinate frame 94 as the head coordinate frame 96 when the head frame determination routine 90 recalculates the head coordinate frame 94. In some embodiments, the head coordinate frame may be the location of the wearable XR device 12.1 relative to the local coordinate frame 72.

[0239] The camera frame system 58 may include camera intrinsic properties 98. The camera intrinsic properties 98 may include the dimensions of the head unit 22, which are characteristic of its design and manufacture. The camera intrinsic properties 98 may be used to calculate a camera coordinate frame 100, which is stored in the origin and destination coordinate frames 34.

[0240] In some embodiments, the camera coordinate frame 100 may include all of the pupil positions of the left eye of the first user 14.1 in Figure 8. As the left eye moves from left to right or up and down, the pupil positions of the left eye are located within the camera coordinate frame 100. In addition, the pupil positions of the right eye are located within the camera coordinate frame 100 for the right eye. In some embodiments, the camera coordinate frame 100 may include the location of the camera relative to a local coordinate frame when an image is taken.

[0241] The origin / destination coordinate frame converter 36 may include a local / world coordinate converter 104, a world / head coordinate converter 106, and a head / camera coordinate converter 108. The local / world coordinate converter 104 may receive the local coordinate frame 72 and transform the local coordinate frame 72 to the world coordinate frame 86. The transformation of the local coordinate frame 72 to the world coordinate frame 86 may be represented as a local coordinate frame that is transformed within the world coordinate frame 86 to a world coordinate frame 110.

[0242] The world / head coordinate converter 106 may transform from the world coordinate frame 86 to the head coordinate frame 96. The world / head coordinate converter 106 may transform a local coordinate frame that is transformed to the world coordinate frame 110 into the head coordinate frame 96. The transformation may be expressed as a local coordinate frame that is transformed to the head coordinate frame 112 in the head coordinate frame 96.

[0243] The head / camera coordinate converter 108 may convert from the head coordinate frame 96 to the camera coordinate frame 100. The head / camera coordinate converter 108 may convert the local coordinate frame, which is converted to the head coordinate frame 112, to a local coordinate frame, which is converted to the camera coordinate frame 114 in the camera coordinate frame 100. The local coordinate frame, which is converted to the camera coordinate frame 114, may be brought into the rendering engine 30. The rendering engine 30 may render image data 68 representing the local content 28 based on the local coordinate frame, which is converted to the camera coordinate frame 114.

[0244] FIG. 10 is a spatial representation of various origin and destination coordinate frames 34. A local coordinate frame 72, a world coordinate frame 86, a head coordinate frame 96, and a camera coordinate frame 100 are represented in the figure. In some embodiments, the local coordinate frame associated with the XR content 28 may have a position and rotation relative to the local and / or world coordinate frames and / or PCFs (e.g., may provide nodal and facing directions) when the virtual content is placed in the real world and thus may be viewed by the user. Each camera may have its own camera coordinate frame 100 that encompasses all pupil positions of one eye. Reference numerals 104A and 106A represent the transformations performed by the local-to-world coordinate converter 104, the world-to-head coordinate converter 106, and the head-to-camera coordinate converter 108 in FIG. 9, respectively.

[0245] FIG. 11 depicts a camera rendering protocol for transforming from a head coordinate frame to a camera coordinate frame, according to some embodiments. In the illustrated example, the pupil for one eye moves from position A to B. A virtual object intended to appear stationary would be projected onto the depth plane at one of two positions A or B, depending on the position of the pupil (assuming the camera is configured to use as a pupil-based coordinate frame). As a result, using a pupil coordinate frame that is transformed into the head coordinate frame would cause jitter in the stationary virtual object as the eye moves from position A to position B. This situation is referred to as view-dependent display or projection.

[0246] As depicted in Figure 12, the camera coordinate frame (e.g., CR) is positioned to encompass all pupil positions, but the object projection will now be consistent regardless of pupil positions A and B. The head coordinate frame transforms to the CR frame, which is referred to as a view-independent display or projection. Image reprojection may be applied to the virtual content to account for changes in eye position, however, the rendering is still in the same position, so jitter is minimized.

[0247] 13 illustrates in more detail the display system 42. The display system 42 includes a stereoscopic analyzer 144 that is connected to the rendering engine 30 and forms part of the visual data and algorithms.

[0248] The display system 42 further includes left and right projectors 166A and 166B and left and right waveguides 170A and 170B. The left and right projectors 166A and 166B are connected to a power supply. Each projector 166A and 166B has an individual input through which image data is provided to the individual projector 166A or 166B. When powered, the individual projector 166A or 166B generates light in a two-dimensional pattern and emits light therefrom. The left and right waveguides 170A and 170B are positioned to receive light from the left and right projectors 166A and 166B, respectively. The left and right waveguides 170A and 170B are transparent waveguides.

[0249] In use, a user mounts head-mountable frame 40 on their head. Components of head-mountable frame 40 may include, for example, a strap (not shown) that wraps around the back of the user's head. Left and right waveguides 170A and 170B are then positioned in front of the user's left and right eyes 220A and 220B.

[0250] The rendering engine 30 takes the image data it receives into the stereoscopic analyzer 144. The image data is three-dimensional image data of the local content 28 in FIG. 8. The image data is projected onto multiple virtual planes. The stereoscopic analyzer 144 analyzes the image data and determines left and right image data sets based on the image data for projection onto each depth plane. The left and right image data sets are data sets representing two-dimensional images that are projected in three dimensions to give the user the perception of depth.

[0251] The stereoscopic analyzer 144 captures left and right image data sets into the left and right projectors 166A and 166B, which then create left and right light patterns. The components of the display system 42 are shown in plan view, but it should be understood that the left and right patterns are two-dimensional patterns when shown in a front elevation view. Each light pattern includes a number of pixels. For illustrative purposes, light rays 224A and 226A from two of the pixels are shown exiting the left projector 166A and entering the left waveguide 170A. The light rays 224A and 226A reflect off the side of the left waveguide 170A. Although rays 224A and 226A are shown propagating through internal reflection from left to right within left waveguide 170A, it should be understood that rays 224A and 226A may also propagate into the plane of the paper using refractive and reflective systems.

[0252] Light rays 224A and 226A exit the left optical waveguide 170A through pupil 228A and then enter the left eye 220A through the pupil 230A of the left eye 220A. Light rays 224A and 226A then impinge on the retina 232A of the left eye 220A. Thus, the left light pattern impinges on the retina 232A of the left eye 220A. The user is given the perception that the pixels formed on the retina 232A are pixels 234A and 236A, which the user perceives as being at a distance on the side of the left waveguide 170A facing the left eye 220A. Depth perception is created by manipulating the focal length of the light.

[0253] Similarly, stereoscopic analyzer 144 captures a right image data set to right projector 166B. Right projector 166B transmits a right light pattern, which is represented by pixels in the form of rays 224B and 226B. Rays 224B and 226B reflect in right waveguide 170B and exit through pupil 228B. Rays 224B and 226B then enter through pupil 230B of right eye 220B and impinge on retina 232B of right eye 220B. The pixels of rays 224B and 226B are perceived as pixels 134B and 236B behind right waveguide 170B.

[0254] The patterns created on the retinas 232A and 232B are perceived as left and right images, respectively, which differ slightly from each other due to the function of the stereoscopic analyzer 144. The left and right images are perceived as a three-dimensional rendering in the user's brain.

[0255] As mentioned, the left and right waveguides 170A and 170B are transparent. Light from a real object, such as a table 16, on the side of the left and right waveguides 170A and 170B facing the eyes 220A and 220B can be projected through the left and right waveguides 170A and 170B and impinge on the retinas 232A and 232B.

[0256] Persistent Coordinate Frame (PCF)

[0257] Described herein are methods and apparatus for providing spatial persistence across user instances in a shared space. Without spatial persistence, virtual content placed in the physical world by a user in a session may not be present or may be misplaced in the view of a user in a different session. Without spatial persistence, virtual content placed in the physical world by one user may not be present or may be misplaced in the view of a second user, even if the second user intends to share an experience of the same physical space with the first user.

[0258] The inventors recognize and understand that spatial persistence may be provided through a Persistent Coordinate Frame (PCF). A PCF may be defined based on one or more points that represent recognized features (e.g., corners, edges) in the physical world. The features may be selected such that they are likely to be the same from one user instance to another user instance of the XR system.

[0259] Furthermore, drift during tracking, which may cause the calculated tracking path (e.g., camera trajectory) to deviate from the actual tracking path, may cause the location of the virtual content to appear out of place when rendered relative to a local map based only on the tracking map. The tracking map for the space may be refined to correct for drift as the XR device gathers more information of the scene over time. However, if the virtual content is placed on real objects and stored relative to the device's world coordinate frame derived from the tracking map before map refinement, the virtual content may appear displaced as if the real objects had moved during map refinement. The PCF may be updated according to map refinement because the PCF is defined based on features and is updated as the features move during map refinement.

[0260] A PCF may have six degrees of freedom with translation and rotation relative to a map coordinate system. A PCF may be stored in a local and / or remote storage medium. The translation and rotation of a PCF may be calculated relative to a map coordinate system, for example, depending on the storage location. For example, a PCF used locally by a device may have translation and rotation relative to the world coordinate frame of the device. A PCF in the cloud may have translation and rotation relative to the reference coordinate frame of a reference map.

[0261] PCFs may provide a sparse representation of the physical world and provide less than all of the available information about the physical world so that they can be efficiently processed and transferred. Techniques for processing persistent spatial information may include creating a dynamic map based on one or more coordinate systems in real space across one or more sessions and generating a persistent coordinate frame (PCF) over the sparse map that can be exposed to XR applications, for example, via an application programming interface (API).

[0262] 14 is a block diagram illustrating the creation of a persistent coordinate frame (PCF) and binding of XR content to the PCF, according to some embodiments. Each block may represent digital information stored in computer memory. In the case of application 1180, the data may represent computer-executable instructions. In the case of virtual content 1170, the digital information may define a virtual object, for example, as defined by application 1180. In the case of other boxes, the digital information may characterize some aspect of the physical world.

[0263] In the illustrated embodiment, one or more PCFs are created from images captured using sensors on the wearable device. In the embodiment of FIG. 14, the sensors are visual image cameras. These cameras may be the same cameras used to form the tracking map. Thus, some of the processing suggested by FIG. 14 may be implemented as part of updating the tracking map. However, FIG. 14 illustrates that information providing persistence is generated in addition to the tracking map.

[0264] To derive the 3D PCF, two images 1110 from two cameras mounted on a wearable device in a configuration that enables stereoscopic image analysis are processed together. Figure 14 illustrates image 1 and image 2, each derived from one of the cameras. A single image from each camera is illustrated for convenience. However, each camera may output a stream of image frames, and the processing illustrated in Figure 14 may be performed for multiple image frames in the stream.

[0265] Thus, image 1 and image 2 may each be one frame in a sequence of image frames. The process as depicted in FIG. 14 may be repeated on successive image frames in the sequence until an image frame containing feature points that provide a suitable image from which to form persistent spatial information is processed. Alternatively, or in addition, the process of FIG. 14 may be repeated as the user moves such that the user is no longer close enough to a previously identified PCF to reliably use that PCF to determine position relative to the physical world. For example, the XR system may maintain a current PCF for the user. When the distance exceeds a threshold, the system may switch to a new current PCF closer to the user, which may be generated according to the process of FIG. 14 using image frames acquired at the user's current location.

[0266] Even when generating a single PCF, the stream of image frames may be processed to identify image frames that depict content in the physical world that is likely to be stable and that may be easily identified by devices in the vicinity of the area of ​​the physical world depicted in the image frames. In the embodiment of Figure 14, the process begins with identifying features 1120 in the image. Features may be identified, for example, by finding locations of gradients or other characteristics in the image that are above a threshold, which may correspond, for example, to corners of an object. In the illustrated embodiment, the features are points, but other recognizable features, such as edges, may alternatively or additionally be used.

[0267] In the illustrated embodiment, a fixed number N of features 1120 are selected for further processing. Those feature points may be selected based on one or more criteria, such as gradient magnitude or proximity to other feature points. Alternatively, or in addition, feature points may be selected heuristically, such as based on characteristics that suggest the feature point is persistent. For example, a heuristic may be defined based on a feature point's characteristics that are likely to correspond to a window or door or a corner of a large piece of furniture. Such a heuristic may take into account the feature point itself and what surrounds it. As a specific example, the number of feature points per image may be 100-500, such as 200, or 150-250.

[0268] Regardless of the number of selected feature points, a descriptor 1130 may be calculated for the feature points. In this embodiment, a descriptor is calculated for each selected feature point, but a descriptor may be calculated for a group of feature points, or for a subset of feature points, or for all features in the image. The descriptor characterizes the feature points such that feature points that represent the same object in the physical world are assigned similar descriptors. The descriptor may facilitate matching of two frames, such as may occur when one map is localized against another map. Rather than searching for a relative orientation of the frames that minimizes the distance between feature points in the two images, an initial matching of the two frames may be done by identifying feature points with similar descriptors. Matching of image frames may be based on matching points with similar descriptors, which may involve less processing than calculating matching of all feature points in an image.

[0269] The descriptor may be calculated as a mapping of the descriptor to the feature point, or in some embodiments, a mapping of a patch of the image around the feature point. The descriptor may be a numerical quantity. U.S. Patent Application No. 16 / 190,948 describes calculating descriptors for feature points and is incorporated herein by reference in its entirety.

[0270] In the example of FIG. 14, a descriptor 1130 is calculated for each feature point in each image frame. Based on the descriptors and / or the feature points and / or the image itself, image frames may be identified as key frames 1140. In the illustrated embodiment, key frames are image frames that meet certain criteria that are then selected for further processing. In creating the tracking map, for example, image frames that add meaningful information to the map may be selected as key frames to be integrated into the map. On the other hand, image frames that substantially overlap regions over which image frames have already been integrated into the map may be discarded so that they do not become key frames. Alternatively or in addition, key frames may be selected based on the number and / or type of feature points in the image frames. In the embodiment of FIG. 14, key frames 1150 selected for inclusion in the tracking map may also be processed as key frames for determining the PCF, although different or additional criteria for selecting key frames for generation of the PCF may be used.

[0271] Although FIG. 14 shows that key frames are used for further processing, information obtained from the images may be processed in other forms. For example, feature points, such as in a key rig, may alternatively or additionally be processed. Furthermore, while key frames are described as being derived from a single image frame, there need not be a one-to-one relationship between the key frames and the obtained image frames. Key frames may be obtained from multiple image frames, such as by stitching or aggregating the image frames together, such that only features that appear in multiple images are retained in the key frame.

[0272] A key frame may include image information and / or metadata associated with the image information. In some embodiments, images captured by cameras 44, 46 (FIG. 9) may be computed into one or more key frames (e.g., key frames 1, 2). In some embodiments, a key frame may include a camera pose. In some embodiments, a key frame may include one or more camera images captured at a camera pose. In some embodiments, the XR system may determine that a portion of a camera image captured at a camera pose is not useful and therefore not include that portion in a key frame. Thus, using key frames to match new images with earlier knowledge of the scene reduces the use of computational resources of the XR system. In some embodiments, a key frame may include images and / or image data at a location with a certain direction / angle. In some embodiments, a key frame may include a location and direction from which one or more map points may be viewed. In some embodiments, a key frame may include a coordinate frame with an ID. US Patent Application No. 15 / 877,359 describes keyframes and is incorporated herein by reference in its entirety.

[0273] Some or all of the key frames 1140 may be selected for further processing, such as generating persistent poses 1150 for the key frames. The selection may be based on characteristics of all or a subset of the feature points in the image frames. These characteristics may be determined from processing the descriptors, features, and / or the image frames themselves. As a specific example, the selection may be based on a cluster of feature points identified as likely to be associated with a persistent object.

[0274] Each keyframe is associated with the camera pose at which the keyframe was acquired. For keyframes selected for processing into persistent poses, that pose information may be stored along with other metadata about the keyframe, such as a WiFi fingerprint and / or GPS coordinates at the time and / or location of acquisition.

[0275] The persistent pose is a source of information that the device may use to orient itself relative to previously obtained information about the physical world. For example, if a key frame from which the persistent pose was created is incorporated into a map of the physical world, the device may orient itself relative to the persistent pose using a sufficient number of feature points in the key frame associated with the persistent pose. The device may match the persistent pose with a current image obtained of its surroundings. This matching may be based on matching the current image with the image 1110, features 1120, and / or descriptors 1130 that gave rise to the persistent pose, or any subset of the image or their features or descriptors. In some embodiments, the current image frame matched to the persistent pose may be another key frame that is incorporated into the tracking map of the device.

[0276] Information about persistent attitudes may be stored in a format that facilitates sharing between multiple applications, which may be running on the same or different devices. In the example of FIG. 14, some or all of the persistent attitudes may be reflected as persistent coordinate frames (PCFs) 1160. Like persistent attitudes, PCFs may also be associated with a map and may comprise a set of features or other information that a device may use to determine its orientation relative to the PCF. A PCF may include a transformation that defines its transformation relative to the origin of the map, such that a device may determine its position relative to any object in the physical world reflected in the map by correlating its position to the PCF.

[0277] Because PCFs provide a mechanism for determining location relative to physical objects, an application, such as application 1180, may define the location of a virtual object relative to one or more PCFs that serve as anchors for virtual content 1170. FIG. 14 illustrates, for example, app 1 associating PCF1.2 with its virtual content 2. Similarly, app 2 associating PCF1.2 with its virtual content 3. App 1 is also shown associating PCF4.5 with its virtual content 1, and app 2 is shown associating PCF3 with its virtual content 4. In some embodiments, PCF3 may be based on image 3 (not shown), and PCF4.5 may be based on image 4 and image 5 (not shown), similar to how PCF1.2 is based on image 1 and image 2. When rendering this virtual content, the device may apply one or more transformations to calculate information such as the location of the virtual content relative to the device's display and / or the location of the physical object relative to the desired location of the virtual content. Using a PCF as a reference may simplify such calculations.

[0278] In some embodiments, the persistent pose may be a coordinate location and / or orientation having one or more associated keyframes. In some embodiments, the persistent pose may be created automatically after the user travels a certain distance, e.g., 3 meters. In some embodiments, the persistent pose may act as a reference point during localization. In some embodiments, the persistent pose may be stored in the passable world (e.g., the passable world module 538).

[0279] In some embodiments, a new PCF may be determined based on a predefined distance allowed between adjacent PCFs. In some embodiments, one or more sustained poses may be calculated into the PCF once the user has progressed a predetermined distance, e.g., 5 meters. In some embodiments, the PCF may be associated with one or more world coordinate frames and / or reference coordinate frames, e.g., within a passable world. In some embodiments, the PCF may be stored in a local and / or remote database, e.g., depending on security settings.

[0280] 15 illustrates a method 4700 of establishing and using a persistent coordinate frame, according to some embodiments. The method 4700 may begin with capturing images (e.g., image 1 and image 2 in FIG. 14) centered on a scene using one or more sensors of the XR device (act 4702). Multiple cameras may be used, and one camera may generate multiple images, for example in a stream.

[0281] The method 4700 may include extracting points of interest (e.g., map points 702 in FIG. 7, features 1120 in FIG. 14) from the captured image (4704), generating descriptors (e.g., descriptors 1130 in FIG. 14) for the extracted points of interest (act 4706), and generating keyframes (e.g., keyframes 1140) based on the descriptors (act 4708). In some embodiments, the method may compare points of interest in the keyframes and form paired keyframes that share a predetermined amount of points of interest. The method may use each paired keyframe to reconstruct a portion of the physical world. The mapped portions of the physical world may be saved as 3D features (e.g., keyrigs 704 in FIG. 7). In some embodiments, selected portions of the paired keyframes may be used to construct the 3D features. In some embodiments, the results of the mapping may be selectively saved. Keyframes that are not used to construct 3D features may be associated with the 3D features through poses that represent the distance between the keyframes, for example, with a covariance matrix between the poses of the keyframes. In some embodiments, paired keyframes may be selected to construct 3D features such that the distance between each of the constructed 3D features is within a predefined distance, which may be determined to balance the amount of computation required and the level of accuracy of the resulting model. Such an approach makes it possible to provide a model of the physical world with an amount of data suitable for efficient and accurate computation with the XR system. In some embodiments, the covariance matrix of the two images may include the covariance between the poses (e.g., 6 degrees of freedom) of the two images.

[0282] Method 4700 may include generating a persistent pose based on the keyframes (act 4710). In some embodiments, the method may include generating a persistent pose based on 3D features reconstructed from the pair of keyframes. In some embodiments, the persistent pose may be bound to the 3D feature. In some embodiments, the persistent pose may include a pose of a keyframe used to construct the 3D feature. In some embodiments, the persistent pose may include an average pose of the keyframes used to construct the 3D feature. In some embodiments, the persistent pose may be generated such that the distance between neighboring persistent poses is within a predetermined value, e.g., within a range of 1 meter to 5 meters, any value therebetween, or any other suitable value. In some embodiments, the distance between neighboring persistent poses may be represented by a covariance matrix of the neighboring persistent poses.

[0283] Method 4700 may include generating a PCF based on the sustained pose (act 4712). In some embodiments, the PCF may be bound to a 3D feature. In some embodiments, the PCF may be associated with one or more sustained poses. In some embodiments, the PCF may include one pose of the associated sustained poses. In some embodiments, the PCF may include an average pose of the poses of the associated sustained poses. In some embodiments, the PCF may be generated such that the distance between neighboring PCFs is within a predetermined value, e.g., a range of 3 meters to 10 meters, any value therebetween, or any other suitable value. In some embodiments, the distance between neighboring PCFs may be represented by a covariance matrix of the neighboring PCFs. In some embodiments, the PCF may be exposed to an XR application, e.g., via an application programming interface (API), such that the XR application may access a model of the physical world through the PCF without accessing the model itself.

[0284] The method 4700 may include associating at least one of the PCFs with image data of the virtual object for display by the XR device (act 4714). In some embodiments, the method may include calculating a translation and orientation of the virtual object relative to the associated PCF. It should be understood that associating the virtual object with the PCF generated by the device on which it is placed is not required. For example, the device may retrieve a stored PCF in a reference map in the cloud and associate the virtual object with the retrieved PCF. It should be understood that the virtual object may move with the associated PCF as the PCF is adjusted over time.

[0285] FIG. 16 illustrates the visual data and algorithms of the first XR device 12.1, the second XR device 12.2, and the server 20 according to some embodiments. The components illustrated in FIG. 16 may operate to perform some or all of the operations associated with generating, updating, and / or using spatial information such as a persistent pose, a persistent coordinate frame, a tracking map, or a reference map as described herein. Although not shown, the first XR device 12.1 may be configured identically to the second XR device 12.2. The server 20 may include a map storage routine 118, a reference map 120, a map transmitter 122, and a map merging algorithm 124.

[0286] The second XR device 12.2, which may be in the same scene as the first XR device 12.1, may include a persistent coordinate frame (PCF) integration unit 1300, an application 1302 that generates image data 68 that may be used to render virtual objects, and a frame embedding generator 308 (see FIG. 21). In some embodiments, the map download system 126, the PCF identification system 128, the map 2, the localization module 130, the reference map embedder 132, the reference map 133, and the map publisher 136 may be grouped into a passable world unit 1304. The PCF integration unit 1300 may be connected to the passable world unit 1304 and other components of the second XR device 12.2 to enable reading, generating, using, uploading, and downloading of PCFs.

[0287] Maps with PCFs can enable more persistence in a changing world. In some embodiments, locating a tracking map, including, for example, matching features for images, may include selecting features representing persistent content from a map constructed by the PCF, which enables fast matching and / or location. For example, a world in which people move in and out of a scene and objects such as doors move relative to a scene requires less storage space and transmission rates, and enables the use of individual PCFs and their relationships to one another (e.g., a unified constellation of PCFs) to map a scene.

[0288] In some embodiments, the PCF integration unit 1300 may include three converters including a PCF 1306 previously stored in a data storage on a storage unit of the second XR device 12.2, a PCF tracker 1308, a persistent attitude obtainer 1310, a PCF verifier 1312, a PCF generation system 1314, a coordinate frame calculator 1316, a persistent attitude calculator 1318, a tracking map and persistent attitude converter 1320, a persistent attitude and PCF converter 1322, and a PCF and image data converter 1324.

[0289] In some embodiments, the PCF tracker 1308 may have an on prompt and an off prompt that are selectable by the application 1302. The application 1302 may be executable by a processor of the second XR device 12.2 to display virtual content, for example. The application 1302 may have a call, via an on prompt, to switch the PCF tracker 1308 on. The PCF tracker 1308 may generate a PCF when the PCF tracker 1308 is switched on. The application 1302 may have a subsequent call, via an off prompt, to switch the PCF tracker 1308 off. The PCF tracker 1308 terminates PCF generation when the PCF tracker 1308 is switched off.

[0290] In some embodiments, the server 20 may include a plurality of sustained attitudes 1332 and a plurality of PCFs 1330 that have been previously stored in association with the reference map 120. The map transmitter 122 may transmit the reference map 120 together with the sustained attitudes 1332 and / or the PCFs 1330 to the second XR device 12.2. The sustained attitudes 1332 and the PCFs 1330 may be stored on the second XR device 12.2 in association with the reference map 133. Once the map 2 is located relative to the reference map 133, the sustained attitudes 1332 and the PCFs 1330 may be stored in association with the map 2.

[0291] In some embodiments, the persistent attitude obtainer 1310 may obtain a persistent attitude for map 2. A PCF verifyer 1312 may be coupled to the persistent attitude obtainer 1310. The PCF verifyer 1312 may read a PCF from the PCF 1306 based on the persistent attitude read by the persistent attitude obtainer 1310. The PCFs read by the PCF verifyer 1312 may form an initial group of PCFs to be used for image display based on the PCF.

[0292] In some embodiments, the application 1302 may request that additional PCFs be generated. For example, if the user moves into an area not previously mapped, the application 1302 may turn on the PCF tracker 1308. The PCF generation system 1314 may be connected to the PCF tracker 1308 and begin generating PCFs based on map 2 as map 2 begins to expand. The PCFs generated by the PCF generation system 1314 may form a second group of PCFs that may be used for PCF-based image display.

[0293] The coordinate frame calculator 1316 may be connected to the PCF checker 1312. After the PCF checker 1312 reads the PCF, the coordinate frame calculator 1316 may retrieve the head coordinate frame 96 and determine the head pose of the second XR device 12.2. The coordinate frame calculator 1316 may also retrieve the sustained pose calculator 1318. The sustained pose calculator 1318 may be directly or indirectly connected to the frame embedding generator 308. In some embodiments, an image frame may be designated as a key frame after a threshold distance, e.g., 3 meters, from the previous key frame has been advanced. The sustained pose calculator 1318 may generate a sustained pose based on multiple, e.g., three, key frames. In some embodiments, the sustained pose may essentially be an average of the coordinate frames of multiple key frames.

[0294] A tracking map and sustained attitude converter 1320 may be coupled to the map 2 and sustained attitude calculator 1318. The tracking map and sustained attitude converter 1320 may convert map 2 to a sustained attitude and determine a sustained attitude at the origin for map 2.

[0295] The persistent attitude and PCF converter 1322 may be connected to the tracking map and persistent attitude converter 1320 and further to the PCF validator 1312 and the PCF generation system 1314. The persistent attitude and PCF converter 1322 may convert the persistent attitude (for which the tracking map has been converted) to a PCF from the PCF validator 1312 and the PCF generation system 1314 and determine the PCF for the persistent attitude.

[0296] The PCF and image data converter 1324 may be connected to the persistent attitude and PCF converter 1322 and the data channel 62. The PCF and image data converter 1324 converts the PCF into image data 68. The rendering engine 30 is connected to the PCF and image data converter 1324 and may display the image data 68 for the PCF to a user.

[0297] The PCF integration unit 1300 may store additional PCFs generated using the PCF generation system 1314 in the PCF 1306. The PCF 1306 may be stored for a sustained attitude. The map publisher 136 may read the PCF 1306 and the sustained attitude associated with the PCF 1306 when the map publisher 136 transmits map 2 to the server 20 and the map publisher 136 also transmits the PCF and sustained attitude associated with map 2 to the server 20. When the map storage routine 118 of the server 20 stores map 2, the map storage routine 118 may also store the sustained attitude and PCF generated by the second viewing device 12.2. The map merge algorithm 124 may create the reference map 120 with the sustained attitude and PCF of map 2 associated with the reference map 120 and stored in the sustained attitude 1332 and PCF 1330, respectively.

[0298] The first XR device 12.1 may include a PCF integration unit similar to the PCF integration unit 1300 of the second XR device 12.2. When the map transmitter 122 transmits the reference map 120 to the first XR device 12.1, the map transmitter 122 may transmit the sustained attitude 1332 and the PCF 1330 associated with the reference map 120 and resulting from the second XR device 12.2. The first XR device 12.1 may store the PCF and sustained attitude in a data storage device on the storage device of the first XR device 12.1. The first XR device 12.1 may then utilize the sustained attitude and the PCF resulting from the second XR device 12.2 for image display on the PCF. Additionally or alternatively, the first XR device 12.1 may read, generate, utilize, upload and download PCFs and persistent postures in a manner similar to the second XR device 12.2, as described above.

[0299] In the illustrated embodiment, the first XR device 12.1 generates a local tracking map (hereinafter referred to as “Map 1”), and the map storage routine 118 receives Map 1 from the first XR device 12.1. The map storage routine 118 then stores Map 1 as a reference map 120 on the storage device of the server 20.

[0300] The second XR device 12.2 includes a map download system 126, an anchor identification system 128, a location determination module 130, a reference map embedder 132, a local content positioning system 134, and a map publisher 136.

[0301] In use, the map transmitter 122 transmits the reference map 120 to the second XR device 12.2, and the map download system 126 downloads and stores the reference map 120 from the server 20 as the reference map 133.

[0302] The anchor identification system 128 is connected to the world surface determination routine 78. The anchor identification system 128 identifies anchors based on objects detected by the world surface determination routine 78. The anchor identification system 128 uses the anchors to generate a second map (Map 2). As shown by cycle 138, the anchor identification system 128 continues to identify anchors and update Map 2. The locations of the anchors are recorded as three-dimensional data based on the data provided by the world surface determination routine 78. The world surface determination routine 78 receives images from the real object detection camera 44 and depth data from the depth sensor 135 and determines the location of the surface and its relative distance from the depth sensor 135.

[0303] A location module 130 is connected to the reference map 133 and map 2. The location module 130 repeatedly attempts to locate map 2 relative to the reference map 133. A reference map embedder 132 is connected to the reference map 133 and map 2. Once the location module 130 locates map 2 relative to the reference map 133, the reference map embedder 132 embeds the reference map 133 into the anchor of map 2. Map 2 is then updated with the missing data contained in the reference map.

[0304] A local content positioning system 134 is connected to map 2. The local content positioning system 134 may be, for example, a system in which a user may locate local content at a particular location in a world coordinate frame. The local content itself is then tied to an anchor in map 2. The local-to-world coordinate converter 104 converts the local coordinate frame to the world coordinate frame based on the settings of the local content positioning system 134. The functionality of the rendering engine 30, the display system 42, and the data channel 62 are described with reference to FIG. 2.

[0305] Map publisher 136 uploads Map 2 to server 20. Map storage routine 118 of server 20 then stores Map 2 in the server 20's storage medium.

[0306] A map merging algorithm 124 merges map 2 with the reference map 120. When more than two maps, for example three or four maps, relating to the same or adjacent regions of the physical world are stored, the map merging algorithm 124 merges all maps into the reference map 120 and renders the new reference map 120. The map transmitter 122 then transmits the new reference map 120 to every device 12.1 and 12.2 within the area represented by the new reference map 120. Once devices 12.1 and 12.2 locate their respective maps against the reference map 120, the reference map 120 becomes a promoted map.

[0307] FIG. 17 illustrates an example of generating keyframes for a map of a scene, according to some embodiments. In the illustrated example, a first keyframe KF1 is generated for a door on the left wall of a room. A second keyframe KF2 is generated for the area in the corner where the floor, left wall, and right wall of the room meet. A third keyframe KF3 is generated for the window area on the right wall of the room. A fourth keyframe KF4 is generated for the area at the edge of a rug on the floor of the wall. A fifth keyframe KF5 is generated for the area of ​​the rug closest to the user.

[0308] FIG. 18 illustrates an example of generating a persistence posture for the map of FIG. 17, according to some embodiments. In some embodiments, a new persistence posture is created when the device measures a threshold distance traveled and / or an application requests a new persistence posture (PP). In some embodiments, the threshold distance may be 3 meters, 5 meters, 20 meters, or any other suitable distance. Selecting a smaller threshold distance (e.g., 1 m) may result in an increase in computational load since a larger number of PPs may be created and managed compared to a larger threshold distance. Selecting a larger threshold distance (e.g., 40 m) may result in fewer PPs being created and fewer PCFs being created, which may result in an increase in virtual content placement error since virtual content tied to a PCF may be a relatively large distance (e.g., 30 m) away from the PCF, meaning that the error may increase with the increase in the distance from the PCF to the virtual content.

[0309] In some embodiments, a PP may be created at the beginning of a new session. This initial PP may be considered zero and visualized as the center of a circle with a radius equal to the threshold distance. When the device reaches the circumference of the circle and, in some embodiments, an application requests a new PP, the new PP may be placed at the device's current location (threshold distance). In some embodiments, a new PP will not be created at the threshold distance if the device is able to find an existing PP within the threshold distance from the device's new location. In some embodiments, when a new PP (PP 1150 in FIG. 14) is created, the device binds one or more of the closest key frames to the PP. In some embodiments, the location of the PP relative to the key frame may be based on the device's location at the time the PP is created. In some embodiments, a PP will not be created even if the device progresses the threshold distance unless an application requests the PP.

[0310] In some embodiments, an application may request a PCF from a device when the application has virtual content to display to a user. A PCF request from an application may trigger a PP request, and a new PP will be created after the device progresses a threshold distance. FIG. 18 illustrates a first sustained posture PP1 that may cause the closest keyframes (e.g., KF1, KF2, and KF3) to be linked, for example, by calculating the relative posture between the keyframes and the sustained posture. FIG. 18 also illustrates a second sustained posture PP2 that may cause the closest keyframes (e.g., KF4 and KF5) to be linked.

[0311] FIG. 19 illustrates an example of generating a PCF for the map of FIG. 17, according to some embodiments. In the illustrated example, PCF1 may include PP1 and PP2. As described above, the PCF may be used to display image data for the PCF. In some embodiments, each PCF may have coordinates in another coordinate frame (e.g., a world coordinate frame) and a PCF descriptor, e.g., that uniquely identifies the PCF. In some embodiments, the PCF descriptor may be calculated based on feature descriptors of features in a frame associated with the PCF. In some embodiments, various constellations of PCFs may be combined to represent the real world in a persistent manner, requiring less data and less transmission of data.

[0312] 20A-20C are schematic diagrams illustrating an example of establishing and using persistent coordinate frames. FIG. 20A shows two users 4802A, 4802B with their respective local tracking maps 4804A, 4804B that are not localized relative to a reference map. The origins 4806A, 4806B for each user are described by a coordinate system (e.g., a world coordinate system) within their respective areas. These origins of each tracking map may be local to each user since the origin depends on the orientation of that respective device when tracking begins.

[0313] As the user device's sensors scan the environment, the device may capture images that may contain features representative of persistent objects such that those images may be classified as keyframes from which persistent poses may be created, as described above in connection with Figure 14. In this example, tracking map 4802A includes persistent pose (PP) 4808A and tracking map 4802B includes PP 4808B.

[0314] Also, as described above in connection with Figure 14, some of the PPs may be classified as PCFs, which are used to determine the orientation of virtual content to render it to the user. Figure 20B shows that XR devices worn by individual users 4802A, 4802B may create local PCFs 4810A, 4810B based on PPs 4808A, 4808B. Figure 20C shows that persistent content 4812A, 4812B (e.g., virtual content) may be bound to the PCFs 4810A, 4810B by individual XR devices.

[0315] In this example, the virtual content may have a virtual content coordinate frame that may be used by the application generating the virtual content regardless of how the virtual content is to be displayed. The virtual content may be defined as surfaces, such as triangles of a mesh, at particular locations and angles relative to the virtual content coordinate frame. To render that virtual content to a user, the locations of those surfaces may be determined relative to the user who will perceive the virtual content.

[0316] Binding virtual content to a PCF may simplify the calculations involved in determining the location of the virtual content relative to a user. The location of the virtual content relative to a user may be determined by applying a series of transformations. Some of those transformations may change and be updated frequently. Others of those transformations may be stable and updated less frequently or not at all. Nonetheless, the transformations may be applied with a relatively low computational burden such that the location of the virtual content is updated frequently relative to the user and may provide a realistic appearance to the rendered virtual content.

[0317] In the example of Figures 20A-20C, user 1's device has a coordinate system that may be related to the coordinate system that defines the map origin by a transformation rig1_T_w1. User 2's device has a similar transformation rig2_T_w2. These transformations may be expressed as 6 degree transformations and define a translation and rotation to align the device and map coordinate systems. In some embodiments, the transformations may be expressed as two separate transformations, one defining the translation and the other defining the rotation. Thus, it should be understood that the transformations may be expressed in a form that simplifies calculations or otherwise provides advantages.

[0318] The transformations between the origin of the tracking map and the PCFs identified by the individual user devices are represented as pcf1_T_w1 and pcf2_T_w2. In this embodiment, the PCFs and PPs are the same, such that the same transformation also characterizes the PP.

[0319] The location of the user device relative to the PCF is therefore rig1_T_pcf1=(rig1_T_w1) * It can be calculated by successive applications of these transformations such as (pcf1_T_w1).

[0320] As shown in Figure 20C, the virtual content is located relative to the PCF using the transformation obj1_T_pcf1. This transformation may be set by the application generating the virtual content, which may receive information from a world reconstruction system that describes the physical objects relative to the PCF. To render the virtual content to the user, a transformation to the coordinate system of the user's device is calculated, which is the transformation obj1_t_w1 = (obj1_T_pcf1). * The virtual content coordinate frame can be calculated by relating it to the origin of the tracking map via (pcf1_T_w1), which can then be related to the user's device via a further transformation rig1_T_w1.

[0321] The location of the virtual content may change based on the output from the application generating the virtual content. As it changes, the end-to-end transformation from the source coordinate system to the destination coordinate system may be recalculated. In addition, the user's location and / or head pose may also change as the user moves. As a result, any end-to-end transformation that depends on the user's location or head pose will also change, just as the transformation rig1_T_w1 may change.

[0322] The transformation rig1_T_w1 may be updated with the user's movement based on tracking the user's position relative to stationary objects in the physical world. Such tracking may be performed by a headphone tracking component that processes a sequence of images, as described above, or by other components of the system. Such updates may be performed by determining the user's pose relative to a stationary reference frame, such as PP.

[0323] In some embodiments, the location and orientation of the user device may be determined relative to the nearest persistent pose, or in this example, the PCF as PP is used as the PCF. Such a determination may be made by identifying feature points that characterize the PP in a current image captured with a sensor on the device. Using image processing techniques such as stereoscopic image analysis, the location of the device relative to those feature points may be determined. From this data, the system calculates the relationship rig1_T_pcf1=(rig1_T_w1) * Based on (pcf1_T_w1), the change in transformation associated with the user's movement may be calculated.

[0324] The system may determine and apply the transformations in an order that is computationally efficient. For example, the need to calculate rig1_T_w1 from the measurements that result in rig1_T_pcf1 may be avoided by both tracking the user pose and defining the location of the virtual content relative to a PP or PCF that is built on a persistent pose. In this way, the transformation from the source coordinate system of the virtual content to the destination coordinate system of the user's device can be calculated using the representation (rig1_T_pcf1) * The end-to-end transformation may be based on a measured transformation according to (obj1_t_pcf1), where the first transformation is measured by the system and the latter transformation is supplied by the application that defines the virtual content for rendering. In embodiments where the virtual content is positioned relative to the map origin, the end-to-end transformation may relate the virtual object coordinate system to the PCF coordinate system based on a further transformation between map coordinates and PCF coordinates. In embodiments where the virtual content is positioned relative to a different PP or PCF than the one for which the user position is tracked, a transformation between the two may be applied. Such a transformation may be fixed or may be determined, for example, from the map in which both appear.

[0325] A transform-based approach may be implemented in a device, for example, with components that process sensor data and build a tracking map. As part of that process, those components may identify feature points that can be used as persistent poses, which may in turn be turned into PCFs. Those components may limit the number of persistent poses generated for the map and provide suitable spacing between persistent poses, as described above in connection with FIGS. 17-19, while allowing a user to be close enough to a persistent pose location to accurately calculate the user's pose, regardless of location in the physical environment. As the persistent pose closest to the user is updated as a result of user movement, refinements to the tracking map, or other causes, any of the transforms used to calculate the location of the virtual content relative to the user, which depend on the location of the PP (or PCF, if used), may be updated and stored for use at least until the user moves away from that persistent pose. Note that by calculating and storing the transforms, the computational burden each time the location of the virtual content is updated may be relatively low such that it may be implemented with relatively low latency.

[0326] 20A-20C illustrate positioning relative to a tracking map, with each device having its own tracking map. However, transformations may be generated relative to any map coordinate system. Persistence of content across user sessions of the XR system may be achieved by using a persistent map. Shared user experiences may also be facilitated by using a map to which multiple user devices may be oriented.

[0327] In some embodiments, described in more detail below, the location of virtual content may be defined relative to coordinates in a reference map, which is formatted so that any of multiple devices may use the map. Each device may maintain a tracking map and may determine changes in the user's pose relative to the tracking map. In this example, the transformation between the tracking map and the reference map may be determined through a process of "localization," which may be performed by matching structures in the tracking map (such as one or more sustained poses) with one or more structures of the reference map (such as one or more PCFs).

[0328] Described further below are techniques for creating and using such reference maps.

[0329] Deep Keyframes

[0330] The technique as described herein relies on the comparison of image frames. For example, to establish the position of the device relative to the tracking map, a new image may be captured using a sensor worn by the user, and the XR system may search for images in the set of images used to create the tracking map that share at least a predetermined amount of points of interest with the new image. As an example of another scenario involving image frame comparison, the tracking map may be located relative to the reference map by first finding an image frame associated with a sustained pose in the tracking map that is similar to an image frame associated with a PCF in the reference map. Alternatively, the transformation between the two reference maps may be calculated by first finding similar image frames in the two maps.

[0331] Deep keyframes provide a method for reducing the amount of processing required to identify similar image frames. For example, in some embodiments, a comparison may be made between image features (e.g., "2D features") in the new 2D image and 3D features in the map. Such a comparison may be made in any suitable manner, such as by projecting the 3D image into a 2D plane. Conventional methods, such as bag of words (BoW), search for the 2D features of the new image in a database that includes all the 2D features in the map, which may require significant computational resources, especially when the map represents a large area. Conventional methods then locate images that share at least one of the 2D features with the new image, which may include images that are not useful for locating meaningful 3D features in the map. Conventional methods then locate 3D features that are not meaningful to the 2D features in the new image.

[0332] The inventors have recognized and understood techniques for retrieving images in a map using fewer memory resources (e.g., one-quarter of the memory resources used by BoW), more efficient (e.g., 2.5 ms processing time per keyframe, 100 μs for comparison against 500 keyframes), and higher accuracy (e.g., 20% better read-out recall than BoW for 1,024 dimensional models, 5% better read-out recall than BoW for 256 dimensional models).

[0333] To reduce computations, a descriptor may be computed for an image frame that may be used to compare the image frame to other image frames. The descriptor may be stored instead of or in addition to the image frame and feature points. In a map from which sustained poses and / or PCFs may be generated from image frames, a descriptor for the image frame or frames from which each sustained pose or PCF was generated may be stored as part of the sustained pose and / or PCF.

[0334] In some embodiments, the descriptor may be calculated as a function of feature points in an image frame. In some embodiments, a neural network is configured to calculate a unique frame descriptor to represent the image. The image may have a resolution higher than 1 megabyte so that sufficient detail of the 3D environment within the field of view of the device worn by the user is captured in the image. The frame descriptor may be much smaller, such as a string of numbers, for example, any number in the range or between 128 bytes and 512 bytes.

[0335] In some embodiments, the neural network is trained such that the calculated frame descriptor indicates the similarity between images. Images in the map may be located by identifying the nearest image in the database that comprises the images used to generate the map, which may have a frame descriptor within a predetermined distance to the frame descriptor for the new image. In some embodiments, the distance between images may be represented by the difference between the frame descriptors of the two images.

[0336] 21 is a block diagram illustrating a system for generating a descriptor for an individual image, according to some embodiments. In the illustrated example, a frame embedding generator 308 is shown. The frame embedding generator 308 may be used in conjunction with the server 20 in some embodiments, but may alternatively or additionally be executed in whole or in part within one of the XR devices 12.1 and 12.2, or any other device that processes images for comparison with other images.

[0337] In some embodiments, the frame embedding generator may be configured to generate a data representation of the image reduced from an initial size (e.g., 76,800 bytes) to a final size (e.g., 256 bytes) that still indicates the content in the image despite the reduced size. In some embodiments, the frame embedding generator may be used to generate a data representation for the image, which may be a key frame or frame used in other methods. In some embodiments, the frame embedding generator 308 may be configured to convert the image at a particular location and orientation into a unique string of numbers (e.g., 256 bytes). In the illustrated example, an image 320 captured by an XR device may be processed by a feature extractor 324 to detect points of interest 322 in the image 320. The points of interest may or may not be derived from identified feature points as described above with respect to features 1120 (FIG. 14) or as otherwise described herein. In some embodiments, the points of interest may be represented by a descriptor, as described above with respect to descriptor 1130 (FIG. 14), which may be generated using deep sparse feature methods. In some embodiments, each point of interest 322 may be represented by a string of numbers (e.g., 32 bytes). For example, there may be n features (e.g., 100), each represented by a string of 32 bytes.

[0338] In some embodiments, the frame embedding generator 308 may include a neural network 326. The neural network 326 may include a multi-layer perceptron unit 312 and a max pooling unit 314. In some embodiments, the multi-layer perceptron (MLP) unit 312 may comprise a multi-layer perceptron, which may be trained. In some embodiments, the points of interest 322 (e.g., descriptors for the points of interest) may be reduced by the multi-layer perceptron 312 and output as a weighted combination of descriptors 310. For example, the MLP may reduce n features to m features, which is less than n features.

[0339] In some embodiments, the MLP unit 312 may be configured to perform matrix multiplication. The multi-layer perceptron unit 312 receives a number of points of interest 322 of the image 320 and converts each point of interest into a separate string of numbers (e.g., 256). For example, there may be 100 features, and each feature may be represented by a string of 256 numbers. The matrix may be created to have, in this example, 100 horizontal rows and 256 vertical columns. Each row may have a series of 256 numbers of varying magnitude, some smaller and some larger. In some embodiments, the output of the MLP may be an n×256 matrix, where n represents the number of features extracted from the image. In some embodiments, the output of the MLP may be an m×256 matrix, where m is the number of points of interest reduced from n.

[0340] In some embodiments, the MLP 312 may have a training phase during which model parameters for the MLP are determined, and a use phase. In some embodiments, the MLP may be trained as illustrated in Figure 25. The input training data may comprise data in three sets, the three sets comprising 1) a query image, 2) positive samples, and 3) negative samples. The query image may be considered a reference image.

[0341] In some embodiments, the positive sample may comprise an image that is similar to the query image. For example, in some embodiments, similar may mean having the same object in both the query and positive sample images, but viewed from different angles. In some embodiments, similar may mean having the same object in both the query and positive sample images, but having an object that is shifted (e.g., left, right, up, down) relative to the other image.

[0342] In some embodiments, the negative samples may comprise images that are dissimilar to the query image. For example, in some embodiments, dissimilar images may not contain any objects salient in the query image, or may contain only a small portion (e.g., <10%, 1%) of objects salient in the query image. Similar images, in contrast, may have, for example, a large portion (e.g., >50%, or >75%) of objects in the query image.

[0343] In some embodiments, points of interest may be extracted from images in the input training data and converted into feature descriptors. These descriptors may be calculated for both the training images, as shown in FIG. 25, and for features extracted during operation of the frame embedding generator 308 of FIG. 21. In some embodiments, a deep sparse feature (DSF) process may be used to generate the descriptors (e.g., DSF descriptors), as described in U.S. Patent Application Serial No. 16 / 190,948. In some embodiments, the DSF descriptors are n×32 in size. The descriptors may then be passed through the model / MLP to create a 256-byte output. In some embodiments, the model / MLP may have the same structure as the MLP 312, such that once the model parameters are set through training, the resulting trained MLP may be used as the MLP 312.

[0344] In some embodiments, the feature descriptors (e.g., the 256 bytes output from the MLP model) may then be sent to a triplet margin loss module (used only during the training phase of the MLP neural network and may not be used during the usage phase). In some embodiments, the triplet margin loss module may be configured to select parameters for the model to reduce the difference between the 256 bytes output from the query image and the 256 bytes output from the positive samples and to increase the difference between the 256 bytes output from the query image and the 256 bytes output from the negative samples. In some embodiments, the training phase may include feeding a plurality of triplet input images into a learning process to determine model parameters. This training process may continue, for example, until the difference for positive images is minimized and the difference for negative images is maximized, or until other suitable termination criteria is reached.

[0345] Referring back to FIG. 21, the frame embedding generator 308 may include a pooling layer, here illustrated as a max pooling unit 314. The max pooling unit 314 may analyze each column and determine the maximum number in the individual columns. The max pooling unit 314 may combine the maximum value of each column of the number of output matrices of the MLP 312 into a global feature column 316 of, for example, 256 numbers. It should be understood that the images processed in the XR system may desirably have high resolution frames, potentially with millions of pixels. The global feature column 316 is a relatively small number that occupies relatively little memory and is easily searchable compared to images (e.g., with a resolution higher than 1 megabyte). Thus, it is possible to search the images without analyzing each original frame from the camera, and it is also cheaper to store 256 bytes instead of a full frame.

[0346] 22 is a flow chart illustrating a method 2200 of calculating image descriptors according to some embodiments. The method 2200 may begin with receiving a plurality of images captured by an XR device worn by a user (act 2202). In some embodiments, the method 2200 may include determining one or more key frames from the plurality of images (act 2204). In some embodiments, act 2204 may be skipped and / or may instead occur after step 2210.

[0347] The method 2200 may include identifying one or more points of interest in the plurality of images using an artificial neural network (act 2206) and computing feature descriptors for each point of interest using the artificial neural network (act 2208). The method may also include computing, for each image, a frame descriptor to represent the image based, at least in part, on the computed feature descriptors for the identified points of interest in the image using the artificial neural network (act 2210).

[0348] FIG. 23 is a flow chart illustrating a method 2300 of localization using image descriptors, according to some embodiments. In this example, a new image frame depicting the current location of the XR device may be compared to stored image frames associated with points in the map (such as a persistent pose or PCF, as described above). The method 2300 may begin with receiving a new image captured by an XR device worn by a user (act 2302). The method 2300 may include identifying one or more nearest key frames in a database (act 2304), which comprise key frames used to generate one or more maps. In some embodiments, the nearest key frames may be identified based on coarse spatial information and / or previously determined spatial information. For example, the coarse spatial information may indicate that the XR device is present within a geographic region represented by a 50m x 50m area of ​​the map. Image matching may be performed only for points within that area. As another example, based on the tracking, the XR system may know that the XR device was previously proximate to a first sustained pose in the map and was moving in the direction of a second sustained pose in the map. That second sustained pose may be considered the nearest sustained pose, and the keyframe stored therewith may be considered the nearest keyframe. Alternatively or in addition, other metadata, such as GPS data or WiFi fingerprints, may also be used to select the nearest keyframe or set of nearest keyframes.

[0349] Regardless of how the nearest keyframe is selected, the frame descriptor may be used to determine whether the new image matches any of the frames selected as being associated with nearby sustained poses. The determination may be made by comparing the frame descriptor of the new image with the frame descriptors of the nearest keyframes or a subset of keyframes in the database selected in any other suitable manner, and selecting the keyframe with a frame descriptor within a predefined distance of the frame descriptor of the new image. In some embodiments, the distance between two frame descriptors may be calculated by taking the difference between two strings of numbers that may represent the two frame descriptors. In embodiments where the strings are treated as a string of multiple quantities, the difference may be calculated as a vector difference.

[0350] Once a matching image frame is identified, the orientation of the XR device relative to that image frame may be determined. Method 2300 may include performing feature matching on 3D features in the map that correspond to the identified closest key frame (act 2306) and calculating a pose of the device worn by the user based on the feature matching results (act 2308). In this manner, computationally intensive matching of feature points in two images may be performed with respect to as little as one image that has already been determined to be a likely match for the new image.

[0351] 24 is a flow chart illustrating a method 2400 of training a neural network, according to some embodiments. The method 2400 may begin with generating a dataset (act 2402) comprising a plurality of image sets. Each of the plurality of image sets may include a query image, a positive sample image, and a negative sample image. In some embodiments, the plurality of image sets may include synthetic record pairs that are configured to teach the neural network basic information, such as shape, for example. In some embodiments, the plurality of image sets may include real record pairs that may be recorded from the physical world.

[0352] In some embodiments, the positive correspondence may be calculated by matching fundamental matrices between the two images. In some embodiments, the sparse overlap may be calculated as the intersection over union (IoU) of points of interest found in both of the images. In some embodiments, a positive sample may include at least 20 points of interest that are identical in the query image and serve as positive correspondences. A negative sample may include less than 10 positive correspondences. A negative sample may have less than half of the sparse points overlapping with analysis points of the query image.

[0353] Method 2400 may include calculating a loss by comparing the query image with the positive and negative sample images for each set of images (act 2404). Method 2400 may also include modifying the artificial neural network based on the calculated loss such that a distance between a frame descriptor generated by the artificial neural network for the query image and a frame descriptor for the positive sample images is less than a distance between the frame descriptor for the query image and a frame descriptor for the negative sample images (act 2406).

[0354] Although methods and apparatus configured to generate a global descriptor for an individual image are described above, it should be understood that the methods and apparatus may also be configured to generate a descriptor for an individual map. For example, a map may include multiple keyframes, each having a frame descriptor as described above. The max pooling unit may analyze the frame descriptors of the keyframes of a map and combine the frame descriptors into a unique map descriptor for the map.

[0355] It should be further understood that other architectures may be used for processing as described above. For example, separate neural networks are described for generating the DSF descriptor and the frame descriptor. Such an approach is computationally efficient. However, in some embodiments, the frame descriptor may be generated from selected feature points without first generating the DSF descriptor. Ranking and merging maps

[0356] Described herein are methods and apparatus for ranking and merging multiple environmental maps within a cross-reality (XR) system. Map merging may allow maps representing overlapping portions of the physical world to be combined to represent a larger area. Ranking the maps may allow for efficient implementation of techniques as described herein, including map merging, which involves selecting a map from a set of maps based on similarity. In some embodiments, a set of reference maps may be maintained by the system, for example, formatted in a manner that may be accessed by any of several XR devices. These reference maps may be formed by merging selected tracking maps from those devices with other tracking maps or previously stored reference maps. The reference maps may be ranked, for example, to select one or more reference maps and merge them with new tracking maps, and / or to select one or more reference maps from the set and use in the device.

[0357] To provide a user with a realistic XR experience, the XR system must know the user's physical surroundings in order to correctly correlate the location of virtual objects in relation to real objects. Information about the user's physical surroundings may be obtained from an environmental map of the user's location.

[0358] The inventors have recognized and appreciated that an XR system may provide an enhanced XR experience to multiple users sharing the same world, comprising real and / or virtual content, regardless of whether those users are present in the world at the same or different times, by enabling efficient sharing of real / physical world environmental maps collected by multiple users. However, significant challenges exist in providing such a system. Such a system may store multiple maps generated by multiple users, and / or the system may store multiple maps generated at different times. For example, as described above, with respect to operations that may be performed with previously generated maps, such as localization, substantial processing may be required to identify relevant environmental maps of the same world (e.g., the same real-world location) from all environmental maps collected within the XR system. In some embodiments, there may be only a small number of environmental maps that a device may access, for example, for localization. In some embodiments, there may be a large number of environmental maps that a device may access. The inventors have recognized and appreciated techniques for quickly and accurately ranking the relevance of environmental maps from all possible environmental maps, such as, for example, the population of all reference maps 120 in Figure 28. Highly ranked maps may then be selected for further processing, such as for rendering virtual objects on a user display to interact realistically with the physical world around the user, or for merging the stored map with map data collected by the user to create a larger or more accurate map.

[0359] In some embodiments, stored maps relevant to a task for a user at a location in the physical world may be identified by filtering the stored maps based on multiple criteria. The criteria may refer to a comparison of a tracked map generated by the user's wearable device at the location with a candidate environment map stored in a database. The comparison may be performed based on metadata associated with the map, such as a Wi-Fi fingerprint detected by the device generating the map, and / or the set of BSSIDs to which the device is connected while forming the map. The comparison may also be performed based on compressed or decompressed content of the map. A comparison based on a compressed representation may be performed, for example, by comparing vectors calculated from the map content. A comparison based on an uncompressed map may be performed, for example, by locating the tracked map within the stored map, or vice versa. Multiple comparisons may be performed in an order based on the calculation time required to reduce the number of candidate maps for consideration, with comparisons involving fewer calculations being performed before other comparisons requiring more calculations.

[0360] FIG. 26 depicts an AR system 800 configured to rank and merge one or more environmental maps according to some embodiments. The AR system may include a passable world model 802 of the AR device. Information for populating the passable world model 802 may come from sensors on the AR device, which may include computer-executable instructions stored in a processor 804 (e.g., local data processing module 570 in FIG. 4) that may perform some or all of the processing for converting sensor data into a map. Such a map may be a tracking map, such as may be built as sensor data is collected as the AR device operates within an area. Along with the tracking map, area attributes may be provided to indicate the area that the tracking map represents. These area attributes may be coordinates presented as latitude and longitude or geographic location identifiers, such as IDs used by the AR system to represent a location. Alternatively or in addition, the area attributes may be measured characteristics that have a high likelihood of being unique for the area. The area attributes may be derived, for example, from parameters of wireless networks detected within the area. In some embodiments, the area attribute may be associated with a unique address of an access point to which the AR system is nearby and / or connected. For example, the area attribute may be associated with a MAC address or Basic Service Set Identifier (BSSID) of a 5G base station / router, Wi-Fi router, and the like.

[0361] 26, the tracking map may be merged with other maps of the environment. The map ranking portion 806 receives the tracking map from the device PW 802, communicates with the map database 808, and selects and ranks the environment maps from the map database 808. The selected maps that are ranked higher are sent to the map merging portion 810.

[0362] The map merging portion 810 may perform a merging process on the maps sent from the map ranking portion 806. The merging process may involve merging some or all of the ranked maps with the tracking maps and transmitting the new merged map to the passable world model 812. The map merging portion may merge maps by identifying maps that depict overlapping portions of the physical world. Those overlapping portions may be matched so that information in both maps may be aggregated in a final map. Reference maps may be merged with other reference maps and / or tracking maps.

[0363] Aggregation may involve augmenting one map with information from another map. Alternatively, or in addition, aggregation may involve adjusting the representation of the physical world in one map based on information in another map. The latter map may represent, for example, that an object causing a feature point has moved so that the map may be updated based on the latter information. Alternatively, the two maps may characterize the same region with different feature points, and aggregation may involve selecting a set of feature points from the two maps to better represent the region. Regardless of the specific processing that occurs in the merging process, in some embodiments, PCFs from all maps that are merged may be retained so that applications that position content relative to them may continue to do so. In some embodiments, merging maps may result in redundant persistent postures, and some of the persistent postures may be deleted. When a PCF is associated with a persistent posture that is to be deleted, merging maps may involve modifying the PCF to be associated with the persistent posture that remains in the map after the merge.

[0364] In some embodiments, as maps are expanded and / or updated, they may be refined. Refinement may involve calculations to reduce internal discrepancies between feature points that are likely to represent the same object in the physical world. Discrepancies may arise from inaccuracies in the poses associated with keyframes that provide feature points that represent the same object in the physical world. Such discrepancies may arise, for example, from the XR device calculating the pose for a tracking map on which it builds upon estimating the pose, such that errors in the pose estimate accumulate and create a "drift" in the pose accuracy over time. Maps may be refined by performing bundle adjustment or other operations to reduce discrepancies in feature points from multiple keyframes.

[0365] Depending on the refinement, the location of the persistence point relative to the map origin may change. Thus, the transformation associated with that persistence point, such as the persistence pose or PCF, may also change. In some embodiments, the XR system may recalculate the transformations associated with any persistence point that has changed in connection with a map refinement (whether performed as part of a merge operation or for other reasons). These transformations may be pushed from the components that compute the transformations to the components that use the transformations, so that any use of the transformations may be based on the updated location of the persistence point.

[0366] The passable world model 812 may be a cloud model, which may be shared by multiple AR devices. The passable world model 812 may store or otherwise have access to the environment map in a map database 808. In some embodiments, when a previously calculated environment map is updated, the previous version of the map may be deleted to remove the outdated map from the database. In some embodiments, when a previously calculated environment map is updated, the previous version of the map may be archived, enabling reading / viewing of the previous version of the environment. In some embodiments, permissions may be set such that only AR systems with certain read / write access may trigger the deletion / archiving of the previous version of the map.

[0367] These environment maps created from the tracking maps provided by one or more AR devices / systems may be accessed by the AR devices in the AR system. The map ranking portion 806 may also be used in providing the environment maps to the AR devices. The AR device may send a message requesting an environment map for its current location, and the map ranking portion 806 may be used to select and rank the environment map associated with the requesting device.

[0368] In some embodiments, the AR system 800 may include a downsampling portion 814 configured to receive a merged map from the cloud PW 812. The merged map received from the cloud PW 812 may be in a storage format for the cloud, which may include high-resolution information, such as multiple image frames or a large set of feature points associated with a large number of PCFs or PCFs per square meter. The downsampling portion 814 may be configured to downsample the cloud format map to a format suitable for storage on the AR device. The device format map may have less data, such as fewer PCFs or less data stored per PCF, to accommodate the limited local computing capabilities and storage space of the AR device.

[0369] 27 is a simplified block diagram illustrating multiple reference maps 120 that may be stored in a remote storage medium, e.g., the cloud. Each reference map 120 may include multiple reference map identifiers that indicate the location of the reference map in physical space, such as anywhere on planet Earth. These reference map identifiers may include one or more of the following identifiers: an area identifier, represented by a longitude and latitude range; a frame descriptor (e.g., global feature column 316 in FIG. 21); a Wi-Fi fingerprint; a feature descriptor (e.g., feature descriptor 310 in FIG. 21); and a device identification, indicating one or more devices that contributed to the map.

[0370] In the illustrated embodiment, reference maps 120 may reside on the surface of the Earth and thus be geographically arranged in a two-dimensional pattern. Reference maps 120 may be uniquely identifiable by their corresponding longitude and latitude, such that any reference maps with overlapping longitudes and latitudes may be merged into a new reference map.

[0371] FIG. 28 is a schematic diagram illustrating a method of selecting a reference map that may be used to locate a new tracking map against one or more reference maps, according to some embodiments. The method may begin with accessing (act 120) a universe of reference maps 120, which may be stored in a database in the passable world (e.g., the passable world module 538), as an example. The universe of reference maps may include reference maps from all previously visited locations. The XR system may filter the universe of all reference maps to a small subset or only a single map. It should be understood that in some embodiments, due to bandwidth limitations, it may be impossible to transmit all reference maps to the viewing device. Selecting a subset selected as likely candidates for matching a tracking map to transmit to the device may reduce bandwidth and latency associated with accessing a remote database of maps.

[0372] The method may include filtering the universe of criteria maps (act 300) based on areas with a predetermined size and shape. In the example illustrated in FIG. 27, each square may represent an area. Each square may cover 50m×50m. Each square may have six neighboring areas. In some embodiments, act 300 may select at least one matching criteria map 120 that covers a longitude and latitude that includes the longitude and latitude of the location identifier received from the XR device, as long as at least one map exists at that longitude and latitude. In some embodiments, act 300 may select at least one neighboring criteria map that covers a longitude and latitude that is adjacent to the matching criteria map. In some embodiments, act 300 may select multiple matching criteria maps and multiple neighboring criteria maps. Act 300 may, for example, reduce the number of criteria maps by about a factor of ten, for example from thousands to hundreds, to form a first filtered selection. Alternatively, or in addition, criteria other than latitude and longitude may be used to identify nearby maps. The XR device may have been previously located using a reference map in the set, for example as part of the same session. The cloud service may retain information about the XR device, including the previously located maps. In this example, the maps selected in act 300 may include those that cover an area adjacent to the map for which the XR device was located.

[0373] The method may include filtering a first filtered selection of reference maps based on the Wi-Fi fingerprint (act 302). Act 302 may determine a latitude and longitude based on the Wi-Fi fingerprint received as part of the location identifier from the XR device. Act 302 may compare the latitude and longitude from the Wi-Fi fingerprint with the latitude and longitude of the reference map 120 to determine one or more reference maps that form a second filtered selection. Act 302 may reduce the number of reference maps by about a factor of 10, e.g., from hundreds to tens of reference maps (e.g., 50), that form the second selection. For example, the first filtered selection may include 130 reference maps, and the second filtered selection may include 50 of the 130 reference maps and not the remaining 80 of the 130 reference maps.

[0374] The method may include filtering a second filtered selection of the reference maps based on the key frames (act 304). Act 304 may compare data representing the image captured by the XR device with data representing the reference map 120. In some embodiments, the data representing the image and / or map may include feature descriptors (e.g., DSF descriptors in FIG. 25) and / or global feature columns (e.g., 316 in FIG. 21). Act 304 may provide a third filtered selection of the reference maps. In some embodiments, the output of act 304 may be, for example, only five of the 50 reference maps identified following the second filtered selection. The map transmitter 122 then transmits one or more reference maps to the viewing device based on the third filtered selection. Act 304 may reduce the number of reference maps by about a factor of ten, for example, from tens to a single order of magnitude of reference maps (e.g., 5), forming a third selection. In some embodiments, the XR device may receive a reference map within the third filtered selection and attempt to locate within the received reference map.

[0375] For example, act 304 may filter the reference map 120 based on the global feature columns 316 of the reference map 120 and the global feature columns 316 based on an image captured by the viewing device (e.g., an image that may be part of a local tracking map for the user). Each one of the reference maps 120 in FIG. 27 thus has one or more global feature columns 316 associated with it. In some embodiments, the global feature columns 316 may be obtained when the XR device submits the image or feature details to the cloud, which processes the image or feature details and generates the global feature columns 316 for the reference map 120.

[0376] In some embodiments, the cloud may receive feature details of live / new / current images captured by the viewing device, and the cloud may generate a global feature sequence 316 for the live image. The cloud may then filter the reference map 120 based on the live global feature sequence 316. In some embodiments, the global feature sequence may be generated on the local viewing device. In some embodiments, the global feature sequence may be generated remotely, for example, on the cloud. In some embodiments, the cloud may transmit the filtered reference map to the XR device along with the global feature sequence 316 associated with the filtered reference map. In some embodiments, the viewing device may do so by matching the global feature sequence 316 of the local tracking map with the global feature sequence of the reference map once it has located its tracking map relative to the reference map.

[0377] It should be understood that the operation of the XR device may not perform all of the acts (300, 302, 304). For example, if the universe of reference maps is relatively small (e.g., 500 maps), the XR device attempting to locate may filter the universe of reference maps based on Wi-Fi fingerprints (e.g., act 302) and key frames (e.g., act 304), but omit filtering based on area (e.g., act 300). Furthermore, maps do not need to be compared in their entirety. In some embodiments, for example, a comparison of two maps may result in the identification of common persistence points, such as persistence poses or PCFs, that appear in both the new map and the map selected from the universe of maps. In that case, a descriptor may be associated with the persistence points, and those descriptors may be compared.

[0378] 29 is a flow chart illustrating a method 900 of selecting one or more ranked environment maps according to some embodiments. In the illustrated embodiment, the ranking is performed for the user's AR device, which creates the tracking map. Thus, the tracking map is available for use in ranking the environment maps. In embodiments where the tracking map is not available, some or all of the parts of the selection and ranking of the environment maps that do not explicitly rely on the tracking map may be used.

[0379] Method 900 may begin at act 902, where a set of maps from a database of environmental maps (which may be formatted as reference maps) in the vicinity of where the tracking map was formed may be accessed and then filtered for ranking. Additionally, at act 902, at least one area attribute is determined for an area in which the user's AR device is operating. In a scenario in which the user's AR device is building a tracking map, the area attribute may correspond to the area over which the tracking map is created. As a specific example, the area attribute may be calculated based on signals received from an access point to a computer network while the AR device was calculating the tracking map.

[0380] 30 depicts an example map ranking portion 806 of an AR system 800, according to some embodiments. The map ranking portion 806 may include portions that execute on the AR device and portions that execute on a remote computing system, such as a cloud, and thus may execute within a cloud computing environment. The map ranking portion 806 may be configured to implement at least a portion of the method 900.

[0381] FIG. 31A depicts an example of area attributes AA1-AA8 of the tracking map (TM) 1102 and the environment maps CM1-CM4 in the database, according to some embodiments. As shown, the environment map may be associated with multiple area attributes. The area attributes AA1-AA8 may include parameters of the wireless network detected by the AR device that calculates the tracking map 1102, such as the basic service set identifier (BSSID) of the network to which the AR device is connected, and / or the received signal strength of an access point to the wireless network, such as through a network tower 1104. The parameters of the wireless network may conform to protocols including Wi-Fi and 5G NR. In the example illustrated in FIG. 32, the area attributes are fingerprints of the areas in which the user AR device collected sensor data and formed the tracking map.

[0382] FIG. 31B depicts an example of a determined geographic location 1106 of a tracking map 1102, according to some embodiments. In the illustrated example, the determined geographic location 1106 includes a centroid point 1110 and an area 1108 surrounding the centroid point. It should be understood that the determination of geographic locations of the present application is not limited to the format shown. The determined geographic location may have any suitable format, including, for example, different area shapes. In this example, the geographic location is determined from the area attributes using a database that associates the area attributes with the geographic location. The database is commercially available, for example, a database that associates Wi-Fi fingerprints with locations, expressed as latitude and longitude, that can be used for the present operation.

[0383] 29 embodiment, the map database containing the environmental maps may also include location data for those maps, including the latitude and longitude covered by the maps. Processing in act 902 may involve selecting from that database a set of environmental maps that cover the same latitude and longitude determined for the area attribute of the tracking map.

[0384] Act 904 is a first filtering of the set of environmental maps accessed in act 902. In act 902, environmental maps are retained in the set based on their proximity to the geographic location of the tracking map. This filtering step may be performed by comparing the latitudes and longitudes associated with the tracking map and the environmental maps in the set.

[0385] FIG. 32 depicts an example of act 904, according to some embodiments. Each area attribute may have a corresponding geographic location 1202. The set of environment maps may include environment maps with at least one area attribute that has a geographic location that overlaps with the determined geographic location of the tracking map. In the illustrated example, the set of identified environment maps includes environment maps CM1, CM2, and CM4, each having at least one area attribute that has a geographic location that overlaps with the determined geographic location of the tracking map 1102. Environment map CM3, associated with area attribute AA6, is not included in the set because it is outside the determined geographic location of the tracking map.

[0386] Other filtering steps may also be performed on the set of environmental maps to reduce / rank the number of environmental maps in the set that are ultimately processed (such as for map merging or for providing passable world information to the user device). The method 900 may include filtering the set of environmental maps (act 906) based on similarity of identifiers of one or more network access points associated with the environmental maps of the set of tracking maps and environmental maps. During formation of the map, the devices that collect sensor data and generate the map may be connected to the network through network access points, such as through Wi-Fi or a similar wireless communication protocol. The access points may be identified by a BSSID. The user device may connect to multiple different access points as it moves through an area, collects data, and forms the map. Similarly, when multiple devices provide information to form a map, the devices may be connected through different access points, and thus, for that same reason, there may be multiple access points used in forming the map. Thus, there may be multiple access points associated with the map, and the set of access points may be an indication of the location of the map. The strength of the signal from the access points, which may be reflected as an RSSI value, may provide further geographic information. In some embodiments, the list of BSSIDs and RSSI values ​​may form an area attribute for a map.

[0387] In some embodiments, filtering the set of environment maps based on similarity of the one or more identifiers of the network access points may include retaining in the set of environment maps the environment map with the highest Jaccard similarity to at least one area attribute of the tracking map based on the one or more identifiers of the network access points. FIG. 33 depicts an example of act 906, according to some embodiments. In the example shown, a network identifier associated with area attribute AA7 may be determined as the identifier for tracking map 1102. The set of environment maps after act 906 includes environment map CM2, which may have an area attribute within a higher Jaccard similarity with AA7, and environment map CM4, which also includes area attribute AA7. Environment map CM1 is not included in the set because it has the lowest Jaccard similarity with AA7.

[0388] The processing in acts 902-906 may be performed without actually accessing the contents of the map stored in the map database based on metadata associated with the map. Other processing may involve accessing the contents of the map. Act 908 illustrates accessing the environmental maps remaining in the subset after filtering based on metadata. It should be understood that this act may be performed either earlier or later in the process, where subsequent operations may be performed using the accessed content.

[0389] The method 900 may include filtering the set of environment maps based on similarity of a metric representative of the content of the environment maps of the set of tracking maps and environment maps (act 910). The metric representative of the content of the tracking maps and environment maps may include a vector of values ​​calculated from the content of the maps. For example, a deep keyframe descriptor as described above, calculated for one or more keyframes used in forming the maps, may provide a metric for comparison of the maps or portions of the maps. The metric may be calculated from the maps retrieved in act 908, or may be pre-calculated and stored as metadata associated with the maps. In some embodiments, filtering the set of environment maps based on similarity of a metric representative of the content of the environment maps of the set of tracking maps and environment maps may include retaining in the set of environment maps an environment map with a minimum vector distance between a vector of the characteristics of the tracking map and a vector representing the environment map in the set of environment maps.

[0390] The method 900 may further include filtering the set of environment maps (act 912) based on a degree of matching between the portion of the tracking map and the portion of the environment map of the set of environment maps. The degree of matching may be determined as part of the localization process. As a non-limiting example, localization may be performed by identifying interest points in the tracking map and the environment map that are similar enough that they may represent the same portion of the physical world. In some embodiments, the interest points may be features, feature descriptors, keyframes, keyrigs, sustained poses, and / or PCFs. The set of interest points in the tracking map may then be matched to produce a best match with the set of interest points in the environment map. The mean squared distance between corresponding interest points may be calculated and, if it is below a threshold for a particular region of the tracking map, is used as an indication that the tracking map and the environment map represent the same region of the physical world.

[0391] In some embodiments, filtering the set of environment maps based on the degree of matching between the portion of the tracking map and the portion of the environment map of the set of environment maps may include calculating a volume of the physical world represented by the tracking map that is also represented in the environment map of the set of environment maps, and retaining in the set of environment maps environment maps with a calculated volume larger than the filtered out environment map of the set. FIG. 34 depicts an example of act 912, according to some embodiments. In the example shown, the set of environment maps after act 912 includes environment map CM4, which has an area 1402 that matches an area of ​​tracking map 1102. Environment map CM1 is not included in the set because it does not have an area that matches an area of ​​tracking map 1102.

[0392] In some embodiments, the set of environment maps may be filtered in the order of act 906, act 910, and act 912. In some embodiments, the set of environment maps may be filtered based on act 906, act 910, and act 912, which may be performed in an order based on the processing required to perform the filtering from lowest to highest. Method 900 may include loading the set of environment maps and data (act 914).

[0393] In the illustrated embodiment, the user database stores an area identification indicating an area in which the AR device was used. The area identification may be an area attribute, which may include parameters of a wireless network detected by the AR device when in use. The map database may store a plurality of environment maps constructed from data provided by the AR device and associated metadata. The associated metadata may include an area identification derived from an area identification of the AR device that provided the data from which the environment map was constructed. The AR device may send a message to the PW module to indicate that a new tracking map is created or is being created. The PW module may calculate an area identifier for the AR device and update the user database based on the received parameters and / or the calculated area identifier. The PW module may also determine an area identifier associated with the AR device requesting the environment map, identify a set of environment maps from the map database based on the area identifier, filter the set of environment maps, and transmit the filtered set of environment maps to the AR device. In some embodiments, the PW module may filter the set of environmental maps based on one or more criteria including, for example, the geographic location of the tracking map, the similarity of one or more identifiers of network access points associated with the tracking map and the environmental maps of the set of environmental maps, the similarity of metrics representing the content of the tracking map and the environmental maps of the set of environmental maps, and the degree of matching between a portion of the tracking map and a portion of the environmental map of the set of environmental maps.

[0394] Although some aspects of some embodiments have been described above, it should be understood that various alterations, modifications, and improvements will be readily apparent to those skilled in the art. As an example, the embodiments are described in relation to an augmented (AR) environment. It should be understood that some or all of the techniques described herein may be applied in MR environments, and more generally, in other XR and VR environments.

[0395] As another example, embodiments are described in connection with devices, such as wearable devices. It should be understood that some or all of the techniques described herein may be implemented via a network (such as the cloud), a discrete application, and / or any suitable combination of devices, networks, and discrete applications.

[0396] Further, FIG. 29 provides examples of criteria that may be used to filter the candidate maps and result in a set of highly ranked maps. Other criteria may be used instead of or in addition to the criteria described. For example, if multiple candidate maps have similar values ​​of a metric used to filter out less desirable maps, characteristics of the candidate maps may be used to determine which maps are retained as candidate maps or filtered out. For example, larger or denser candidate maps may be prioritized over smaller candidate maps. In some embodiments, FIGS. 27-28 may describe all or part of the systems and methods described in FIGS. 29-34.

[0397] 35 and 36 are schematic diagrams illustrating an XR system configured to rank and merge multiple environment maps, according to some embodiments. In some embodiments, a passable world (PW) may determine when to trigger ranking and / or merging the maps. In some embodiments, determining which map to use may be based, at least in part, on deep keyframes, as described above in connection with FIGS. 21-25, according to some embodiments.

[0398] FIG. 37 is a block diagram illustrating a method 3700 of creating an environment map of a physical world according to some embodiments. The method 3700 may begin with locating (act 3702) a tracking map captured by an XR device worn by a user against a set of reference maps (e.g., a reference map selected by the method of FIG. 28 and / or the method 900 of FIG. 29). Act 3702 may include locating a key rig of the tracking map within the set of reference maps. The localization results for each key rig may include a localized pose of the key rig and a set of 2D / 3D feature correspondences.

[0399] In some embodiments, method 3700 may include splitting the tracking map into connected components (act 3704), which may allow for robust merging of the maps by merging connected fragments. Each connected component may include a key rig that is within a predetermined distance. Method 3700 may include merging connected components greater than a predetermined threshold into one or more reference maps (act 3706) and removing the merged connected components from the tracking map.

[0400] In some embodiments, the method 3700 may include merging the group of reference maps that are merged with the same connected components of the tracking map (act 3708). In some embodiments, the method 3700 may include promoting the remaining connected components of the tracking map that are not merged with any reference map to the reference map (act 3710). In some embodiments, the method 3700 may include merging the reference map that is merged with the persistent attitude and / or PCF of the tracking map and at least one connected component of the tracking map (act 3712). In some embodiments, the method 3700 may include completing the reference map (act 3714), for example, by fusing map points and pruning redundant key rigs.

[0401] 38A and 38B illustrate an environment map 3800 created by updating a reference map 700, which may be leveled up from the tracking map 700 (FIG. 7) with a new tracking map, according to some embodiments. As shown and described with respect to FIG. 7, the reference map 700 may provide a floor plan 706 of a reconstructed physical object in a corresponding physical world, represented by points 702. In some embodiments, the map points 702 may represent features of the physical object, which may include multiple features. The new tracking map may be captured centered on the physical world, uploaded to the cloud, and merged with the map 700. The new tracking map may include the map points 3802 and key rigs 3804, 3806. In the illustrated example, the key rig 3804 represents a key rig that has been successfully located relative to the reference map, for example, by establishing a correspondence with the key rig 704 of the map 700 (as illustrated in FIG. 38B). Key rig 3806, on the other hand, represents a key rig that is not located relative to map 700. Key rig 3806 may, in some embodiments, be elevated to a separate reference map.

[0402] 39A-39F are schematic diagrams illustrating an example of a cloud-based persistent coordinate system that provides a shared experience for users in the same physical space. FIG. 39A shows, for example, a reference map 4814 from the cloud being received by XR devices worn by users 4802A and 4802B of FIG. 20A-20C. The reference map 4814 may have a reference coordinate frame 4806C. The reference map 4814 may have a PCF 4810C with multiple associated PPs (e.g., 4818A, 4818B in FIG. 39C).

[0403] 39B shows that the XR device has established a relationship between its individual world coordinate system 4806A, 4806B and the reference coordinate frame 4806C. This may be done, for example, by locating the reference map 4814 on the individual device. Locating the tracking map relative to the reference map may result in a transformation for each device between its local world coordinate system and the coordinate system of the reference map.

[0404] FIG. 39C shows that as a result of the localization, a transformation can be calculated between the local PCFs (e.g., PCFs 4810A, 4810B) on the respective devices and the respective persistent poses (e.g., PPs 4818A, 4818B) on the reference map (e.g., transformations 4816A, 4816B). Using these transformations, each device can use its local PCF, which can be detected locally on the device by processing images detected with sensors on the device, determining a location relative to the local device, and displaying virtual content that is tied to PPs 4818A, 4818B or other persistent points on the reference map. Such an approach can accurately localize the virtual content to each user, allowing each user to have the same experience of the virtual content within physical space.

[0405] FIG. 39D shows a persistent attitude snapshot from the reference map to the local tracking map. As can be seen, the local tracking maps are connected to each other through persistent attitudes. FIG. 39E shows that the PCF 4810A on the device worn by the user 4802A is accessible in the device worn by the user 4802B through the PP 4818A. FIG. 39F shows that the tracking maps 4804A, 4804B and the reference 4814 may be merged. In some embodiments, some PCFs may be removed as a result of the merging. In the illustrated example, the merged map includes the PCF 4810C of the reference map 4814, but does not include the PCFs 4810A, 4810B of the tracking maps 4804A, 4804B. The PP previously associated with the PCFs 4810A, 4810B may be associated with the PCF 4810C after the map merging. EXAMPLES

[0406] Figures 40 and 41 illustrate examples of using a tracking map by the first XR device 12.1 of Figure 9. Figure 40 is a two-dimensional representation of a three-dimensional first local tracking map (Map 1) that may be generated by the first XR device of Figure 9, according to some embodiments. Figure 41 is a block diagram illustrating uploading Map 1 from the first XR device to the server of Figure 9, according to some embodiments.

[0407] FIG. 40 illustrates a map 1 and virtual content (content 123 and content 456) on a first XR device 12.1. Map 1 has an origin (origin 1). Map 1 includes several PCFs (PCFa-PCFd). From the perspective of the first XR device 12.1, PCFa is, as an example, located at the origin of map 1 and has X, Y, and Z coordinates of (0,0,0), and PCFb has X, Y, and Z coordinates of (-1,0,0). Content 123 is associated with PCFa. In this example, content 123 has an X, Y, and Z relationship to PCFa of (1,0,0). Content 456 has a relationship to PCFb. In this example, content 456 has an X, Y, and Z relationship to PCFb of (1,0,0).

[0408] In FIG. 41, the first XR device 12.1 uploads Map 1 to the server 20. In this example, the tracking map was stored as an initial reference map because the server does not store a reference map for the same area of ​​the physical world represented by the tracking map. The server 20 now has a reference map based on Map 1. The first XR device 12.1 has a reference map that is empty at this stage. The server 20, for purposes of discussion, does not include any other maps than Map 1 in some embodiments. No maps are stored on the second XR device 12.2.

[0409] The first XR device 12.1 also transmits its Wi-Fi signature data to the server 20. The server 20 may use the Wi-Fi signature data to determine a rough location of the first XR device 12.1 based on intelligence gathered from other devices that have previously connected to the server 20 or other servers, along with recorded GPS locations of such other devices. The first XR device 12.1 may now end the first session (see FIG. 8) and disconnect from the server 20.

[0410] FIG. 42 is a schematic diagram illustrating the XR system of FIG. 16 according to some embodiments, showing that after a first user 14.1 ends a first session, a second user 14.2 starts a second session using a second XR device of the XR system.

[0411] FIG. 43A is a block diagram showing the start of a second session by the second user 14.2. The first user 14.1 is shown in phantom because the first session by the first user 14.1 has ended. The second XR device 12.2 begins to record objects. Various systems with variable granularity may be used by the server 20 to determine that the second session by the second XR device 12.2 is within the same vicinity of the first session by the first XR device 12.1. For example, Wi-Fi signature data, Global Positioning System (GPS) positioning data, GPS data based on Wi-Fi signature data, or any other data indicating location may be included in the first and second XR devices 12.1 and 12.2 to record the location. Alternatively, the PCF identified by the second XR device 12.2 may show similarity to the PCF of map 1.

[0412] As shown in FIG. 43B, the second XR device boots up and begins to collect data, such as images 1110, from one or more cameras 44, 46. As shown in FIG. 14, in some embodiments, the XR device (e.g., the second XR device 12.2) may collect one or more images 1110, perform image processing, and extract one or more features / points of interest 1120. Each feature may be converted into a descriptor 1130. In some embodiments, the descriptor 1130 may be used to describe a key frame 1140 that may have a location and orientation of an associated image that is tied to it. The one or more key frames 1140 may correspond to a single sustained pose 1150, which may be automatically generated after a threshold distance, e.g., 3 meters, from a previous sustained pose 1150. The one or more sustained poses 1150 may correspond to a single PCF 1160, which may be automatically generated after a predetermined distance, e.g., every 5 meters. Over time, as the user continues to move around the user's environment and the XR device continues to collect more data, such as images 1110, additional PCFs (e.g., PCF3 and PCFs 4, 5) may be created. One or more applications 1180 may be launched on the XR device and provide virtual content 1170 to the XR device for presentation to the user. The virtual content may have an associated content coordinate frame, which may be placed relative to one or more PCFs. As shown in FIG. 43B, the second XR device 12.2 creates three PCFs. In some embodiments, the second XR device 12.2 may attempt to locate relative to one or more reference maps stored on the server 20.

[0413] In some embodiments, the second XR device 12.2 may download the reference map 120 from the server 20, as shown in FIG. 43C. Map 1 on the second XR device 12.2 includes PCFa-d and origin 1. In some embodiments, the server 20 may have multiple reference maps for various locations and may determine that the second XR device 12.2 is within the same neighborhood as the neighborhood of the first XR device 12.1 during the first session and send the reference map for that neighborhood to the second XR device 12.2.

[0414] FIG. 44 shows that the second XR device 12.2 begins identifying PCFs for the purpose of generating Map 2. The second XR device 12.2 has identified only a single PCF, i.e., PCF1, 2. The X, Y, and Z coordinates of PCF1, 2 for the second XR device 12.2 may be (1,1,1). Map 2 has its own origin (origin 2), which may be based on the head pose of device 2 at device start-up for the current head pose session. In some embodiments, the second XR device 12.2 may immediately attempt to locate Map 2 against the reference map. In some embodiments, Map 2 may be impossible to locate against the reference map (Map 1) (i.e., localization may fail) because the system does not recognize any or sufficient overlap between the two maps. Localization may be performed by identifying portions of the physical world represented in a first map that are also represented in a second map, and calculating a transformation required to align those portions between the first and second maps. In some embodiments, the system may localize based on a PCF comparison between the local map and a reference map. In some embodiments, the system may localize based on a sustained pose comparison between the local map and a reference map. In some embodiments, the system may localize based on a keyframe comparison between the local map and a reference map.

[0415] FIG. 45 shows map 2 after the second XR device 12.2 has identified additional PCFs for map 2 (PCF1, 2, PCF3, PCF4, 5). The second XR device 12.2 again attempts to locate map 2 against the reference map. Because map 2 has been extended to overlap with at least a portion of the reference map, the localization attempt will be successful. In some embodiments, the overlap between the local tracking map, map 2, and the reference map may be represented by PCFs, sustained poses, key frames, or any other suitable intermediate or derived constructs.

[0416] Additionally, the second XR device 12.2 associates content 123 and content 456 with PCFs 1, 2 and PCF 3 of map 2. Content 123 has X, Y and Z coordinates for PCFs 1, 2 of (1, 0, 0). Similarly, the X, Y and Z coordinates of content 456 for PCF 3 in map 2 are also (1, 0, 0).

[0417] 46A and 46B illustrate successful localization of map 2 relative to the reference map. Localization may be based on matching features in one map and the other map. Here, using appropriate transformations, involving both translation and rotation of one map relative to the other, the overlapping area / volume / section of map 1410 represents the intersection with map 1 and the reference map. Map 2 created PCFs 3, 4, 5 before localization, and the reference map created PCFs a and c before map 2 was created, so different PCFs were created to represent the same volume in real space (e.g., different maps).

[0418] As shown in FIG. 47, the second XR device 12.2 extends map 2 to include PCF a-d from the reference map. The inclusion of PCF a-d represents the localization of map 2 relative to the reference map. In some embodiments, the XR system may perform an optimization step to remove the PCFs in 1410, i.e., PCF 3 and duplicate PCFs such as PCF 4, 5, etc., from the overlapping area. After map 2 is localized, placement of virtual content such as content 456 and content 123 will be relative to the closest updated PCF in the updated map 2. The virtual content will appear in the same real-world location to the user despite the changed PCF binding for the content and despite the updated PCF for map 2.

[0419] As shown in Figure 48, the second XR device 12.2 continues to expand map 2 as additional PCFs (PCFe, f, g, and h) are identified by the second XR device 12.2, e.g., as the user walks around the real world. Also, note that map 1 is not expanded in Figures 47 and 48.

[0420] 49, the second XR device 12.2 uploads map 2 to the server 20. The server 20 stores map 2 along with the reference map. In some embodiments, map 2 may be uploaded to the server 20 when the session for the second XR device 12.2 ends.

[0421] The reference map in the server 20 now includes PCFi, which is not included in map 1 on the first XR device 12.1. The reference map on the server 20 may be expanded to include PCFi when a third XR device (not shown) uploads a map to the server 20 and such map includes PCFi.

[0422] In Figure 50, the server 20 merges map 2 with the reference map to form a new reference map. The server 20 determines that PCFa-d are common to the reference map and map 2. The server extends the reference map to include PCFe-h and PCF1,2 from map 2 to form a new reference map. The reference maps on the first and second XR devices 12.1 and 12.2 are based on map 1 and become out of date.

[0423] In Figure 51, the server 20 transmits a new reference map to the first and second XR devices 12.1 and 12.2. In some embodiments, this may occur when the first and second XR devices 12.1 and 12.2 attempt to locate during a different or new or subsequent session. The first and second XR devices 12.1 and 12.2 proceed to locate their respective local maps (Map 1 and Map 2, respectively) against the new reference map, as described above.

[0424] As shown in FIG. 52, the head coordinate frame 96 or "head pose" is related to the PCF in map 2. In some embodiments, the map's origin, i.e., origin 2, is based on the head pose of the second XR device 12.2 at the beginning of the session. As PCFs are created during the session, they are placed relative to the world coordinate frame, i.e., origin 2. The PCF of map 2 serves as a persistent coordinate frame relative to the reference coordinate frame, which may be the world coordinate frame of the previous session (e.g., origin 1 of map 1 in FIG. 40). These coordinate frames are related by the same transformation used to locate map 2 relative to the reference map, as discussed above in connection with FIG. 46B.

[0425] The transformation from the world coordinate frame to the head coordinate frame 96 was described above with reference to Figure 9. The head coordinate frame 96 shown in Figure 52 has only two orthogonal axes, at a particular coordinate location relative to the PCF of map 2 and at a particular angle relative to map 2. However, it should be understood that the head coordinate frame 96 is in a 3D location relative to the PCF of map 2 and has three orthogonal axes in 3D space.

[0426] In FIG. 53, the head coordinate frame 96 has moved relative to the PCF of map 2. The head coordinate frame 96 has moved because the second user 14.2 has moved his head. The user can move his head in six degrees of freedom (6dof). The head coordinate frame 96 can therefore move in 6dof, i.e. in three dimensions from its previous location in FIG. 52, and in approximately three orthogonal axes relative to the PCF of map 2. The head coordinate frame 96 is adjusted as the real object detection camera 44 and the inertial measurement unit 48 in FIG. 9 detect the movement of the real object and the head unit 22, respectively. For more information on head pose tracking, see "Enhanced Pose Tracking" in the "Enhanced Pose Tracking" section of the IEEE 802.11 Symposium on Head Pose Tracking. No. 16 / 221,065, entitled "Determination of a Display Device," which is incorporated herein by reference in its entirety.

[0427] FIG. 54 shows that a sound may be associated with one or more PCFs. A user may, for example, wear headphones or earphones with stereophonic sound. The location of the sound through the headphones can be simulated using conventional techniques. The location of the sound may be located at a constant position such that when the user rotates his / her head to the left, the location of the sound rotates to the right, and thus the user perceives the sound originating from the same location in the real world. In this example, the location of the sound is represented by sound 123 and sound 456. For the purposes of discussion, FIG. 54 is similar in its analysis to FIG. 48. When a first and a second user 14.1 and 14.2 are located in the same room, at the same or different times, they perceive sound 123 and sound 456 originating from the same location in the real world.

[0428] 55 and 56 illustrate further implementations of the techniques described above. The first user 14.1 started a first session as described with reference to FIG. 8. As shown in FIG. 55, the first user 14.1 ended the first session as indicated by the phantom line. At the end of the first session, the first XR device 12.1 uploaded map 1 to the server 20. The first user 14.1 now started a second session at a time after the first session. The first XR device 12.1 does not download map 1 from the server 20 because map 1 is already stored on the first XR device 12.1. If map 1 is lost, the first XR device 12.1 downloads map 1 from the server 20. The first XR device 12.1 then proceeds to build a PCF for map 2, locate for map 1, and further develop a reference map as described above. Map 2 of the first XR device 12.1 is then used to correlate local content, head coordinate frames, local sounds etc. as explained above.

[0429] With reference to Figures 57 and 58, it may also be possible for more than one user to interact with the server in the same session. In this example, the first user 14.1 and the second user 14.2 are joined by a third user 14.3 with a third XR device 12.3. Each XR device 12.1, 12.2, and 12.3 starts to generate its own map, namely map 1, map 2, and map 3, respectively. As the XR devices 12.1, 12.2, and 12.3 continue to develop maps 1, 2, and 3, the maps are progressively uploaded to the server 20. The server 20 merges maps 1, 2, and 3 to form a reference map. The reference map is then transmitted from the server 20 to each of the XR devices 12.1, 12.2, and 12.3.

[0430] FIG. 59 illustrates aspects of a viewing method for restoring and / or resetting head pose, according to some embodiments. In the illustrated example, in act 1400, the viewing device is powered on. In act 1410, in response to being powered on, a new session is started. In some embodiments, the new session may include establishing a head pose. One or more capture devices on a head-mounted frame affixed to the user's head capture surfaces of the environment by first capturing images of the environment and then determining the surfaces from the images. In some embodiments, the surface data may also be combined with data from a gravity sensor to establish the head pose. Other suitable methods of establishing head pose may be used.

[0431] In act 1420, the viewing device's processor enters a routine for head pose tracking. The capture device continues to capture surfaces of the environment and determine the orientation of the head-mounted frame relative to the surfaces as the user moves their head.

[0432] In act 1430, the processor determines whether the head pose has been lost. The head pose may be lost due to "edge" cases such as too many reflective surfaces, low light, bare walls, outdoors, etc., which may result in low feature acquisition, or due to dynamic cases such as a crowd that moves and forms part of the map. The routine at 1430 allows a certain amount of time to pass, for example 10 seconds, to allow sufficient time to determine whether the head pose has been lost. If the head pose has not been lost, the processor returns to 1420 and again goes into tracking the head pose.

[0433] If the head pose is lost in act 1430, the processor enters a routine to restore the head pose in 1440. If the head pose is lost due to low light, a message such as the following message is displayed to the user through the display of the viewing device:

[0434] The system is detecting low light conditions. Move to an area with more light.

[0435] The system will continue to monitor whether sufficient light is available and whether the head pose can be restored. The system may alternatively determine that low texture on the surface is causing the head pose to be lost, in which case the user is given the following prompt in the display as a suggestion to improve the capture of the surface:

[0436] The system is unable to detect surfaces with sufficient fine texture. Move to an area where the surface has less coarse texture and more fine texture.

[0437] In act 1450, the processor enters a routine to determine whether head pose reconstruction has failed. If head pose reconstruction has not failed (i.e., head pose reconstruction was successful), the processor returns to act 1420 by again entering head pose tracking. If head pose reconstruction has failed, the processor returns to act 1410 and establishes a new session. As part of the new session, all cached data is invalidated and the head pose is newly established thereafter. Any suitable method of head tracking may be used in combination with the process described in FIG. 59. U.S. Patent Application No. 16 / 221,065 describes head tracking and is incorporated herein by reference in its entirety.

[0438] Remote Location

[0439] Various embodiments may utilize remote resources to facilitate sustained and consistent cross-reality experiences between individual users and / or groups of users. The inventors recognize and understand that the advantages of operating an XR device with a reference map as described herein may be achieved without downloading a set of reference maps, for example as illustrated in FIG. 30. The advantages may be achieved, for example, by transmitting feature and pose information to a remote service that maintains a set of reference maps. A device that seeks to use the reference maps to position virtual content at a location defined relative to the reference maps may receive from the remote service one or more transformations between the features and the reference maps. Those transformations may be used on a device that maintains information about the location of those features in the physical world to position virtual content at a location defined relative to one or more reference maps, or otherwise identify a location in the physical world defined relative to the reference maps.

[0440] In some embodiments, spatial information is captured by the XR device and communicated to a remote service, such as a cloud-based service, which uses the spatial information to locate the XR device relative to a reference map used by applications or other components of the XR system to define the location of virtual content relative to the physical world. Once located, a transformation can be communicated to the device that links a tracking map maintained by the device to the reference map. The transformation, in conjunction with the tracking map, may be used to determine a location within which to render virtual content defined relative to the reference map, or otherwise identify a location in the physical world defined relative to the reference map.

[0441] The inventors recognize that the data required to be exchanged between the device and a remote location service may be very minimal compared to communicating map data such as may occur when the device communicates tracking maps to a remote service and receives a set of reference maps for device-based location from the service. In some embodiments, performing location functions on cloud resources requires that only a small amount of information be transmitted from the device to the remote service. For example, it is not a requirement that a complete tracking map be communicated to the remote service to perform location. In some embodiments, feature and attitude information such as may be stored in association with a persistent attitude, as described above, may be transmitted to the remote server. In embodiments in which features are represented by descriptors, as described above, the uploaded information may be even less.

[0442] The results returned from the location service to the device may be one or more transformations that relate the uploaded features to a matching portion of the reference map. Those transformations may be used in the XR system in conjunction with the tracking map to identify the location of the virtual content or otherwise identify a location in the physical world. As described above, in embodiments where persistent spatial information such as a PCF is used to define a location relative to the reference map, the location service may download to the device the transformations between the features and one or more PCFs after successful location.

[0443] As a result, the network bandwidth consumed by communication between the XR device and the remote service for performing the localization may be small. The system may therefore support frequent localization, enabling each device interacting with the system to quickly obtain information for locating virtual content or performing other location-based functions. As the device moves through the physical environment, it may repeat requests for updated localization information. In addition, the device may frequently obtain updates to the localization information to expand the map or increase its accuracy, such as through the merging of additional tracking maps, such as when the reference map changes.

[0444] Furthermore, uploading features and downloading transformations may improve privacy in XR systems that share map information among multiple users by increasing the difficulty of obtaining a map through deception. Unauthorized users may be prevented from obtaining maps from the system, for example, by sending a fake request for a reference map that represents a portion of the physical world in which the unauthorized user is not located. Unauthorized users will be unlikely to have access to features in an area of ​​the physical world for which they are requesting map information if they are not physically present in that area. In embodiments where the feature information is formatted as a feature description, the difficulty of deceiving feature information in a request for map information will be increased. Furthermore, when the system returns transformations that are intended to be applied to a tracking map of a device operating in an area for which location information is requested, the information returned by the system is likely to be of little or no use to an imposter.

[0445] According to one embodiment, the location service is implemented as a cloud-based microservice. In some examples, implementing a cloud-based location service can help conserve device computational resources and enable computations required for location to be performed with very low latency. Their operations can be supported by the nearly infinite computational power or other computing resources available by providing additional cloud resources, ensuring the scalability of the XR system to support a large number of devices. In one example, many reference maps can be kept in memory for near-instantaneous access, or alternatively stored in a highly available device, reducing system latency.

[0446] Additionally, implementing location determination for multiple devices within a cloud service can enable refinements to the process. Location determination telemetry and statistics can provide information regarding which criteria maps are in active memory and / or highly available storage. Statistics for multiple devices may be used, for example, to identify the most frequently accessed criteria maps.

[0447] Additional accuracy may also be achieved as a result of processing in a cloud environment or other remote environment with substantial processing resources for the remote device. For example, localization may be performed on a denser reference map in the cloud versus processing performed on the local device. The map may be stored in the cloud, for example, with more PCFs or a higher density of feature descriptors per PCF, increasing the accuracy of the match between the set of features from the device and the reference map.

[0448] FIG. 61 is a schematic diagram of an XR system 6100. During a user session, a user device that displays cross-reality content can appear in a variety of forms. For example, the user device can be a wearable XR device (e.g., 6102) or a handheld mobile device (e.g., 6104). As discussed above, these devices can be configured with software, such as applications or other components, and / or wired to generate local location information (e.g., a tracking map) that can be used to render virtual content on its respective display.

[0449] The virtual content positioning information may be defined relative to global location information, which may be formatted, for example, as a reference map that contains one or more PCFs. According to some embodiments, such as the embodiment shown in Figure 61, the system 6100 is configured with cloud-based services that support the functionality and display of virtual content on user devices.

[0450] In one embodiment, the location functionality is provided as a cloud-based service 6106, which may be a microservice. The cloud-based service 6106 may be implemented on any of a number of computing devices from which computing resources may be allocated to one or more services running in the cloud. The computing devices may be accessible and interconnected to each other and to devices such as the wearable XR device 6102 and the handheld device 6104. Such connectivity may be provided via one or more networks.

[0451] In some embodiments, the cloud-based service 6106 is configured to receive the descriptor information from the individual user devices and "locate" the devices against a matching reference map or maps. For example, the cloud-based location service matches the received descriptor information to the descriptor information for the individual reference maps. The reference maps may be created using techniques such as those described above to create the reference maps by merging maps provided by one or more devices having image sensors or other sensors that obtain information about the physical world. However, it is not a requirement that the reference maps be created by the devices that access them, and thus maps may be created by map developers who may publish them, for example, by making them available to the location service 6106.

[0452] According to some embodiments, the cloud service may include operations for handling the reference map identification and filtering the repository of reference maps to a set of potential matches. Filtering may be performed as illustrated in FIG. 29 or by using any subset of the filter criteria and other filter criteria instead of or in addition to the filter criteria shown in FIG. 29. In one embodiment, geographic data may be used to limit the search for matching reference maps to maps representing areas proximate to the device requesting location. For example, area attributes such as Wi-Fi signal data, Wi-Fi fingerprint information, GPS data, and / or other device location information may be used as a coarse filter stored on the reference map, thereby limiting the analysis of the descriptors to reference maps known or likely to be proximate to the user device. Similarly, the location history of each device may be maintained by the cloud service such that reference maps within the vicinity of the device's most recent location are preferentially searched. In some examples, filtering may include the functionality discussed above with respect to FIGS. 31B, 32, 33, and 34.

[0453] FIG. 62 is an example process flow that may be executed by a device to use a cloud-based service to locate a device's position using a reference map and receive transformation information that defines one or more transformations between the device local coordinate system and the coordinate system of the reference map. Various embodiments and examples may describe one or more transformations as defining a transformation from a first coordinate frame to a second coordinate frame. Other embodiments include a transformation from a second coordinate frame to a first coordinate frame. In still other embodiments, the transformation enables a transition from one coordinate frame to another, with the resulting coordinate frame depending only on the desired coordinate frame output (e.g., including the coordinate frame in which the content is to be displayed). In still further embodiments, the coordinate system transformation may enable a frame determination from the second coordinate frame to the first coordinate frame and from the first coordinate frame to the second coordinate frame.

[0454] According to some embodiments, information reflecting a transformation for each sustained pose defined relative to the reference map can be communicated to the device.

[0455] According to one embodiment, process 6200 can start at 6202 with a new session. Starting a new session on the device can begin capturing image information and building a tracking map for the device. In addition, the device can send a message to register with a server for location services, prompting the server to create a session for the device.

[0456] In some embodiments, initiating a new session on a device may optionally include transmitting adjustment data from the device to a location service. The location service returns to the device one or more transformations calculated based on the set of features and associated poses. If the pose of the features is adjusted based on device-specific information before the calculation of the transformations and / or the transformations are adjusted based on device-specific information after the calculation of the transformations, rather than performing those calculations on the device, the device-specific information may be transmitted to the location service so that the location service may apply the adjustments. As a specific example, transmitting the device-specific adjustment information may include capturing calibration data for the sensor and / or the display. The calibration data may be used, for example, to adjust the location of the feature points relative to the measured location. Alternatively or additionally, the calibration data may be used to adjust the location where the display is commanded to render the virtual content so that it appears accurately positioned for that particular device. This calibration data may be derived, for example, from multiple images of the same scene obtained using sensors on the device. The locations of features detected within those images may be expressed as a function of the sensor location such that multiple images result in a set of equations that can be solved for the sensor locations. The calculated sensor locations may be compared to nominal positions, and calibration data may be derived from any differences. In some embodiments, inherent information about the structure of the device may also enable calibration data to be calculated for the display in some embodiments.

[0457] In embodiments where calibration data is generated for a sensor and / or display, the calibration data may be applied at any point in the measurement or display process. In some embodiments, the calibration data may be transmitted to a location server, which may store the calibration data in a data structure established for each device that is registered with the location server and therefore in session with the server. The location server may apply the calibration data to any transformations calculated as part of the location process for the device that supplies that calibration data. The computational burden of using the calibration data for greater accuracy of sensed and / or displayed information is thus borne by the calibration service, providing a further mechanism for reducing the processing burden on the device.

[0458] Once a new session is established, process 6200 may continue at 6204 with the capture of a new frame of the device's environment. Each frame may be processed at 6206 to generate a descriptor for the captured frame (e.g., including the DSF values ​​discussed above). These values ​​may be calculated using some or all of the techniques described above, including those discussed above with respect to Figures 14, 22, and 23. As discussed, the descriptor may be calculated as a mapping of feature points to the descriptor, or in some embodiments, a mapping of patches of the image around the feature points. The descriptor may have a value that enables efficient matching between the newly acquired frame / image and the stored map. Additionally, the number of features extracted from the image may be limited to a maximum number of feature points per image, such as 200 feature points per image. The feature points may be selected to represent points of interest, as described above. Thus, acts 6204 and 6206 may be performed as part of a device process that forms a tracking map or otherwise periodically collects images of the physical world around the device, or may be performed separately for localization purposes, but need not be.

[0459] The feature extraction at 6206 may include appending pose information to the features extracted at 6206. The pose information may be a pose in a local coordinate system of the device. In some embodiments, the pose may be relative to a reference point in a tracking map, such as a persistent pose, as discussed above. Alternatively, or in addition, the pose may be relative to an origin of the tracking map of the device. Such an embodiment may enable a location service, as described herein, to provide location services for a wide range of devices, even if they do not utilize persistent pose. Regardless, pose information may be appended to each feature or each set of features such that the location service may use the pose information to compute a transformation that may be returned to the device in response to matching the features to features in the stored map.

[0460] Process 6200 may continue to decision block 6207, where a determination is made whether to request location. One or more criteria may be applied to determine whether to request location. The criteria may include the passage of time, such that the device may request location after a threshold amount of time. For example, if location has not been attempted within a threshold amount of time, the process may continue from decision block 6207 to act 6208, where location is requested from the cloud. The threshold amount of time may be, for example, 10-30 seconds, such as 25 seconds. Alternatively, or in addition, location may be triggered by movement of the device. A device executing process 6200 may track its movement using an IMU and / or its tracking map, and initiate location upon detection of movement exceeding a threshold distance from where the device last requested location. The threshold distance may be, for example, 1-10 meters, such as 3-5 meters. As a still further alternative, position determination may be triggered in response to an event, such as when the device creates a new persistent pose or when a current persistent pose for the device changes, as described above.

[0461] In some embodiments, decision block 6207 may be implemented such that the threshold for triggering localization may be established dynamically. For example, in an environment in which features are primarily homogeneous, such that there may be low confidence in matching a set of extracted features to features in a stored map, localization may be requested more frequently, increasing the chance that at least one attempt at localization will be successful. In such a scenario, the threshold applied in decision block 6207 may be decreased. Similarly, in an environment in which there are relatively few features, the threshold applied in decision block 6207 may be decreased to increase the frequency of localization attempts.

[0462] Regardless of how the location is triggered, once triggered, process 6200 may proceed to act 6208, where the device transmits a request for a location service including data used by the location service to perform the location. In some embodiments, data from multiple image frames may be provided for the location attempt. The location service may not be deemed a location success unless, for example, features in multiple image frames result in consistent location results. In some embodiments, process 6200 may include saving the feature descriptors and the appended pose information in a buffer. The buffer may be, for example, a circular buffer that stores a set of features extracted from the most recently captured frame. Thus, the location request may be transmitted with several sets of features accumulated in the buffer. In some settings, the buffer size is implemented to accumulate several sets of data that would be more likely to result in a location success. In some embodiments, the buffer size may be set to accumulate features from, for example, 2, 3, 4, 5, 6, 7, 8, 9, or 10 frames). Optionally, the buffer size can have a baseline setting, which can be increased in response to a location failure, in some embodiments, increasing the buffer size and the corresponding number of transmitted feature sets reduces the likelihood that a subsequent location function will fail to return a result.

[0463] Regardless of how the buffer size is set, the device may transfer the contents of the buffer to the location service as part of the location request. Other information may also be transmitted along with the feature points and the appended attitude information. For example, in some embodiments, geographic information may be transmitted. The geographic information may include, for example, GPS coordinates or a radio signature associated with a device tracking map or a current sustained attitude.

[0464] In response to the request transmitted at 6208, the cloud location service may analyze the feature descriptors and locate the device in a reference map or other persistent map maintained by the service. For example, the descriptors are matched to a set of features in the map to which the device is located. The cloud-based location service may perform location as described above with respect to device-based location (e.g., may rely on any of the features discussed above for location (including map ranking, map filtering, place estimation, filtered map selection, examples, PCF and / or PP identification and matching, etc., discussed in Figures 44-46 and / or with respect to the location module)). However, instead of communicating the identified reference map to the device (e.g., in device location), the cloud-based location service may proceed to generate a transformation based on the relative orientation of the feature set transmitted from the device and the matching features of the reference map. The location service may return these transformations to the device, which may be received at block 6210.

[0465] In some embodiments, the reference map maintained by the location service may employ PCFs, as described above. In such embodiments, feature points of the reference map that match feature points transmitted from the device may have locations defined relative to one or more PCFs. Thus, the location service may identify one or more reference maps and compute a transformation between the coordinate frame represented in the pose transmitted with the request for location and one or more PCFs. In some embodiments, the identification of one or more reference maps is aided by filtering potential maps based on geographic data about the individual device. For example, once filtered to a candidate set (e.g., by GPS coordinates, among other options), the candidate set of reference maps may be analyzed in detail, as described above, to determine matching feature points or PCFs.

[0466] The data returned to the requesting device in act 6210 may be formatted as a table of persistent attitude transformations. The table may be accompanied by one or more reference map identifiers and indicate the reference map for which the device was located by the location service. However, it should be understood that the location information may be formatted in other manners, including as a list of transformations with associated PCFs and / or reference map identifiers.

[0467] Regardless of how the transforms are formatted, in act 6212 the device may use these transforms to calculate locations for which to render the virtual content, the locations of which have been specified to any of the PCFs by the application or other components of the XR system. This information may alternatively, or in addition, be used to implement any location-based actions on the device in which the locations are specified based on the PCF.

[0468] In some scenarios, the location service may not be able to match features transmitted from the device to any stored criteria maps, or may be unable to match a sufficient number of sets of features communicated with a request for the location service to consider a successful location to have occurred. In such scenarios, rather than returning a transformation to the device as described above in connection with act 6210, the location service may indicate to the device that the location failed. In such scenarios, process 6200 may branch at decision block 6209 to act 6230, and the device may take one or more actions to handle the failure. These actions may include increasing the size of a buffer that holds the feature sets transmitted for location. For example, if the location service does not consider a location successful unless three sets of features match, the buffer size may be increased from five to six to increase the likelihood that three of the transmitted sets of features can be matched to the criteria maps maintained by the location service.

[0469] Alternatively or additionally, failure handling may include adjusting the device's operating parameters to trigger more frequent location attempts. The threshold time and / or threshold distance between location attempts may, for example, be decreased. As another example, the number of feature points in each set of features may be increased. A match between a set of features and features stored in the reference map may be deemed to occur when a sufficient number of features in the set transmitted from the device match features in the map. Increasing the number of features transmitted may increase the chances of a match. As a specific example, the initial feature set size may be 50, which may be increased to 100, 150, then 200 upon each successive location failure. Upon successful matching, the set size may then be returned to its initial value.

[0470] Failure handling may also include obtaining location information other than from a location service. According to some embodiments, a user device can be configured to cache a reference map. The cached map allows the device to access and display content if the cloud is unavailable. For example, the cached reference map allows device-based location in the event of communication failure or other unavailability.

[0471] According to various embodiments, Figure 62 illustrates a high level flow for a device to initiate cloud-based location. In other embodiments, various one or more of the illustrated steps may be combined, omitted, or call other processes to accomplish location and ultimately visualization of the virtual content within an individual device's view.

[0472] Additionally, while process 6200 shows the device determining whether to initiate location at decision block 6207, it should be understood that the trigger for initiating location may come from outside the device, including from the location service. The location service may, for example, maintain information about each of the devices that are in session with it. That information may include, for example, an identifier of the reference map to which each device was most recently located. The location service or other components of the XR system may update the reference map, including using techniques such as those described...

Claims

1. 1. A system for supporting specification of a location of virtual content relative to one or more persistent maps in a database of persistent maps, the system comprising: a location service, the location service comprising: a communication component configured to receive, from a portable electronic device, information about a plurality of sets of features within an image of a three-dimensional (3D) environment of the portable electronic device; At least one processor configured to execute computer-executable instructions to implement a location determination component coupled to the communication component, the location determination component comprising: With respect to a first set of features of the plurality of sets of features, performing a coarse location of the first set of features relative to one or more persistent maps in the database of persistent maps, where performing the coarse location includes calculating a first coarse transformation of the portable electronic device relative to a persistent map in the database of persistent maps; performing a refined localization of the first set of features relative to the persistent map, where performing a refined localization of the first set of features includes calculating a first candidate localization of the portable electronic device based on the first coarse transformation; With respect to a second set of features of the plurality of sets of features, performing a refined localization of the second set of features relative to the persistent map, the performing a refined localization of the second set of features including calculating a second candidate localization of the portable electronic device based on the first coarse transformation; calculating a plurality of candidate localizations for the plurality of sets of features by selecting a location location based on a commonality among the plurality of candidate location locations including the first candidate location location and the second candidate location location; At least one processor configured to A system comprising:

2. To perform a rough localization, identifying a first set of features within the first collection of features; identifying a second set of features in the persistent map, wherein features in the first set of features correspond to individual features of the second set of features based on similarity of at least one characteristic; The system of claim 1 further comprising:

3. The system of claim 2, wherein the at least one characteristic of the feature comprises a feature descriptor calculated from pixels in an image depicting the feature.

4. The system of claim 3 , wherein the feature descriptor comprises deep sparse features (DSF).

5. To perform a rough localization, For multiple iterations, multiple candidate rough transformations and associated measures of error are calculated. selecting a first subset of the first set of features and a second subset of the second set of features; determining a candidate coarse transformation between the first subset of features and the second subset of features based on a measure of alignment error between the first subset transformed using the candidate coarse transformation and the second subset; selecting a selected candidate coarse transformation from among the plurality of candidate coarse transformations based on an associated measure of alignment error; providing the first coarse transformation based on the selected candidate coarse transformations; and The system of claim 2 further comprising:

6. the selected localization comprises a localization transformation between a local coordinate frame of the portable electronic device and a coordinate frame of the persistent map; The system of claim 1 , wherein the localization component is further configured to transmit the localization transformation to the portable electronic device.

7. The system of claim 1 , wherein each of the multiple sets of features comprises a pose-based feature rig.

8. The system of claim 7 , wherein the pose-based feature rig comprises a plurality of feature descriptors and a pose associated with each of the plurality of feature descriptors.

9. The computer-executable instructions implementing the location component include: determining whether to perform a refined localization of the second set of features based on satisfaction of one or more criteria; The system of claim 1 further comprising instructions for:

10. The computer-executable instructions implementing the location component include: When the one or more criteria are not satisfied, performing a refined localization of the second set of features based on a second coarse transformation calculated from the second set of features. The system of claim 9 further comprising instructions for:

11. 11. The system of claim 10, wherein the one or more criteria comprises a disparity above a threshold amount between an image from which the first set of features was extracted and an image from which the second set of features was extracted.

12. The one or more criteria are: The system of claim 10 , further comprising calculating a confidence in the first candidate location location above a first threshold.

13. 13. The system of claim 12, wherein calculating the first candidate location location includes rejecting a location location with a confidence level less than a second threshold as the first candidate location location, the first threshold being higher than the second threshold.

14. The computer executable instructions for selecting a location location based on commonalities among the candidate location locations calculated for the plurality of sets of features include: determining whether there are shared terms above a first threshold among the plurality of candidate localizations calculated for the plurality of sets of features including the second candidate localization; recalculating candidate localizations for a set of features among the plurality of sets of features where the candidate localizations were calculated based on rough transformations from different sets of features based on the determined commonality terms being below the first threshold. The system of claim 10.

15. The computer executable instructions for selecting a location location based on commonalities among the candidate location locations calculated for the plurality of sets of features include: determining whether there are any shared terms among the plurality of candidate localizations calculated for the plurality of sets of features that exceed a second threshold; The system of claim 14 , wherein the plurality of candidate location locations comprises the recalculated candidate location locations, and the first threshold is higher than the second threshold.

16. The system of claim 14 , wherein determining whether there are any common terms that exceed the first threshold comprises comparing a most frequent value of the plurality of candidate location locations to the first threshold.

17. Performing a rough location determination for the first set of features includes: selecting a sub-region of the persistent map based on location metadata associated with the first set of features and the selected sub-region of the persistent map. The system of claim 1 further comprising:

18. In combination with a portable electronic device configured to operate within a cross reality system, the portable electronic device having a device coordinate frame, the portable electronic device comprising: one or more sensors configured to capture information about the 3D environment, the captured information comprising a plurality of images; At least one second processor configured to execute second computer-executable instructions, the second computer-executable instructions comprising: extracting a plurality of features for each of the plurality of images; Associating pose information in the device coordinate frame with each of the plurality of features; generating information about the plurality of sets of features by transmitting information about the plurality of sets of features over a network to a location service, including transmitting information about the plurality of extracted features and the associated pose information; receiving from the location service a transformation relating the device coordinate frame to a second coordinate frame; at least one second processor having instructions for performing The system of claim 1 , comprising:

19. The portable electronic device further comprises a buffer; The computer executable instructions for the portable electronic device include: storing, for each of the plurality of images in the buffer, information about the set of extracted features and the associated pose information; transmitting information about the plurality of sets of characteristics over the network, including transmitting the contents of the buffer; and 20. The system of claim 18, further comprising instructions for:

20. the portable electronic device comprises a display; 20. The system of claim 18, wherein the computer-executable instructions of the portable electronic device further comprise instructions for rendering virtual content having a location defined in the second coordinate frame on the display at a position calculated based at least in part on the received transformation.

21. The system of claim 18 , wherein the information about the plurality of extracted features comprises a descriptor for each feature.

22. 2. The system of claim 1, wherein the refined localization of the second set of features is performed using at least four times the number of features used to perform the coarse localization of the first set of features.

23. 1. A method of computing a pose between a first component and a map, the pose of the first component being represented by a plurality of sets of features posed relative to a coordinate frame of the first component, the method comprising: calculating a plurality of candidate localizations based on respective sets of features from the plurality of sets of features; determining a location location based on the plurality of candidate location locations; and Including, Calculating the plurality of candidate localizations includes: With respect to a first portion of the plurality of sets of features, calculating a first coarse localization for a set of features within the first portion of the plurality of sets of features; calculating a first refined localization based on the first coarse localization as a first candidate localization of the plurality of candidate localizations; With respect to a second portion of the plurality of sets of features, evaluating one or more criteria to determine whether to calculate a second refined location determination based on the first coarse location determination as a second candidate location determination of the plurality of candidate location determinations; A method comprising:

24. for a set of features in the second portion of the plurality of sets of features for which it is determined to calculate the second refined localization based on the first coarse localization, calculating the second refined location determination based on the first coarse location determination as the second candidate location determination of the plurality of candidate location determinations; 24. The method of claim 23, further comprising:

25. for a set of features in the second portion of the plurality of sets of features for which it is determined not to calculate the second refined localization based on the first rough localization, calculating a second coarse localization for the set of features in the second portion of the plurality of sets of features for which it has been determined not to calculate the second refined localization based on the first coarse localization; calculating the second refined location determination based on the second coarse location determination as the second candidate location determination of the plurality of candidate location determinations; 25. The method of claim 24, further comprising:

26. Evaluating the one or more criteria to determine whether to calculate the second refined location based on the first coarse location includes: determining whether to calculate the second refined localization based on the first coarse localization based on a disparity between an image from which the set of features in the first portion was extracted and an image from which the set of features in the second portion was extracted exceeding a threshold; 25. The method of claim 24, comprising:

27. Evaluating the one or more criteria to determine whether to calculate the second refined location based on the first coarse location includes: determining whether to calculate the second refined localization based on the first coarse localization based on a measure of error for the first refined localization for the set of features within the first portion being below a threshold; 25. The method of claim 24, comprising:

28. Determining the location location based on the plurality of candidate location locations includes: evaluating the plurality of candidate localizations for common terms based on a common term criterion depending on whether the second candidate localization for the set of features within the second portion was calculated as the second refined localization based on the first rough localization.

24. The method of claim 23, comprising:

29. The method of claim 23 , wherein the multiple sets of features comprise batches of posed feature rigs received from a portable electronic device.

30. The method of claim 23 , wherein the plurality of sets of features comprises a posed feature rig in a tracking map formed by a portable electronic device.

31. 30. The method of claim 29, further comprising selecting the map from a database of stored maps based on location metadata included with the pose feature rig of the batch.

32. Calculating the first coarse localization with respect to a set of features within the first portion includes: identifying a set of features in the collection of features in the first portion that correspond to features in the map; iteratively forming subsets of the set of corresponding features and determining a candidate rough transformation and associated accuracy for each subset; selecting a coarse transformation from among said iteratively formed candidate coarse transformations based on the determined accuracy of the selected coarse transformation; 24. The method of claim 23, comprising:

33. Calculating the first refined localization based on the first coarse localization includes: searching for a transformation that matches the set of features identified in the feature collection in the first portion with corresponding features in the map, the searching being constrained around the coarse transformation calculated for the first coarse localization; 33. The method of claim 32, comprising:

34. determining the plurality of candidate location determinations further includes evaluating a confidence metric for the first candidate location determination; Evaluating the one or more criteria and determining whether to calculate the second refined location location based on the first coarse location location as the second candidate location location of the plurality of candidate location locations includes: determining whether to calculate the second candidate location location of the plurality of location locations based on the confidence metric; and the second candidate location location of the plurality of location locations comprises: calculating a second coarse location for a set of features in the second portion of the plurality of sets of features and calculating the second refined location as the second candidate location based on the second coarse location; or calculating the second refined location location based on the first coarse location location as the second candidate location location; The method of claim 32, wherein the calculation is performed by:

35. 1. A non-transitory computer readable medium comprising computer executable instructions that, when executed by at least one processor, are configured to perform a method for calculating a pose between a first component and a map, the pose of the first component being represented by a plurality of sets of features posed relative to a coordinate frame of the first component, the method comprising: calculating a plurality of candidate localizations based on respective sets of features of the plurality of sets of features; determining a location location based on the plurality of candidate location locations; and Including, Calculating the plurality of candidate localizations includes: With respect to a first portion of the plurality of sets of features, calculating a first coarse localization for a set of features within the first portion of the plurality of sets of features; calculating a first refined localization based on the first coarse localization as a first candidate localization of the plurality of candidate localizations; With respect to a second portion of the plurality of sets of features, evaluating one or more criteria to determine whether to calculate a second refined location determination based on the first coarse location determination as a second candidate location determination of the plurality of candidate location determinations; 16. A non-transitory computer readable medium comprising:

36. determining the plurality of candidate location determinations further includes evaluating a confidence metric for the first candidate location determination; Evaluating the one or more criteria and determining whether to calculate the second refined location location based on the first coarse location location as the second candidate location location of the plurality of candidate location locations includes: determining whether to calculate the second candidate location location of the plurality of location locations based on the confidence metric; and the second candidate location location of the plurality of location locations comprises: calculating a second coarse location for a set of features in the second portion of the plurality of sets of features and calculating the second refined location as the second candidate location based on the second coarse location; or calculating the second refined location location based on the first coarse location location as the second candidate location location; 36. The non-transitory computer readable medium of claim 35, wherein the non-transitory computer readable medium is calculated by:

Images