Mapping and Localizing the Pathable World

Abstract

An augmented reality (AR) device can be configured to generate a virtual representation of a user's physical environment. The AR device can capture an image of the user's physical environment and generate or identify the user's location. The AR device can project graphics at designated locations within the user's environment and guide the user to capture an image of the user's physical environment. The AR device can provide visual, audible, or tactile guidance to direct the user of the AR device to explore the user's environment.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of priority to U.S. Patent Application No. 62 / 937056, entitled "MAPPING AND LOCALIZATION OF A PASSABLE WORLD," filed November 18, 2019, and U.S. Patent Application No. 62 / 968751, entitled "MAPPING AND LOCALIZATION OF A PASSABLE WORLD," filed January 31, 2020. This application is related to U.S. Patent Application No. 14 / 690,401, entitled "SYSTEMS AND METHOD FOR AUGMENTED AND VIRTUAL REALITY," filed April 18, 2015. This application is further related to U.S. Patent Application No. 15 / 859,277, entitled "SYSTEMS AND METHODS FOR AUGMENTED REALITY," filed December 29, 2017, and U.S. Patent No. 9,767,616, entitled "RECOGNIZING OBJECTS IN A PASSABLE WORLD MODEL IN AN AUGMENTED OR VIRTUAL REALITY SYSTEM," filed May 8, 2015. This application is further related to U.S. Patent Application No. 16 / 448,743, entitled "WAYPOINT CREATION IN MAP DETECTION," filed June 21, 2019. This application is further related to U.S. Design Patent Application No. 29 / 716,368, entitled "PORTION OF A DISPLAY SCREEN WITH GUIDING GRAPHICS," filed December 9, 2019. This application is further related to U.S. Design Patent Application No. 29 / 716,361, entitled "PORTION OF A DISPLAY SCREEN WITH GUIDING GRAPHICS," filed December 9, 2019. This application is further related to U.S. Design Patent Application No. 29 / 716,375, entitled "PORTION OF A DISPLAY SCREEN WITH GUIDING GRAPHICS," filed December 9, 2019.This application is further related to U.S. Design Patent Application No. 29 / 716363, entitled "PORTION OF A DISPLAY SCREEN WITH GUIDING GRAPHICS" and filed December 9, 2019. This application is further related to U.S. Design Patent Application No. 29 / 716367, entitled "PORTION OF A DISPLAY SCREEN WITH GUIDING GRAPHICS" and filed December 9, 2019. This application is further related to U.S. Design Patent Application No. 29 / 716364, entitled "PORTION OF A DISPLAY SCREEN WITH GUIDING GRAPHICS" and filed December 9, 2019. The contents of the aforementioned patent documents are expressly incorporated herein by reference in their entireties.

[0002] The present disclosure relates generally to systems and methods for facilitating an interactive virtual or augmented reality environment for one or more users. [Background technology]

[0003] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality", "augmented reality", or "mixed reality" experiences, in which digitally reproduced images or portions thereof are presented to a user in a manner that appears or may be perceived as real. Virtual reality, or "VR", scenarios typically involve the presentation of digital or virtual image information without transparency to other actual real-world visual inputs. Augmented reality, or "AR", scenarios typically involve the presentation of digital or virtual image information as an augmentation to the visualization of the real world around the user. Mixed reality or "MR" refers to the merging of real and virtual worlds to generate new environments in which physical and virtual objects coexist and interact in real time. In conclusion, the human visual perception system is highly complex, making it difficult to create VR, AR, or MR technologies that facilitate comfortable, natural-feeling, and rich presentation of virtual image elements in other virtual or real-world image elements. The systems and methods disclosed herein address various challenges associated with VR, AR, and MR technologies. Summary of the Invention [Means for solving the problem]

[0004] Embodiments of the present disclosure are directed to devices, systems, and methods for facilitating virtual or augmented reality interactions for one or more users.

[0005] Further details of the features, objects, and advantages of the present disclosure are set forth in the following detailed description, drawings, and claims. Both the foregoing general description and the following detailed description are exemplary and explanatory and are not intended to be limiting as to the scope of the present disclosure.

[0006] In some examples, an augmented reality (AR) system may include an AR display configured to present virtual content to a user of the AR system; an outward-facing camera configured to capture one or more images of the user's environment, where a portion of the user's environment visible to the user comprises the user's field of view (FOV); and a hardware processor in communication with the AR display and the outward-facing camera, the hardware processor programmed to: generate a plurality of ray castings from an origin comprising a location associated with the user's head; analyze the plurality of ray castings; identify ray castings associated with unmapped locations of the environment; update the virtual content to include graphics associated with the unmapped locations; collect data associated with the user's environment; and, in response to collecting sufficient data associated with the unmapped locations, signal to the user that the data has been successfully collected.

[0007] In some examples, an augmented reality (AR) system includes an AR display configured to present virtual content to a user of the AR system, an outward facing camera configured to capture one or more images of the user's environment, where a portion of the user's environment visible to the user comprises the user's field of view (FOV), the outward facing camera, and a hardware processor in communication with the AR display and the outward facing camera, for receiving a user selection of a map, comprising a location associated with the user's head, generating a plurality of ray castings from an origin, analyzing the plurality of ray castings, and identifying ray castings associated with unmapped locations of the environment. and a hardware processor programmed to: update the virtual content to include a graphic that indicates the location of the map being mapped, instruct the user to view the graphic, collect data associated with the user's environment, and in response to collecting sufficient data associated with the unmapped location, signal to the user that the data was successfully collected, and in response to determining that at least one marker in the collected data corresponds to at least one marker in the map, display virtual reality content associated with the map, or in response to determining that no marker in the collected data corresponds to a marker in the map, repeat said updating, instructing, collecting, and signaling for another unmapped location.

[0008] In some embodiments, a method of creating or updating an augmented reality map of a user's environment implemented by a computing system having one or more hardware computer processors and one or more non-transitory computer readable storage devices storing software instructions executable by the computing system to implement the computerized method, comprising the steps of obtaining distance data from each of a plurality of directions around a user, the distance data indicating a distance at which a physical object was detected in the respective directions; and creating a ranked list of waypoints associated with the distance data having distances exceeding a threshold distance, the waypoint associated with the greatest distance being ranked. displaying a graphic positioned with reference to the highest ranked waypoint for viewing on an augmented reality (AR) device worn by the user, the graphic configured to indicate to the user an area of ​​the user's environment for which the augmented reality map is incomplete; detecting user movement enabling collection of data by the AR device associated with the waypoint; determining whether additional waypoints remain in the ranked list; and in response to a determination that additional waypoints remain in the ranked list, performing the displaying, detecting, and determining with the next highest ranked waypoint.

[0009] In some examples, an augmented reality (AR) system can include an AR display configured to present virtual content to a user of the AR system, an outwardly facing camera configured to capture one or more images of the user's three-dimensional (3D) environment, and a hardware processor in communication with the AR display and the outwardly facing camera, which can be programmed to receive invitations to access a shared map, facilitate user access to the shared map, guide the user to observe the user's 3D environment, collect data associated with the user's 3D environment, and load content associated with the shared map if the data is found within the user's 3D environment.

[0010] In some examples, an augmented reality (AR) system can include an AR display configured to present virtual content to a user of the AR system, an outwardly facing camera configured to capture one or more images of the user's three-dimensional (3D) environment, and a hardware processor in communication with the AR display and the outwardly facing camera, which can be programmed to receive user input to update the map, guide the user to observe the user's 3D environment, collect data associated with the user's 3D environment, determine a map quality index associated with the map, display an indication of the map quality index on the AR display, update the map based on the collected data, and stop guiding the user in response to identifying a stop condition comprising a user input to stop or a determination that the map quality index reaches a threshold. [Brief description of the drawings]

[0011] Throughout the drawings, reference numbers are reused to indicate correspondence between referenced elements. The following drawings and the associated description are provided to illustrate embodiments of the present disclosure and are not intended to limit the scope of the claims.

[0012] The drawings illustrate the design and utility of various embodiments of the present disclosure. It should be noted that the drawings are not drawn to scale, and elements of similar structure or function are represented by similar reference numerals throughout the drawings. In order to better understand how to obtain the above-listed advantages and objects and others of the various embodiments of the present disclosure, a more detailed description of the present disclosure, briefly described above, will be given with reference to specific embodiments thereof, as illustrated in the accompanying drawings. With the understanding that these drawings depict only exemplary embodiments of the present disclosure and therefore should not be considered as limiting its scope, the present disclosure will be explained and described with additional specificity and detail through the use of the accompanying drawings. The present invention provides, for example, the following: (Item 1) 1. An augmented reality (AR) system, comprising: an AR display configured to present virtual content to a user of the AR system; an outwardly facing camera configured to capture one or more images of the user's environment, the portion of the user's environment visible to the user comprising the user's field of view (FOV); a hardware processor in communication with the AR display and the outward facing camera, the hardware processor comprising: generating a plurality of ray castings from an origin having a location associated with the user's head; analyzing the plurality of ray castings and identifying ray castings associated with unmapped locations of the environment; updating the virtual content to include a graphic associated with the unmapped location; collecting data associated with an environment of the user; In response to collecting sufficient data associated with the unmapped location, signaling to the user that the data has been successfully collected. a hardware processor programmed to An AR system equipped with (Item 2) 2. The AR system of claim 1, wherein the hardware processor is configured to instruct the user to observe the graphic. (Item 3) 3. The AR system of claim 1, wherein the hardware processor is configured to repeat the updating, indicating, collecting, and signaling regarding the additional unmapped locations in response to determining that additional unmapped locations exist within the environment. (Item 4) The AR system of any one of items 1-3, wherein the updating, instructing, collecting, and signaling are repeated at least until a minimum threshold of data associated with unmapped locations is obtained. (Item 5) The hardware processor further comprises: The AR system of any one of items 1-4, configured to determine the presence of at least one persistent coordinate frame (PCF) in the user's environment based on the collected data, and the step of determining whether there are additional unmapped locations in the environment is based on the determination of whether at least one PCF has been identified. (Item 6) The hardware processor further comprises: 6. The AR system of any one of items 1-5, configured to load content associated with a stored map that includes at least one PCF in response to determining the presence of the at least one PCF. (Item 7) To analyze the plurality of ray castings, the hardware processor performs, with respect to the ray castings: determining the presence of a mesh along said ray casting; calculating the distance of the mesh along the ray casting; determining a ray casting type comprising a content type, a no content type, or an unobserved type based on the presence of the mesh, the presence of the content, and the distance of the mesh; 7. The AR system according to any one of items 1 to 6, configured to perform the following: (Item 8) 8. The AR system of claim 7, wherein the content type comprises ray casting with content on a mesh within a maximum distance from at least one of the user and other unobserved locations. (Item 9) 9. The AR system of claim 8, wherein the maximum distance is 2 meters. (Item 10) 10. The AR system of any one of items 7-9, wherein the no-content type comprises ray casting with no content on meshes within the maximum distance. (Item 11) 11. The AR system of any one of items 7-10, wherein the unobserved type comprises ray casting with unknown content within the maximum distance. (Item 12) 13. The AR system of any one of items 1-12, wherein the hardware processor is configured to display graphics in the center of the user's FOV. (Item 13) Item 13. The AR system of item 12, wherein the hardware processor is configured to display a graphic in a center of the user's FOV prior to generating the graphic at the unobserved location. (Item 14) The hardware processor includes: instructing the user to view the graphic in a center of the user's FOV; signaling to the user that data has been successfully collected near the at least one unobserved location; and

[0023] 14. The AR system of any one of claims 12-13, wherein the collected data comprises data associated with a center of the user's FOV. (Item 15) 15. The AR system of any one of items 1-14, wherein the hardware processor is configured to display an animation associated with the graphic at the unobserved location to signal to the user that data has been successfully collected at the unobserved location. (Item 16) To display the animation, the AR system: generating at least one bubble associated with said graphic; determining a collision trajectory associated with the at least one bubble; and animating the at least one bubble to move along the collision trajectory towards a collision location on a mesh associated with a map of the user's environment; displaying a collision between said at least one bubble and said mesh; Item 16. The AR system of item 15, configured to perform the following: (Item 17) 17. The AR system of any one of items 1-16, wherein to instruct a user to observe the graphic, the hardware processor is configured to display an indicator ring based on the unobserved location relative to the user's position. (Item 18) 1. An augmented reality (AR) system, comprising: an AR display configured to present virtual content to a user of the AR system; an outwardly facing camera configured to capture one or more images of the user's environment, the portion of the user's environment visible to the user comprising the user's field of view (FOV); a hardware processor in communication with the AR display and the outward facing camera, the hardware processor comprising: receiving a user selection of a map; generating a plurality of ray castings from an origin comprising a location associated with a head of the user; analyzing the plurality of ray castings and identifying ray castings associated with unmapped locations of the environment; updating the virtual content to include a graphic associated with the unmapped location; instructing the user to view the graphic; collecting data associated with an environment of said user; in response to collecting sufficient data associated with the unmapped location, signaling to the user that data has been successfully collected; in response to determining that at least one marker in the collected data corresponds to at least one marker in the map, displaying virtual reality content associated with the map; or and repeating the updating, indicating, collecting, and signaling for another unmapped location in response to determining that a marker in the collected data does not correspond to a marker in the map. a hardware processor programmed to An AR system equipped with (Item 19) 20. The AR system of claim 18, wherein the at least one marker comprises at least one PCF. (Item 20) To analyze the plurality of ray castings, the hardware processor performs, with respect to the ray castings: determining the presence of a mesh along said ray casting; calculating the distance of the mesh along the ray casting; determining a ray casting type comprising a content type, a no content type, or an unobserved type based on the presence of the mesh, the presence of the content, and the distance of the mesh; 20. The AR system according to any one of items 18-19, configured to: (Item 21) 21. The AR system of claim 20, wherein the content type comprises ray casting with content on a mesh within a maximum distance from at least one of the user and other unobserved locations. (Item 22) 22. The AR system of claim 21, wherein the maximum distance is 2 meters. (Item 23) 23. The AR system of any one of items 20-22, wherein the no-content type comprises ray casting with no content on meshes within the maximum distance. (Item 24) 24. The AR system of any one of claims 20-23, wherein the unobserved type comprises ray casting with unknown content within the maximum distance. (Item 25) 25. The AR system of any one of items 18-24, wherein the hardware processor is configured to display graphics in the center of the user's FOV. (Item 26) 26. The AR system of claim 25, wherein the hardware processor is configured to display the graphic in a center of the user's FOV prior to generating the graphic at the unobserved location. (Item 27) The hardware processor includes: instructing the user to view the graphic in a center of the user's FOV; signaling to the user that data has been successfully collected near the at least one unobserved location; and

[0023] 27. The AR system of any one of claims 25-26, wherein the collected data comprises data associated with a center of the user's FOV. (Item 28) 28. The AR system of any one of items 18-27, wherein the hardware processor is configured to display an animation associated with the graphic at the unobserved location to signal to the user that data has been successfully collected at the unobserved location. (Item 29) To display the animation, the AR system: generating at least one bubble associated with said graphic; determining a collision trajectory associated with the at least one bubble; and animating the at least one bubble to move along the collision trajectory towards a collision location on a mesh associated with a map of the user's environment; displaying a collision between said at least one bubble and said mesh; 29. The AR system of item 28, configured to perform the following: (Item 30) 30. The AR system of any one of items 18-29, wherein to instruct a user to observe the graphic, the hardware processor is configured to display an indicator ring based on the unobserved location relative to the user's position. (Item 31) 1. A method for creating or updating an augmented reality map of a user environment, the method being performed by a computing system having one or more hardware computer processors and one or more non-transitory computer readable storage devices storing software instructions executable by the computing system to perform the computerized method; obtaining distance data from each of a plurality of directions around a user, the distance data indicating a distance at which a physical object is detected in each of the respective directions; creating a ranked list of waypoints associated with distance data having distances exceeding a threshold distance, the waypoint associated with the greatest distance being first in the ranked list; displaying a graphic located with reference to a highest ranked waypoint for viewing on an augmented reality (AR) device worn by a user, the graphic configured to indicate to the user areas of the user environment where the augmented reality map is incomplete; Detecting a movement of the user that enables collection of data by the AR device associated with a waypoint; determining whether additional waypoints remain in the ranked list; and in response to determining that additional waypoints remain in said ranked list, performing said indicating, detecting, and determining using a next highest ranked waypoint; A method comprising: (Item 32) Item 32. The method of item 31, wherein the ranked list comprises six waypoints. (Item 33) 33. The method of claim 31 or 32, wherein the indicating, detecting, and determining are repeated until at least a minimum threshold of data has been collected. (Item 34) 34. The method of any one of items 31-33, further comprising determining the presence of at least one persistent coordinate frame (PCF) in the user's environment based on the collected data. (Item 35) 35. The method of any one of claims 31-34, comprising, in response to determining the presence of at least one PCF, loading content associated with a stored map that includes the at least one PCF. (Item 36) 36. The method of any one of claims 31-35, comprising displaying a graphic in the center of the user's FOV. (Item 37) Item 37. The method of item 36, wherein the hardware processor is configured to display a graphic in the center of the user's FOV prior to generating the graphic positioned with reference to the highest ranked waypoint. (Item 38) 38. The method of claim 36 or 37, comprising detecting a movement of the user that enables collection of data by the AR device associated with a center of the user's FOV. (Item 39) 39. The method of any one of items 31-38, comprising displaying an animation associated with a graphic that is positioned with reference to the highest ranked waypoint. (Item 40) Displaying the animation includes: generating at least one bubble associated with said graphic; determining a collision trajectory associated with the at least one bubble; and animating the at least one bubble to move along the collision trajectory towards a collision location on a mesh associated with a map of the user's environment; displaying a collision between said at least one bubble and said mesh; 40. The method according to item 39, comprising: (Item 41) 41. The method of any one of items 31-40, further comprising guiding the user to the waypoint. (Item 42) Item 42. The method of item 41, wherein guiding the user includes displaying an indicator ring. (Item 43) 1. An augmented reality (AR) system, comprising: an AR display configured to present virtual content to a user of the AR system; an outwardly facing camera configured to capture one or more images of the user's three-dimensional (3D) environment; a hardware processor in communication with the AR display and the outward facing camera, the hardware processor comprising: receiving an invitation to access a shared map; facilitating user access to the shared map; and inducing a user to observe a 3D environment of said user; collecting data associated with the user's 3D environment; loading content associated with the shared map if data associated with the shared map is found within the user's 3D environment; a hardware processor programmed to An AR system equipped with (Item 44) The hardware processor further comprises: Item 44. The AR system of item 43, configured to determine the presence of at least one persistent coordinate frame (PCF) within the user's environment based on the collected data, and determining whether data associated with the shared map is found within the user's 3D environment is based on determining whether at least one PCF associated with the shared map is identified within the collected data. (Item 45) 45. The AR system of any one of items 43-44, wherein, to load content, the hardware processor is configured to load content associated with the shared map including the at least one PCF in response to determining the presence of the at least one PCF. (Item 46) 46. ​​The AR system of any one of items 43-45, wherein the hardware processor is configured to display graphics in the center of the user's FOV. (Item 47) Item 47. The AR system of item 46, wherein the hardware processor is configured to display the graphic in a center of the user's FOV prior to generating the graphic in an unobserved location. (Item 48) The hardware processor includes: instructing the user to view the graphic in a center of the user's FOV; signaling to the user that data has been successfully collected near the at least one unobserved location; and

[0023] 48. The AR system of any one of claims 43-47, wherein the collected data comprises data associated with a center of the user's FOV. (Item 49) 49. The AR system of any one of items 43-48, wherein the hardware processor is configured to display an animation associated with the graphic at the unobserved location to signal to the user that data has been successfully collected at the unobserved location. (Item 50) To display the animation, the AR system: generating at least one bubble associated with said graphic; determining a collision trajectory associated with the at least one bubble; and animating the at least one bubble to move along the collision trajectory towards a collision location on a mesh associated with a map of the user's environment; displaying a collision between said at least one bubble and said mesh; 50. The AR system of item 49, configured to: (Item 51) 51. The AR system of any one of items 43-50, wherein the shared map comprises a map stored in a location external to the AR display. (Item 52) 52. The AR system of any one of claims 43-51, wherein the shared map comprises a map accessible to multiple AR devices or multiple users. (Item 53) 53. The AR system of any one of items 43-52, wherein to instruct a user to observe the graphic, the hardware processor is configured to display an indicator ring based on the unobserved location relative to the user's position. (Item 54) The hardware processor includes: identifying a map for storage; determining a storage location comprising on-device or off-device storage; determining one or more user permissions associated with the identified map, the one or more user permissions comprising authorization for one or more users to access the identified map; storing the identified map based on the determined storage location and the determined one or more user permissions. 54. The AR system of any one of items 43-53, configured to: (Item 55) 1. An augmented reality (AR) system, comprising: an AR display configured to present virtual content to a user of the AR system; an outwardly facing camera configured to capture one or more images of the user's three-dimensional (3D) environment; a hardware processor in communication with the AR display and the outward facing camera, the hardware processor comprising: receiving user input for updating the map; inducing a user to observe a 3D environment of said user; collecting data associated with the user's 3D environment; determining a map quality index associated with the map; displaying an indication of the map quality index on the AR display; and updating the map based on the collected data; and ceasing to guide the user in response to identifying a stopping condition comprising a user input to stop or a determination that the map quality index reaches a threshold. a hardware processor programmed to An AR system equipped with (Item 56) 56. The AR system of item 55, wherein the map quality index is based on a percentage of the 3D environment that has collected data associated with it. (Item 57) 57. The AR system of any one of items 55-56, wherein the hardware processor is configured to display graphics in the center of the user's FOV. (Item 58) Item 58. The AR system of item 57, wherein the hardware processor is configured to display the graphic in a center of the user's FOV prior to generating the graphic in an unobserved location. (Item 59) The hardware processor includes: instructing the user to view the graphic in a center of the user's FOV; signaling to the user that data has been successfully collected near at least one unobserved location;

[0023] 59. The AR system of any one of items 55-58, wherein the collected data comprises data associated with a center of the user's FOV. (Item 60) 60. The AR system of any one of items 55-59, wherein the hardware processor is configured to display an animation associated with the graphic at the unobserved location to signal to the user that data has been successfully collected at the unobserved location. (Item 61) To display the animation, the AR system: generating at least one bubble associated with said graphic; determining a collision trajectory associated with the at least one bubble; and animating the at least one bubble to move along the collision trajectory towards a collision location on a mesh associated with a map of the user's environment; displaying a collision between said at least one bubble and said mesh; Item 61. The AR system of item 60, configured to perform the following: (Item 62) 62. The AR system of any one of items 55-61, wherein to instruct a user to observe the graphic, the hardware processor is configured to display an indicator ring based on the unobserved location relative to the user's position.

[0013] [Figure 1] FIG. 1 depicts an illustration of a mixed reality scenario involving a virtual reality object and a physical object viewed by a person.

[0014] [Diagram 2] FIG. 2 diagrammatically illustrates an example of a wearable system in which an embodiment of the exemplary waypoint system may be implemented.

[0015] [Diagram 3] FIG. 3 diagrammatically illustrates aspects of an approach for simulating a three-dimensional image using multiple depth planes.

[0016] [Figure 4] FIG. 4 diagrammatically illustrates an embodiment of a waveguide stack for outputting image information to a user.

[0017] [Diagram 5]FIG. 5 shows an exemplary output beam that may be output by a waveguide.

[0018] [Figure 6] FIG. 6 is a schematic diagram showing an optical system including a waveguide device, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem for use in generating a multifocal volumetric display, image, or light field.

[0019] [Figure 7] FIG. 7 is a block diagram of an embodiment of a wearable system.

[0020] [Figure 8] FIG. 8 is a process flow diagram of an embodiment of a method for rendering virtual content in relation to recognized objects.

[0021] [Figure 9] FIG. 9 is a block diagram of another example of a wearable system including an embodiment of an exemplary waypoint system.

[0022] [Figure 10] FIG. 10 is a process flow diagram of an example method for determining user input to a wearable system.

[0023] [Figure 11] FIG. 11 is a process flow diagram of an embodiment of a method for interacting with a virtual user interface.

[0024] [Figure 12] FIG. 12 diagrammatically illustrates an overall system diagram depicting multiple user devices interacting with each other.

[0025] [Figure 13A]FIG. 13A is a schematic diagram illustrating a user wearing an AR display system that renders AR content as the user moves through a physical world environment, according to some embodiments.

[0026] [Figure 13B] FIG. 13B is a schematic diagram illustrating a viewing optics assembly and associated components, according to some embodiments.

[0027] [Figure 13C] FIG. 13C is a schematic diagram illustrating an AR system using a world reconstruction system, according to some embodiments.

[0028] [Figure 13D] FIG. 13D is a schematic diagram illustrating components of an AR system that maintains a model of a passable world, according to some embodiments.

[0029] [Figure 14] FIG. 14 is a block diagram illustrating the creation of a persistent coordinate frame (PCF) and binding of XR content to the PCF according to some embodiments.

[0030] [Figure 15] FIG. 15 is a flowchart illustrating a method of establishing and using a PCF, according to some embodiments.

[0031] [Figure 16] FIG. 16 is a block diagram of the XR system of FIG. 8 including a second XR device, according to some embodiments.

[0032] [Figure 17] FIG. 17 is a schematic diagram illustrating a room and key frames established for various areas within the room, according to some embodiments.

[0033] [Figure 18] FIG. 18 is a schematic diagram illustrating establishment of sustained poses based on key frames, according to some embodiments.

[0034] [Figure 19] FIG. 19 is a schematic diagram illustrating establishment of a persistent coordinate frame (PCF) based on a persistent attitude, according to some embodiments.

[0035] [Figure 20] 20A-20C are schematic diagrams illustrating examples of creating a PCF, according to some embodiments.

[0036] [Figure 21] FIG. 21 illustrates an example block diagram of the map creation process.

[0037] [Figure 22A] FIG. 22A illustrates an example map creation flow for a new user.

[0038] [Figure 22B] FIG. 22B illustrates an example map creation flow for an existing user.

[0039] [Figure 23A] FIG. 23A illustrates an example flow chart of an example relocation process.

[0040] [Figure 23B] FIG. 23B illustrates an example relocation flow.

[0041] [Figure 24A] FIG. 24A illustrates an example map permission and / or storage flow.

[0042] [Figure 24B-1] 24B-1, 24B-2, and 24B-3 illustrate various examples of map permissions and / or storage configurations. [Figure 24B-2] 24B-1, 24B-2, and 24B-3 illustrate various examples of map permissions and / or storage configurations. [Figure 24B-3] 24B-1, 24B-2, and 24B-3 illustrate various examples of map permissions and / or storage configurations.

[0043] [Figure 25-1] 25A-25H illustrate example user interactions with a local map, including local relocation and map creation. [Figure 25-2] 25A-25H illustrate example user interactions with a local map, including local relocation and map creation. [Figure 25-3] 25A-25H illustrate example user interactions with a local map, including local relocation and map creation. [Figure 25-4] 25A-25H illustrate example user interactions with a local map, including local relocation and map creation.

[0044] [Figure 26A] FIG. 26A illustrates an example environment synchronization flow.

[0045] [Figure 26B] 26B and 26C illustrate an exemplary process of location into a shared map. [Figure 26C] 26B and 26C illustrate an exemplary process of location into a shared map.

[0046] [Figure 27] FIG. 27 illustrates an example map curation flow.

[0047] [Figure 28] FIG. 28 illustrates an example scanning process that may be implemented by the AR system.

[0048] [Figure 29] FIG. 29 illustrates an example waypoint spawning process.

[0049] [Diagram 30] FIG. 30 illustrates an example flow chart of an example waypoint guidance process.

[0050] [Diagram 31] 31A and 31B illustrate an exemplary indicator ring.

[0051] [Diagram 32] FIG. 32 illustrates an example waypoint activation process.

[0052] [Diagram 33] FIG. 33 illustrates an example waypoint animation process.

[0053] [Figure 34A] 34A-34C illustrate example stages of an example waypoint animation process. [Figure 34B] 34A-34C illustrate example stages of an example waypoint animation process. [Figure 34C] 34A-34C illustrate example stages of an example waypoint animation process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0054] Detailed Description A. Introduction Virtual and augmented reality environments are generated, in part, by a computer using data that describes the environment. This data may, for example, describe various objects with which a user may sense and interact. Examples of these objects include objects that are rendered and displayed for the user to see, audio that is played for the user to hear, and haptic (or tactile) feedback for the user to feel. Users may sense and interact with virtual and augmented reality environments through a variety of visual, auditory, and tactile methods.

[0055] Virtual or augmented reality (AR) systems can be useful for many applications, spanning the fields of scientific visualization, medical and military training, engineering design and prototyping, remote operation and telepresence, and personal entertainment. Augmented reality, in contrast to virtual reality, comprises one or more virtual objects in association with real objects in the physical world. Such an experience greatly enhances a user's experience and enjoyment with an augmented reality system, and also opens the door for a variety of applications that allow a user to experience real and virtual objects simultaneously.

[0056] However, there are significant challenges in providing such a system. In order to provide a user with a realistic augmented reality experience, the AR system must understand the user's physical surroundings at all times in order to correctly correlate the location of virtual objects in relation to real objects. Additionally, the AR system must correctly know how to position virtual objects in relation to the user's head, body, etc. This requires extensive knowledge of the user's position in relation to the world at all times. Additionally, these functions should advantageously be implemented in a manner such that speed and performance are maintained while costs (e.g., energy costs, etc.) are kept low.

[0057] Therefore, there is a need for an improved system for providing a user with a realistic augmented reality experience. B. Example of a 3D display for a wearable system

[0058] The wearable system (also referred to herein as an augmented reality (AR) system) can be configured to present 2D or 3D virtual images to a user. The images may be still images, frames of video, or videos, in combination or the like. At least a portion of the wearable system can be implemented on a wearable device that may present a VR, AR, or MR environment, alone or in combination, for user interaction. The wearable device can be a head-mounted device (HMD), which is used synonymously with the AR device (ARD). Additionally, for purposes of this disclosure, the term "AR" is used synonymously with the term "MR."

[0059] Figure 1 depicts an illustration of a mixed reality scenario with certain virtual reality objects and certain physical objects viewed by a person. In Figure 1, an MR scene 100 is depicted in which a user of the MR technology sees a real-world park-like setting 110 featuring people, trees, buildings in the background, and a concrete platform 112. In addition to these items, the user of the MR technology also perceives that they "see" a robotic figure 114 standing on the real-world platform 112, and a flying cartoon-like avatar character 140 that appears to be an anthropomorphic bumblebee, although these elements do not exist in the real world.

[0060] In order for a 3D display to produce a true depth sensation, and more specifically, a simulated sensation of surface depth, it may be desirable to generate, for each point in the display's field of view, an accommodation response that corresponds to that point's virtual depth. If the accommodation response to a display point does not correspond to that point's virtual depth as determined by the binocular depth cues of convergence and stereopsis, the human eye may experience accommodation conflicts, resulting in unstable imaging, deleterious eye strain, headaches, and, in the absence of accommodative information, a near-complete lack of surface depth.

[0061] VR, AR, and MR experiences can be provided by a display system having a display in which images corresponding to multiple depth planes are provided to a viewer. The images may be different for each depth plane (e.g., providing slightly different presentations of a scene or object) and may be focused separately by the viewer's eyes, thereby serving to provide depth cues to the user based on the ocular accommodation required to focus on different image features of a scene located on different depth planes, or based on observing different image features on different depth planes that are out of focus. As discussed elsewhere herein, such depth cues provide a believable perception of depth.

[0062] FIG. 2 illustrates an example of a wearable system 200, which can be configured to provide an AR / VR / MR scene and can include an embodiment of an exemplary waypoint system described herein. The wearable system 200 can also be referred to as an AR system 200. The wearable system 200 includes a display 220 and various mechanical and electronic modules and systems to support the functionality of the display 220. The display 220 can be coupled to a frame 230, which is wearable by a user, wearer, or viewer 210. The display 220 can be positioned in front of the eyes of the user 210. The display 220 can present AR / VR / MR content to the user. The display 220 can comprise a head-mounted display worn on the user's head. In some embodiments, a speaker 240 is coupled to the frame 230 and positioned adjacent to the user's ear canal (in some embodiments, another speaker, not shown, is positioned adjacent to the user's other ear canal to provide stereo / shapeable sound control). The display 220 may include an audio sensor (e.g., a microphone) 232 to detect audio streams from the environment and capture ambient sounds. One or more other audio sensors, not shown, may be positioned to provide stereo sound reception. Stereo sound reception may be used to determine the location of a sound source. The wearable system 200 may perform voice or speech recognition on the audio streams.

[0063] The wearable system 200 may include an outwardly facing imaging system 464 (shown in FIG. 4) that observes the world in the user's surrounding environment. The wearable system 200 may also include an inwardly facing imaging system 462 (shown in FIG. 4) that may track the user's eye movements. The inwardly facing imaging system may track either one eye's movements or both eyes' movements. The inwardly facing imaging system 462 may be mounted to the frame 230 and may be in electrical communication with a processing module 260 or 270 that may process image information obtained by the inwardly facing imaging system and determine, for example, the pupil diameter or orientation of the user's 210 eyes, eye movements, or eye posture.

[0064] As an example, the wearable system 200 can obtain an image of the user's posture using an outward facing imaging system 464 or an inward facing imaging system 462. The image may be a still image, a frame of a video, or a video.

[0065] The display 220 can be operably coupled (250) to a local data processing module 260, which can be mounted in a variety of configurations, such as fixedly attached to the frame 230, by wired leads or a wireless connection, fixedly attached to a helmet or hat worn by the user, built into headphones, or otherwise removably attached to the user 210 (e.g., in a backpack configuration, in a belt-attached configuration).

[0066] The local processing and data module 260 may comprise a hardware processor and digital memory such as non-volatile memory (e.g., flash memory), both of which may be utilized to aid in the processing, caching, and storage of data. The data may include a) data captured from sensors (e.g., that may be operatively coupled to the frame 230 or otherwise attached to the user 210), such as an image capture device (e.g., a camera in an inward-facing or outward-facing imaging system), audio sensor (e.g., a microphone), an inertial measurement unit (IMU), an accelerometer, a compass, a global positioning system (GPS) unit, a wireless device, or a gyroscope, or b) data obtained or processed using the remote processing module 270 or remote data repository 280, possibly for passing to the display 220 after processing or retrieval. The local processing and data module 260 may be operatively coupled to a remote processing module 270 or a remote data repository 280, such as via a wired or wireless communication link 262 or 264, such that these remote modules are available as resources to the local processing and data module 260. In addition, the remote processing module 270 and the remote data repository 280 may be operatively coupled to each other.

[0067] The remote processing module 270 may include one or more processors configured to analyze and process the data or image information. The remote data repository 280 may include a digital data storage facility, which may be available through the Internet or other networking configurations in a "cloud" resource configuration. Data may be stored and computations performed in the local processing and data module, allowing for fully autonomous use from the remote module.

[0068] The human visual system is complex and difficult to provide a realistic perception of depth. Without being limited by theory, it is believed that a viewer of an object may perceive the object as three-dimensional due to a combination of vergence and accommodation. Vergence movements of the two eyes relative to one another (e.g., rotational movement of the pupils toward or away from one another to converge the gaze of the eyes and fixate on an object) are closely coupled to the focusing (or "accommodation") of the eye's lenses. Under normal conditions, a change in focus of the eye's lenses or accommodation of the eye to change focus from one object to another at a different distance will automatically produce a matching change in vergence at the same distance, a relationship known as the "accommodation-vergence reflex." Similarly, a change in vergence will induce a matching change in accommodation under normal conditions. A display system that provides a better match between accommodation and vergence may produce a more realistic and comfortable simulation of a three-dimensional image.

[0069] FIG. 3 illustrates aspects of an approach for simulating a three-dimensional image using multiple depth planes. With reference to FIG. 3, objects at various distances from the eyes 302 and 304 on the z-axis are accommodated by the eyes 302 and 304 such that the objects are in focus. The eyes 302 and 304 assume a particular accommodated state and focus objects at different distances along the z-axis. As a result, a particular accommodated state may be said to be associated with a particular one of the depth planes 306 having an associated focal length such that an object or part of an object at a particular depth plane is in focus when the eye is in an accommodated state relative to that depth plane. A three-dimensional image may be simulated by providing a different presentation of an image for each eye 302 and 304, and by providing a different presentation of an image corresponding to each of the depth planes. Although shown as separate for clarity of illustration, it should be understood that the fields of view of the eyes 302 and 304 may overlap, for example, as the distance along the z-axis increases. In addition, it should be understood that the contours of the depth plane, shown as flat for ease of illustration, may be curved in physical space such that all features within the depth plane are in focus with the eye in a particular accommodative state. Without being limited by theory, it is believed that the human eye is typically capable of interpreting a finite number of depth planes to provide depth perception. As a result, a highly realistic simulation of perceived depth may be achieved by providing the eye with different presentations of images corresponding to each of these limited number of depth planes. C. Waveguide Stack Assembly

[0070] FIG. 4 illustrates an example of a waveguide stack for outputting image information to a user. The wearable system 400 includes a stack of waveguides or a stacked waveguide assembly 480 that can be utilized to provide a three-dimensional perception to the eye / brain using multiple waveguides 432b, 434b, 436b, 438b, 4400b. The wearable system 400 can correspond to the wearable system 200 of FIG. 2, of which FIG. 4 diagrammatically illustrates some portions in more detail. For example, the waveguide assembly 480 can be integrated into the display 220 of FIG. 2.

[0071] 4, the waveguide assembly 480 may also include a number of features 458, 456, 454, 452 between the waveguides. The features 458, 456, 454, 452 may be lenses. The features 458, 456, 454, 452 may not be lenses. Rather, they may simply be spacers (e.g., cladding layers or structures to form air gaps).

[0072] The waveguides 432b, 434b, 436b, 438b, 440b or multiple lenses 458, 456, 454, 452 may be configured to transmit image information to the eye with various levels of wavefront curvature or light beam divergence. Each waveguide level may be associated with a particular depth plane and configured to output image information corresponding to that depth plane. Image injection devices 420, 422, 424, 426, 428 may be utilized to inject image information into the waveguides 440b, 438b, 436b, 434b, 432b, respectively, which may be configured to disperse incident light across each individual waveguide for output toward the eye 410. Light exits output surfaces of image launch devices 420, 422, 424, 426, 428 and is launched into corresponding input edges of waveguides 440b, 438b, 436b, 434b, 432b. A single beam of light (e.g., a collimated beam) may be launched into each waveguide to output an entire field of cloned collimated beams that are directed towards the eye 410 at a particular angle (and divergence) that corresponds to the depth plane associated with the particular waveguide.

[0073] The image input devices 420, 422, 424, 426, 428 may each be a discrete display that generates image information for input into a respective corresponding waveguide 440b, 438b, 436b, 434b, 432b. Additionally or alternatively, the image input devices 420, 422, 424, 426, 428 may be the output of a single multiplexed display that may send image information to each of the image input devices 420, 422, 424, 426, 428 via, for example, one or more optical conduits (such as fiber optic cables).

[0074] A controller 460 controls the operation of the stacked waveguide assembly 480 and the image input devices 420, 422, 424, 426, 428. The controller 460 may include programming (e.g., instructions in a non-transitory computer-readable medium) that coordinates the timing and provision of image information to the waveguides 440b, 438b, 436b, 434b, 432b. The controller 460 may be a single integrated device or a distributed system connected by a wired or wireless communication channel. The controller 460 may be part of the processing module 260 or 270 (shown in FIG. 2) in some embodiments.

[0075] The waveguides 440b, 438b, 436b, 434b, 432b may be configured to propagate light within each individual waveguide by total internal reflection (TIR). The waveguides 440b, 438b, 436b, 434b, 432b may each be planar or have another shape (e.g., curved) with major top and bottom surfaces and edges extending between the major top and bottom surfaces. In the illustrated configuration, the waveguides 440b, 438b, 436b, 434b, 432b may each include a light extraction optical element 440a, 438a, 436a, 434a, 432a configured to extract light from the waveguide by redirecting light propagating within each individual waveguide out of the waveguide and outputting image information to the eye 410. The extracted light may also be referred to as out-coupled light, and the light extraction optical element may also be referred to as out-coupling optical element. The extracted light beam may be output by a waveguide to where the light propagating in the waveguide strikes a light redirecting element. The light extraction optical element (440a, 438a, 436a, 434a, 432a) may be, for example, a reflective or diffractive optical feature. Although shown disposed on the bottom major surface of the waveguides 440b, 438b, 436b, 434b, 432b for ease of explanation and clarity of the drawings, the light extraction optical element 440a, 438a, 436a, 434a, 432a may be disposed on the top or bottom major surface, or may be disposed directly within the volume of the waveguides 440b, 438b, 436b, 434b, 432b. The light extraction optical elements 440a, 438a, 436a, 434a, 432a may be formed in a layer of material that is attached to a transparent substrate and forms the waveguides 440b, 438b, 436b, 434b, 432b. The waveguides 440b, 438b, 436b, 434b, 432b may be a monolithic piece of material and the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be formed on a surface of and / or within that piece of material.

[0076] Continuing with reference to FIG. 4, as discussed herein, each waveguide 440b, 438b, 436b, 434b, 432b can be configured to output light and form an image corresponding to a particular depth plane. For example, the waveguide 432b closest to the eye may be configured to deliver collimated light to the eye 410 as it is launched into such waveguide 432b. The collimated light may represent an optical infinity focal plane. The next waveguide 434b may be configured to send collimated light that passes through a first lens 452 (e.g., a negative lens) before it can reach the eye 410. The first lens 452 may be configured to generate a slight convex wavefront curvature so that the eye / brain interprets the light originating from the next upper waveguide 434b as originating from a first focal plane closer inward from optical infinity toward the eye 410. Similarly, the third upper waveguide 436b passes its output light through both a first lens 452 and a second lens 454 before reaching the eye 410. The combined refractive power of the first and second lenses 452 and 454 may be configured to produce another incremental amount of wavefront curvature such that the eye / brain interprets the light emerging from the third waveguide 436b as originating from a second focal plane that is even closer inward from optical infinity towards the person than was the light from the next upper waveguide 434b.

[0077] Other waveguide layers (e.g., waveguides 438b, 440b) and lenses (e.g., lenses 456, 458) are similarly configured, with the highest waveguide 440b in the stack sending its output through all of the lenses between it and the eye for an aggregate focal force that represents the focal plane closest to the person. To compensate for the stack of lenses 458, 456, 454, 452 when viewing / interpreting light originating from the world 470 on the other side of the stacked waveguide assembly 480, a compensating lens layer 430 may be placed on top of the stack to compensate for the aggregate force of the lens stacks 458, 456, 454, 452 below. Such a configuration provides as many perceived focal planes as there are waveguide / lens pairs available. Both the light extraction optical elements of the waveguides and the focusing sides of the lenses may be static (e.g., not dynamic or electroactive). Additionally or alternatively, one or both may be dynamic using electroactive features.

[0078] Continuing with reference to FIG. 4, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be configured to both redirect light from its respective waveguide and output this light with the appropriate amount of divergence or collimation for the particular depth plane associated with the waveguide. As a result, waveguides with different associated depth planes may have different configurations of light extraction optical elements that output light with different amounts of divergence depending on the associated depth plane. As discussed herein, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be solid or surface features that may be configured to output light at a specific angle. For example, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be solid holograms, surface holograms, or diffraction gratings. Light extraction optical elements such as diffraction gratings are described in U.S. Patent Publication No. 2015 / 0178939, published June 25, 2015, which is incorporated by reference in its entirety.

[0079] In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a are diffractive features, i.e., "diffractive optical elements" (also referred to herein as "DOEs"), that form a diffraction pattern. Preferably, the DOEs have a relatively low diffraction efficiency, such that only a portion of the light of the beam is deflected toward the eye 410 at each intersection of the DOE, while the remainder continues traveling through the waveguide via total internal reflection. The light carrying the image information is thus split into several related exit beams that exit the waveguide at multiple locations, which can result in a very uniform pattern of exit emission toward the eye 304 for this particular collimated beam bouncing within the waveguide.

[0080] One or more DOEs may be switchable between an "on" state in which they actively diffract and an "off" state in which they do not significantly diffract. For example, a switchable DOE may comprise a layer of polymer dispersed liquid crystal in which the microdroplets comprise a diffractive pattern within a host medium, and the refractive index of the microdroplets can be switched to substantially match that of the host material (in which case the pattern does not significantly diffract incident light), or the microdroplets can be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts incident light).

[0081] The number and distribution of depth planes or depths of field may be dynamically varied based on the pupil size or orientation of the viewer's eye. The depth of field may vary inversely with the pupil size of the viewer. As a result, as the size of the pupil of the viewer's eye decreases, the depth of field increases so that one plane that is indistinguishable because its location exceeds the focal depth of the eye may become distinguishable and appear more in focus with a reduction in pupil size and a corresponding increase in depth of field. Similarly, the number of spaced depth planes used to present different images to the viewer may be reduced with a reduced pupil size. For example, the viewer may not be able to clearly perceive the details of both the first and second depth planes at one pupil size without adjusting the accommodation of the eye from one depth plane to the other. However, these two depth planes may be simultaneously well-focused for the user at another pupil size without changing accommodation.

[0082] The display system may vary the number of waveguides that receive image information based on a determination of pupil size or orientation, or in response to receiving an electrical signal indicative of a particular pupil size or orientation. For example, if the user's eye is unable to distinguish between two depth planes associated with two waveguides, the controller 460 (which may be an embodiment of the local processing and data module 260) may be configured or programmed to stop providing image information to one of these waveguides. Advantageously, this may reduce the processing burden on the system, thereby increasing the responsiveness of the system. In an embodiment in which the DOE for a waveguide is switchable between on and off states, the DOE may be switched to the off state when the waveguide receives image information.

[0083] It may be desirable to have the exit beam satisfy the condition of having a diameter less than the diameter of the viewer's eye. However, satisfying this condition may be difficult in light of the variability in the size of the viewer's pupil. This condition may be satisfied over a wide range of pupil sizes by varying the size of the exit beam in response to a determination of the size of the viewer's pupil. For example, as the pupil size decreases, the size of the exit beam may also decrease. The exit beam size may be varied using a variable aperture.

[0084] The wearable system 400 may include an outwardly facing imaging system 464 (e.g., a digital camera) that images a portion of the world 470. This portion of the world 470 may be referred to as the field of view (FOV) of the world camera, and the imaging system 464 is sometimes also referred to as the FOV camera. The world camera FOV may or may not be the same as the viewer 210 FOV, which encompasses the portion of the world 470 that the viewer 210 perceives at a given time. For example, in some situations, the world camera FOV may be larger than the viewer 210's field of view of the wearable system 400. The entire area available for viewing or imaging by the viewer may be referred to as the ocular field of view (FOR). The FOR may include a solid angle of 4π steradians surrounding the wearable system 400, since the wearer can move their body, head, or eyes and perceive virtually any direction in space. In other contexts, the wearer's movements may be more constrained and the wearer's FOR may correspondingly subtend a smaller solid angle. Images obtained from the outwardly facing imaging system 464 can be used to track gestures (e.g., hand or finger gestures) made by the user, detect objects in the world 470 in front of the user, etc.

[0085] The wearable system 400 can include an audio sensor 232, e.g., a microphone, to capture ambient sound. As described above, one or more other audio sensors can be positioned to provide stereo sound reception useful in determining the location of the speech source. The audio sensor 232 can, as another example, comprise a directional microphone, which can also provide such useful directional information regarding where the audio source is located. The wearable system 400 can use information from both the outwardly facing imaging system 464 and the audio sensor 232 in locating the speech source or to determine the active speaker at a particular moment, etc. For example, the wearable system 400 can use voice recognition, alone or in combination with a reflected image of the speaker (e.g., as seen in a mirror), to determine the identity of the speaker. As another example, the wearable system 400 can determine the location of the speaker in the environment based on sounds obtained from the directional microphone. The wearable system 400 can use speech recognition algorithms to analyze sounds originating from the speaker's location, determine the content of the speech, and use voice recognition techniques to determine the speaker's identity (e.g., name or other demographic information).

[0086] The wearable system 400 may also include an inwardly facing imaging system 462 (e.g., a digital camera) that observes user movements such as eye and face movements. The inwardly facing imaging system 462 may be used to capture images of the eye 410 and determine the size or orientation of the pupil of the eye 304. The inwardly facing imaging system 462 may be used to obtain images for use in determining the direction the user is looking (e.g., eye pose) or for biometric identification of the user (e.g., via iris identification). At least one camera may be utilized for each eye independently to separately determine the pupil size or eye pose of each eye, thereby allowing the presentation of image information to each eye to be dynamically adjusted for that eye. The pupil diameter or orientation of only a single eye 410 (e.g., using only a single camera per pair of eyes) may be determined and assumed to be similar for both eyes of the user. Images obtained by inwardly facing imaging system 462 may be analyzed to determine the user's eye posture or mood, which may be used by wearable system 400 to determine audio or visual content to be presented to the user. Additionally or alternatively, wearable system 400 may determine head pose (e.g., head position or head orientation) using sensors such as an IMU, accelerometer, gyroscope, etc.

[0087] The wearable system 400 may include a user input device 466 through which a user may input commands into the controller 460 and interact with the wearable system 400. For example, the user input device 466 may include a trackpad, a touch screen, a joystick, a multi-degree-of-freedom (DOF) controller, a capacitive sensing device, a game controller, a keyboard, a mouse, a directional pad (D-pad), a wand, a tactile device, a totem (e.g., serving as a virtual user input device), and the like. A multi-DOF controller may sense user input in possible translation (e.g., left / right, forward / backward, or up / down) or rotation (e.g., yaw, pitch, or roll) of some or all of the controller. A multi-DOF controller that supports translation may be referred to as 3DOF, while a multi-DOF controller that supports translation and rotation may be referred to as 6DOF. A user may use a finger (e.g., thumb) to press or swipe across the touch-sensitive input device to provide input to the wearable system 400 (e.g., to provide user input to a user interface provided by the wearable system 400). The user input device 466 may be held by the user's hand during use of the wearable system 400. The user input device 466 may communicate with the wearable system 400 in wired or wireless communication.

[0088] FIG. 5 shows an example of an output beam output by a waveguide. Although one waveguide is shown, it should be understood that other waveguides in the waveguide assembly 480 may function similarly and that the waveguide assembly 480 includes multiple waveguides. Light 520 is launched into the waveguide 432b at the input edge 432c of the waveguide 432b and propagates within the waveguide 432b by TIR. At the point where the light 520 impinges on the DOE 432a, a portion of the light may exit the waveguide as an output beam 510. Although the output beams 510 are shown as approximately parallel, they may also be redirected to propagate to the eye 410 at an angle (e.g., divergent output beam formation) depending on the depth plane associated with the waveguide 432b. It should be understood that a nearly collimated exit beam may represent a waveguide with light extraction optics that couples out light and forms an image that appears to be set at a depth plane at a far distance (e.g., optical infinity) from the eye 410. Other waveguides or other sets of light extraction optics may output a more divergent exit beam pattern, which would require the eye 410 to accommodate to a closer distance and focus on the retina, and would be interpreted by the brain as light from a distance closer to the eye 410 than optical infinity.

[0089] FIG. 6 is a schematic diagram showing an optical system including a waveguide device, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem used in generating a multifocal volumetric display, image, or light field. The optical system can include a waveguide device, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem. The optical system can be used to generate a multifocal volumetric display, image, or light field. The optical system can include one or more primary planar waveguides 632a (only one is shown in FIG. 6) and one or more DOEs 632b associated with each of at least some of the primary waveguides 632a. The planar waveguides 632b can be similar to the waveguides 432b, 434b, 436b, 438b, 440b discussed with reference to FIG. 4. The optical system may employ a dispersive waveguide device to relay light along a first axis (vertical or Y-axis in the illustration of FIG. 6) and expand the effective exit pupil of the light along the first axis (e.g., Y-axis). The dispersive waveguide device may include, for example, a dispersive planar waveguide 622b and at least one DOE 622a (illustrated by a double dashed line) associated with the dispersive planar waveguide 622b. The dispersive planar waveguide 622b may be similar or the same in at least some respects as the primary planar waveguide 632b, which has a different orientation therefrom. Similarly, the at least one DOE 622a may be similar or the same in at least some respects as the DOE 632a. For example, the dispersive planar waveguide 622b or the DOE 622a may be made of the same material as the primary planar waveguide 632b or the DOE 632a, respectively. The embodiment of the optical display system 600 shown in FIG. 6 can be integrated into the wearable system 200 shown in FIG.

[0090] The relayed and exit pupil expanded light may be optically coupled from the dispersive waveguide device into one or more primary planar waveguides 632b. The primary planar waveguides 632b may relay the light along a second axis (e.g., the horizontal or X-axis in the diagram of FIG. 6) that is preferably orthogonal to the first axis. It should be noted that the second axis may be a non-orthogonal axis to the first axis. The primary planar waveguides 632b may expand the effective exit pupil of the light along its second axis (e.g., the X-axis). For example, the dispersive planar waveguide 622b may relay and expand the light along the vertical or Y-axis and pass the light to the primary planar waveguide 632b, which may relay and expand the light along the horizontal or X-axis.

[0091] The optical system may include one or more colored light sources (e.g., red, green, and blue laser light) 610, which may be optically coupled into a proximal end of a single mode optical fiber 640. The distal end of the optical fiber 640 may be threaded or received through a hollow tube 642 of piezoelectric material. The distal end protrudes from the tube 642 as an unfixed flexible cantilever 644. The piezoelectric tube 642 may be associated with four quadrant electrodes (not shown). The electrodes may be, for example, plated on the outside, outer surface or circumference, or diameter of the tube 642. A core electrode (not shown) may also be located in the core, center, inner circumference, or inner diameter of the tube 642.

[0092] For example, drive electronics 650, electrically coupled via wires 660, drive opposing pairs of electrodes to bend the piezoelectric tube 642 independently in two axes. The protruding distal tip of the optical fiber 644 has a mechanical resonant mode. The frequency of the resonance may depend on the diameter, length, and material properties of the optical fiber 644. By oscillating the piezoelectric tube 642 near the first mechanical resonant mode of the fiber cantilever 644, the fiber cantilever 644 may be caused to oscillate and sweep through a large deflection.

[0093] By stimulating resonant vibrations in two axes, the tip of the fiber cantilever 644 is scanned biaxially within an area filling a two-dimensional (2-D) scan. By modulating the intensity of the light source 610 synchronously with the scanning of the fiber cantilever 644, the light emitted from the fiber cantilever 644 can form an image. A description of such a setup is provided in U.S. Patent Publication No. 2014 / 0003762, which is incorporated herein by reference in its entirety.

[0094] The optical coupler subsystem components can collimate the light emitted from the scanning fiber cantilever 644. The collimated light can be reflected by the mirrored surface 648 into a narrow dispersive planar waveguide 622b, which contains at least one diffractive optical element (DOE) 622a. The collimated light can propagate perpendicularly (with respect to the view of FIG. 6) along the dispersive planar waveguide 622b by TIR, and in so doing repeatedly intersect with the DOE 622a. The DOE 622a preferably has a low diffraction efficiency. This causes a portion of the light (e.g., 10%) to diffract toward the edge of the larger primary planar waveguide 632b at each point of intersection with the DOE 622a, and via TIR, a portion of the light can continue on its original trajectory down the length of the dispersive planar waveguide 622b.

[0095] At each point of intersection with the DOE 622a, additional light can be diffracted toward the entrance of the primary planar waveguide 632b. By splitting the incident light into multiple outcoupled sets, the exit pupil of the light can be vertically expanded by the DOE 622a in the dispersive planar waveguide 622b. This vertically expanded light that is outcoupled from the dispersive planar waveguide 622b can enter the edge of the primary planar waveguide 632b.

[0096] Light entering the primary waveguide 632b can propagate horizontally (with respect to the view of FIG. 6) along the primary waveguide 632b via TIR. The light propagates horizontally along at least a portion of the length of the primary waveguide 632b via TIR as it intersects with the DOE 632a at multiple points. The DOE 632a can advantageously be designed or configured to have a phase profile that is the sum of a linear diffraction pattern and a radially symmetric diffraction pattern, producing both deflection and focusing of the light. The DOE 632a can advantageously have a low diffraction efficiency (e.g., 10%) such that only a portion of the light in the beam is deflected toward the viewer's eye at each intersection of the DOE 632a, while the remainder of the light continues to propagate through the primary waveguide 632b via TIR.

[0097] At each point of intersection between the propagating light and the DOE 632a, a portion of the light is diffracted toward the adjacent face of the primary waveguide 632b, allowing the light to escape the TIR and emerge from the face of the primary waveguide 632b. The radially symmetric diffraction pattern of the DOE 632a can additionally impart a focal level to the diffracted light, both shaping (e.g., imparting curvature) the optical wavefronts of the individual beams and steering the beams to angles matching the designed focal level.

[0098] Thus, these different paths can couple light out of the primary planar waveguide 632b by resulting in different fill patterns at the DOE 632a's multiplicity, focus level, or exit pupil at different angles. Different fill patterns at the exit pupil can be advantageously used to generate light field displays with multiple depth planes. Each layer or set of layers in a stack (e.g., three layers) in a waveguide assembly may be employed to generate individual colors (e.g., red, blue, green). Thus, for example, a first set of three adjacent layers may be employed to generate red, blue, and green light, respectively, at a first focal depth. A second set of three adjacent layers may be employed to generate red, blue, and green light, respectively, at a second focal depth. Multiple sets may be employed to generate full 3D or 4D color image light fields with various focal depths. D. Other Components of the Wearable System

[0099] In many implementations, the wearable system may include other components in addition to or as an alternative to the components of the wearable system described above. The wearable system may include, for example, one or more tactile devices or components. The tactile device or component may be operable to provide a haptic sensation to the user. For example, the tactile device or component may provide a haptic sensation of pressure or texture upon touching the virtual content (e.g., a virtual object, virtual tool, other virtual structure). The haptic sensation may replicate the sensation of a physical object that the virtual object represents, or may replicate the sensation of an imaginary object or character (e.g., a dragon) that the virtual content represents. In some implementations, the tactile device or component may be worn by the user (e.g., a user wearable glove). In some implementations, the tactile device or component may be held by the user.

[0100] A wearable system may include one or more physical objects that can be manipulated by a user, for example, to enable input to or interaction with the wearable system. These physical objects may be referred to herein as totems. Some totems may take the form of inanimate objects, e.g., pieces of metal or plastic, walls, the surface of a table. In some implementations, a totem may not actually have any physical input structures (e.g., keys, triggers, joysticks, trackballs, rocker switches). Instead, a totem may simply provide a physical surface, and the wearable system may render a user interface to appear to the user as being on one or more surfaces of the totem. For example, the wearable system may render an image of a computer keyboard and trackpad to appear to reside on one or more surfaces of the totem. For example, the wearable system may render a virtual computer keyboard and virtual trackpad to appear on the surface of a thin rectangular plate of aluminum that serves as the totem. The rectangular plate itself does not have any physical keys or trackpads or sensors. However, the wearable system may detect user manipulation or interaction or touch with the rectangular plate as a selection or input made via a virtual keyboard or virtual trackpad. User input device 466 (shown in FIG. 4) may be an embodiment of the totem, which may include a trackpad, touchpad, trigger, joystick, trackball, rocker or virtual switch, mouse, keyboard, multi-degree-of-freedom controller, or another physical input device. A user may use the totem alone or in combination with posture to interact with the wearable system or other users.

[0101] Examples of tactile devices and totems usable with the wearable devices, HMDs, and display systems of the present disclosure are described in U.S. Patent Publication No. 2015 / 0016777, which is incorporated by reference in its entirety. E. Exemplary Wearable Systems, Environments, and Interfaces

[0102] The wearable system may employ various mapping-related techniques to achieve a high depth of field in the rendered light field. When mapping a virtual world, it is advantageous to capture all features and points in the real world and accurately depict the virtual objects in relation to the real world. To this end, FOV images captured from a user of the wearable system can be added to the world model by including new photos that convey information about various points and features of the real world. For example, the wearable system can collect a set of map points (such as 2D or 3D points), find new map points, and render a more accurate version of the world model. The world model of the first user can be communicated (e.g., via a network such as a cloud network) to a second user so that the second user can experience the world surrounding the first user.

[0103] 7 is a block diagram of an example of a MR environment 700. The MR environment 700 may be configured to receive inputs (e.g., visual inputs 702 from a user's wearable system, stationary inputs 704 such as room cameras, sensory inputs 706 from various sensors, gestures, totems, eye tracking, user inputs from user input devices 466, etc.) from one or more user wearable systems (e.g., wearable system 200 or display system 220) or stationary room systems (e.g., room cameras, etc.). The wearable systems can use various sensors (e.g., accelerometers, gyroscopes, temperature sensors, movement sensors, depth sensors, GPS sensors, inward-facing imaging systems, outward-facing imaging systems, etc.) to determine the location and various other attributes of the user's environment. This information may be further supplemented with information from stationary cameras in the room, which may provide images from different perspectives or various cues. Image data acquired by cameras (e.g., room cameras or cameras of outward-facing imaging systems) may be reduced to a set of mapping points.

[0104] One or more object recognizers 708 can crawl through the received data (e.g., a collection of points), recognize or map the points, tag the images, and associate semantic information with the objects using a map database 710. The map database 710 may comprise various points and their corresponding objects collected over time. The various devices and the map database may be interconnected through a network (e.g., LAN, WAN, etc.) and accessible to the cloud.

[0105] Based on this information and the set of points in the map database, the object recognizers 708a-708n may recognize objects in the environment. For example, the object recognizers may recognize faces, people, windows, walls, user input devices, televisions, documents (e.g., travel documents, driver's licenses, passports as described in the security embodiments herein), other objects in the user's environment, etc. One or more object recognizers may be specialized for objects with certain characteristics. For example, the object recognizer 708a may be used to recognize faces, while another object recognizer may be used to recognize documents.

[0106] Object recognition may be performed using various computer vision techniques. For example, the wearable system may analyze images acquired by the outward facing imaging system 464 (shown in FIG. 4) and perform scene reconstruction, event detection, video tracking, object recognition (e.g., people or documents), object pose estimation, face recognition (e.g., from images of people in the environment or on documents), learning, indexing, motion estimation, or image analysis (e.g., identifying indicia in documents, such as photographs, signatures, identification information, travel information, etc.), etc. One or more computer vision algorithms may be used to perform these tasks. Non-limiting examples of computer vision algorithms include Scale Invariant Feature Transform (SIFT), Speed ​​Up Robust Features (SURF), Orientation FAST and Rotation BRIEF (ORB), Binary Robust Invariant Scalable Keypoints (BRISK), Fast Retinal Keypoints (FREAK), Viola-Jones algorithm, Eigenfaces approach, Lucas-Kanade algorithm, Horn-Schunk algorithm, Mean-shift algorithm, visual simultaneous localization and mapping (vSLAM) techniques, sequential Bayes estimators (e.g., Kalman filter, extended Kalman filter, etc.), bundle adjustment, adaptive thresholding (and other thresholding techniques), iterative nearest neighbor (ICP), semi-global matching (SGM), semi-global block matching (SGBM), feature point histograms, various machine learning algorithms (e.g., support vector machines, k-nearest neighbor algorithms, naive Bayes, neural networks (including convolutional or deep neural networks), or other supervised / unsupervised models, etc.), and the like.

[0107] Object recognition can additionally or alternatively be performed by various machine learning algorithms. Once trained, the machine learning algorithms can be stored by the HMD. Some examples of machine learning algorithms can include supervised or unsupervised machine learning algorithms, including regression algorithms (e.g., ordinary least squares regression, etc.), instance-based algorithms (e.g., learning vector quantization, etc.), decision tree algorithms (e.g., classification and regression trees, etc.), Bayesian algorithms (e.g., naive Bayes, etc.), clustering algorithms (e.g., k-means clustering, etc.), association rule learning algorithms (e.g., a priori algorithm, etc.), artificial neural network algorithms (e.g., Perceptron, etc.), deep learning algorithms (e.g., Deep Boltzmann Machine, i.e., deep neural networks, etc.), dimensionality reduction algorithms (e.g., principal component analysis, etc.), ensemble algorithms (e.g., stacked generalization, etc.), or other machine learning algorithms. Individual models can be customized for individual datasets. For example, the wearable device can generate or store a base model. The base model may be used as a starting point to generate additional models specific to a data type (e.g., a particular user in a telepresence session), a data set (e.g., a set of additional images acquired of a user in a telepresence session), a conditional situation, or other variations. The wearable HMD can be configured to generate models for analysis of the aggregated data using a number of techniques. Other techniques may include using predefined thresholds or data values.

[0108] Based on this information and the set of points in the map database, the object recognizer 708a-708n may recognize the object, complement the object with semantic information, and bring the object to life. For example, if the object recognizer recognizes that the set of points is a door, the system may attach some semantic information (e.g., a door has a hinge and has 90 degree movement around the hinge). If the object recognizer recognizes that the set of points is a mirror, the system may attach semantic information that the mirror has a reflective surface that can reflect an image of the object in the room. The semantic information may include the affordances of the object as described herein. For example, the semantic information may include the object's normal. The system may assign a vector, the direction of which indicates the object's normal. Over time, the map database grows as the system (which may reside locally or be accessible through a wireless network) accumulates more data from the world. Once the object is recognized, the information may be transmitted to one or more wearable systems. For example, the MR environment 700 may contain information about a scene taking place in California. The environment 700 may be transmitted to one or more users in New York. Based on the data received from the FOV cameras and other inputs, the object recognizer and other software components can map points collected from the various images, recognize objects, etc., so that the scene can be accurately "passed" to a second user who may be in a different part of the world. The environment 700 may also use a topology map for location purposes.

[0109] 8 is a process flow diagram of an example method 800 for rendering virtual content in relation to recognized objects. Method 800 illustrates how a virtual scene may be presented to a user of a wearable system. The user may be geographically remote from the scene. For example, a user may be in New York but may wish to view a scene currently occurring in California, or may wish to go for a walk with a friend who is in California.

[0110] In block 810, the wearable system may receive input from the user and other users regarding the user's environment. This may be accomplished through various input devices and knowledge already held in the map database. The user's FOV camera, sensors, GPS, eye tracking, etc. communicate information to the system in block 810. The system may determine sparse points based on this information in block 820. The sparse points may be used in determining pose data (e.g., head pose, eye pose, body pose, or hand gestures) that may be used in displaying and understanding the orientation and position of various objects in the user's surroundings. The object recognizers 708a-708n may crawl through these collected points in block 830 and recognize one or more objects using the map database. This information may then be communicated to the user's respective wearable system in block 840, and the desired virtual scene may be displayed to the user in block 850 as appropriate. For example, a desired virtual scene (eg, a user in CA) may be displayed in the proper orientation, position, etc., relative to various objects and other surroundings of the user in New York.

[0111] FIG. 9 is a block diagram of another example of a wearable system. In this example, the wearable system 900 comprises a map 920, which may include a map database 710, containing map data about the world. The map may reside in part locally on the wearable system and in part in a networked storage location (e.g., in a cloud system) accessible by a wired or wireless network. An attitude process 910 may run on the wearable computing architecture (e.g., processing module 260 or controller 460) and utilize data from the map 920 to determine the position and orientation of the wearable computing hardware or the user. The attitude data may be calculated from data collected on the fly as the user experiences the system and operates within the world. The data may comprise images, data from sensors (such as inertial measurement units, which generally include accelerometer and gyroscope components), and surface information about objects in the real or virtual environment.

[0112] The sparse representation may be the output of a simultaneous localization and mapping (e.g., SLAM or vSLAM, which refers to configurations where the input is image / vision only) process. The system can be configured to find not only the location of various components in the world, but also what the world is composed of. Poses can be building blocks that accomplish many goals, including capturing in and using data from maps.

[0113] In one embodiment, the sparse point locations may not be completely correct by themselves, and further information may be required to generate a multifocal AR, VR, or MR experience. A dense representation, generally referring to depth map information, may be utilized to fill this gap, at least in part. Such information may be calculated from a process called stereoscopic vision 940, where the depth information is determined using techniques such as triangulation or time-of-flight sensing. Image information and active patterns (such as infrared patterns generated using an active projector), images obtained from an image camera, or hand gestures / totems 950 may serve as inputs to the stereoscopic vision process 940. A significant amount of depth map information may be fused together, some of which may be summarized using a surface representation. For example, a mathematically definable surface may be an efficient (e.g., for large point clouds) and easy to summarize input to other processing devices such as a game engine. Thus, the outputs of the stereoscopic vision process (e.g., depth maps) 940 may be combined in a fusion process 930. The pose 910 may also be input to this fusion process 930, the output of which becomes the input to the map capture process 920. Sub-surfaces may interconnect to form larger surfaces, such as in topographic mapping, and the map becomes a large-scale hybrid of points and surfaces.

[0114] Various inputs may be utilized to resolve various aspects in the mixed reality process 960. For example, in the embodiment depicted in FIG. 9, game parameters may be inputs to determine that a user of the system is playing a monster battle game with one or more monsters at various locations, whether the monster is dead or fleeing under various conditions (such as when the user shoots the monster), walls or other objects at various locations, and the like. A world map may contain information about the location of objects or semantic information of objects, and the world map can be another useful input for mixed reality. Posture relative to the world is an input as well, and plays an important role for almost any interactive system.

[0115] Controls or inputs from the user are another input to the wearable system 900. As described herein, user inputs can include visual inputs, gestures, totems, audio inputs, sensory inputs, and the like. To move around or play a game, for example, the user may need to command the wearable system 900 as to what they want to do. There are various forms of user control that can be utilized beyond just moving around in space themselves. A totem (e.g., a user input device), or an object such as a toy gun, may be held by the user and tracked by the system. The system would preferably be configured to know that the user is holding an item and understand the type of interaction the user is having with the item (e.g., if the totem or object is a gun, the system may be configured to understand not only the location and orientation, but also whether the user is clicking a trigger or other sensitive button or element, which may be equipped with sensors such as an IMU that can help determine the situation that is occurring even when such activity is not within the field of view of any of the cameras).

[0116] Hand gesture tracking or recognition may also provide input information. The wearable system 900 may be configured to track and interpret hand gestures to gesture left or right, stop, grab, hold, etc. for a button press. For example, in one configuration, a user may want to flip through an email or calendar in a non-gaming environment, or to "fist bump" with another person or player. The wearable system 900 may be configured to utilize a minimal amount of hand gestures, which may or may not be dynamic. For example, gestures may be simple static gestures, such as spreading the hand to indicate stop, thumbs up to indicate OK, thumbs down to indicate not OK, or flipping the hand left and right or up and down to indicate a directional command.

[0117] Eye tracking is another input (e.g., tracking where the user is looking, controlling display technology, rendering to a specific depth or range). Eye vergence may be determined using triangulation, and then accommodation may be determined using a vergence / accommodation model developed for that particular person. An eye camera can perform eye tracking and determine eye gaze (e.g., direction or orientation of one or both eyes). Other techniques can also be used for eye tracking, such as measurement of electrical potentials with electrodes placed near the eyes (e.g., electro-oculography).

[0118] Speech tracking may be another input that may be used alone or in combination with other inputs (e.g., totem tracking, eye tracking, gesture tracking, etc.). Speech tracking may include speech recognition, voice recognition, alone or in combination. System 900 may include an audio sensor (e.g., a microphone) that receives an audio stream from the environment. System 900 may incorporate voice recognition technology to determine who is speaking (e.g., whether the speech is from the wearer of the ARD or another person or voice (e.g., recorded voice transmitted by a loudspeaker in the environment)) and to determine what is being said. The local data and processing module 260 or the remote processing module 270 can process the audio data from the microphone (or audio data in another stream, such as, for example, a video stream being viewed by the user) and recognize the content of the speech by applying various speech recognition algorithms, such as, for example, hidden Markov models, dynamic time warping (DTW) based speech recognition, neural networks, deep learning algorithms such as deep feed-forward and recurrent neural networks, end-to-end automatic speech recognition, machine learning algorithms (described with reference to FIG. 7), or other algorithms using acoustic or language modeling, etc. In some cases, the speech will come from multiple sources, for example, from another person in the vicinity of the user, from an announcer on a television broadcast in the person's vicinity, and from speech content being played to the user of the ARD via the speaker 240. As described further below, these different speech sources (e.g., in this example, people, television announcers, and audio streams) may be content analyzed and different topics may be presented differently to the user by the ARD's user interface (e.g., different topics organized into different threads, speech by different speakers organized into different threads, or a combination of these).

[0119] The local data and processing module 260 or the remote processing module 270 can also apply speech recognition algorithms, which can identify the identity of the speaker, such as whether the speaker is the user 210 of the wearable system 900 or another person with whom the user is conversing. Some example speech recognition algorithms can include frequency estimation, hidden Markov models, Gaussian mixture models, pattern matching algorithms, neural networks, matrix representations, vector quantization, speaker diarization, decision trees, and dynamic time warping (DTW) techniques. Speech recognition techniques can also include anti-speaker techniques such as cohort models and world models. Spectral features may be used in representing speaker characteristics. The local data and processing module or the remote data processing module 270 can perform speech recognition using various machine learning algorithms described with reference to FIG. 7.

[0120] An implementation of the waypoint mapping system 970 may use these user controls or inputs via a user interface (UI). UI elements (e.g., controls, pop-up windows, bubbles, data entry fields, etc.) may be used, for example, to close the display of auxiliary information or add a word to a general word dictionary. Examples of such implementations of the waypoint mapping system 970 and their use are described with reference to Figures 13-24.

[0121] With regard to the camera system, the exemplary wearable system 900 shown in FIG. 9 may include three pairs of cameras: a relative wide FOV or passive SLAM pair of cameras arranged on either side of the user's face, and a different pair of cameras oriented in front of the user to handle the stereoscopic imaging process 940 and also capture hand gestures and totem / object trajectories in front of the user's face. The FOV cameras and pair of cameras for the stereo process 940 may be part of the outward facing imaging system 464 (shown in FIG. 4). The wearable system 900 may include an eye-tracking camera (which may be part of the inward facing imaging system 462 shown in FIG. 4) oriented towards the user's eyes to triangulate eye vectors and other information. The wearable system 900 may also include one or more textured light projectors (such as infrared (IR) projectors) to inject texture into the scene.

[0122] FIG. 10 is a process flow diagram of an example of a method 1000 for determining user input to a wearable system. In this example, a user may interact with a totem. A user may have multiple totems. For example, a user may have one totem designated for social media applications, another totem for playing games, etc. In block 1010, the wearable system may detect the movement of the totem. The movement of the totem may be recognized through an outward facing imaging system or may be detected through sensors (e.g., tactile gloves, image sensors, hand tracking devices, eye tracking cameras, head pose sensors, etc.).

[0123] Based at least in part on the detected gestures, eye poses, head poses, or inputs through the totem, the wearable system detects the position, orientation, or movement of the totem (or the user's eyes or head or gestures) relative to a reference frame, at block 1020. The reference frame may be a set of map points based on which the wearable system translates the totem's (or user's) movements into actions or commands. At block 1030, the user's interactions with the totem are mapped. Based on the mapping of the user interactions to the reference frame 1020, the system determines the user inputs at block 1040.

[0124] For example, a user may move a totem or physical object back and forth, turn a virtual page, indicate moving to the next page, or move from one user interface (UI) display screen to another UI screen. As another example, a user may move their head or eyes to look at different real or virtual objects in the user's FOR. If the user's gaze at a particular real or virtual object is longer than a threshold time, that real or virtual object may be selected as the user input. The vergence-divergence movement of the user's eyes can be tracked and an accommodation / vergence-divergence movement model can be used to determine the accommodation state of the user's eyes, which provides information about the depth plane the user is focusing on. The wearable system can use ray casting techniques to determine real or virtual objects that are in line with the direction of the user's head pose or eye pose. Ray casting techniques can include casting a thin bundle of rays with virtually no lateral width, or casting a ray (e.g., a cone or a truncated cone) with substantial lateral width.

[0125] The user interface may be projected by a display system as described herein (such as display 220 in FIG. 2 ). It may also be displayed using a variety of other techniques, such as one or more projectors. A projector may project an image onto a physical object, such as a canvas or a sphere. Interactions with the user interface may be tracked using one or more cameras outside or part of the system (e.g., using inward facing imaging system 462 or outward facing imaging system 464).

[0126] 11 is a process flow diagram of an example of a method 1100 for interacting with a virtual user interface. Method 1100 may be performed by a wearable system described herein. An embodiment of method 1100 can be used by a wearable system to detect a person or document within the FOV of the wearable system.

[0127] In block 1101, the wearable system may identify a particular UI. The type of UI may be determined by the user. The wearable system may identify that a particular UI needs to be captured based on user input (e.g., gestures, visual data, audio data, sensory data, direct commands, etc.). The UI may be specific to a security scenario, where a wearer of the system observes a user presenting a document to the wearer (e.g., at a passenger checkpoint). In block 1105, the wearable system may generate data for a virtual UI. For example, data associated with the UI's boundaries, general structure, shape, etc. may be generated. In addition, the wearable system may determine map coordinates of the user's physical location so that the wearable system may display the UI in relation to the user's physical location. For example, if the UI is body-centered, the wearable system may determine coordinates of the user's physical standing position, head pose, or eye pose so that a ring UI may be displayed around the user, or a planar UI may be displayed on a wall or in front of the user. In the security context described herein, the UI may be displayed as if it were surrounding the traveler presenting the document to the wearer of the system, so that the wearer may easily view the UI while viewing the traveler and the traveler's documents. If the UI is hand-centered, map coordinates of the user's hand may be determined. These map points may be derived through the FOV camera, data received through sensory input, or any other type of collected data.

[0128] In block 1115, the wearable system may send data from the cloud to the display, or data may be sent from a local database to the display component. In block 1125, a UI is displayed to the user based on the sent data. For example, a light field display may project the virtual UI into one or both of the user's eyes. Once the virtual UI is generated, the wearable system may simply wait for commands from the user in block 1135 and generate more virtual content on the virtual UI. For example, the UI may be a body-centered ring around the user's body or the body of a person (e.g., a traveler) in the user's environment. The wearable system may then wait for a command (gesture, head or eye movement, voice command, input from a user input device, etc.) and, if recognized (block 1165), the virtual content associated with the command may be displayed to the user (block 1175).

[0129] Additional examples of wearable systems, UI, and user experience (UX) are described in U.S. Patent Publication No. 2015 / 0016777, which is incorporated by reference in its entirety. F. Exemplary Communication Between Wearable Systems

[0130] FIG. 12 diagrammatically illustrates an overall system diagram depicting multiple user devices interacting with each other. The computing environment 1200 includes user devices 1230a, 1230b, 1230c. The user devices 1230a, 1230b, and 1230c can communicate with each other through a network 1290. Each of the user devices 1230a-1230c includes a network interface and can communicate with a remote computing system 1220 (which may also include a network interface 1271) via the network 1290. The network 1290 may be a LAN, a WAN, a peer-to-peer network, wireless, Bluetooth, or any other network. The computing environment 1200 may also include one or more remote computing systems 1220. The remote computing systems 1220 may include server computer systems that are clustered and located in different geographic locations. The user devices 1230 a , 1230 b , and 1230 c may communicate with the remote computing system 1220 via a network 1290 .

[0131] The remote computing system 1220 may include a remote data repository 1280, which may maintain information about a particular user's physical or virtual world. The remote data repository may be an embodiment of the remote data repository 280 shown in FIG. 2. The remote computing system 1220 may also include a remote processing module 1270. The remote processing module 1270 may be an embodiment of the remote processing module 270 shown in FIG. 2. The remote processing module 1270 may include one or more processors, which may be in communication with the user devices (1230a, 1230b, 1230c) and the remote data repository 1280. The processors may process information obtained from the user devices and other sources. At least a portion of the processing or storage may be provided by the local processing and data module 260 (as shown in FIG. 2). The remote computing system 1220 may enable a given user to share information about the particular user's own physical or virtual world with another user.

[0132] The user devices may be, alone or in combination, a wearable device (such as an HMD or ARD), a computer, a mobile device, or any other device. For example, user devices 1230b and 1230c may be an embodiment of the wearable system 200 shown in FIG. 2 (or the wearable system 400 shown in FIG. 4), which can be configured to present AR / VR / MR content.

[0133] One or more of the user devices can be used in conjunction with the user input device 466 shown in FIG. 4. The user device can acquire information about the user and the user's environment (e.g., using the outwardly facing imaging system 464 shown in FIG. 4). The user device or the remote computing system 1220 can use the information acquired from the user device to build, update, and construct a collection of images, points, and other information. For example, the user device may process the acquired raw information and send the processed information to the remote computing system 1220 for further processing. The user device may also send the raw information to the remote computing system 1220 for processing. The user device may receive the processed information from the remote computing system 1220 and provide final processing before projecting it to the user. The user device may also process the acquired information and pass the processed information to other user devices. The user device may communicate with the remote data repository 1280 while processing the acquired information. Multiple user devices or multiple server computer systems may be involved in building or processing the acquired images.

[0134] Information about the physical world may evolve over time and may be based on information collected by different user devices. Models of the virtual world may also evolve over time and be based on inputs of different users. Such information and models may sometimes be referred to herein as world maps or world models. As described with reference to Figures 7 and 9, information obtained by user devices may be used to build a world map 1210. The world map 1210 may include at least a portion of the map 920 illustrated in Figure 9. Various object recognizers (e.g., 708a, 708b, 708c...708n) may be used to recognize objects and tag images and to add semantic information to objects. These object recognizers are also described in Figure 7.

[0135] The remote data repository 1280 can be used to store data and facilitate the construction of the world map 1210. The user device can constantly update information about the user's environment and receive information about the world map 1210. The world map 1210 may be created by the user or another person. As discussed herein, the user device (e.g., 1230a, 1230b, 1230c) and the remote computing system 1220 may build or update the world map 1210, alone or in combination. For example, the user device may communicate with the remote processing module 1270 and the remote data repository 1280. The user device may obtain or process information about the user and the user's environment. The remote processing module 1270 may communicate with the remote data repository 1280 and the user device (e.g., 1230a, 1230b, 1230c) and process information about the user and the user's environment. The remote computing system 1220 can modify information obtained by a user device (e.g., 1230a, 1230b, 1230c), for example, by selectively cropping an image of the user, modifying the user's background, adding virtual objects to the user's environment, annotating the user's speech with auxiliary information, etc. The remote computing system 1220 can transmit the processed information to the same or a different user device. G. Example Maps

[0136] FIG. 13A illustrates a user 1331 wearing an AR display system that renders AR content as the user 1331 moves through a physical world environment 1333 (hereafter referred to as “environment 1333”). Information captured by the AR system along the user's path of movement may be processed into one or more tracking maps. The user 1331 positions the AR display system at a location 1334, and the AR display system records ambient information of the passable world (e.g., digital representations of real objects in the physical world that may be stored and updated as the real objects in the physical world change) relative to the location 1334. That information may be stored as a pose in combination with images, features, directional audio input, or other desired data. The location 1334 is aggregated against data input 1336, e.g., as part of a tracking map, and processed by at least a passable world module 1338, which may be implemented by processing on a remote processing module. In some embodiments, the passable world module 1338 may include a head pose component and a world reconstruction component such that the processed information, in combination with other information about the physical objects used in the rendering virtual content, may indicate the location of the objects in the physical world.

[0137] The passable world module 1338 determines, at least in part, where and how the AR content 1340 may be placed in the physical world, as determined from the data input 1336. The AR content is "placed" in the physical world by presenting both a representation of the physical world and the AR content via the user interface, where the AR content is rendered as if interacting with objects in the physical world, and the objects in the physical world are presented as if the AR content obscures the user's view of those objects, when appropriate. In some embodiments, the AR content may be placed by appropriately selecting a portion of a fixed element 1342 (e.g., a table) from the reconstruction and determining the shape and location of the AR content 1340. As an example, the fixed element may be a table, and the virtual content may be positioned to appear on the table. In some embodiments, the AR content may be placed in a structure in the field of view 1344, which may be the current field of view or an estimated future field of view. In some embodiments, the AR content may be persisted to a model 1346 (e.g., a mesh) of the physical world.

[0138] As depicted, the fixed element 1342 serves as a proxy (e.g., a digital copy) for any fixed element in the physical world that may be stored in the passable world module 1338 such that the user 1331 may perceive content on the fixed element 1342 without the system having to map it to the fixed element 1342 each time the user 1331 sees it. The fixed element 1342 may thus be a mesh model determined from a previous modeling session or from a separate user, but stored by the passable world module 1338 for future reference by multiple users. Thus, the passable world module 1338 may recognize the environment 1333 from a previously mapped environment and display the AR content without the user's 1331 device having to first map all or part of the environment 1333, saving computational processes and cycles and avoiding latency of any rendered AR content.

[0139] A mesh model 1346 of the physical world may be created by the AR display system, and appropriate surfaces and metrics for interacting with and displaying the AR content 1340, either fully or partially, can be stored by the passable world module 1338 for future retrieval by the user 1331 or other users, without the need to recreate the model. In some embodiments, the data inputs 1336 are inputs such as geographic location, user identification, and current activity to indicate to the passable world module 1338 which of the one or more fixed elements 1342 is available, the AR content 1340 last placed on the fixed element 1342, and whether that same content should be displayed (such AR content is “persistent” content, regardless of whether the user is viewing a particular passable world model).

[0140] Even in embodiments where objects are considered fixed (e.g., a kitchen table), the passable world module 1338 may update those objects in the model of the physical world from time to time to account for possible changes in the physical world. The model of fixed objects may be updated very infrequently. Other objects in the physical world may be moving or otherwise not considered fixed (e.g., a kitchen chair). To render the AR scene with a realistic feel, the AR system may update the positions of these non-fixed objects much more frequently than those used to update fixed objects. To enable accurate tracking of all of the objects in the physical world, the AR system may draw information from multiple sensors, including one or more image sensors.

[0141] 13B is a schematic illustration of a viewing optics assembly 1348 and associated components. In some embodiments, two eye tracking cameras 1350 are pointed towards the user's eye 1349 to detect metrics of the user's eye 1349, such as eye shape, eyelid occlusion, pupil direction, and glints on the user's eye 1349.

[0142] In some embodiments, one of the sensors may be a depth sensor 1351, such as a time-of-flight sensor, that emits signals into the world and detects reflections of those signals from nearby objects to determine the distance to a given object. The depth sensor may, for example, quickly determine whether objects have entered the user's field of view, either as a result of the objects' motion or a change in the user's posture. However, information about the location of objects in the user's field of view may alternatively or additionally be collected using other sensors. Depth information may be obtained, for example, from a stereoscopic image sensor or a plenoptic sensor.

[0143] In some embodiments, the world camera 1352 records, maps, and / or otherwise models the environment 1333 and detects inputs that may affect the AR content. In some embodiments, the world camera 1352 and / or the camera 1353 may be grayscale and / or color image sensors that may output grayscale and / or color image frames at fixed time intervals. The camera 1353 may also capture a physical world image within the user's field of view at a specific time. The pixels of a frame-based image sensor may be repeatedly sampled even if their values ​​are immutable. The world camera 1352, the camera 1353, and the depth sensor 1351 each have a separate field of view 1354, 1355, and 1356, and collect and record data from the physical world scene.

[0144] The inertial measurement unit 1357 may determine the movement and orientation of the viewing optics assembly 1348. In some embodiments, each component is operably coupled to at least one other component. For example, the depth sensor 1351 is operably coupled to the eye tracking camera 1350 as a check of the measured accommodation to the actual distance the user's eye 1349 is looking at.

[0145] In some embodiments, for example, the viewing optics assembly 1348 may include two world cameras 1352 instead of four. Alternatively, or in addition, the cameras 1352 and 1353 need not capture a visible light image of its full field of view. The viewing optics assembly 1348 may include other types of components. In some embodiments, the viewing optics assembly 1348 may include one or more dynamic visual sensors (DVS), whose pixels may respond asynchronously to relative changes in light intensity exceeding a threshold.

[0146] In some embodiments, the viewing optics assembly 1348 may not include a depth sensor 1351 based on time-of-flight information. In some embodiments, for example, the viewing optics assembly 1348 may include one or more plenoptic cameras, whose pixels may capture the light intensity and angle of incident light, from which depth information can be determined. For example, the plenoptic camera may include an image sensor overlaid with a transmissive diffraction mask (TDM). Alternatively, or in addition, the plenoptic camera may include an image sensor containing angle-sensing pixels and / or phase-detection autofocus pixels (PDAF) and / or a microlens array (MLA). Such sensors may serve as a depth information source instead of, or in addition to, the depth sensor 1351.

[0147] It should also be understood that the configuration of components in FIG. 13B is provided as an example. The viewing optics assembly 1348 may include components with any suitable configuration, which may be configured to provide the user with the maximum field of view practical for a particular set of components. For example, if the viewing optics assembly 1348 has one world camera 1352, the world camera may be located in a central region of the viewing optics assembly instead of on the side.

[0148] Information from sensors in the viewing optics assembly 1348 may be coupled to one or more of the processors in the system. The processor may generate data that may be rendered to cause the user to perceive the virtual content as interacting with objects in the physical world. The rendering may be implemented in any suitable manner, including generating image data that depicts both the physical and virtual objects. In other embodiments, the physical and virtual content may be depicted in a scene by modulating the opacity of a display device through which the user sees through the physical world. The opacity may be controlled to create the appearance of the virtual object and block the user from seeing objects in the physical world that are occluded by the virtual object. In some embodiments, the image data may include only the virtual content, which may be modified (e.g., clipping the content and taking into account occlusion) to be perceived by the user as realistically interacting with the physical world when the virtual content is viewed through the user interface.

[0149] The locations on the viewing optics assembly 1348 where content may be displayed to create the impression of an object at a particular location may depend on the physics of the viewing optics assembly. Additionally, the pose of the user's head relative to the physical world and the direction the user's eyes are looking will affect the location within the physical world content that is displayed at a particular location on the viewing optics assembly where the content will appear. Sensors as described above may provide information from which this information may be collected and / or calculated so that a processor receiving the sensor input may calculate where objects should be rendered on the viewing optics assembly 1348 to create a desired appearance for the user.

[0150] Regardless of how content is presented to a user, a model of the physical world may be used so that characteristics of virtual objects that may be affected by physical objects, including the shape, position, motion, and visibility of the virtual objects, may be correctly calculated. In some embodiments, the model may include a reconstruction of the physical world, such as reconstruction 1318.

[0151] The model may be created from data collected from sensors on the user's wearable device, although in some embodiments the model may be created from data collected by multiple users, which may be aggregated in a computing device remote from all users (and may be "in the cloud").

[0152] The model may be created, at least in part, by a world reconstruction system, for example, depicted in further detail in FIG. 6A. The world reconstruction component 516 may include a perception module 1360, which may generate, update, and store representations for portions of the physical world. In some embodiments, the perception module 1360 may represent a portion of the physical world within the reconstruction range of the sensor as a plurality of voxels. Each voxel corresponds to a 3D cube of a predetermined volume within the physical world and may include surface information and indicate whether a surface exists within the volume represented by the voxel. A voxel may be assigned a value indicating whether its corresponding volume has been determined to include a surface of a physical object, has been determined to be empty, or has not yet been measured with a sensor, and thus its value is unknown. It should be understood that values ​​indicating voxels determined to be empty or unknown need not be explicitly stored, and that the values ​​of voxels may be stored in computer memory in any suitable manner, including not storing information about voxels determined to be empty or unknown.

[0153] In addition to generating information for the persisted world representation, perception module 1360 may identify and output indications of changes in the area surrounding a user of the AR system. Such indications of changes may trigger updates to volumetric data stored as part of the persisted world, or trigger other functionality, such as triggering generate AR content, update AR content, component 604.

[0154] In some embodiments, the perception module 1360 may identify changes based on a signed distance function (SDF) model. The perception module 1360 may be configured to receive sensor data, such as, for example, a depth map 1360a and a head pose 1360b, and then fuse the sensor data into an SDF model 1360c. The depth map 1360a may provide SDF information directly, or images may be processed to arrive at the SDF information. The SDF information represents distances from sensors used to capture that information. Because those sensors may be part of the wearable unit, the SDF information may represent the physical world from the perspective of the wearable unit, and therefore the user. The head pose 1360b may enable the SDF information to be related to voxels in the physical world.

[0155] In some embodiments, the perception module 1360 may generate, update, and store representations for portions of the physical world that are within a perception range. The perception range may be determined at least in part based on the reconstruction range of the sensor, which may be determined at least in part based on the limits of the observation range of the sensor. As a specific example, an active depth sensor that operates using active IR pulses may operate reliably over a range of distances, creating an observation range for the sensor that may be a few centimeters or tens of centimeters to several meters.

[0156] The world reconstruction component 516 may include additional modules that may interact with the perception module 1360. In some embodiments, the persisted world module 1362 may receive a representation for the physical world based on data obtained by the perception module 1360. The persisted world module 1362 may also include representations of the physical world in various formats. For example, volumetric metadata 1362b, such as voxels, may be stored along with the meshes 1362c and planes 1362d. In some embodiments, other information, such as depth maps, may also be saved.

[0157] In some embodiments, a representation of the physical world such as that illustrated in FIG. 6A may provide relatively dense information about the physical world compared to a sparse map such as a tracking map based on feature points, as described above.

[0158] In some embodiments, the perception module 1360 may include modules that generate representations for the physical world in various formats, including, for example, meshes 1360d, planes, and semantics 1360e. The representations for the physical world may be stored across local and remote storage media. The representations for the physical world may be described in different coordinate frames, for example, depending on the location of the storage media. For example, a representation for the physical world stored in a device may be described in a coordinate frame local to the device. The representation for the physical world may have a counterpart stored in the cloud. The counterpart in the cloud may be described in a coordinate frame shared by all devices in the XR system.

[0159] In some embodiments, these modules may generate the representation based on data within the perception range of one or more sensors at the time the representation is generated and data captured at earlier times and information in the persisted world module 1362. In some embodiments, these components may operate on depth information captured using a depth sensor. However, the AR system may also include a vision sensor and generate such representations by analyzing monocular or binocular visual information.

[0160] In some embodiments, these modules may operate on regions of the physical world. They may be triggered to update a subregion of the physical world when the perception module 1360 detects a change in the physical world in that subregion. Such a change may be detected, for example, by detecting a new surface in the SDF model 1360c, or by other criteria, such as a change in the values ​​of a sufficient number of voxels representing the subregion.

[0161] The world reconstruction components 516 may include components 1364 that may receive a representation of the physical world from the perception module 1360. Information about the physical world may be pulled by these components, for example, according to usage requests from applications. In some embodiments, information may be pushed to the usage components, such as via indications of changes in pre-identified areas or changes in the physical world representation within a perception range. Components 1364 may include game programs and other components that implement processing for, for example, visual occlusion, physics-based interactions, and environmental inference.

[0162] In response to a query from component 1364, perception module 1360 may transmit a representation for the physical world in one or more formats. For example, when component 1364 indicates the use is for visual occlusion or physics-based interaction, perception module 1360 may transmit a representation of a surface. When component 1364 indicates the use is for environmental inference, perception module 1360 may transmit meshes, planes, and semantics of the physical world.

[0163] In some embodiments, perception module 1360 may include a component that provides formatting information to component 1364. An example of such a component may be ray casting component 1360f. A usage component (e.g., component 1364) may, for example, query information about the physical world from a particular viewpoint. Ray casting component 1360f may select from one or more representations of the physical world data within a field of view from that viewpoint.

[0164] As should be understood from the foregoing description, the perception module 1360 or another component of the AR system may process data and create a 3D representation of a portion of the physical world. The data to be processed may be reduced, at least in part, by thinning a portion of the 3D reconstruction volume based on the camera frustum and / or depth image, extracting and persisting planar data, capturing, persisting, and updating the 3D reconstruction data in a block that allows local updates while maintaining neighborhood consistency, occlusion data is derived from a combination of one or more depth data sources, providing occlusion data to an application that generates such a scene, and / or performing multi-stage mesh simplification. The reconstruction may contain data of different levels of sophistication, including, for example, raw data such as live depth data, fused volumetric data such as voxels, and calculated data such as meshes.

[0165] In some embodiments, components of the passable world model may be distributed, with some parts running locally on the XR device and some parts running remotely, such as on a network connected to a server or otherwise in the cloud. The distribution of information processing and storage between the local XR device and the cloud may affect the functionality and user experience of the XR system. For example, reducing processing on the local device by distributing processing to the cloud may enable longer battery life and reduce heat generated on the local device. However, distributing much more processing to the cloud may create undesirable latency that causes an unacceptable user experience.

[0166] FIG. 13D depicts a distributed component architecture 1370 configured for spatial computing, according to some embodiments. The distributed component architecture 1370 may include a passable world component 1372 (e.g., PW 538 in FIG. 5A), a Lumin OS 1374, an API 1376, an SDK 1378, and applications 1380. The Lumin OS 1374 may include a Linux-based kernel with custom drivers compatible with XR devices. The API 1376 may include an application programming interface that gives XR applications (e.g., applications 1380) access to the spatial computing features of the XR device. The SDK 1378 may include a software development kit that enables the creation of XR applications.

[0167] One or more components in the architecture 1370 may create and maintain a model of the passable world. In this example, sensor data is collected on the local device. Processing of the sensor data may be performed partially locally on the XR device and partially in the cloud. The PW 538 may include an environment map that is created at least in part based on data captured by AR devices worn by multiple users. During a session of an AR experience, individual AR devices (such as the wearable devices described above in connection with FIG. 4) may create a tracking map, which is one type of map.

[0168] In some embodiments, a device may include components that build both sparse and dense maps. The tracking map may serve as a sparse map and may include information about the head pose of the AR device scanning the environment and the objects detected in the environment at each head pose. Those head poses may be maintained locally per device. For example, the head poses on each device may be relative to the initial head pose when the device is turned on for the session. As a result, each tracking map may be local to the device that creates it. The dense map may include surface information, which may be represented by meshes or depth information. Alternatively or in addition, the dense map may include higher level information derived from surface or depth information, such as the location and / or properties of planes and / or other objects.

[0169] The creation of the dense map may be independent from the creation of the sparse map in some embodiments. The creation of the dense map and the sparse map may be performed, for example, in separate processing pipelines in the AR system. Separating the processing may, for example, allow the generation or processing of different types of maps to be performed at different rates. The sparse map may, for example, be refreshed at a faster rate than the dense map. However, in some embodiments, the processing of the dense and sparse maps may be related even if performed in different pipelines. Changes in the physical world exposed in the sparse map may, for example, trigger an update of the dense map, or vice versa. Furthermore, even if created independently, the maps may be used together. For example, a coordinate system derived from the sparse map may be used to define the position and / or orientation of objects in the dense map.

[0170] The sparse and / or dense maps may be persisted for reuse by the same device and / or for sharing with other devices. Such persistence may be achieved by storing the information in a cloud. The AR device may send the tracking map to the cloud and merge it with an environment map selected from the persisted maps previously stored in the cloud, for example. In some embodiments, the selected persisted map may be sent from the cloud to the AR device for merging. In some embodiments, the persisted map may be oriented with respect to one or more persistent coordinate frames. Such maps may serve as reference maps since they may be used by any of multiple devices. In some embodiments, a model of the passable world may consist of or be created from one or more reference maps. Although the device performs some operations based on a coordinate frame local to the device, it may use the reference map by determining a transformation between its coordinate frame local to the device and the reference map.

[0171] The reference map may originate as a tracking map (TM) (e.g., TM 1102 in FIG. 31A), which may be promoted to the reference map. The reference map may be persisted such that a device accessing the reference map, once it has determined the transformation between its local coordinate system and the coordinate system of the reference map, may use the information in the reference map to determine the location of objects represented in the reference map in the physical world around the device. In some embodiments, the TM may be a head pose sparse map created by the XR device. In some embodiments, the reference map may be created when the XR device sends one or more TMs to a cloud server for merging with additional TMs captured by the XR device at different times or by other XR devices.

[0172] The reference map or other map may provide information about the portion of the physical world represented by the processed data to create the individual map. For example, the tracking map may provide a plan view of the physical objects in the corresponding physical world represented by points. In some embodiments, the map points may represent features of the physical objects, which may include multiple features. For example, each corner of a table may be a feature represented by a point on the map. The features may be derived from the processed images, such as may be obtained using a sensor of a wearable device in an augmented reality system. The features may be derived, for example, by processing the image frames output by the sensor and identifying features based on large gradients in the images or other suitable criteria. Further processing may limit the number of features in each frame. For example, the processing may select features that are likely to represent persistent objects. One or more heuristics may be applied for this selection.

[0173] The tracking map may include data about points collected by the device. For each image frame with data points included in the tracking map, a pose may be stored. The pose may represent the orientation from which the image frame was captured so that feature points in each image frame may be spatially correlated. The pose may be determined by positioning information, such as may be derived from a sensor, such as an IMU sensor, on the wearable device. Alternatively, or in addition, the pose may be determined from matching an image frame with another image frame that depicts an overlapping portion of the physical world. By finding such positional correlation, which may be accomplished by matching a subset of feature points in the two frames, a relative pose between the two frames may be calculated. The relative pose may be appropriate for the tracking map because the map may be relative to a coordinate system local to the device, which is established based on the initial pose of the device when construction of the tracking map began.

[0174] Because much of the information collected with the sensor is likely to be redundant, not all of the feature points and image frames collected by the device may be retained as part of the tracking map. Rather, only certain frames may be added to the map. These frames may be selected based on one or more criteria, such as the degree of overlap with image frames already in the map, the number of new features they contain, or a quality metric for the features in the frame. Image frames that are not added to the tracking map may be discarded or used to revise the location of features. As a further alternative, all or most of the image frames, represented as a set of features, may be retained, but a subset of those frames may be designated as key frames, which are used for further processing.

[0175] The keyframes may be processed to produce a keyrig. The keyframes may be processed to produce a three-dimensional set of feature points and stored as a keyrig. Such processing may involve, for example, comparing image frames derived simultaneously from two cameras and stereoscopically determining the 3D positions of the feature points. Metadata such as pose may be associated with these keyframes and / or keyrigs.

[0176] The environment map may have any of a number of formats, depending on, for example, the storage location of the environment map, including, for example, the local storage and the remote storage of the AR device. For example, the map in the remote storage may have a higher resolution than the map in the local storage on the wearable device when memory is limited. To transmit the higher resolution map from the remote storage to the local storage, the map may be downsampled or otherwise converted to a suitable format, such as by reducing the number of poses per area of ​​the physical world stored in the map and / or the number of feature points stored per pose. In some embodiments, a slice or portion of the high resolution map from the remote storage may be transmitted to the local storage, and the slice or portion is not downsampled.

[0177] The database of environmental maps may be updated as new tracking maps are created. To determine which of the potentially numerous environmental maps in the database should be updated, the updating step may include efficiently selecting one or more environmental maps to be stored in the database associated with the new tracking map. The selected one or more environmental maps may be ranked by relevance, and one or more of the highest ranked maps may be selected for processing to merge the new tracking map with the higher ranked selected environmental map and create one or more updated environmental maps. When the new tracking map represents a portion of the physical world over which there is no existing environmental map to update, the tracking map may be stored in the database as a new environmental map. H. Persistent Coordinate Frames

[0178] The inventors recognize and understand that spatial persistence may be provided through a Persistent Coordinate Frame (PCF). A PCF may be defined based on one or more points that represent recognized features (e.g., corners, edges) in the physical world. The features may be selected such that they are likely to be the same from one user instance to another user instance of the XR system.

[0179] Furthermore, drift during tracking, which may cause the calculated tracking path (e.g., camera trajectory) to deviate from the actual tracking path, may cause the location of the virtual content to appear out of place when rendered relative to a local map based only on the tracking map. The tracking map for the space may be refined to correct for drift as the XR device gathers more information of the scene over time. However, if the virtual content is placed on real objects and stored relative to the device's world coordinate frame derived from the tracking map before map refinement, the virtual content may appear displaced as if the real objects had moved during map refinement. The PCF may be updated according to map refinement because the PCF is defined based on features and is updated as the features move during map refinement.

[0180] A PCF may have six degrees of freedom with translation and rotation relative to a map coordinate system. A PCF may be stored in a local and / or remote storage medium. The translation and rotation of a PCF may be calculated relative to a map coordinate system, for example, depending on the storage location. For example, a PCF used locally by a device may have translation and rotation relative to the world coordinate frame of the device. A PCF in the cloud may have translation and rotation relative to the reference coordinate frame of a reference map.

[0181] PCFs may provide a sparse representation of the physical world and provide less than all of the available information about the physical world so that they can be efficiently processed and transferred. Techniques for processing persistent spatial information may include creating a dynamic map based on one or more coordinate systems in real space across one or more sessions and generating a persistent coordinate frame (PCF) over the sparse map that can be exposed to XR applications, for example, via an application programming interface (API).

[0182] 14 is a block diagram illustrating the creation of a persistent coordinate frame (PCF) and binding of XR content to the PCF, according to some embodiments. Each block may represent digital information stored in computer memory. In the case of application 1180, the data may represent computer-executable instructions. In the case of virtual content 1170, the digital information may define a virtual object, for example, as defined by application 1180. In the case of other boxes, the digital information may characterize some aspect of the physical world.

[0183] In the illustrated embodiment, one or more PCFs are created from images captured using sensors on the wearable device. In the embodiment of FIG. 14, the sensors are visual image cameras. These cameras may be the same cameras used to form the tracking map. Thus, some of the processing suggested by FIG. 14 may be implemented as part of the step of updating the tracking map. However, FIG. 14 illustrates that information providing persistence is generated in addition to the tracking map.

[0184] To derive the 3D PCF, two images 1110 from two cameras mounted on a wearable device in a configuration that enables stereoscopic image analysis are processed together. Figure 14 illustrates image 1 and image 2, each derived from one of the cameras. A single image from each camera is illustrated for convenience. However, each camera may output a stream of image frames, and the processing illustrated in Figure 14 may be performed for multiple image frames in the stream.

[0185] Thus, image 1 and image 2 may each be one frame in a sequence of image frames. The process as depicted in FIG. 14 may be repeated on successive image frames in the sequence until an image frame containing feature points that provide a suitable image from which to form persistent spatial information is processed. Alternatively, or in addition, the process of FIG. 14 may be repeated as the user moves such that the user is no longer close enough to a previously identified PCF to reliably use that PCF to determine position relative to the physical world. For example, the XR system may maintain a current PCF for the user. When the distance exceeds a threshold, the system may switch to a new current PCF closer to the user, which may be generated according to the process of FIG. 14 using image frames acquired at the user's current location.

[0186] Even when generating a single PCF, the stream of image frames may be processed to identify image frames that depict content in the physical world that is likely to be stable and that may be easily identified by devices in the vicinity of the area of ​​the physical world depicted in the image frames. In the embodiment of Figure 14, the process begins with identifying features 1120 in the image. Features may be identified, for example, by finding locations of gradients or other characteristics in the image that are above a threshold, which may correspond, for example, to corners of an object. In the illustrated embodiment, the features are points, but other recognizable features, such as edges, may alternatively or additionally be used.

[0187] In the illustrated embodiment, a fixed number N of features 1120 are selected for further processing. Those feature points may be selected based on one or more criteria, such as gradient magnitude or proximity to other feature points. Alternatively, or in addition, feature points may be selected heuristically, such as based on characteristics that suggest the feature point is persistent. For example, a heuristic may be defined based on a feature point's characteristics that are likely to correspond to a window or door or a corner of a large piece of furniture. Such a heuristic may take into account the feature point itself and what surrounds it. As a specific example, the number of feature points per image may be 100-500, such as 200, or 150-250.

[0188] Regardless of the number of selected feature points, a descriptor 1130 may be calculated for the feature points. In this embodiment, a descriptor is calculated for each selected feature point, but a descriptor may be calculated for a group of feature points, or for a subset of feature points, or for all features in the image. The descriptor characterizes the feature points such that feature points that represent the same object in the physical world are assigned similar descriptors. The descriptor may facilitate matching of two frames, such as may occur when one map is localized against another map. Rather than searching for a relative orientation of the frames that minimizes the distance between the feature points of the two images, an initial matching of the two frames may be done by identifying feature points with similar descriptors. Matching of image frames may be based on matching points with similar descriptors, which may involve less processing than calculating matching of all feature points in the image.

[0189] The descriptors may be calculated as a mapping of the descriptors to the feature points, or in some embodiments, a mapping of patches of the image around the feature points. The descriptors may be numerical quantities. U.S. Patent Application No. 16 / 190,948 describes steps for calculating descriptors for feature points and is incorporated herein by reference in its entirety.

[0190] In the example of FIG. 14, a descriptor 1130 is calculated for each feature point in each image frame. Based on the descriptors and / or the feature points and / or the image itself, image frames may be identified as key frames 1140. In the illustrated embodiment, key frames are image frames that meet certain criteria that are then selected for further processing. In creating the tracking map, for example, image frames that add meaningful information to the map may be selected as key frames to be integrated into the map. On the other hand, image frames that substantially overlap regions over which image frames have already been integrated into the map may be discarded so that they do not become key frames. Alternatively or in addition, key frames may be selected based on the number and / or type of feature points in the image frames. In the embodiment of FIG. 14, key frames 1150 selected for inclusion in the tracking map may also be processed as key frames for determining the PCF, although different or additional criteria for selecting key frames for generation of the PCF may be used.

[0191] Although FIG. 14 shows that key frames are used for further processing, information obtained from the images may be processed in other forms. For example, feature points, such as in a key rig, may alternatively or additionally be processed. Furthermore, while key frames are described as being derived from a single image frame, there need not be a one-to-one relationship between the key frames and the obtained image frames. Key frames may be obtained from multiple image frames, such as by stitching or aggregating the image frames together, such that only features that appear in multiple images are retained in the key frame.

[0192] A key frame may include image information and / or metadata associated with the image information. In some embodiments, images captured by cameras 44, 46 (FIG. 9) may be computed into one or more key frames (e.g., key frames 1, 2). In some embodiments, a key frame may include a camera pose. In some embodiments, a key frame may include one or more camera images captured at a camera pose. In some embodiments, the XR system may determine that a portion of a camera image captured at a camera pose is not useful and therefore not include that portion in a key frame. Thus, using key frames to match new images with earlier knowledge of the scene reduces the use of computational resources of the XR system. In some embodiments, a key frame may include images and / or image data at a location with a certain direction / angle. In some embodiments, a key frame may include a location and direction from which one or more map points may be viewed. In some embodiments, a key frame may include a coordinate frame with an ID. US Patent Application No. 15 / 877,359 describes keyframes and is incorporated herein by reference in its entirety.

[0193] Some or all of the key frames 1140 may be selected for further processing, such as generating persistent poses 1150 for the key frames. The selection may be based on characteristics of all or a subset of the feature points in the image frames. These characteristics may be determined from processing the descriptors, features, and / or the image frames themselves. As a specific example, the selection may be based on a cluster of feature points identified as likely to be associated with a persistent object.

[0194] Each keyframe is associated with the camera pose at which the keyframe was acquired. For keyframes selected for processing into persistent poses, that pose information may be stored along with other metadata about the keyframe, such as a WiFi fingerprint and / or GPS coordinates at the time and / or location of acquisition.

[0195] The persistent pose is a source of information that the device may use to orient itself relative to previously obtained information about the physical world. For example, if a key frame from which the persistent pose was created is incorporated into a map of the physical world, the device may orient itself relative to the persistent pose using a sufficient number of feature points in the key frame associated with the persistent pose. The device may match the persistent pose with a current image obtained of its surroundings. This matching may be based on matching the current image with the image 1110, features 1120, and / or descriptors 1130 that gave rise to the persistent pose, or any subset of the image or their features or descriptors. In some embodiments, the current image frame matched to the persistent pose may be another key frame that is incorporated into the tracking map of the device.

[0196] Information about persistent attitudes may be stored in a format that facilitates sharing between multiple applications, which may be running on the same or different devices. In the example of FIG. 14, some or all of the persistent attitudes may be reflected as persistent coordinate frames (PCFs) 1160. Like persistent attitudes, PCFs may also be associated with a map and may comprise a set of features or other information that a device may use to determine its orientation relative to the PCF. A PCF may include a transformation that defines its transformation relative to the origin of the map, such that a device may determine its position relative to any object in the physical world reflected in the map by correlating its position to the PCF.

[0197] Because PCFs provide a mechanism for determining location relative to physical objects, an application, such as application 1180, may define the location of a virtual object relative to one or more PCFs that serve as anchors for virtual content 1170. FIG. 14 illustrates, for example, app 1 associating PCFs 1, 2 with its virtual content 2. Similarly, app 2 associating PCFs 1, 2 with its virtual content 3. App 1 is also shown associating PCFs 4, 5 with its virtual content 1, and app 2 is shown associating PCF 3 with its virtual content 4. In some embodiments, PCF 3 may be based on image 3 (not shown), and PCFs 4, 5 may be based on images 4 and 5 (not shown), similar to how PCFs 1, 2 are based on images 1 and 2. When rendering this virtual content, the device may apply one or more transformations to calculate information such as the location of the virtual content relative to the device's display and / or the location of the physical object relative to the desired location of the virtual content. Using PCFs as a reference may simplify such calculations.

[0198] In some embodiments, the persistent pose may be a coordinate location and / or orientation having one or more associated keyframes. In some embodiments, the persistent pose may be created automatically after the user travels a certain distance, e.g., 3 meters. In some embodiments, the persistent pose may act as a reference point during localization. In some embodiments, the persistent pose may be stored in the passable world (e.g., the passable world module 538).

[0199] In some embodiments, a new PCF may be determined based on a predefined distance allowed between adjacent PCFs. In some embodiments, one or more sustained poses may be calculated into the PCF once the user has progressed a predetermined distance, e.g., 5 meters. In some embodiments, the PCF may be associated with one or more world coordinate frames and / or reference coordinate frames, e.g., within a passable world. In some embodiments, the PCF may be stored in a local and / or remote database, e.g., depending on security settings.

[0200] 15 illustrates a method 4700 of establishing and using a persistent coordinate frame, according to some embodiments. The method 4700 may begin with capturing images (e.g., image 1 and image 2 in FIG. 14) centered on a scene using one or more sensors of the XR device (act 4702). Multiple cameras may be used, and one camera may generate multiple images, for example in a stream.

[0201] The method 4700 may include extracting points of interest (e.g., features 1120 in FIG. 14 ) from the captured image (4704), generating a descriptor (e.g., descriptor 1130 in FIG. 14 ) for the extracted points of interest (act 4706), and generating a keyframe (e.g., keyframe 1140) based on the descriptor (act 4708). In some embodiments, the method may compare points of interest in the keyframes and form pairs of keyframes that share a predetermined amount of points of interest. The method may use each pair of keyframes to reconstruct a portion of the physical world. The mapped portion of the physical world may be saved as a 3D feature (e.g., a keyrig). In some embodiments, selected portions of the pair of keyframes may be used to construct a 3D feature. In some embodiments, the results of the mapping may be selectively saved. Keyframes that are not used to construct a 3D feature may be associated with the 3D feature through a pose that represents the distance between the keyframes, for example, with a covariance matrix between the poses of the keyframes. In some embodiments, the pair of keyframes may be selected to construct the 3D features such that the distance between each of the constructed 3D features is within a predetermined distance, which may be determined to balance the amount of calculations required and the level of accuracy of the resulting model. Such an approach makes it possible to provide a model of the physical world with an amount of data suitable for efficient and accurate calculations using the XR system. In some embodiments, the covariance matrix of the two images may include the covariance between the poses (e.g., 6 degrees of freedom) of the two images.

[0202] Method 4700 may include generating sustained poses based on the keyframes (act 4710). In some embodiments, the method may include generating sustained poses based on 3D features reconstructed from the pair of keyframes. In some embodiments, the sustained poses may be bound to the 3D features. In some embodiments, the sustained poses may include poses of keyframes used to construct the 3D features. In some embodiments, the sustained poses may include average poses of keyframes used to construct the 3D features. In some embodiments, the sustained poses may be generated such that the distance between nearby sustained poses is within a predetermined value, for example, within a range of 1 meter to 5 meters, any value therebetween, or any other suitable value. In some embodiments, the distance between nearby sustained poses may be represented by a covariance matrix of the nearby sustained poses.

[0203] Method 4700 may include generating a PCF based on the sustained pose (act 4712). In some embodiments, the PCF may be bound to a 3D feature. In some embodiments, the PCF may be associated with one or more sustained poses. In some embodiments, the PCF may include one pose of the associated sustained poses. In some embodiments, the PCF may include an average pose of the poses of the associated sustained poses. In some embodiments, the PCF may be generated such that the distance between neighboring PCFs is within a predetermined value, e.g., a range of 3 meters to 10 meters, any value therebetween, or any other suitable value. In some embodiments, the distance between neighboring PCFs may be represented by a covariance matrix of the neighboring PCFs. In some embodiments, the PCF may be exposed to an XR application, e.g., via an application programming interface (API), such that the XR application may access a model of the physical world through the PCF without accessing the model itself.

[0204] The method 4700 may include associating at least one of the PCFs with image data of the virtual object for display by the XR device (act 4714). In some embodiments, the method may include calculating a translation and orientation of the virtual object relative to the associated PCF. It should be understood that associating the virtual object with the PCF generated by the device on which it is placed is not required. For example, the device may retrieve a stored PCF in a reference map in the cloud and associate the virtual object with the retrieved PCF. It should be understood that the virtual object may move with the associated PCF as the PCF is adjusted over time.

[0205] FIG. 16 illustrates the visual data and algorithms of the first XR device 12.1, the second XR device 12.2, and the server 20 according to some embodiments. The components illustrated in FIG. 16 may operate to perform some or all of the operations associated with generating, updating, and / or using spatial information such as a persistent pose, a persistent coordinate frame, a tracking map, or a reference map as described herein. Although not shown, the first XR device 12.1 may be configured identically to the second XR device 12.2. The server 20 may include a map storage routine 118, a reference map 120, a map transmitter 122, and a map merging algorithm 124.

[0206] The second XR device 12.2, which may be in the same scene as the first XR device 12.1, may include a persistent coordinate frame (PCF) integration unit 1300, an application 1302 that generates image data 68 that may be used to render virtual objects, and a frame embedding generator 308. In some embodiments, the map download system 126, the PCF identification system 128, the map 2, the localization module 130, the reference map embedder 132, the reference map 133, and the map publisher 136 may be grouped into a passable world unit 1304. The PCF integration unit 1300 may be connected to the passable world unit 1304 and other components of the second XR device 12.2 to enable reading, generating, using, uploading, and downloading of PCFs.

[0207] A map comprising PCFs may enable more persistence in a changing world. In some embodiments, locating a tracking map, including, for example, matching features for images, may include selecting features representing persistent content from a map constructed by the PCF, which enables fast matching and / or location. For example, a world in which people move in and out of a scene and objects such as doors move relative to a scene requires less storage space and transmission rates, and enables the use of individual PCFs and their relationships to one another (e.g., a unified constellation of PCFs) to map a scene.

[0208] In some embodiments, the PCF integration unit 1300 may include three converters including a PCF 1306 previously stored in a data storage on a storage unit of the second XR device 12.2, a PCF tracker 1308, a persistent attitude obtainer 1310, a PCF verifier 1312, a PCF generation system 1314, a coordinate frame calculator 1316, a persistent attitude calculator 1318, a tracking map and persistent attitude converter 1320, a persistent attitude and PCF converter 1322, and a PCF and image data converter 1324.

[0209] In some embodiments, the PCF tracker 1308 may have an on prompt and an off prompt that are selectable by the application 1302. The application 1302 may be executable by a processor of the second XR device 12.2 to display virtual content, for example. The application 1302 may have a call, via an on prompt, to switch the PCF tracker 1308 on. The PCF tracker 1308 may generate a PCF when the PCF tracker 1308 is switched on. The application 1302 may have a subsequent call, via an off prompt, to switch the PCF tracker 1308 off. The PCF tracker 1308 terminates PCF generation when the PCF tracker 1308 is switched off.

[0210] In some embodiments, the server 20 may include a plurality of sustained attitudes 1332 and a plurality of PCFs 1330 that have been previously stored in association with the reference map 120. The map transmitter 122 may transmit the reference map 120 together with the sustained attitudes 1332 and / or the PCFs 1330 to the second XR device 12.2. The sustained attitudes 1332 and the PCFs 1330 may be stored on the second XR device 12.2 in association with the reference map 133. Once the map 2 is located relative to the reference map 133, the sustained attitudes 1332 and the PCFs 1330 may be stored in association with the map 2.

[0211] In some embodiments, the persistent attitude obtainer 1310 may obtain a persistent attitude for map 2. A PCF verifyer 1312 may be coupled to the persistent attitude obtainer 1310. The PCF verifyer 1312 may read a PCF from the PCF 1306 based on the persistent attitude read by the persistent attitude obtainer 1310. The PCFs read by the PCF verifyer 1312 may form an initial group of PCFs to be used for image display based on the PCF.

[0212] In some embodiments, the application 1302 may request that additional PCFs be generated. For example, if the user moves into an area not previously mapped, the application 1302 may turn on the PCF tracker 1308. The PCF generation system 1314 may be connected to the PCF tracker 1308 and begin generating PCFs based on map 2 as map 2 begins to expand. The PCFs generated by the PCF generation system 1314 may form a second group of PCFs that may be used for PCF-based image display.

[0213] The coordinate frame calculator 1316 may be connected to the PCF checker 1312. After the PCF checker 1312 reads the PCF, the coordinate frame calculator 1316 may retrieve the head coordinate frame 96 and determine the head pose of the second XR device 12.2. The coordinate frame calculator 1316 may also retrieve the sustained pose calculator 1318. The sustained pose calculator 1318 may be directly or indirectly connected to the frame embedding generator 308. In some embodiments, an image frame may be designated as a key frame after a threshold distance, e.g., 3 meters, from the previous key frame has been advanced. The sustained pose calculator 1318 may generate a sustained pose based on multiple, e.g., three, key frames. In some embodiments, the sustained pose may essentially be an average of the coordinate frames of multiple key frames.

[0214] A tracking map and sustained attitude converter 1320 may be coupled to the map 2 and sustained attitude calculator 1318. The tracking map and sustained attitude converter 1320 may convert map 2 to a sustained attitude and determine a sustained attitude at the origin for map 2.

[0215] The persistent attitude and PCF converter 1322 may be connected to the tracking map and persistent attitude converter 1320 and further to the PCF validator 1312 and the PCF generation system 1314. The persistent attitude and PCF converter 1322 may convert the persistent attitude (for which the tracking map has been converted) to a PCF from the PCF validator 1312 and the PCF generation system 1314 and determine the PCF for the persistent attitude.

[0216] The PCF and image data converter 1324 may be connected to the persistent attitude and PCF converter 1322 and the data channel 62. The PCF and image data converter 1324 converts the PCF into image data 68. The rendering engine 30 is connected to the PCF and image data converter 1324 and may display the image data 68 for the PCF to a user.

[0217] The PCF integration unit 1300 may store additional PCFs generated using the PCF generation system 1314 in the PCF 1306. The PCF 1306 may be stored for a sustained attitude. The map publisher 136 may read the PCF 1306 and the sustained attitude associated with the PCF 1306 when the map publisher 136 transmits map 2 to the server 20 and the map publisher 136 also transmits the PCF and sustained attitude associated with map 2 to the server 20. When the map storage routine 118 of the server 20 stores map 2, the map storage routine 118 may also store the sustained attitude and PCF generated by the second viewing device 12.2. The map merge algorithm 124 may create the reference map 120 with the sustained attitude and PCF of map 2 associated with the reference map 120 and stored in the sustained attitude 1332 and PCF 1330, respectively.

[0218] The first XR device 12.1 may include a PCF integration unit similar to the PCF integration unit 1300 of the second XR device 12.2. When the map transmitter 122 transmits the reference map 120 to the first XR device 12.1, the map transmitter 122 may transmit the sustained attitude 1332 and the PCF 1330 associated with the reference map 120 and resulting from the second XR device 12.2. The first XR device 12.1 may store the PCF and sustained attitude in a data storage device on the storage device of the first XR device 12.1. The first XR device 12.1 may then utilize the sustained attitude and the PCF resulting from the second XR device 12.2 for image display on the PCF. Additionally or alternatively, the first XR device 12.1 may read, generate, utilize, upload and download PCFs and persistent postures in a manner similar to the second XR device 12.2, as described above.

[0219] In the illustrated embodiment, the first XR device 12.1 generates a local tracking map (hereinafter referred to as “Map 1”), and the map storage routine 118 receives Map 1 from the first XR device 12.1. The map storage routine 118 then stores Map 1 as a reference map 120 on the storage device of the server 20.

[0220] The second XR device 12.2 includes a map download system 126, an anchor identification system 128, a location determination module 130, a reference map embedder 132, a local content positioning system 134, and a map publisher 136.

[0221] In use, the map transmitter 122 transmits the reference map 120 to the second XR device 12.2, and the map download system 126 downloads and stores the reference map 120 from the server 20 as the reference map 133.

[0222] The anchor identification system 128 is connected to the world surface determination routine 78. The anchor identification system 128 identifies anchors based on objects detected by the world surface determination routine 78. The anchor identification system 128 uses the anchors to generate a second map (Map 2). As shown by cycle 138, the anchor identification system 128 continues to identify anchors and update Map 2. The locations of the anchors are recorded as three-dimensional data based on the data provided by the world surface determination routine 78. The world surface determination routine 78 receives images from the real object detection camera 44 and depth data from the depth sensor 135 and determines the location of the surface and its relative distance from the depth sensor 135.

[0223] A location module 130 is connected to the reference map 133 and map 2. The location module 130 repeatedly attempts to locate map 2 relative to the reference map 133. A reference map embedder 132 is connected to the reference map 133 and map 2. Once the location module 130 locates map 2 relative to the reference map 133, the reference map embedder 132 embeds the reference map 133 into the anchor of map 2. Map 2 is then updated with the missing data contained in the reference map.

[0224] A local content positioning system 134 is connected to map 2. The local content positioning system 134 may be, for example, a system in which a user may locate local content at a particular location in a world coordinate frame. The local content itself is then tied to an anchor in map 2. The local-to-world coordinate converter 104 converts the local coordinate frame to the world coordinate frame based on the settings of the local content positioning system 134. The functionality of the rendering engine 30, the display system 42, and the data channel 62 are described with reference to FIG. 2.

[0225] Map publisher 136 uploads Map 2 to server 20. Map storage routine 118 of server 20 then stores Map 2 in the server 20's storage medium.

[0226] A map merging algorithm 124 merges map 2 with the reference map 120. When more than two maps, for example three or four maps, relating to the same or adjacent regions of the physical world are stored, the map merging algorithm 124 merges all maps into the reference map 120 and renders the new reference map 120. The map transmitter 122 then transmits the new reference map 120 to every device 12.1 and 12.2 within the area represented by the new reference map 120. Once devices 12.1 and 12.2 locate their respective maps against the reference map 120, the reference map 120 becomes a promoted map.

[0227] FIG. 17 illustrates an example of generating keyframes for a map of a scene, according to some embodiments. In the illustrated example, a first keyframe KF1 is generated for a door on the left wall of a room. A second keyframe KF2 is generated for the area in the corner where the floor, left wall, and right wall of the room meet. A third keyframe KF3 is generated for the window area on the right wall of the room. A fourth keyframe KF4 is generated for the area at the far end of the rug on the floor of the wall. A fifth keyframe KF5 is generated for the area of ​​the rug closest to the user.

[0228] FIG. 18 illustrates an example of generating a persistence posture for the map of FIG. 17, according to some embodiments. In some embodiments, a new persistence posture is created when the device measures a threshold distance traveled and / or an application requests a new persistence posture (PP). In some embodiments, the threshold distance may be 3 meters, 5 meters, 20 meters, or any other suitable distance. Selecting a smaller threshold distance (e.g., 1 m) may result in an increase in computational load since a larger number of PPs may be created and managed compared to a larger threshold distance. Selecting a larger threshold distance (e.g., 40 m) may result in fewer PPs being created and fewer PCFs being created, which may result in an increase in virtual content placement error since virtual content tied to a PCF may be a relatively large distance (e.g., 30 m) away from the PCF, meaning that the error may increase with the increase in the distance from the PCF to the virtual content.

[0229] In some embodiments, a PP may be created at the beginning of a new session. This initial PP may be considered zero and visualized as the center of a circle with a radius equal to the threshold distance. When the device reaches the circumference of the circle and, in some embodiments, an application requests a new PP, the new PP may be placed at the device's current location (threshold distance). In some embodiments, a new PP will not be created at the threshold distance if the device is able to find an existing PP within the threshold distance from the device's new location. In some embodiments, when a new PP (PP 1150 in FIG. 14) is created, the device binds one or more of the closest key frames to the PP. In some embodiments, the location of the PP relative to the key frame may be based on the device's location at the time the PP is created. In some embodiments, a PP will not be created even if the device progresses the threshold distance unless an application requests the PP.

[0230] In some embodiments, an application may request a PCF from a device when the application has virtual content to display to a user. A PCF request from an application may trigger a PP request, and a new PP will be created after the device progresses a threshold distance. FIG. 18 illustrates a first sustained posture PP1 that may cause the closest keyframes (e.g., KF1, KF2, and KF3) to be linked, for example, by calculating the relative posture between the keyframes and the sustained posture. FIG. 18 also illustrates a second sustained posture PP2 that may cause the closest keyframes (e.g., KF4 and KF5) to be linked.

[0231] FIG. 19 illustrates an example of generating a PCF for the map of FIG. 17, according to some embodiments. In the illustrated example, PCF1 may include PP1 and PP2. As described above, the PCF may be used to display image data for the PCF. In some embodiments, each PCF may have coordinates in another coordinate frame (e.g., a world coordinate frame) and a PCF descriptor, e.g., that uniquely identifies the PCF. In some embodiments, the PCF descriptor may be calculated based on feature descriptors of features in a frame associated with the PCF. In some embodiments, various constellations of PCFs may be combined to represent the real world in a persistent manner, requiring less data and less transmission of data.

[0232] 20A-20C are schematic diagrams illustrating an example of establishing and using persistent coordinate frames. FIG. 20A shows two users 4802A, 4802B with their respective local tracking maps 4804A, 4804B that are not localized relative to a reference map. The origins 4806A, 4806B for each user are described by a coordinate system (e.g., a world coordinate system) within their respective areas. These origins of each tracking map may be local to each user since the origin depends on the orientation of that respective device when tracking begins.

[0233] As the user device's sensors scan the environment, the device may capture images that may contain features representative of persistent objects such that those images may be classified as keyframes from which persistent poses may be created, as described above in connection with Figure 14. In this example, tracking map 4804A includes persistent poses (PP) 4808A and tracking map 4804B includes PP 4808B.

[0234] Also, as described above in connection with Figure 14, some of the PPs may be classified as PCFs, which are used to determine the orientation of virtual content to render it to the user. Figure 20B shows that XR devices worn by individual users 4802A, 4802B may create local PCFs 4810A, 4810B based on PPs 4808A, 4808B. Figure 20C shows that persistent content 4812A, 4812B (e.g., virtual content) may be bound to the PCFs 4810A, 4810B by individual XR devices.

[0235] In this example, the virtual content may have a virtual content coordinate frame that may be used by the application generating the virtual content regardless of how the virtual content is to be displayed. The virtual content may be defined as surfaces, such as triangles of a mesh, at particular locations and angles relative to the virtual content coordinate frame. To render that virtual content to a user, the locations of those surfaces may be determined relative to the user who will perceive the virtual content.

[0236] Binding virtual content to a PCF may simplify the calculations involved in determining the location of the virtual content relative to a user. The location of the virtual content relative to a user may be determined by applying a series of transformations. Some of those transformations may change and be updated frequently. Others of those transformations may be stable and updated less frequently or not at all. Nonetheless, the transformations may be applied with a relatively low computational burden such that the location of the virtual content is updated frequently relative to the user and may provide a realistic appearance to the rendered virtual content.

[0237] In the example of Figures 20A-20C, user 1's device has a coordinate system that may be related to the coordinate system that defines the map origin by a transformation rig1_T_w1. User 2's device has a similar transformation rig2_T_w2. These transformations may be expressed as 6 degree transformations and define a translation and rotation to align the device and map coordinate systems. In some embodiments, the transformations may be expressed as two separate transformations, one defining the translation and the other defining the rotation. Thus, it should be understood that the transformations may be expressed in a form that simplifies calculations or otherwise provides advantages.

[0238] The transformations between the origin of the tracking map and the PCFs identified by the individual user devices are represented as pcf1_T_w1 and pcf2_T_w2. In this embodiment, the PCFs and PPs are the same, such that the same transformation also characterizes the PP.

[0239] The location of the user device relative to the PCF is therefore rig1_T_pcf1=(rig1_T_w1) * It can be calculated by successive applications of these transformations such as (pcf1_T_w1).

[0240] As shown in Figure 20C, the virtual content is located relative to the PCF using the transformation obj1_T_pcf1. This transformation may be set by the application generating the virtual content, which may receive information from a world reconstruction system that describes the physical objects relative to the PCF. To render the virtual content to the user, a transformation to the coordinate system of the user's device is calculated, which is the transformation obj1_t_w1 = (obj1_T_pcf1). * The virtual content coordinate frame can be calculated by relating it to the origin of the tracking map via (pcf1_T_w1), which can then be related to the user's device via a further transformation rig1_T_w1.

[0241] The location of the virtual content may change based on the output from the application generating the virtual content. As it changes, the end-to-end transformation from the source coordinate system to the destination coordinate system may be recalculated. In addition, the user's location and / or head pose may also change as the user moves. As a result, any end-to-end transformation that depends on the user's location or head pose will also change, just as the transformation rig1_T_w1 may change.

[0242] The transformation rig1_T_w1 may be updated with the user's movement based on tracking the user's position relative to stationary objects in the physical world. Such tracking may be performed by a headphone tracking component that processes a sequence of images, as described above, or by other components of the system. Such updates may be performed by determining the user's pose relative to a stationary reference frame, such as PP.

[0243] In some embodiments, the location and orientation of the user device may be determined relative to the nearest persistent pose, or in this example, the PCF as PP is used as the PCF. Such a determination may be made by identifying feature points that characterize the PP in a current image captured with a sensor on the device. Using image processing techniques such as stereoscopic image analysis, the location of the device relative to those feature points may be determined. From this data, the system calculates the relationship rig1_T_pcf1=(rig1_T_w1) * Based on (pcf1_T_w1), the change in transformation associated with the user's movement may be calculated.

[0244] The system may determine and apply the transformations in an order that is computationally efficient. For example, the need to calculate rig1_T_w1 from the measurements that result in rig1_T_pcf1 may be avoided by both tracking the user pose and defining the location of the virtual content relative to a PP or PCF that is built on a persistent pose. In this way, the transformation from the source coordinate system of the virtual content to the destination coordinate system of the user's device can be calculated using the representation (rig1_T_pcf1) * The end-to-end transformation may be based on a measured transformation according to (obj1_t_pcf1), where the first transformation is measured by the system and the latter transformation is supplied by the application that defines the virtual content for rendering. In embodiments where the virtual content is positioned relative to the map origin, the end-to-end transformation may relate the virtual object coordinate system to the PCF coordinate system based on a further transformation between map coordinates and PCF coordinates. In embodiments where the virtual content is positioned relative to a different PP or PCF than the one for which the user position is tracked, a transformation between the two may be applied. Such a transformation may be fixed or may be determined, for example, from the map in which both appear.

[0245] A transform-based approach may be implemented in a device, for example, with components that process sensor data and build a tracking map. As part of that process, those components may identify feature points that can be used as persistent poses, which may in turn be turned into PCFs. Those components may limit the number of persistent poses generated for the map and provide suitable spacing between persistent poses, as described above in connection with FIGS. 17-19, while allowing a user to be close enough to a persistent pose location to accurately calculate the user's pose, regardless of location in the physical environment. As the persistent pose closest to the user is updated as a result of user movement, refinements to the tracking map, or other causes, any of the transforms used to calculate the location of the virtual content relative to the user, which depend on the location of the PP (or PCF, if used), may be updated and stored for use at least until the user moves away from that persistent pose. Note that by calculating and storing the transforms, the computational burden each time the location of the virtual content is updated may be relatively low such that it may be implemented with relatively low latency.

[0246] 20A-20C illustrate positioning relative to a tracking map, with each device having its own tracking map. However, transformations may be generated relative to any map coordinate system. Persistence of content across user sessions of the XR system may be achieved by using a persistent map. Shared user experiences may also be facilitated by using a map to which multiple user devices may be oriented.

[0247] In some embodiments, described in more detail below, the location of virtual content may be defined relative to coordinates in a reference map, which is formatted so that any of multiple devices may use the map. Each device may maintain a tracking map and may determine changes in the user's pose relative to the tracking map. In this example, the transformation between the tracking map and the reference map may be determined through a process of "localization," which may be performed by matching structures in the tracking map (such as one or more sustained poses) with one or more structures of the reference map (such as one or more PCFs).

[0248] Described further below are techniques for creating and using such reference maps. I. Map Creation Process

[0249] A user may create a map of the current environment if the user has not previously interacted with or visited the environment, has not previously scanned the current environment, or the AR system fails to recognize the user's environment. FIG. 21 illustrates an example flow chart of a map creation process 2100. An example of the map creation process 2100 may include a new user map creation flow 2200 as illustrated in FIG. 22A or an existing user map creation flow 2201 as illustrated in FIG. 22B. The map creation process 2100 may include, but is not limited to, a start block 2110, a scan block 2112, a map recognition block 2114, a map quality block 2116, a decision block 2118, a content restoration block 2120, a map save block 2122, and a landscape entry block 2124.

[0250] In a mapping initiation block 2110, the AR system can determine whether to begin scanning or mapping the user's environment. For example, the AR system can determine whether a start condition is met to begin scanning the environment. In some examples, the start condition can include detection by the system of the user's movement into a new and / or unfamiliar location, input from one or more sensors, and / or user input. The user input can include a positive or negative response to one or more prompts. The one or more prompts can differ based on any number of AR system conditions, such as whether the user is a new or existing user, whether the user has previously scanned and created a map of the environment, or the type of program used to initiate the prompt.

[0251] For example, as illustrated in FIG. 22A, the start mapping prompt for a new user may include one or more user interfaces (as shown in blocks 2210, 2212, as labeled as screen 00-screen 6) specific to the new user. For example, the user interface may include a welcome interface. The welcome interface may include a dialog that prompts the user to scan or continue scanning the user's environment. In some examples, the welcome interface may receive user input. The AR system may start or not start scanning based on the user input. Additionally or alternatively, the AR system may present another prompt to the user based on the user input. Additionally or alternatively, the start mapping prompt may include a start mapping prompt in block 2212. The start mapping prompt may include information associated with the process of scanning or mapping the user's environment. Additionally or alternatively, the start mapping prompt may include a prompt to start mapping or scanning the user's environment or to skip mapping or scanning the user's environment. The AR system may receive user input in response to the prompt. Based on that input, the AR system may or may not initiate mapping.

[0252] In another example, as illustrated in FIG. 22B, the start mapping prompt for an existing user can include one or more user interfaces (as shown in blocks or screens 2230, 2232 labeled as screen 00-screen 6) associated with the existing user. For example, the user interface 2230 can include a recognition interface. The recognition interface can include a dialog that prompts the user to select a previous map and restore the content to the user's environment. Additionally or alternatively, the interface 2230 can include a prompt to skip the selection of a map or to generate a new map. Thus, the interface can start mapping or not start, go into a saved map, or move the user to another prompt based on the user input. Additionally or alternatively, the start mapping prompt can include a start mapping prompt separate from or as part of the map selection prompt. The start mapping prompt can include a prompt to select a name for the map that the user will create during the scanning process. The AR system can then start mapping or scanning based on or using that input. For example, the user may select that the map be of an office, home, school, or other location, and the AR system may tag the newly created map based on the user input.

[0253] If the AR system determines that mapping will be initiated, the system may proceed to block 2112 and scan or map the user's environment. If the AR system determines that active mapping will not be initiated, the system may enter an AR landscape at block 2124. As illustrated in Figures 22A and 22B, the AR landscape may include applications or user interfaces, such as those shown in blocks 2224, 2244, that are associated with augmented reality content.

[0254] Additionally or alternatively, the AR system may passively scan or map the user's environment. For example, in instances where the AR system determines that active mapping will not be initiated (or the user exits from active mapping prior to completion), the AR system may map or scan the user's environment after entering the AR landscape. Thus, advantageously, if the map quality reaches a sufficient threshold before the session ends in the AR landscape, user content associated with the user's current environment may be associated with and retrieved from that environment even if not actively mapped via a gamified flow or active scanning process. If the environment is recognized in the future, the AR system may prompt the user to name or tag the location at a later time. However, if the map quality does not reach a sufficient threshold, the AR system may not save or restore the content for later use.

[0255] In the scanning block 2112, the AR system may initiate a scanning process, such as one or more processes described in further detail with reference to FIG. 28-34C or shown in blocks 2214, 2216, 2218, 2220 of FIG. 22A or blocks 2234, 2236, 2238, 2240 of FIG. 22B. Advantageously, the scanning process in the scanning block 2112 may be a process that has a gamified element and serves to instruct the user to move around the environment and collect data within the space. For example, as described with reference to FIG. 28-34C below, the AR system may generate and display one or more graphics (also referred to as waypoints) of the environment around the user and instruct the user to interact with the graphics until an exit criterion is met. As used herein, a waypoint may refer to a particular location in a map and / or a graphic (or other indication) of a particular location in a map. In some examples, a waypoint may include a graphic that marks a particular location in a map and / or directs the user toward the waypoint location. During the user's interaction with one or more graphics, the AR system may collect data about the user's environment.

[0256] In some implementations, the AR system may check whether the user's space is known or recognized in the map recognition block 2114. The AR system may perform this check during or after the scanning block 2112. For example, the AR system may perform a scanning process in block 2112, and the AR system may check at intervals during the scanning process whether the user's environment matches the known environment (e.g., the AR system may match one or more PCFs found in the user's current environment with one or more PCFs in the user's stored map). If the map is recognized by the AR system, the AR system may restore the AR content associated with the recognized map in block 2120 before entering the landscape in block 2124. If the map is not recognized by the AR system, the system may check the map quality in block 2116.

[0257] In map quality block 2116, the AR system can check whether the map generated based on the data collected during scanning block 2112 (and / or combined with the data stored in the virtual world map) is of sufficiently high quality to provide a high quality user experience during current and / or future use. The quality criteria can be any suitable criteria for assessing map quality, such as the number of key frames, PCFs, or other data associated with a mesh in the user's environment. For example, the AR system may determine whether sufficient PCFs have been found or generated based on the collected data to make the user's space identifiable in future scans. The number of PCFs may be a suitable number, such as one, two, three, or five PCFs in the user's environment. However, other numbers may also be possible. For example, the number of PCFs required for a particular environment may be dynamically determined by the AR system, such as based on an analysis of the collected scanning data and / or map data previously associated with the environment. Once the AR system determines that the map reaches a quality threshold, the AR system may save the map using the data collected in block 2122.

[0258] In a save block 2122, the AR system may save the map in a remote or local memory for retrieval by the user or a third party. Additionally or alternatively, the system may prompt the user to enter other information associated with the map, such as a name or geophysical location, to be stored with the map as metadata. As illustrated in FIG. 22A, the system may display an interface and prompt the user to select or enter a map location in block 2222. Additionally or alternatively, as illustrated in FIG. 22B, the AR system may display an acknowledgement interface or message to the user in block 2242 signaling that the new map has been created or saved.

[0259] If the map quality fails to reach the map quality threshold or is otherwise not sufficient to provide a quality user experience, the AR system can determine whether the user wants to continue scanning or mapping the user's environment in decision block 2118. For example, the AR system can prompt the user to continue scanning or to stop the scanning process. The AR system may receive user input in response to the prompt in block 2112 to continue scanning the environment or enter the landscape in block 2124.

[0260] Additionally or alternatively, the AR system can stop the map creation process 2100 at any point and enter the landscape at block 2124. For example, a user can input an end or skip command during the scanning process at block 2112. The AR system can then abort the scanning process at block 2112 or enter the landscape at block 2124. J. Relocation Process

[0261] The AR system may attempt to relocate or place the user in a stored or otherwise recognized space to restore augmented reality content (e.g., from a stored world map) within the user's environment. Figure 23A illustrates an example relocation process 2300, which will be described with reference to the example AR image shown in Figure 23B. The relocation process 2300 may include a map determination step 2310, a relocation check 2312, a scan block 2314, a relocation check 2316, a decision block 2318, a content restoration block 2310, and a landscape entry block 2322. Depending on the embodiment, the method of Figure 23 may include fewer or additional blocks and / or the blocks may be performed in a different order than those illustrated.

[0262] In a map determination block 2310, the AR system may determine a map to restore for the user. For example, the AR system may automatically determine a map associated with the user's detected geographic location. In another example, as illustrated in block 2410 of FIG. 23B, the AR system may display a prompt for the user to select an existing map (and / or confirm a map automatically selected by the AR system). Additionally or alternatively, the AR system may display a prompt for the user to create a new map or to skip the scanning or mapping process. If the user selects an existing map, the AR system may attempt to relocate the user into a known environment, which may include the selected map, and move to block 2312. If the user selects to create a new map, the AR system may move to a map creation process as described with reference to FIG. 21-22B. If the user selects to skip the mapping or scanning process, the AR system may enter the landscape in block 2322. Additionally or alternatively, if the user selects to skip the mapping or scanning process, the AR system may passively collect data about the user's environment and relocate using the passively collected data. 23B, the AR landscape may include an application or user interface associated with the augmented reality content, such as the interface shown in block 2420. After receiving a user input to skip the scanning process, the AR system may launch the application associated with the AR landscape.

[0263] In a relocation check 2312, the AR system may determine whether relocation has already occurred in the map determined in block 2310. The AR system may determine that relocation has occurred, for example, by determining whether at least one PCF (or in some implementations, some other minimum amount of PCFs) stored in the known environment is found in the user's environment. For example, the known environment may have one or more PCFs. During the course of using the AR system, the AR system may passively or actively collect data about the user's environment. The collected data may include PCFs in the user's current environment. In block 2312, the AR system may determine whether any PCFs in the user's current environment match those in the known environment. If the PCFs match, the AR system may determine that relocation has occurred. If the PCFs do not match, the AR system may determine that relocation has not occurred. If the AR system determines that relocation is occurring, the AR system may restore the content associated with the recognized map in block 2320 before entering the landscape in block 2322. If the map is not recognized by the AR system, the AR system may navigate and scan or map the environment in block 2314.

[0264] In a scanning block 2314, the AR system may initiate a scanning process, such as one or more processes described with reference to FIG. 28-34C or shown in blocks 2412, 2414, 2416 of FIG. 24. Advantageously, as described above with reference to map creation, the scanning process in scanning block 2314 may be a process that has a gamified element and serves to instruct the user to move around the environment and collect data within the space. For example, as described with reference to FIG. 28-34C below, the AR system may generate and display one or more graphics of the environment around the user and instruct the user to interact with waypoints until an exit criterion is met. During the user's interaction with the one or more graphics, the AR system may collect data about the user's environment to build a map of the user's environment.

[0265] In the relocation check 2316, the AR system can check whether the user's space is known or recognized. The AR system can perform this check during or after the scanning block 2314. For example, the AR system can perform a scanning process in block 2314, and the AR system can check whether the user's environment matches the known environment at intervals during the scanning process, and the AR system can match one or more PCFs found in the user's current environment with one or more PCFs in the user's selected map or another stored map. However, other methods of relocation are also possible. If the map is recognized by the AR system, the AR system can restore the AR content associated with the recognized map in block 2320 before entering the landscape in block 2322. If the map is not recognized by the AR system, the AR system can proceed to block 2318.

[0266] In decision block 2318, the AR system may determine whether to continue scanning. For example, if relocation has not occurred, the AR system may need more data to generate a new map or to find one or more PCFs in the user's environment. However, the AR system may utilize a timeout, user input, or other termination criteria to stop the scanning process (e.g., during block 2314). The AR system may then prompt the user for input regarding whether to continue scanning the user's environment (e.g., by the scanning process 2314). If the AR system determines that the scanning should continue, for example, due to user input or other criteria, the AR system may continue scanning in block 2314. If the AR system determines that the scanning should not continue, the AR system may enter the landscape in block 2322.

[0267] Additionally or alternatively, the AR system can stop one or more aspects of the map creation process 2300 at any time and enter the landscape at block 2322. In some examples, the AR system may stop one or more aspects of the map creation process 2300 in response to user input. For example, a user can input an end or skip command during the scanning process at block 2314. The AR system can then abort the scanning process at block 2314 or enter the landscape at block 2322.

[0268] Additionally or alternatively, the AR system may passively scan or map the user's environment. For example, in instances where the AR system does not successfully relocate the user into a known map, the AR system may map or scan the user's environment after entering the AR landscape. Thus, advantageously, if the user does not find a known environment and the map quality of the user's current environment does not reach a sufficient threshold before the session ends in the AR landscape, the AR system may prompt the user to save the environment as a new map. K. Map Permissions and Storage

[0269] A map generated by a user may be stored in a number of ways for accessibility by one or more users. For example, a user may store the map on its device or on an external memory (e.g., "cloud" storage on a remote server). A user may set permissions for access to the map when stored either on its device or on cloud storage such that only authorized users with access to the cloud storage may access and interact with the map content. In some examples, a map may be stored on a user device and accessible only to the user who generated the map. In some examples, a map may be stored on a user device and accessible to all users of that user device. In some examples, a map may be stored in an external memory (e.g., a locally networked device or cloud storage). In some examples, a stored map may be accessible only to the user who generated the map. In some examples, a map may be stored in an external memory (e.g., a locally networked device or cloud storage). In some examples, a stored map may be accessible to users with permission to access that external memory. FIG. 24A illustrates an example process 3400 for setting storage settings and access permissions for user-generated maps.

[0270] In block 3402, the AR system may determine map permissions. The map permissions may include one or more map-specific user settings associated with access and storage of a currently being generated or newly generated map. In some examples, the map-specific user settings may include a desired location for storing the map data. The desired location may include device-specific storage, personal world storage, shared world storage, some combination thereof, or other storage for the map data. The device-specific storage may include internal or on-device storage of the map. The device-specific storage may limit access to the map data to the user of the device that stores the generated map. The personal world storage may include external or cloud-based storage of the map data. The personal world storage may limit access to the map data to a specific user, such as the user who generated the map. The shared world storage may include external or cloud-based storage of the map data. The shared world storage may allow access to the map data to multiple users.

[0271] Access to and / or visibility of the shared world data may be determined based on permissions set by the AR system, the user, or a third party. Map data, as may be stored according to user settings, may include any amount of information associated with the map, such as map point data, PCFs, environmental mesh data, acoustic data, lighting data, other model information, recognized objects, the like, or some combination thereof. In some examples, map permissions may include some combination of visibility and / or interactivity of at least a portion of the map data. For example, map permissions may include that the map data is visible, invisible, selectable, non-selectable, interactable, non-interactable, the like, or some combination thereof to the user.

[0272] Additionally or alternatively, the AR system may allow a user to choose different map storage locations for different types of map data. Advantageously, this may allow a user to personalize their map sharing based on perceived privacy and / or other needs. For example, a user may choose to store world features (such as PCF and / or one or more sparse maps) in a shared world store, but store the world model (such as dense mesh, lighting, plane detection, or acoustic data) only on the device. World understanding (such as object recognition and / or scene understanding) may be another option for a user to select storage location or settings. In some embodiments, the world understanding may be stored by default in the same location as the world features. In other embodiments, the user may select the location where the world understanding is stored.

[0273] In some examples, the AR system may determine map permissions based on one or more prompts to the user. For example, the AR system may display one or more prompts in a user interface and receive user input regarding one or more permission features of the map. With reference to FIGS. 24B-1, 24B-2, and 24B-3, the AR system may present one or more options regarding store and / or map permissions and determine store and / or map permissions associated with the map data. In some examples, the user may view and / or select one or more options based on the map categorization. For example, the user may be presented with options to store the world features and / or world model on the device, in a personal world, or in a shared world. Additionally or alternatively, the user may be presented with an option to store the world understanding. In some examples, the world understanding may be stored in the same location as the world model by default. In such a case, the user may be presented with an option to either save or not save the world understanding. In some examples, the world understanding may be stored in a different location than the world model. When presenting the options to the user, the AR system may make the option visible and selectable, visible but not selectable, visible and selected, or in another manner. If the option is selected, the AR system may indicate the selection. The AR system may indicate the selection, for example, by showing a check mark next to the option or through other highlighting of the option.

[0274] As illustrated in FIG. 24B-1, if the AR system determines that the user wants to store the world features on the device, the AR system may allow the user to only store the world model on the device and / or may not allow the user to store the world understanding. The user may then select some combination of those options. For example, referring to FIG. 24B-1, in a first option configuration, the world features may be stored on the device, the world model may be stored on the device, and the world understanding may not be stored.

[0275] As illustrated in FIG. 24B-2, if the AR system determines that the user wants to store the world features in the personal world, the AR system may allow the user to store the world model on the device or in the personal world and / or may not allow the user to store the world understanding based on a selection of a storage option for the world model. The user may then select some combination of those options. For example, referring to FIG. 24B-2, in a second option configuration, the world features may be stored in the personal world, the world model may be stored on the device, and the world understanding may not be stored. In a third configuration, the world features may be stored in the personal world, the world model may be stored in the personal world, and the world understanding may not be stored. In a fourth configuration, the world features may be stored in the personal world, the world model may be stored on the device, and the world understanding may be stored on the personal world. In a fifth configuration, the world features may be stored in the personal world, the world model may be stored in the personal world, and the world understanding may be stored in the personal world.

[0276] As illustrated in FIG. 24B-3, if the AR system determines that the user desires to store the world features in the shared world, the AR system may allow the user to store the world model on the device or in the shared world and / or may not allow the user to store the world understanding based on a selection of a storage option for the world model. The user may then select some combination of those options. For example, referring to FIG. 24B-3, in a sixth option configuration, the world features may be stored in the shared world, the world model may be stored on the device, and the world understanding may not be stored. In a seventh configuration, the world features may be stored in the shared world, the world model may be stored in the shared world, and the world understanding may not be stored. In an eighth configuration, the world features may be stored in the shared world, the world model may be stored on the device, and the world understanding may be stored on the shared world. In a ninth configuration, the world features may be stored in the shared world, the world model may be stored in the shared world, and the world understanding may be stored on the shared world.

[0277] Although certain combinations of map data and storage and / or permissions are described herein, any combination of map data storage and / or permissions is contemplated as possible. For example, map data may be categorized as world features, world models, or world understanding.

[0278] 24A , in block 3404, the AR system can determine whether to begin scanning or mapping the user's environment. For example, the AR system can determine whether a start condition is met to begin scanning the environment. In some examples, the start condition can include detection by the system of a user's movement into a new and / or unfamiliar location, input from one or more sensors, and / or user input indicating that additional mapping should be performed. The user input can include an affirmative or negative response to one or more prompts. The one or more prompts can differ based on any number of AR system conditions, such as whether the user is a new or existing user, whether the user has previously scanned and created a map of the environment, or the type of program used to initiate the prompt.

[0279] In block 3406, the AR system may map and / or scan the user's environment. The mapping or scanning process may include a scanning process as described with reference to FIG. 28-34C. The mapping or scanning process in block 3406 may include a step for recognizing a map, for example, by matching one or more PCFs found in the user's environment with one or more PCFs in the stored map. The mapping or scanning process in block 3406 may include a step for checking whether the map generated based on the collected data (and / or combined with the data stored in the virtual world map) is of sufficiently high quality to provide a high-quality user experience during current and / or future use. The quality criterion may be any suitable criterion for assessing map quality, such as the number of key frames, PCFs, or other data associated with meshes in the user's environment. For example, the AR system may determine whether sufficient PCFs have been found or generated based on the collected data to make the user's space identifiable in future scans.

[0280] In block 3408, the AR system may store the map according to the map permissions. For example, the AR system may save the map to a remote or local memory according to the permissions determined in block 3402. Additionally or alternatively, the system may prompt the user to enter other information associated with the map, such as a name or geophysical location, to be stored with the map as metadata. The system may display an interface in block 2222 (e.g., FIG. 22A) and prompt the user to select or enter a map location. Additionally or alternatively, the AR system may display an acknowledgement interface or message in block 2242 (e.g., FIG. 22B) to the user signaling that a new map has been created or saved.

[0281] In block 3410, the AR system may enter the landscape. For example, the AR system may allow the user to access content associated with a shared map. Applications may allow the user to collaboratively experience augmented reality content with other users. L. Example Map Interactions 1. Local Map Interaction

[0282] In some examples, a user may interact with private or locally stored maps. For example, a user may save a private home or office map for future use by the user. Figures 25A-25H illustrate example user interactions with local maps, including local relocation and map creation.

[0283] FIG. 25A illustrates an example relocation into a selected map. For example, a user may select or name a map, such as "Home," with the intent to allow location and / or future relocation into the map. This process may occur, for example, during block 2310 of FIG. 23. The AR system may actively or passively discover PCFs by active or passive collection of data in the user's environment. The AR system may then match those discovered PCFs to the selected map and relocate the user into the selected map.

[0284] FIG. 25B illustrates an example relocation using an unselected map into a stored map. For example, the AR system may automatically scan for PCFs as the user moves into a new environment, such as entering another room in the user's home. The user may not reject or otherwise select a map for locating therein. The AR system may actively or passively discover PCFs by active or passive collection of data in the user's environment. The AR system may then match those discovered PCFs to PCFs in a stored or otherwise known map and relocate the user into the stored or otherwise known map.

[0285] FIG. 25C illustrates an example relocation using a selected map and a failure to relocate into the selected map. For example, a user may select or name a map such as "Home" with the intent to locate in the selected map. The AR system may actively or passively find a PCF by active or passive collection of data in the user's environment. The AR system may not find a match between the collected PCF and the PCF in the stored or selected map. The AR system may then save the PCF as a second map using the same or a different name, or as an unconnected part of the selected or stored map. In some examples, further data collection by the AR system may result in a subsequent successful relocation into the selected map, as illustrated in FIG. 25D.

[0286] In some implementations, the AR system may attempt to relocate the user into multiple maps simultaneously. Thus, even if the user selects a map (e.g., "Home") in the relocation phase (or if the user does not select a map), the AR system may instead identify content associated with a different room (e.g., "Office") due to the multiple room relocation attempt that may be associated with the user's current location. The AR system may then inform the user that they are being relocated into the other room (e.g., "Welcome to your office").

[0287] FIG. 25D illustrates an example successful relocation into the selected map after a failed relocation. For example, the AR system may save, generate, or otherwise interact with a map at approximately the same location of a different map. The AR system may, in some examples, connect two maps, effectively extending the map to include data from both maps. The AR system may locate in a first map and then connect the maps as a result of determining the presence of a PCF from a second map in the user's environment. The AR system may look for a PCF in the located map to merge the two maps, for example, if the AR system was not able to locate in the selected map initially. In such an instance, the AR system may then save the created map as a version of the selected map for later location or for extension of the selected map. For example, as illustrated in FIG. 25C, the user may select a "home" map. The AR system may be unable to relocate the user into the previously saved "home" map. As a result, the AR system may save the newly created map as a version of "home." The AR system may subsequently connect the original version of "home" with the newly created version of "home" after more of the user's environment has been scanned or PCFs from the original version of "home" have been found in the user's environment.

[0288] Advantageously, merging maps can enable an AR system to locate within a large map or environment. For example, an AR system may find a PCF from a named map and tag the named map with the found PCF, such that the system cannot relocate it into the named map, but can still build on the named map without creating a new location.

[0289] FIG. 25E illustrates another example of successful relocation. For example, a user may select a map for relocation. During relocation, when scanning the user's environment, the AR system may find a PCF for a second map. The AR system may then connect the maps. The AR system may look for a PCF in the located map to connect the two maps, for example, if the AR system was not able to locate in the initially selected map (as illustrated in FIG. 25D) or if the AR system finds a PCF associated with the second map while scanning a different selected map.

[0290] 25F illustrates an example relocation using a selected map and an added PCF. For example, during a relocation process as described in FIGS. 23 and 24, the AR system may identify one or more PCFs that have not been previously stored in the selected map. In some embodiments, the AR system may add or store the one or more identified PCFs in the selected map.

[0291] FIG. 25G illustrates an example map creation using a new PCF. For example, if the user decides to skip scanning or does not select a map for relocation, the AR system may passively determine the PCFs in the user's environment. In some embodiments, the AR system may store the found PCFs in an unnamed or untagged map. As illustrated in FIG. 25H, the unnamed or untagged map may be merged or connected with a named map during a subsequent scan of the user's environment. 2. Interaction with externally stored maps

[0292] In some examples, a user may wish to interact with a map that the user did not create. For example, the user may be invited or seek to participate in an externally stored map created by a third party or other user of the AR system. In another example, the user may wish to interact with a publicly available map, such as a map in a public or semi-public environment. In some examples, one or more users may locate in the externally stored map. Once one or more users locate in the externally stored map, the users may be able to interact with virtual elements of the map. For example, changes to virtual objects by one user may be perceptible by another user located in the map. For example, two users may be able to move virtual chess pieces in a shared virtual chess game. In another example, two users in a chat session may be able to show or share virtual content in one user's environment that is visible to other users in the shared environment. FIG. 26A shows a flowchart of an example process 3300 for locating into a shared map. 26B and 26C illustrate example graphical user interfaces for user location into a shared map. Advantageously, the process described below can enable a user to access AR content associated with a shared map or space that the user did not map or create.

[0293] Referring to FIG. 26A, in block 3302, the AR system may receive, identify, transmit, or generate an invitation to join a shared session or to locate in an externally stored map. For example, a user may be invited by a second user to join a shared session. A shared session may include a situation in which multiple users interact with virtual content that may be shared by two or more of the multiple users or otherwise associated with them. In some examples, a user or users may be prompted or otherwise decide to locate in the map. Advantageously, locating in the shared map may allow a user to better interact in the shared session through content that may be associated with the map, access information, or be associated with the map. The map may be a map stored in an external storage system or cloud. The invitation may include a prompt to join or locate in the map. For example, one or more users may utilize a single real-world space. A first user may invite a second user to access a map of a real-world space that both users are currently occupying so that the users may access virtual content associated with the map. In another example, a user may receive an invitation by the AR system to locate in a map recognized by the AR system. For example, a user may enter a public space, such as a movie theater or art museum. The AR system may recognize the public space as matching a map in cloud storage. The AR system may then invite or prompt the user to locate, join, or sync in the map. In some examples, the AR system may receive or transmit more than one map invitation. For example, if more than one user is mapping a space and inviting the user to locate in a map, the AR system may transmit more than one invitation or option of a map to locate in.

[0294] In block 3304, the AR system may determine whether the user is accepting the invitation to join the map from block 3302. For example, the user may provide an input to the AR system to be associated with the map invitation. The input may include an acceptance, a denial, or other response to the map invitation. If the user provides an acceptance, the AR system may proceed to determining user authorization in block 3306 or to scanning the environment in block 3308. If the user provides a denial, the AR system may proceed to entering the landscape in block 3316.

[0295] In addition to or as an alternative to blocks 3302 or 3304, the user may proceed to block 3306 and scan the environment based on user input or conditions recognized by the AR system. For example, the user may provide input to the AR system to access a map. The user input may include an indication to improve a map, such as a personal world map or other map, by, for example, scanning the user's environment and adding information shown on the map. In another example, the user input may include an indication to locate within the map. Additionally or alternatively, the AR system may initiate scanning of the user's environment for location within the map without direct user input. For example, the AR system may identify that the user is within a geographic location that contains a map. The AR system may initiate mapping or scanning of the user's environment and attempt to locate within the map without waiting for user input.

[0296] In block 3306, the AR system may determine whether the user has authorization to access the map. For example, as discussed above with reference to FIG. 24, the map may be stored with a set of permissions. The permissions may help determine whether the user can access the map. In the case of a map, the permissions may include whether the user is an authorized user to access and / or update the map. The user's authorization state may include whether the currently authorized user or other authorization source has authorized the user to access or edit the map. In some examples, the map may be a map that is stored externally on the user device but is personal to the user such that only the user who created the map may access it. In another example, the map may be public such that all users with access to the cloud storage in which the map is stored may access the map. In some examples, there may be a hierarchy or different types of map permissions such that some users may have access permissions and some users may have edit permissions. If it is determined that the user is authorized to access or edit the map, the AR system may proceed to block 3308 and scan the user's environment. If it is determined that the user does not have the required authorization to access or edit the map, the AR system may proceed to block 3316 and enter the landscape without loading or accessing the map. In some examples, the user may have access to a map but not have permission to edit it. In such a case, the AR system may allow the user to traverse the environment but prevent the user from saving any changes to the map.

[0297] In block 3308, the AR system may map and / or scan the user's environment. The mapping or scanning process may include a scanning process as described with reference to Figures 28-34C. For example, the AR system may guide the user through the environment to find one or more PCFs in the user's environment. The AR system may guide the user differently through the environment based on the user's identified objective when scanning the space. For example, if the user indicates that they want to update the map, the AR system may preferentially guide the user through areas of the user's environment that have little information (e.g., by the scanning process described with reference to Figures 28-34C) to help add information to the map that is not already in the map. In another example, if the user indicates that they are located in the map or are synchronized into the map, the AR system may preferentially guide the user through areas of the environment that have more information to help locate the user. The mapping or scanning process at block 3308 may include steps for recognizing the map, for example, by matching one or more PCFs found in the user's environment with one or more PCFs in an externally stored or saved map.

[0298] In block 3310, the AR system may determine whether the user has been located in the map. For example, the AR system may determine whether sufficient PCFs have been found or generated based on the collected data to identify the user's space or map data associated with the user's space. In some embodiments, localization may be defined by the user finding one or more PCFs (also referred to as shared PCFs) associated with an externally stored map. However, other types of localization may also be possible. In some embodiments, the user may find a single shared PCF and successfully locate in the shared map. In another embodiment, the user may find multiple PCFs to successfully locate in the map. The success of the localization may be determined by the user, the AR system, or a specific application associated with the map data. For example, the application may require a more precise environmental synchronization of the physical world data and the virtual data. The AR system may therefore increase the number of PCFs the user is required to find to help ensure better or more accurate localization. If the AR system determines that localization has occurred, the AR system may load map data associated with the localized map in block 3312. If the AR system determines that localization has not occurred, the AR system may determine whether to continue scanning in block 3314.

[0299] Additionally or alternatively, if the AR system determines that localization has not occurred by one or more of the users who may be attempting to locate in the same map, the AR system may load map data associated with the map. However, the map data may not be aligned across users due to the failure of localization. In some examples, rough alignment of map data for multiple users may be accomplished through user-manual or semi-manual alignment of a common frame of reference through manipulation of content associated with the map. For example, the shared session may include handles or other content for users to manipulate to help achieve content alignment. The rough alignment mechanism may be applicable to users who fail to locate in the shared real-world space and / or for users in remote real-world spaces. In some examples, the rough alignment mechanism may be used to correct for errors or perceived errors in localization when the AR system determines that localization has occurred.

[0300] In block 3314, the AR system may determine whether to continue scanning. For example, if localization has not occurred, the AR system may need more data to generate a new map or to find one or more PCFs in the user's environment. However, the AR system may utilize a timeout, user input, or other termination criteria to stop the scanning process. The AR system may then prompt the user for input regarding whether to continue scanning the user's environment (e.g., by the scanning process in block 3308). If the AR system determines that the scanning should continue, for example due to user input or other criteria, the AR system may continue scanning in block 3308. If the AR system determines that the scanning should not continue, the AR system may enter the landscape in block 3308.

[0301] Additionally or alternatively, the AR system can stop process 3300 at any point and enter the landscape at block 3316. For example, a user can input an end or skip command during the scanning process at block 3308. The AR system can then abort the scanning process or enter the landscape.

[0302] In block 3318, the AR system may inform the user of the status of the location. For example, if the user is successfully located, the AR system may inform the user who was located in the map or other users (such as those currently located or attempting to locate in the same map) of the successful location. In another example, if the user is not successfully located, the AR system may inform the user who attempted to locate in the map or other users (such as those currently located or attempting to locate in the same map) of the failure of the location.

[0303] FIG. 26B illustrates an exemplary graphical user interface during location 2500 in a public or world map. For example, in an invitation block 2510, the AR system may receive or send an invitation for the user to join a shared map. The shared map may be a map accessible to other users. If the user accepts the invitation in block 2518, the AR system may then determine whether the AR system is located in the shared map. If the AR system is not located in the shared map, the AR system may prompt the user to begin a scanning process of the environment in block 2512. If the AR system is located in the shared map, the AR system may prompt the user to begin a scanning process of the user's environment in block 2512. The AR system may then scan the user's environment in block 2514, for example, by guiding the user through a gamified scanning process. The AR system may then check whether the location is successful in block 2516. If the location is successful, the AR system may restore the content in block 2520. If localization is not successful, the AR system may prompt the user to continue scanning the environment or generate a new map. Once the AR system determines that localization is complete or other termination criteria have been met, the AR system may enter the AR landscape in block 2522. For example, the AR system may hand off to an application that may allow the user to access content associated with the shared map. The application may allow the user to collaboratively experience augmented reality content with other users.

[0304] FIG. 26C illustrates an exemplary graphical user interface during localization 2501 in a shared private map. For example, in an invitation block 2530, a user may send or receive an invitation for the user to join a shared map. The shared map may be a map accessible to other users. If the user accepts the invitation, the AR system may receive a marker associated with the shared map in block 2532. The marker may include one or more PCFs associated with the shared map. The AR system may prompt the user to enable permission to scan the user's environment in block 2534. In response to receiving the user permission, the AR system may scan the user's environment in block 2536. Once the AR system recognizes the marker in the user's environment, the AR system may indicate successful completion of the localization in block 2538, for example, by visual, audible, or tactile feedback. Once the AR system determines that the localization is complete or other termination criteria have been met, the AR system may enter the AR landscape in block 2540. For example, the AR system may hand off to an application that may enable the user to access content associated with a shared map. The application may enable the user to collaboratively experience augmented reality content with other users. 3. Map curation

[0305] In some examples, the AR system may utilize a scanning process as described with reference to FIGS. 28-34C to improve map quality. For example, an already generated map may lack complete information, such as mesh data or PCF data. The AR system may use a scanning process to guide a user to areas of the map that lack high quality data to improve map quality by mapping those areas. Improving map quality may be referred to as map curation. Such map curation may be performed by a user who previously built or generated that or another map. For example, a software developer and / or user may enter a map curation mode to improve map quality. In other examples, multiple users may curate a map and data may be merged to improve map quality. Related U.S. Patent Application No. 62 / 966,267, filed Jan. 27, 2020, entitled “AUGMENTED REALITY MAP CURATION,” provides further details regarding map curation processes that may be implemented in the embodiments discussed herein and incorporated by reference in its entirety for all purposes.

[0306] In block 3502, the AR system may determine a map to curate. For example, the AR system may automatically determine a map associated with the user's detected geographic location. In another example, the AR system may display a prompt for the user to select an existing map to curate (and / or confirm a map automatically selected by the AR system). If the user selects a map, the AR system may determine in block 3504 whether the map requires further curation. In some examples, if the user selects a map, the AR system may proceed to block 3506 and scan the environment without determining in block 3504 whether the selected map meets the map quality threshold.

[0307] In block 3504, the AR system may determine whether the selected map meets a map quality threshold. For example, the AR system may determine whether the map quality is sufficient such that the map requires refinement or does not require refinement. If the map quality does not meet the threshold quality, the AR system may proceed to block 3506 and scan the user's environment. In some examples, the map quality threshold may include a threshold percentage of map data available for the environment, a number of stored PCFs in the environment, the like, or some combination thereof.

[0308] In addition to or as an alternative to block 3504, the AR system may allow the user to curate the map regardless of quality. For example, the user may want to expand the size of the map into a new area. In some implementations, the map may be of high quality but not as large or complete as the user desires. The AR system may therefore continue to scan the environment and curate the map until the user desires to end the curation process.

[0309] In block 3506, the AR system may map and / or scan the user's environment. The mapping or scanning process may include a scanning process as described with reference to Figures 28-34C. For example, the AR system may guide the user through the environment and find one or more PCFs in the user's environment. For example, if the user indicates that they want to update the map, the AR system may guide the user preferentially to areas of the user's environment that have little information (e.g., by a scanning process described with reference to Figures 28-34C), which may help add information to the shared map that is not already in the shared map. In another example, the AR system may guide the user throughout the user's space according to one or more criteria as described with reference to Figures 28-34C. The mapping or scanning process in block 3308 may include a step for recognizing the map, for example, by matching one or more PCFs found in the user's environment with one or more PCFs in the shared or stored map.

[0310] In some examples, during the scanning process at block 3506, the AR system may guide the user based on the identified map quality. For example, areas of the map may have a higher map quality than other areas of the map. The AR system may guide the user to areas of the map with lower quality to collect data in those areas and improve the map quality therein. Map quality may be defined by any number of criteria, including, but not limited to, the amount of data collected in that area of ​​the map. In some examples, as the user moves around the space, the AR system may notify the user of the current map quality to let the user know how much map curation is needed or has been completed. The notification may involve graphical, tactile, audible, or other feedback to the user of the current map quality, as described in more detail in the above-referenced related application. In one example, the notification may include a progress bar associated with a normalized map quality index or value. In another example, the notification may include a color coding of the map.

[0311] In block 3508, the AR system may determine whether to continue the scan. For example, the AR system may identify whether the map quality is sufficient to stop the scan or whether the scan should continue. If the map quality is still low, the AR system may need more data and guide the user to help find one or more PCFs in the user's environment. However, the AR system may utilize a timeout, user input, or other termination criteria to stop the scanning process. The AR system may then prompt the user for input regarding whether to continue scanning the user's environment (e.g., by the scanning process in block 3506). If the AR system determines that the scan should continue, for example, due to user input or other criteria, the AR system may continue the scan in block 3506. If the AR system determines that the scan should not continue, the AR system may end the curation in block 3512. In some implementations, the only termination criteria may be user input, such that the user may continue mapping or scanning the space until they decide to end. Advantageously, this allows users to get as much detail as they want in the map they are curating, without the AR system having to determine termination criteria for the mapping or scanning process. M. Environmental Scanning Process

[0312] As referenced above, when relocating or locating a user in a map or creating a map, the AR system may need to scan the user's environment and recognize PCFs or other markers in the user's environment. In some embodiments, the AR system may passively collect data and scan the user's environment as the user naturally moves around the environment. However, in some embodiments, the AR system may actively scan the environment by prompting the user to walk around and observe the environment. The AR system may prompt the user to perform this scan using waypoints or displayed graphics throughout the user's environment. In addition, the AR system may display an animation associated with successful data collection. FIG. 26 illustrates an example scanning process 2600 that may be implemented by the AR system to scan the user's environment. For example, the scanning process may include a waypoint determination block 2610, a user instruction block 2612, a data collection block 2614, and an end criteria determination block 2616.

[0313] In waypoint determination block 2610, the AR system can determine locations for placing one or more graphics (e.g., visualizations) in the user's environment. For example, the AR system can determine the locations of the graphics according to a process as described with reference to FIG. 29. In some implementations, the AR system can determine locations for one or more graphics to be distributed in the user's environment. The AR system may select locations based on any number of criteria, including, but not limited to, the presence or absence of collected data or meshes in an area of ​​the user's environment. Additionally or alternatively, the AR system may select locations based on distance from the user or other selected waypoint locations. Advantageously, the AR system may select locations further away from the user or other waypoint locations to encourage the user to explore more of the environment.

[0314] The AR system may determine the waypoint locations by any suitable means for analyzing data collected about the user's environment. For example, the AR system may collect data about the user's environment using one or more outward-facing cameras or sensors. The AR system may analyze the data according to one or more criteria in determining the potential waypoint locations. For example, the AR system may determine 32 potential waypoint locations based on data from the outward-facing cameras. The AR system may order the 32 potential locations based on distance from the user and other potential or determined waypoint locations. The AR system may then select a subset of the ordered locations to display to the user.

[0315] In user instruction block 2612, the AR system can instruct the user to explore the environment using one or more graphics at the waypoint locations determined in block 2610. For example, the AR system may display one or more graphics at one or more locations determined in block 2610. The AR system may display the graphics sequentially or at one time, display parts of the graphics sequentially or at one time, or some combination thereof. The AR system may guide the user to one or more of the waypoint locations, for example, by displaying a guidance animation (see, for example, the guidance animation process described with reference to FIG. 30 below). The AR system may use guidance animation to guide the user to multiple waypoint locations in the user's environment. For example, if there are six waypoint locations, the AR system can guide the user to each of the six waypoint locations sequentially. In another example, the AR system can determine a single waypoint location at a time and guide the user to each of the selected waypoint locations. The AR system may then determine a subsequent waypoint location based on the already completed waypoint location and guide the user to the subsequent waypoint location.

[0316] In data collection block 2614, the AR system can collect data associated with the user's environment. For example, the AR system can collect data using one or more sensors associated with the AR system, such as one or more outward-facing cameras on the head-mounted display of the AR system. The AR system may collect data before, during, or after any of the steps of the scanning process 2600. For example, the AR system may collect data during waypoint location determination block 2610 or user instruction block 2612. Additionally or alternatively, the AR system may collect data during interaction with a graphic at the waypoint location determined in block 2610. For example, when the user interacts with the graphic, for example, by coming within a threshold distance of the waypoint location, the AR system may collect data about the user's environment near the waypoint location. Advantageously, the AR system can collect data while guiding the user to the waypoint location in block 2612 and collect a greater amount of detailed data while interacting with the graphic at the waypoint location determined in block 2610 in block 2614. Thus, the AR system may collect more detailed data in areas of the user's environment that may be identified as locations where data needs to be collected.

[0317] In a termination criteria determination block 2616, the AR system may determine whether more data needs to be collected or whether the scanning process 2600 should be terminated. The AR system may determine to terminate the scanning process by analyzing whether one or more scan termination criteria have been met. For example, the AR system may collect data at waypoint locations in block 2614. The AR system may determine whether the user has collected data at a threshold number of waypoint locations, identified a minimum amount of PCFs, and / or met other criteria for determining an adequate amount of mesh in the user's environment. In some embodiments, the termination criteria may match the criteria for the localization process to occur. If the AR system determines that the user has collected data at a threshold number of waypoint locations, the AR system may terminate the scanning process. If the AR system determines that the user has not collected data at a threshold number of waypoint locations, the AR system may continue or resume the scanning process 2600 in block 2610. In another example, the AR system may receive an input from the user to stop or skip the scanning process at any point during the process 2600. If the user stops the scanning process 2600, the AR system may terminate the scanning process. If the AR system does not stop the scanning process 2600, the AR system may continue the scanning process. In another embodiment, the AR system may determine whether a threshold amount of data has been collected within a threshold time frame. If the AR system determines that sufficient data has not been collected within the threshold time frame, the AR system may terminate the process 2600. If the AR system determines that the threshold time frame has not elapsed and more data should be collected, the AR system may continue the scanning process. N. Example Waypoint Generation

[0318] As discussed above, aspects of the scanning process as described with reference to Figure 28 may include determining waypoints or graphic locations within the user's environment. Figure 27 illustrates an example graphics spawn process 2700. For example, the graphics spawn process 2700 may include a ray casting block 2712, a ray casting type block 2714, an order determination block 2716, a selection block 2718, a display block 2720, and an exit criteria block 2722.

[0319] In ray casting block 2712, the AR system can send ray castings around the user. In various embodiments, ray casting techniques may include casting a thin digital ray of light with little to no substantial lateral width, or a ray with substantial lateral width (e.g., a cone or frustum) into the digital version of the user's environment. For example, the ray may be a digital line that extends away from the user and may intersect with one or more real or virtual objects in the digital copy of the real world. In some embodiments, a real object may be represented by a mesh in the digital copy of the real world. In some embodiments, a virtual object may be represented by a renderable 3D asset (e.g., a 3D model, e.g., with a mesh and corresponding mesh renderer data). Ray casting may be used to quickly obtain limited data collection in a particular direction into the user's environment. For example, the limited data collection may include the presence of a surface or content in the direction of the mesh, whether a mesh exists in the direction of the ray casting, the distance to any found content or mesh, the like, or some combination thereof. The ray castings may extend horizontally, vertically, and / or radially from an origin associated with the user. For example, the ray castings may extend horizontally from the user's head such that one or more ray castings are perpendicular to the head mounted display worn by the user. The AR system may transmit any number of ray castings. For example, the AR system may transmit 32 ray castings centered on the user's head. However, other numbers of ray castings are also possible. For example, the number of ray castings required for a particular environment may be dynamically determined by the AR system based on, for example, analysis of collected scan data and / or map data previously associated with the environment.

[0320] In ray casting type block 2714, the AR system can assign, determine, or categorize a type for one or more ray castings from block 2712. For example, the ray casting may include information associated with the user's environment, a mesh, or other AR content. The AR system may categorize one or more of the ray castings according to information obtained from the individual ray castings. For example, the AR system may categorize each of the ray castings as a hit, non-hit, or unobserved type of ray casting. In some embodiments, the system may divide the environment into a series of cells by dividing the space by a grid. Hit ray castings can include ray castings that intersect with content within a threshold distance from the origin of the ray casting, or intersect with cells that contain content such as meshes or PCFs. Non-hit ray castings can include ray castings that intersect with cells that contain no content, or intersect with content such as meshes or PCFs, within a threshold distance from the origin of the ray casting. Unobserved ray casting may include ray casting where it is unknown whether content exists within a threshold distance from the origin of the ray casting. Unobserved ray casting may indicate that a user is not likely to be in a location associated with the ray casting for collecting data. The AR system may then select the unobserved ray casting type and / or other ray casting types for further analysis when determining a location for displaying graphics to prompt user movement that would allow for acquisition of further mesh data.

[0321] In the order determination block 2616, the AR system can order the ray castings according to one or more criteria associated with the ray castings. For example, the AR system can order the unobserved ray castings according to the distance from the furthest point to the closest point away from the origin of the ray casting. Advantageously, ordering the ray castings according to distance can allow the AR system to determine or preferentially select a location for displaying the graphics further away from the user. Displaying the graphics further away from the user can allow the AR system to guide the user to explore more of the environment than if the location was closer to the user. In some implementations, other ordering criteria may also be used.

[0322] In a selection block 2718, the AR system may determine a location in the user's environment or display a graphic at a location. The AR system may select a location for displaying the graphic that is approximately centered or at another location within the user's FOV based on ray casting. In another example, the AR system may display the graphic at a location outside the user's FOV. In some examples, the AR system may analyze data from the ray casting and determine a location in the user's environment that may use more data collection. For example, the AR system may determine that a location is not meshed. The AR system may preferentially select locations that are not meshed. Additionally or alternatively, the AR system may select a location within or outside a threshold distance of the user. For example, the AR system may determine that an initial waypoint location is at a threshold distance from the user. In some examples, the threshold distance can be 1 meter, 2 meters, 5 meters, or other distance. In some examples, the AR system may use more than one criterion in determining a spawn location for the waypoint graphic. For example, the AR system may determine locations or meshed areas that are greater than a threshold distance (e.g., 2 meters) from the user and greater than a threshold distance (e.g., 2 meters) from another selected location (e.g., a previous waypoint). Advantageously, placing an initial waypoint location at a threshold distance from the user can encourage the user to move around the environment and interact with the graphics.

[0323] In some implementations, the AR system may select a waypoint location according to the number of graphics displayed to the user. For example, if the AR system has not displayed any graphics to the user, the scanning process may begin first, or the scanning process may resume, and the AR system may display the graphics within the user's FOV. Additionally or alternatively, the AR system may display the graphics in the best location according to the order determined in the order determination block 2716. The AR system may display the graphics in the best location if the AR system has already displayed the graphics to the user.

[0324] At a display graphics block 2720, the AR system may display one or more graphics at one or more of the determined waypoint locations. For example, the AR system may display a graphic at a single waypoint location. Once data has been collected at the waypoint location, the AR system may determine a second waypoint location and repeat process 2700 from block 2712. Additionally or alternatively, the AR system may display a graphic at more than one location.

[0325] In an exit criteria block 2722, the AR system may determine whether more graphics need to be displayed or whether the graphics display process 2700 should be terminated. The AR system may determine to terminate the process 2700 by analyzing whether one or more exit criteria are met. For example, the AR system may determine waypoint locations and collect data at the waypoint locations. The AR system may determine whether the user has collected data at a threshold number of waypoint locations. If the AR system determines that the user has collected data at a threshold number of waypoint locations, the AR system may terminate the scanning process. If the AR system determines that the user has not collected data at a threshold number of waypoint locations, the AR system may continue or resume the process 2700 and display more graphics. In another example, the AR system may receive an input from the user to stop or skip the scanning process at any point during the process 2700. If the user has stopped the scanning process, the AR system may terminate the process 2700. O. Example Waypoint Guidance

[0326] To prompt the user to interact with a displayed graphic or waypoint spawned or generated by the AR system, the AR system may guide the user to the spawned waypoint in the user's environment. FIG. 30 illustrates an example guidance process 2800 for guiding a user to a waypoint or multiple waypoints in the user's environment using an indicator ring graphic. An example indicator ring stage is illustrated in FIGS. 31A and 31B. The indicator ring graphic is one example of a graphic that may be used to identify a waypoint and prompt the user to move toward the waypoint. In other implementations, any other graphic and / or audio guidance may be provided to the user. For example, the process 2800 may include a field of view determination block 2810, an out-of-FOV guidance block 2812, an in-FOV guidance block 2814, an update block 2816, an activation determination block 2818, and an end block 2820.

[0327] In the field of view determination block 2810, the AR system may determine whether the waypoint is within the user's field of view (FOV). The waypoint may be considered to be within the user's FOV if some or all of the graphics are within the user's FOV. For example, the AR system may determine that the waypoint is within the user's FOV if a threshold portion of the waypoint is within the user's FOV. The threshold portion may include 10 percent, 25 percent, 50 percent, or another percentage. Additionally or alternatively, the AR system may determine whether the waypoint is occluded within the user's FOV (e.g., behind an object in the user's environment). If the AR system determines that the waypoint is occluded within the user's FOV, the AR system may treat the waypoint as being outside the user's FOV. Additionally or alternatively, the AR system may relocate the occluded waypoint to an unoccluded location within the user's FOV and guide the user to the new location using an animation, etc., that moves the waypoint to an unoccluded location. If the AR system determines that the waypoint is within the user's FOV, the AR system may navigate the user to the waypoint using different graphics than if the AR system determines that the waypoint is outside the user's FOV. For example, if the AR system determines that the waypoint is outside the user's FOV, the AR system may navigate the user to the waypoint using partial rings as illustrated in FIG. 29A in block 2812. If the AR system determines that the waypoint is within the user's FOV, the AR system may navigate the user to the waypoint using full rings as illustrated in FIG. 29B in block 2814.

[0328] In out-of-FOV guidance block 2812, the AR system may guide the user to a waypoint outside the user's FOV as determined in block 2810. For example, the AR system may visually, audibly, or otherwise indicate a direction for the user to travel to and find the displayed waypoint. For example, the AR system may display a partial ring graphic 2910 as illustrated in FIG. 31A. The partial ring graphic 2910 may have a curvature. The curvature may have a peak. The peak may indicate a direction to travel, look, or scan to find the waypoint, such as up, down, left, or right. Additionally or alternatively, the partial ring graphic 2910 may include several bubbles or particles arranged in an arc. The arc may have an increased density of bubbles or particles at locations along the arc to indicate a direction to travel, look, or scan to find the waypoint.

[0329] Returning to FIG. 28 , in an within FOV guidance block 2814, the AR system may guide the user to a waypoint within the user's FOV as determined in block 2810. For example, the AR system may visually, audibly, or otherwise indicate directions for the user to navigate to and find the displayed waypoint. For example, the AR system may display a ring graphic 2914 around the displayed waypoint 2916, as illustrated in FIG. 31B . The ring 2914 may have a diameter greater than or equal to the width of the waypoint graphic 2916. For example, the ring 2914 may have a diameter twice the width of the waypoint graphic 2916. In some examples, the AR system may display the ring 2914 at a size that depends on the user's distance from the waypoint location. In some examples, the AR system may display the ring 2914 at a fixed size until the user is within a threshold distance of the waypoint location. In some embodiments, the AR system may display a ring 2914 around the waypoint 2916 while the waypoint 2916 is within the user's FOV.

[0330] Returning to FIG. 28, in update block 2816, the AR system may update a guidance graphic, such as a ring. For example, the AR system may determine a user's proximity to a waypoint location. The AR system may change the appearance of the guidance graphic based on the user's proximity. For example, the AR system may change the size of the ring to gradually approach the waypoint or displayed graphic. Additionally or alternatively, the AR system may play audio or generate tactile feedback to indicate the proximity to the waypoint.

[0331] In an activation decision block 2818, the AR system may determine whether the target waypoint is activated. For example, the AR system may determine that the target waypoint is activated by the user reaching a threshold proximity to the waypoint location. When the waypoint is activated, the AR system may display or show an acknowledgment to the user, such as described with reference to FIGS. 32-34C. For example, the AR system may display an animation, an audible acknowledgment, and / or a tactile acknowledgment, such as described with reference to FIG. 31. When the waypoint is activated or the guidance process is otherwise stopped, the AR system may determine a new target waypoint and start the process 2800 with the new target waypoint. In another embodiment, the AR system may end the guidance process 2800. When the waypoint is not activated, the AR system may continue the guidance process 2800 with the current waypoint, for example, by starting the process again at block 2810. P. Exemplary Waypoint Activation Process

[0332] Once the AR system successfully guides the user to the waypoint location (e.g., by the guidance process described with reference to FIG. 29), the AR system may trigger an activation process associated with the waypoint activation. The activation process may include storing information associated with the user's environment near the waypoint and indicating successful completion of the waypoint to the user through audiovisual or tactile feedback. FIG. 32 illustrates an example waypoint activation process 3000. The waypoint activation process 3000 may include an interaction determination block 3010, an animation block 3012, and a removal block 3014.

[0333] In an interaction determination block 3010, the AR system can determine a user interaction with the waypoint. In some examples, the interaction can include a user line of sight or proximity to the waypoint. In another example, the interaction can include a selection of the waypoint by actuation of a user input device or other user input. If the interaction exceeds a threshold interaction, the AR system can animate the waypoint in block 3012.

[0334] In animation block 3012, the AR system can display an indication of successful interaction or activation of the waypoint. For example, as discussed with reference to FIGS. 33-34C, the AR system can display an animation or sequence of graphics, or generate audible or tactile feedback associated with activating or interacting with the waypoint.

[0335] In a remove block 3014, the AR system may remove the waypoint from a list of waypoints to spawn or display, which may have been generated as part of a waypoint spawning process as described with reference to Figure 29. In embodiments where the AR system displays a single waypoint at a time or determines a single waypoint location at a time, there may not be a list of waypoints and the AR system may dynamically determine the next waypoint to display to the user. Additionally or alternatively, the AR system may save the current waypoint location as an already visited location.

[0336] FIG. 33 illustrates an exemplary waypoint animation process 3100 that may be implemented to indicate activation of a waypoint. For example, the AR system may generate a bubble in block 3110, activate the bubble in block 3112 by determining a collision trajectory, animating the bubble (e.g., to progress to a collision location in the user's environment) or animating a collision with an object or mesh in the user's environment in block 3116, and remove the bubble from display in block 3114. In other implementations, any other graphic and / or audio prompts may be provided to the user as an indication of waypoint activation. In one exemplary implementation, when the user activates their first waypoint, the AR system informs them to "look around for particles of light" (e.g., bubbles or some other visual effect). By looking at them, those particular bubbles will activate. This has the beneficial side effect of causing the user to look further around their environment, thereby increasing their map quality. Thus, the use of such bubbles (and / or other visual effects) not only provides entertaining visual / auditory rewards for the user, but may also provide practical benefits.

[0337] In block 3110, the AR system may generate one or more bubbles or other graphics. The bubbles or other graphics may be generated from the waypoint. For example, as illustrated in FIG. 34A, the waypoint 3210 may include a graphic. Once the waypoint is activated as described above, the AR system may pop or explode the waypoint into multiple smaller bubbles 3220 as illustrated in FIG. 34B. The bubbles may be displayed at different points within the user's FOV. For example, the bubbles may be displayed randomly within the user's FOV. In another embodiment, the bubbles may be displayed such that the bubbles 3220 appear as if the waypoint bubble 3210 had popped. Advantageously, locating the bubbles at distributed points within the user's FOV may encourage the user to look more around the environment near the waypoint location than they would without the bubble animation. The bubbles may be static or dynamic. In addition to or as an alternative to the bubbles 3220, the AR system may display an indication 3214 of activation. The indication 3214 can include a checkmark or other indication of successful activation. Additionally or alternatively, the AR system can play audio or generate tactile feedback associated with the indication.

[0338] In some examples, the AR system may prompt the user to look at a bubble (or other graphic surrounding or associated with the waypoint). When the AR system detects that the user is interacting with the bubble or other graphic, the AR system may proceed to block 3116 and activate the bubble. For example, the AR system may determine that the user is directing their gaze toward a bubble. The AR system may then activate that bubble.

[0339] As discussed further below, in some implementations, when a bubble is activated (e.g., by the user looking at it), the AR system calculates a mesh that is proximate to that location (and generally away from the user). The bubble then flies directly to that proximate mesh location and pops (e.g., triggering another animation). If there are no neighboring meshes for the bubble, it may simply fly away from the user and fade out without the larger popping animation. In some implementations, some or all bubbles that are activated may be moved in a random trajectory.

[0340] In block 3116, the AR system may determine a trajectory for one or more bubbles 3210. For example, the AR system may determine a direction of travel of one or more bubbles 3210 in the user's environment. In some implementations, the direction of travel of the bubbles may trace a path from an origin, e.g., a waypoint, to content in the user's environment, such as a mesh. For example, at least some of the bubbles may travel to an area of ​​the mesh that was generated and / or updated through user activation of a waypoint. The AR system may select a trajectory for the bubble 3210 toward the mesh or other content in the user's environment. In block 3118, the AR system may animate the bubble 3210 to travel along the determined trajectory. In block 3120, the AR system may animate a collision between the bubble 3210 and a mesh 3216 in the user's environment. For example, the AR system may display the collision 3218 and a portion of the mesh 3216 around which the collision 3218 occurred. The AR system may continue colliding the bubble 3210 until a termination criterion is met. For example, the termination criterion may include the display of a threshold portion of the calculated mesh 3216 within the user's environment. In another example, the termination criterion may include a threshold number of bubble collisions displayed. In block 3114, once a bubble collision occurs, the AR system may remove the bubble from display. For example, the AR system may animate the bubble to dissipate by fading it out or moving it out of the user's FOV. As illustrated in FIG. 34C, the result of the animation process 3100 may include a mesh 3216 that may be displayed within the area of ​​the completed or activated waypoint 3214. Q. Additional Examples

[0341] The systems, methods, and devices described herein each have several aspects, no single one of which is solely responsible for its desirable attributes. Without limiting the scope of the present disclosure, some non-limiting features will now be briefly discussed. The following paragraphs describe various exemplary implementations of the devices, systems, and methods described herein.

[0342] Example 1: An augmented reality (AR) system comprising: an AR display configured to present virtual content to a user of the AR system; an outward-facing camera configured to capture one or more images of the user's environment, where a portion of the user's environment visible to the user comprises the user's field of view (FOV); and a hardware processor in communication with the AR display and the outward-facing camera, the hardware processor programmed to: generate a plurality of ray castings from an origin having a location associated with the user's head; analyze the plurality of ray castings; identify ray castings associated with unmapped locations of the environment; update the virtual content to include graphics associated with the unmapped locations; collect data associated with the user's environment; and, in response to collecting sufficient data associated with the unmapped locations, signal to the user that the data has been successfully collected.

[0343] Example 2: The AR system of example 1, wherein the hardware processor is configured to instruct the user to view the graphic.

[0344] Example 3: The AR system of Example 1 or 2, wherein the hardware processor is configured to repeat the updating, indicating, collecting, and signaling regarding additional unmapped locations in response to determining that additional unmapped locations exist within the environment.

[0345] Example 4: An AR system as described in any one of Examples 1-3, wherein the updating, instructing, collecting, and signaling are repeated at least until a minimum threshold of data associated with unmapped locations is obtained.

[0346] Example 5: An AR system described in any one of Examples 1-4, wherein the hardware processor is further configured to determine the presence of at least one persistent coordinate frame (PCF) in the user's environment based on the collected data, and the step of determining whether additional unmapped locations exist in the environment is based on a determination of whether at least one PCF has been identified.

[0347] Example 6: The AR system of any one of Examples 1-5, wherein the hardware processor is further configured to load content associated with the stored map, including the at least one PCF, in response to determining the presence of the at least one PCF.

[0348] Example 7: An AR system described in any one of Examples 1-6, wherein to analyze multiple ray castings, the hardware processor is configured to, for the ray casting, determine the presence of a mesh along the ray casting, calculate the distance of the mesh along the ray casting, and determine a ray casting type comprising a content type, a no content type, or an unobserved type based on the presence of the mesh, the presence of content, and the distance of the mesh.

[0349] Example 8: The AR system of example 7, wherein the content type comprises ray casting with content on a mesh within a maximum distance from at least one of the user and other unobserved locations.

[0350] Example 9: The AR system of example 8, wherein the maximum distance is 2 meters.

[0351] Example 10: The AR system of any one of Examples 7-9, wherein the no-content type comprises ray casting with no content on meshes within a maximum distance.

[0352] Example 11: The AR system of any one of Examples 7-10, wherein the unobserved type comprises ray casting with unknown content within a maximum distance.

[0353] Example 12: The AR system of any one of Examples 1-12, wherein the hardware processor is configured to display graphics in the center of the user's FOV.

[0354] Example 13: The AR system of example 12, wherein the hardware processor is configured to display the graphic in the center of the user's FOV prior to generating the graphic in an unobserved location.

[0355] Example 14: An AR system described in any one of Examples 12-13, comprising data, wherein the hardware processor is configured to instruct the user to observe a graphic in the center of the user's FOV and signal to the user that data has been successfully collected near at least one unobserved location, the collected data being associated with the center of the user's FOV.

[0356] Example 15: An AR system described in any one of Examples 1-14, wherein the hardware processor is configured to display an animation associated with the graphic in the unobserved location to signal to the user that data has been successfully collected in the unobserved location.

[0357] Example 16: The AR system described in Example 15, wherein to display the animation, the AR system is configured to generate at least one bubble associated with the graphic, determine a collision trajectory associated with the at least one bubble, animate the at least one bubble to move along the collision trajectory toward a collision location on a mesh associated with the map of the user's environment, and display a collision between the at least one bubble and the mesh.

[0358] Example 17: An AR system described in any one of Examples 1-16, wherein the hardware processor is configured to display an indicator ring based on an unobserved location relative to the user's position to instruct the user to observe the graphic.

[0359] Example 18: An augmented reality (AR) system configured to present virtual content to a user of the AR system; an AR display configured to capture one or more images of the user's environment; an outward facing camera configured to capture one or more images of the user's environment, where a portion of the user's environment visible to the user comprises the user's field of view (FOV); a hardware processor in communication with the AR display and the outward facing camera, receiving a user selection of a map, comprising a location associated with the user's head, generating a plurality of ray castings from an origin, analyzing the plurality of ray castings, and identifying ray castings associated with unmapped locations of the environment, and generating a ray casting image associated with the unmapped location. and a hardware processor programmed to: update the virtual content to include a graphic; instruct the user to observe the graphic; collect data associated with the user's environment; and in response to collecting sufficient data associated with the unmapped location, signal to the user that the data was successfully collected; and in response to a determination that at least one marker in the collected data corresponds to at least one marker in the map, display virtual reality content associated with the map, or in response to a determination that no marker in the collected data corresponds to a marker in the map, repeat the updating, instructing, collecting, and signaling for another unmapped location.

[0360] Example 19: The AR system of Example 18, wherein at least one marker comprises at least one PCF.

[0361] Example 20: An AR system described in any one of Examples 18-19, wherein to analyze multiple ray castings, the hardware processor is configured to, for the ray casting, determine the presence of a mesh along the ray casting, calculate the distance of the mesh along the ray casting, and determine a ray casting type comprising a content type, a no content type, or an unobserved type based on the presence of the mesh, the presence of content, and the distance of the mesh.

[0362] Example 21: The AR system of Example 20, wherein the content type comprises ray casting with content on a mesh within a maximum distance from at least one of the user and other unobserved locations.

[0363] Example 22: The AR system of example 21, wherein the maximum distance is 2 meters.

[0364] Example 23: An AR system described in any one of Examples 20-22, wherein the no-content type comprises ray casting with no content on meshes within a maximum distance.

[0365] Example 24: An AR system described in any one of Examples 20-23, wherein the unobserved type comprises ray casting with unknown content within a maximum distance.

[0366] Example 25: The AR system of any one of Examples 18-25, wherein the hardware processor is configured to display graphics in the center of the user's FOV.

[0367] Example 26: The AR system of Example 25, wherein the hardware processor is configured to display the graphic in the center of the user's FOV prior to generating the graphic in an unobserved location.

[0368] Example 27: An AR system described in any one of Examples 25-26, comprising data, wherein the hardware processor is configured to instruct the user to observe a graphic in the center of the user's FOV and signal to the user that data has been successfully collected near at least one unobserved location, the collected data being associated with the center of the user's FOV.

[0369] Example 28: An AR system described in any one of Examples 18-27, wherein the hardware processor is configured to display an animation associated with the graphic in the unobserved location to signal to the user that data has been successfully collected in the unobserved location.

[0370] Example 29: The AR system described in Example 28, wherein to display the animation, the AR system is configured to generate at least one bubble associated with the graphic, determine a collision trajectory associated with the at least one bubble, animate the at least one bubble to move along the collision trajectory toward a collision location on a mesh associated with the map of the user's environment, and display a collision between the at least one bubble and the mesh.

[0371] Example 30: An AR system described in any one of Examples 18-29, wherein the hardware processor is configured to display an indicator ring based on an unobserved location relative to the user's position to instruct the user to observe the graphic.

[0372] Example 31: A method of creating or updating an augmented reality map of a user's environment implemented by a computing system having one or more hardware computer processors and one or more non-transitory computer readable storage devices storing software instructions executable by the computing system to implement the computerized method, comprising the steps of obtaining distance data from each of a plurality of directions around the user, the distance data indicating a distance at which a physical object was detected in the respective directions; and creating a ranked list of waypoints associated with the distance data having a distance exceeding a threshold distance, the waypoint associated with the greatest distance being ranked. displaying a graphic positioned with reference to the highest ranked waypoint for viewing on an augmented reality (AR) device worn by the user, the graphic configured to indicate to the user an area of ​​the user's environment for which the augmented reality map is incomplete; detecting user movement enabling collection of data by the AR device associated with the waypoint; determining whether additional waypoints remain in the ranked list; and in response to a determination that additional waypoints remain in the ranked list, performing the displaying, detecting, and determining with the next highest ranked waypoint.

[0373] Example 32: The method described in example 31, wherein the ranked list comprises six waypoints.

[0374] Example 33: The method of example 31 or 32, wherein the indicating, detecting, and determining are repeated until at least a minimum threshold of data is collected.

[0375] Example 34: The method described in any one of Examples 31-33, further comprising a step of determining the presence of at least one persistent coordinate frame (PCF) in the user's environment based on the collected data.

[0376] Example 35: A method according to any one of Examples 31-34, comprising, in response to determining the presence of at least one PCF, loading content associated with a stored map that includes at least one PCF.

[0377] Example 36: The method of any one of Examples 31-35, comprising displaying the graphic in the center of the user's FOV.

[0378] Example 37: The method described in Example 36, wherein the hardware processor is configured to display the graphic in the center of the user's FOV prior to generating the graphic that is positioned with reference to the highest ranked waypoint.

[0379] Example 38: The method described in Example 36 or 37, comprising a step of detecting a user's movement that enables the AR device to collect data associated with the center of the user's FOV.

[0380] Example 39: A method according to any one of Examples 31-38, comprising a step of displaying an animation associated with a graphic that is positioned with reference to the highest ranked waypoint.

[0381] Example 40: The step of displaying an animation includes:

[0382] A method as described in Example 39, comprising the steps of generating at least one bubble associated with a graphic, determining a collision trajectory associated with the at least one bubble, animating the at least one bubble to move along the collision trajectory toward a collision location on a mesh associated with a map of the user's environment, and displaying a collision between the at least one bubble and the mesh.

[0383] Example 41: A method according to any one of Examples 31-40, comprising a step of guiding a user to a waypoint.

[0384] Example 42: The method described in Example 41, wherein the step of guiding the user includes a step of displaying an indicator ring.

[0385] Example 43: An augmented reality (AR) system, comprising: an AR display configured to present virtual content to a user of the AR system; an outwardly facing camera configured to capture one or more images of a user's three-dimensional (3D) environment; a hardware processor in communication with the AR display and the outward facing camera, Receive an invitation to access a shared map, Facilitate user access to shared maps, Inducing the user to observe his / her 3D environment; Collect data that is associated with the user's 3D environment; If data associated with the shared map is found within the user's 3D environment, load the content associated with the shared map; a hardware processor programmed to An AR system equipped with

[0386] Example 44: The hardware processor further comprises: The AR system of Example 43 is configured to determine the presence of at least one persistent coordinate frame (PCF) within the user's environment based on the collected data, and the step of determining whether data associated with the shared map is found within the user's 3D environment is based on a determination of whether at least one PCF associated with the shared map is identified within the collected data.

[0387] Example 45: An AR system described in any one of Examples 43-44, wherein, to load content, the hardware processor is configured to load content associated with a shared map including at least one PCF in response to determining the presence of at least one PCF.

[0388] Example 46: An AR system described in any one of Examples 43-45, wherein the hardware processor is configured to display graphics in the center of the user's FOV.

[0389] Example 47: The AR system of Example 46, wherein the hardware processor is configured to display the graphic in the center of the user's FOV prior to generating the graphic in an unobserved location.

[0390] Example 48: The hardware processor includes: instructing the user to observe a graphic in the center of the user's FOV; signaling to a user that data has been successfully collected near at least one unobserved location; An AR system as described in any one of Examples 43-47, configured so that the collected data comprises data associated with the center of the user's FOV.

[0391] Example 49: An AR system described in any one of Examples 43-48, wherein the hardware processor is configured to display an animation associated with the graphic in the unobserved location to signal to the user that data has been successfully collected in the unobserved location.

[0392] Example 50: To display an animation, the AR system generating at least one bubble associated with the graphic; determining a collision trajectory associated with at least one bubble; animating at least one bubble to move along a collision trajectory toward a collision location on a mesh associated with the map of the user's environment; Show at least one bubble-mesh collision, The AR system of Example 49, configured as follows:

[0393] Example 51: An AR system described in any one of Examples 43-50, comprising a map, the shared map being stored in a location external to the AR display.

[0394] Example 52: An AR system described in any one of Examples 43-51, wherein the shared map comprises a map accessible to multiple AR devices or multiple users.

[0395] Example 53: An AR system described in any one of Examples 43-52, wherein to instruct the user to observe the graphic, the hardware processor is configured to display an indicator ring based on the unobserved location relative to the user's position.

[0396] Example 54: The hardware processor includes: Identify the map to memorize, determining a storage location comprising on-device or off-device storage; determining one or more user permissions associated with the identified map, the one or more user permissions comprising authorization for the one or more users to access the identified map; storing the identified map based on the determined storage location and the determined one or more user permissions; The AR system according to any one of Examples 43-53, configured as follows:

[0397] Example 55: An augmented reality (AR) system, comprising: an AR display configured to present virtual content to a user of the AR system; an outwardly facing camera configured to capture one or more images of a user's three-dimensional (3D) environment; a hardware processor in communication with the AR display and the outward facing camera, Receive user input for updating the map; Inducing the user to observe his / her 3D environment; Collect data that is associated with the user's 3D environment; determining a map quality index associated with the map; Shows an indication of the map quality index on the AR display, Update the map based on the collected data, ceasing to guide the user in response to identifying a stopping condition comprising a user input to stop or a determination that the map quality index reaches a threshold; a hardware processor programmed to An AR system equipped with

[0398] Example 56: An AR system as described in Example 55, wherein the map quality index is based on the percentage of the 3D environment that has collected data associated with it.

[0399] Example 57: An AR system described in any one of Examples 55-56, wherein the hardware processor is configured to display graphics in the center of the user's FOV.

[0400] Example 58: An AR system as described in Example 57, wherein the hardware processor is configured to display the graphic in the center of the user's FOV prior to generating the graphic in an unobserved location.

[0401] Example 59: The hardware processor includes: instructing the user to observe a graphic in the center of the user's FOV; signaling to a user that data has been successfully collected near at least one unobserved location; An AR system as described in any one of Examples 55-58, configured so that the collected data comprises data associated with the center of the user's FOV.

[0402] Example 60: An AR system described in any one of Examples 55-59, wherein the hardware processor is configured to display an animation associated with the graphic in the unobserved location to signal to the user that data has been successfully collected in the unobserved location.

[0403] Example 61: To display an animation, the AR system generating at least one bubble associated with the graphic; determining a collision trajectory associated with at least one bubble; animating at least one bubble to move along a collision trajectory toward a collision location on a mesh associated with the map of the user's environment; Show at least one bubble-mesh collision, The AR system of Example 60, configured as follows:

[0404] Example 62: An AR system described in any one of Examples 55-61, wherein the hardware processor is configured to display an indicator ring based on an unobserved location relative to the user's position to instruct the user to observe the graphic.

[0405] Any of the above embodiments may be combined in any suitable combination. R. Other Considerations

[0406] Each of the processes, methods, and algorithms described herein or depicted in the accompanying figures may be embodied in code modules executed by one or more physical computing systems, hardware computer processors, application-specific circuits, or electronic hardware configured to execute specific and particular computer instructions, and may be fully or partially automated thereby. For example, the computing system may include a general-purpose computer (e.g., a server) or a special-purpose computer programmed with specific computer instructions, special-purpose circuits, etc. The code modules may be compiled and linked into an executable program, installed in a dynamic link library, or written in an interpreted programming language. In some implementations, certain operations and methods may be performed by circuitry specific to a given function.

[0407] Furthermore, certain implementations of the functionality of the present disclosure may be sufficiently mathematically, computationally, or technically complex that special purpose hardware (utilizing appropriate specialized executable instructions) or one or more physical computing devices may be required to implement the functionality, e.g., due to the amount or complexity of the calculations involved, or to provide results in substantially real-time. For example, a moving picture or video may contain many frames, each frame having millions of pixels, and specifically programmed computer hardware is required to process the video data to provide the desired image processing task or application in a commercially reasonable amount of time.

[0408] The code modules or any type of data may be stored on any type of non-transitory computer readable medium, such as physical computer storage devices, including hard drives, solid state memory, random access memory (RAM), read only memory (ROM), optical disks, volatile or non-volatile storage devices, combinations of the same, or the like. The methods and modules (or data) may also be transmitted as data signals (e.g., as part of a carrier wave or other analog or digital propagating signal) generated over a variety of computer readable transmission media, including wireless-based and wired / cable-based media, and may take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). The results of the disclosed processes or process steps may be stored persistently or otherwise in any type of non-transitory tangible computer storage device, or communicated via a computer readable transmission medium.

[0409] Any process, block, state, step, or functionality in the flow diagrams described herein or depicted in the accompanying figures should be understood as potentially representing a code module, segment, or portion of code that includes one or more executable instructions for implementing a specific function (e.g., logical or arithmetic) or step in the process. Various processes, blocks, states, steps, or functionality can be combined, rearranged, added to, removed from, modified, or otherwise altered from the illustrative examples provided herein. In some embodiments, additional or different computing systems or code modules may perform some or all of the functionality described herein. The methods and processes described herein are also not limited to any particular sequence, and the blocks, steps, or states associated therewith can be performed in other sequences as appropriate, e.g., serially, in parallel, or in some other manner. Tasks or events can be added to or removed from the disclosed exemplary embodiments. Furthermore, the separation of various system components in the implementations described herein is for illustrative purposes and should not be understood as requiring such separation in all implementations. It should be understood that the described program components, methods, and systems may generally be integrated together in a single computer product or packaged into multiple computer products. Many implementation variations are possible.

[0410] The process, method, and system may be implemented in a network (or distributed) computing environment. Network environments include enterprise-wide computer networks, intranets, local area networks (LANs), wide area networks (WANs), personal area networks (PANs), cloud computing networks, crowdsourced computing networks, the Internet, and the World Wide Web. The network may be a wired or wireless network or any other type of communication network.

[0411] The systems and methods of the present disclosure each have several innovative aspects, none of which is solely responsible for or required for the desirable attributes disclosed herein. The various features and processes described above may be used independently of one another or may be combined in various ways. All possible combinations and subcombinations are intended to fall within the scope of the present disclosure. Various modifications of the implementations described in the present disclosure may be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other implementations without departing from the spirit or scope of the present disclosure. Thus, the claims are not intended to be limited to the implementations or embodiments shown herein, but should be accorded the widest scope consistent with the present disclosure, the principles and novel features disclosed herein.

[0412] Certa...

Claims

1. 1. An augmented reality (AR) system, comprising: an AR display configured to present virtual content to a user of the AR system; an outwardly facing camera configured to capture one or more images of the user's environment, where a portion of the user's environment that is visible to the user comprises the user's field of view (FOV); a hardware processor in communication with the AR display and the outward facing camera, the hardware processor comprising: generating a ray casting from an origin having a location associated with the user's head; accessing a map containing mesh data representing real objects within the environment; determining the presence of a mesh along said ray casting; calculating the distance of the mesh along the ray casting; determining a presence of content in the direction of the mesh; determining a ray casting type comprising a content type, a no content type, or an unobserved type based on the presence of the mesh, the presence of the content, and the distance of the mesh; determining unmapped locations of the map based at least on the ray casting type; and updating the virtual content to include a graphic associated with the unmapped location; collecting data associated with the environment of the user; In response to collecting sufficient data associated with the unmapped location, signaling to the user that the data has been successfully collected. a hardware processor programmed to An AR system comprising:

2. The AR system of claim 1 , wherein the hardware processor is configured to instruct the user to view the graphic.

3. 3. The AR system of claim 2, wherein the hardware processor is configured to, in response to determining that additional unmapped locations exist within the environment, repeat the updating, indicating, collecting, and signaling with respect to the additional unmapped locations.

4. 3. The AR system of claim 2, wherein the updating, indicating, collecting, and signaling are repeated until at least a minimum threshold of data associated with unmapped locations is obtained.

5. 2. The AR system of claim 1, wherein the hardware processor is further configured to determine the presence of at least one Persistent Coordinate Frame (PCF) within the environment of the user based on the collected data, and determining whether additional unmapped locations exist within the environment is based on the determination of whether at least one PCF has been identified.

6. The AR system of claim 1 , wherein the hardware processor is further configured to, in response to determining a presence of at least one PCF, load content associated with a stored map that includes the at least one PCF.

7. The AR system of claim 1 , wherein the content type comprises ray casting with content on the mesh within a maximum distance from at least one of the user and other unobserved locations.

8. The AR system of claim 1 , wherein the no-content type comprises ray casting with no content on the mesh within a maximum distance.

9. The AR system of claim 1 , wherein the unobserved type comprises ray casting with unknown content within a maximum distance.

10. 1. A computerized method implemented by an augmented reality (AR) system, the AR system comprising: an AR display configured to present virtual content to a user of the AR system; an outwardly facing camera configured to capture one or more images of the user's environment, where a portion of the user's environment that is visible to the user comprises the user's field of view (FOV); A hardware processor, the hardware processor comprising: generating a ray casting from an origin having a location associated with the user's head; accessing a map containing mesh data representing real objects within the environment; determining the presence of a mesh along said ray casting; calculating the distance of the mesh along the ray casting; determining a presence of content in the direction of the mesh; determining a ray casting type comprising a content type, a no content type, or an unobserved type based on the presence of the mesh, the presence of the content, and the distance of the mesh; determining unmapped locations of the map based at least on the ray casting type; and updating the virtual content to include a graphic associated with the unmapped location; collecting data associated with the environment of the user; In response to collecting sufficient data associated with the unmapped location, signaling to the user that the data has been successfully collected. a hardware processor programmed to A computerized method comprising:

Images