Physical map geolocation and virtual augmentation

Abstract

The present disclosure relates to entity map geolocation and virtual augmentation. Various implementations augment the appearance of and / or add interactive functionality to real-world maps in extended reality (XR) experiences. For example, a head-mounted device (HMD) user can view a real-world map on a wall, and the device can identify the map (e.g., using a classifier), determine the geographic boundaries represented in the map, and augment the view of the map and / or add interactive features based on region-specific identifying information. Real-world maps can be examined to identify boundaries using shape matching, OCR, visual inspection, ML, etc. (e.g., by identifying features, landmarks, edges, etc.), and thus the maps need not be Euclidean, e.g., can be for tourist maps, subway maps, curved maps / globe, etc. The augmentations can show points of interest, previously visited locations or routes (e.g., how did I get here), new / future route information, etc. The real-world map can be made interactive and / or altered to have a different format or style.

Description

Technical Field

[0001] This disclosure relates in its entirety to electronic devices that provide extended reality (XR) experiences involving physical maps existing within the physical environment of the device user. Background Technology

[0002] Existing user interaction systems can be improved in a way that efficiently and effectively promotes an enhanced user experience in which entity maps exist within the physical environment of the device user. Summary of the Invention

[0003] The various specific implementations disclosed herein include devices, systems, and methods for enhancing the appearance of real-world maps (i.e., physical maps as opposed to virtual / electronic maps) in XR experiences and / or adding interactive features to real-world maps. For example, a head-mounted device (HMD) user can use an HMD to view an environment including a real-world map on a wall, and the device can (e.g., using a classifier) ​​recognize the map, determine the geographic boundaries represented in the map, and enhance the map's view and / or add interactive features based on region-specific identification information. Shape matching, optical character recognition (OCR), visual inspection, machine learning (ML), etc., can be used to inspect (e.g., by identifying features, landmarks, edges, etc.) real-world maps to identify boundaries. The map does not need to be Euclidean; for example, the map can have a tourist map format, a subway map format, other formats where distances, directions, and locations are not strictly accurate, curved map / globe formats, etc.

[0004] Enhancements based on map information may show or otherwise provide information about points of interest, previously visited locations or routes (e.g., how I got here), new / future route information, etc. Enhancements may additionally or alternatively show the user's location on the map. Enhancements may be positioned on the map based on or otherwise relative to the user's location shown on the map (e.g., using lines or arrows from the user's location to a destination). In some implementations, fixed, immutable, or otherwise non-interactive real-world maps are made variable and / or interactive, for example, by allowing the user to provide input to select landmarks or routes on the map (e.g., gaze or touch detected by device sensors), and subsequently see enhancements in the XR environment that provide arrows, identifiers, etc., on and / or separately from the map, and indicate where to go, points to real-world landmarks, etc. Maps may be visually altered to have different formats, styles, expanded on one or more sides, and / or enhanced using overlays based on preferences, plans, inputs, etc.

[0005] In some implementations, a method is executed by a processor that carries out instructions stored on a non-transitory computer-readable medium. Such methods may involve identifying a real-world map in image data, which includes one or more images captured by sensors on a device. The method may involve determining the boundaries (e.g., one or more sides of a bounded rectangle or other shaped region) of a real-world area depicted by the real-world map. The boundaries may be identified based on features of the real-world map in the image data. For example, one or more boundaries may be determined based on identifying top / northeast landmarks, right / west landmarks, etc. Boundaries may be based on: identifying which landmarks are easternmost, westernmost, etc.; identifying the southernmost geographical features, such as the tip of a peninsula, buildings visible at the southernmost point on the map, etc. The method may also involve providing an XR experience with supplementary capabilities based on the boundaries of a real-world area depicted by a real-world map. As an example, this may involve providing augmented and / or interactive features on a real-world map based on the area, corresponding to real-world objects within the area.

[0006] According to some embodiments, an apparatus includes one or more processors, non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing to perform any of the methods described herein. According to some embodiments, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors of the apparatus, cause the apparatus to perform or cause to perform any of the methods described herein. According to some embodiments, an apparatus includes: one or more processors, non-transitory memory, and components for performing or causing to perform any of the methods described herein. Attached Figure Description

[0007] To enable those skilled in the art to understand this disclosure, more detailed descriptions can be made with reference to aspects of some exemplary embodiments, some of which are shown in the accompanying drawings.

[0008] Figure 1 Exemplary electronic devices operating in a physical environment according to some specific implementations are illustrated.

[0009] Figure 2 It shows the specific implementation targeting Figure 1 The boundaries of the physical environment's entity map identifiers.

[0010] Figure 3 It shows the implementation of some specific methods by Figure 1 The view provided by the electronic device, which enhances the XR experience based on a physical map.

[0011] Figure 4 It shows the implementation of some specific methods by Figure 1 The electronic device provides another view, which enhances the XR experience based on a physical map.

[0012] Figure 5 It shows the implementation of some specific methods by Figure 1 The electronic device provides another view, which enhances the XR experience based on a physical map.

[0013] Figure 6 It shows the implementation of some specific methods by Figure 1 Another view provided by electronic devices, in which interactive features are used to enhance the XR experience based on physical maps.

[0014] Figure 7 It shows the implementation of some specific methods by Figure 1 The electronic device provides another view, which enhances the XR experience based on a physical map.

[0015] Figure 8 This is a flowchart illustrating methods for enhancing XR experiences based on entity maps according to some specific implementations.

[0016] Figure 9 It is a block diagram based on some specific implementations of electronic devices.

[0017] As is customary practice, various features illustrated in the accompanying drawings may not be drawn to scale. Therefore, for clarity, the dimensions of various features may be arbitrarily expanded or reduced. Furthermore, some drawings may not depict all components of a given system, method, or apparatus. Finally, similar reference numerals may be used throughout the specification and drawings to denote similar features. Detailed Implementation

[0018] Numerous details have been described to provide a thorough understanding of the exemplary embodiments illustrated in the accompanying drawings. However, the drawings illustrate only some exemplary aspects of this disclosure and should not be considered limiting. Those skilled in the art will understand that other effective aspects and / or variations do not include all the specific details described herein. Furthermore, well-known systems, methods, components, devices, and circuits have not been described exhaustively so as not to obscure further relevant aspects of the exemplary embodiments described herein.

[0019] Figure 1 An exemplary electronic device 105 operating in physical environment 100 is shown. Figure 1In the example, physical environment 100 is a room including map 120, door 130, and potted plants 150. The physical environment may include one or more additional devices (not shown) communicating with electronic device 105. Such devices may be separate and distinct devices, for example, using different processing units, different input mechanisms, different output mechanisms, etc., and may be able to operate independently of each other. However, the devices may communicate with each other and / or may be configured to share information with each other, including but not limited to sensor data about map 120 and other contextual aspects of physical environment 100 and / or user 102.

[0020] Electronic device 105 (and / or other devices communicating therewith) may include one or more cameras, microphones, depth sensors, or other sensors that can be used to capture information about physical environment 100 and objects within it (e.g., images, sound, lighting characteristics, etc.) and information about user 102 of electronic device 105, and to evaluate the physical environment and objects within it. Information about physical environment 100 and / or user 102 can be used to supplement the user experience.

[0021] In some embodiments, sensor information from electronic device 105 provides information about the physical environment 100 and / or user 102 that can be used to identify the location of electronic device 105 and / or user 102, such as GPS coordinates, the postal address of the building where the user is located, or which room in a multi-room building the user is in. In some embodiments, sensor information from electronic device 105 provides information about user 102 relative to elements in the physical environment 100. For example, sensor data from one or more outward-facing and / or user-facing cameras can be used to identify elements within the physical environment, the user's positional relationship to such elements, and / or how the user interacts with these elements (e.g., looking at map 120 or otherwise demonstrating an intention to interact with map 120). In some embodiments, sensors on electronic device 105 capture sensor data for determining and tracking the gaze direction and / or hand position / configuration of user 102. Such gaze data and / or hand data can be used to identify the user's intention to interact with map 120 via gaze- and / or hand-based input modalities.

[0022] In some embodiments, the electronic device 105 does not include a display, and the user views the physical environment 100 directly, for example, using an optical see-through component and / or through a transparent lens on the electronic device 105. In some embodiments, such lenses are not configured to display content. In some embodiments, such lenses are configured to display content (e.g., by displaying augmented or other virtual content (its augmented physical environment 100 user view) using an optical waveguide that uses transmitted light to display content on the lens to present an extended reality (XR) environment). Augmentation may alternatively or additionally include audio content generated via one or more speakers on the electronic device 105.

[0023] In some embodiments, the electronic device 105 includes a display that presents a view of the physical environment 100 based on images captured by an outward-facing camera on the electronic device 105, for example, by providing pass-through video with or without added virtual content. In some embodiments, the display presents an XR environment by displaying augmented or other virtual content that enhances the displayed depiction of the physical environment 100.

[0024] In various specific implementations, a view of the XR environment may be provided via electronic device 105. Such an XR environment may include a pass-through video view of a 3D environment (e.g., adjacent to physical environment 100) generated from camera images and / or depth camera images of physical environment 100. Such an XR environment may include virtual content positioned in 3D relative to a 3D coordinate system associated with the XR environment (which may correspond to the 3D coordinate system of physical environment 100) (e.g., positioning the virtual content to appear as if it were on or adjacent to map 120). The virtual content may enhance the appearance of the displayed content (e.g., as a highlight appearing around map elements depicted on map 120).

[0025] In some implementations, a view of the XR environment may be provided to one or more participants (e.g., user 102 and / or other participants not shown) via a first electronic device 105 (e.g., a wearable device such as an HMD). Such an XR environment may include a view of a 3D environment generated based on camera images and / or depth camera images of the physical environment 100. Such an XR environment may include virtual content positioned at a 3D location relative to a 3D coordinate system (i.e., 3D space) associated with the XR environment, which may correspond to the 3D coordinate system of the physical environment 100.

[0026] In some embodiments, video (e.g., transparent video depicting the physical environment) is received from the image sensor of a device (e.g., device 105). In some embodiments, the 3D representation of the virtual environment is aligned with the 3D coordinate system of the physical environment. The size of the 3D representation of the virtual environment can be generated based on, for example, the scale of the physical environment or the positioning of open spaces, floors, walls, etc., such that the 3D representation is configured to align with corresponding features of the physical environment. In some embodiments, a viewpoint (e.g., that of user 102) in the 3D coordinate system can be determined based on the position of the electronic device within the physical environment. This viewpoint can be determined based on, for example, image data, depth sensor data, motion sensor data, etc., which can be retrieved via a virtual inertial odometry (VIO) system, a simultaneous localization and mapping (SLAM) system, etc.

[0027] In some implementations, the view of the XR environment is provided via an HMD (e.g., AR glasses) that includes a see-through display, in which virtual content is presented within the view. For example, such a device may include one or more lenses through which a user 102 observes the physical environment 100, and presentation components (e.g., one or more light sources and / or one or more waveguides) that allow the virtual content to be viewed by the user within the view seen through the one or more lenses.

[0028] In some implementations, electronic device 105 provides an XR experience, such as a view of an environment (e.g., physical environment 100, virtual environment, environment that mixes real and virtual elements, etc.). Electronic device 105 may enhance the physical environment experience that provides content (e.g., visual content and / or audio content). Electronic device 105 may enhance the appearance of a real-world (i.e., physical) map (such as map 120) in the XR experience and / or add interactive features to the real-world map. For example, user 105 can use electronic device 105 to view the physical environment 100, including a real-world map 120 on a wall, and electronic device 105 may (e.g., using a classifier) ​​identify map 120, determine the geographic boundaries represented in map 120, and enhance the view of the map and / or add interactive features based on region-specific identification information. Shape matching, optical character recognition (OCR), visual inspection, machine learning (ML), etc., may be used to inspect (e.g., by identifying features, landmarks, edges, etc.) the real-world map 120 to identify boundaries. Map 120 does not need to be Euclidean. For example, Map 120 can be in tourist map format, subway map format, other formats where distance, direction and / or location are not strictly accurate, curved map / globe format, etc.

[0029] Enhancements based on map 120 information may show or otherwise provide information about: points of interest (e.g., the user's current location within map 120), previously visited locations or routes (e.g., how I got here), new / future route information, etc. In some implementations, map 120 is fixed, immutable, or otherwise non-interactive, but is made variable and / or interactive within the XR experience. For example, it allows user 105 to provide input to select landmarks or routes on map 105 (e.g., gaze or touch detected by device sensors), and subsequently see enhancements in the view of the XR environment. These enhancements provide arrows, identifiers, etc., on and / or separately from the map 120, and indicate where to go, points to real-world landmarks, etc. Map 120 may be visually changed within the view of the XR experience to have different formats, styles, expand on one or more sides, and / or be enhanced using overlays based on preferences, plans, inputs, etc.

[0030] In some implementations, electronic device 105 uses sensor data from one or more sensors to evaluate (e.g., continuously, periodically, under triggering conditions, etc.) the physical environment 105 to detect the presence of a map (such as map 120) and / or to detect that the user 105 is looking at a scenario in which they may expect to supplement the experience based on such a map, focusing on the scenario, inquiring about the scenario, needing information about the scenario, or otherwise detecting the scenario. In some implementations, electronic device 105 examines sensor data and / or other information to understand the map, such as what geographic area is depicted, the location of the geographic area relative to world coordinate space and / or relative to a larger map including the surrounding area, what features are mapped and what features are not mapped, what the boundaries of the map are, where the user's current location would be if represented on the map, what features or parts of the map the user is interested in (e.g., based on where the user is looking or other contextual factors), what supplements the user might be interested in based on current contextual factors and / or the user's future plans, etc.

[0031] In some implementations, electronic device 105 identifies the map, for example, through a Stock Unit (SKU), code, or other information that uniquely identifies the map. In some implementations, electronic device 105 determines that the map's identity is unknown or undeterminable and therefore examines the map to determine what it depicts. Electronic device 105 can identify geographic areas and / or locations of interest associated with the boundaries surrounding the geographic areas depicted by the map. Given the boundaries of the areas depicted by the map and / or locations of interest associated with such boundaries, electronic device 105 can access associated information (e.g., other maps, datasets, previous user information, etc.) and supplement the user's XR experience accordingly.

[0032] Visual inspection of maps (e.g., captured in images and / or other sensor data via electronic device 105) may involve algorithms or machine learning models for identifying boundaries. Such computer-implemented processes may involve identifying map edges, land / water boundaries depicted on the map, political boundaries depicted on the map, cities, towns, buildings, landmarks, and other locations depicted within the map. In one example, a set of landmarks or other features is identified based on image analysis, and one or more boundaries are selected to encompass an area including all landmarks or other features.

[0033] Some implementations involve examining maps via decision trees or similar processes, where a series of evaluations are performed to identify the map's type, its boundaries, what is depicted on the map, what other information is available, etc. Such decision trees or similar processes may involve algorithmic and / or ML-based evaluations, for example, using an ML classifier to determine whether a map exists, whether the map is Euclidean, whether the map corresponds to the user's current location, etc. Map evaluations can be gated to conserve device resources (e.g., power, processing, etc.), thus determining map existence only periodically, and in some cases only detecting boundaries and / or map features.

[0034] In some implementations, electronic device 105 determines that one or more physical environmental elements outside the boundaries marked on the map may be of interest to the user; for example, there may be a restaurant that the user likes but is just outside the visible content depicted by the map. Electronic device 105 may use information about elements inside and / or outside the boundaries marked on the map 120 to supplement the user's XR experience.

[0035] Some specific implementations enhance or otherwise supplement the user's XR experience by providing information about paths or routes based on identifying entity maps (e.g., Map 120) and / or user interactions with the entity maps. For example, an overlay may be displayed above the entity map showing one or more routes taken by the user while navigating within an area / boundary depicted by such a map. As another example, information showing a route from the user's current location to a desired location, which is outside or beyond the map boundaries, may be provided.

[0036] The terms “physical map” and “real-world map” are used interchangeably herein and refer to any map that exists in a user environment separate from the user’s personal electronic device. A physical map or real-world map can be a tangible map, such as on paper, a bulletin board, a wall, etc., or any other form of map (e.g., on an electronic display) within a physical environment separate from the user’s own device, such as electronic maps displayed on bulletin boards, kiosks, etc.

[0037] Some implementations involve identifying physical maps, such as identifying objects within the user's physical environment as maps, identifying the content depicted by the map, identifying specific roads, sidewalks, buildings, etc., within the map, identifying the type of map, and / or identifying specific map identifiers (e.g., SKU or other unique map identifiers). In some implementations, direction (e.g., walking direction, driving direction, cycling direction, etc.) is determined based on: identifying physical maps, the user's current location within the physical map, and / or user information, such as information about where the user is going, upcoming appointments based on calendar data, etc. Maps may include various levels of information; for example, some information may have a great deal of highly detailed and dense information about buildings and other locations within the map area, while other information will only show the outline of a geographical area or political boundary, and still others will provide an intermediate level of detail. Direction can be used to complement the user's XR experience, for example, by providing visual and / or audio cues indicating to the user where the route is and / or where to go next. Visual cues can be provided on the map or in the user's view around the map and / or relative to the environment. Audio cues can be spatialized, for example, appearing to come from a map and / or indicating directions in the environment. For instance, a “go this way” audio cue sounds as if it comes from a specific direction from which the user should proceed along the identified route.

[0038] In some implementations, enhancements are provided relative to a physical map (e.g., on top of or in place of a map or a portion thereof). A copy of the physical map (e.g., in image data or from another source) can be captured and / or saved for later use. Similarly, a copy of the enhanced physical map can be captured and / or saved for later use. For example, a hiking user might look at a map of trails in an area at the park entrance, see the enhancement (which shows them a specific route through a network of multiple trails on the map), and save a copy of the map with the enhanced route for later use, e.g., when the user has walked away from the physical map along the trails and wants to see the map again. The map can be redisplayed (e.g., as a virtual content item in an XR experience) and can be further updated or enhanced, e.g., showing the user's new current location on a copy of the map displayed in the XR experience. In some implementations, a virtual map is continuously displayed, presenting the user's current location, e.g., as the user walks through a trail system, seeing their current location indicated on the map. It can display alternative or additional information, such as the total route distance, the distance traveled along the route so far, the distance to go along the route, alternative routes available based on the current location, etc.

[0039] Virtual content providing direction can offer guidance relative to the physical environment. For example, a user can look at a map near an opening leading to three paths. The user can provide input to identify one of the paths (e.g., by looking at a point along that path and then making a gesture such as pinching), and supplementary content can be displayed accordingly. For example, the user's view of the map can be enhanced to highlight that path. As another example, a user can look away from the map at the opening leading to three paths and see what appears to be an arrow (or other visual indicator) floating above the ground at the opening corresponding to the path identified by the user's input.

[0040] Electronic device 105 can identify a user's map and intended route, track the user's progress along the route, and provide visual or audio information as the user makes progress. For example, a user may come to a fork in the path and ask "Where do I go?", and electronic device 105 can determine (e.g., via image analysis used to identify multiple paths / routes in a view, GPS, or other location tracking) that the user is located at a fork, divergence, or otherwise presented alternative route. Electronic device 105 can provide feedback, for example, using arrows pointing to paths corresponding to the user's pre-selected route, information indicating options (e.g., following the left route would mean walking 0.5 miles to the destination, while taking the right route would mean walking 2.3 miles to the destination), and information indicating the destination of the route (e.g., indicating that the left route leads to a lookout point across the river in the park, the right route leads to a waterfall, etc.).

[0041] In some specific implementations, the view of the physical map is supplemented by changing the map style (e.g., changing from a road map to an elevation map, from a map without geographical boundaries to a map with geographical boundaries, or from a black and white map to a color map with colors indicating map features, terrain, etc.).

[0042] Some implementations provide a view of a physical map that shows the user’s current location and the direction the user is facing relative to a direction on the map. For example, if the user is facing east and east is on the right side of the map, an indication of the user’s location on the map can be shown, and the user is shown facing right relative to the map content from that location.

[0043] Information about a user's previous experiences can be displayed. For example, when revisiting a park using the trails described in the examples above, the physical map can be enhanced with information about specific trails the user took last year or during a previous visit. This information can highlight all previously taken routes, allowing adventurous hikers to identify and explore trails / routes they haven't previously explored.

[0044] In some implementations, physical maps include paper or otherwise immutable or non-interactive maps, and supplementary features are provided to allow users to experience the map as if it were a digital, dynamic, and / or interactive map. Interactions with the map can be identified and interpreted based on an understanding of the content displayed on the map and / or other contextual information (e.g., via user gaze, touch, pointing, other gestures, audio commands, etc.). Users can provide input to identify features of interest, obtain more information about the displayed items, zoom in and out of the map (e.g., by replacing the map with virtual content showing zoomed-in or zoomed-out map areas respectively), expand or reduce the map size, add user customizations to the map (e.g., images, links, etc.), and change the map view / format (e.g., change to satellite map, traffic map, topographic map, elevation map, etc.).

[0045] Contexts can be used to customize map supplements, such as providing different information during the daytime relative to night, during peak hours relative to weekends, when the user has an upcoming appointment, or when the user's destination is known. In one example, a user views an amusement park map showing the locations of rides and attractions offered by the park. Electronic devices display the map and present information to the user during their experience in the park, such as providing navigation to rides selected by the user, or identifying similar rides (e.g., "If you like this ride, consider this nearby ride next; it also has a loop and there is currently a 10-minute wait").

[0046] Figure 2 Showing the target Figure 1 The physical environment 100 is defined by the boundary identified by map 120. In this example, a portion of the image data of images captured within the physical environment 100 is analyzed, and the presence of entity map 120 is detected. If multiple images include entity map 120, one or more best images can be selected for further evaluation (e.g., based on criteria such as image integrity, sharpness, viewpoint angle, etc.). One or more images of map 120 can be compiled and used for further evaluation. Such one or more images are, for example, images from which image fragment 220 is selected via a cropping algorithm. The cropping algorithm can be used to select portions of an image corresponding to the map data. This can be based on identifying one or more map edges, boundaries, borders, etc.

[0047] Figure 2A portion 220 of the physical environment image corresponding to entity map 120 is shown. In this example, portion 220 is evaluated via one or more processes (e.g., algorithms or machine learning models) to determine the boundaries of the real world depicted on the map. Boundaries may correspond to east, west, north, and south boundaries. In this example, the westernmost point 230 is identified and used to identify the western boundary 232, the northernmost point 250 is identified and used to identify the northernmost boundary 252, the easternmost point 240 is identified and used to identify the easternmost boundary 242, and the southernmost feature 260 is identified and used to identify the southernmost boundary 262. Points and features used to identify boundaries may include, but are not limited to, land / sea intersections, road intersections, buildings, landmarks, etc. Boundaries may be selected to cover an entire area, even if the area represented by the map is smaller than such an entire area. For example, the southernmost feature could be a city park whose map only shows the northern half. The boundary could be selected to cover the entire park. Alternatively, features can be examined to determine where the map intersects with it (e.g., at the midpoint), and the boundaries selected based on that determination.

[0048] Figure 3 It shows the result of Figure 1 The electronic device 105 provides a view 300, in which the XR experience is enhanced based on map 120. View 300 includes a depiction 340 of map 120, a depiction 350 of potted plants 150, and a depiction 330 of door 130. In this example, the boundaries identified based on map 120 (e.g., as shown in the reference) are... Figure 2 An additional map 334 (described) is used to identify areas within these boundaries (i.e., the areas covered by boundaries 232, 242, 252, 262). In view 300, the additional map 334 is positioned adjacent to the depiction 340 of map 120. The additional map 334 can be given a 3D position corresponding to an adjacent position of map 120 in the physical environment 100. Thus, the additional map 334 can be presented at a fixed relative position, for example, appearing to be anchored to a wall in the physical environment adjacent to map 120. Furthermore, link indicators such as indicator 332 are displayed to provide graphical indication that a given feature of the depiction 340 of map 120 corresponds to a given feature of the additional map 334. Such indications can be made by identifying one or more of the most salient features within the boundaries, for example, which features are of highest priority according to general or user-specific criteria.

[0049] Figure 4 It shows the result of Figure 1 The electronic device 105 provides another view 400, in which the XR experience is enhanced based on map 120. View 400 includes a depiction 440 of map 120. In this example, the boundaries identified by map 120 (e.g., as referenced) are used to enhance the XR experience. Figure 2 The enhancements (described) are used to identify areas within these boundaries (i.e., the areas covered by boundaries 232, 242, 252, 262). In view 400, these enhancements are positioned relative to the depiction 340 of map 120, for example, where indicator dots are overlaid with locations on the depiction 340 of map 120, and explanatory indicators 460, 462, 464, and 468 are added to mark and point to the locations of these indicator dots. The enhancements can be positioned at 3D locations corresponding to fixed locations in the physical environment 100, so that they consistently appear in view 400 provided by the electronic device 400 even as the user 102 moves around and changes the viewpoint of the XR experience.

[0050] Figure 5 It shows the result of Figure 1 The electronic device 105 provides another view 500, in which the XR experience is enhanced based on a map. View 500 includes a map depiction 540. In this example, map-based boundaries are used to identify enhancements for the XR experience. In view 500, these enhancements are positioned relative to the map depiction 540 and individually within the XR environment. In this example, a first indicator dot and accompanying text 562 identify a location on the map depiction 540 corresponding to the user's location in the world; that is, the user is currently at a building in San Francisco, located at the position of the first indicator dot and accompanying text 562. Furthermore, a second indicator dot and accompanying text 564 identify the location the user intends to travel to (i.e., to Lombard Street in San Francisco). A route indicator 566 is also presented, showing the route from the user's current location to the user's intended location on the map depiction 540. Virtual content (such as overlays and other graphics) is used to provide these enhancements. In addition, another enhancement provides an arrow 568 separate from the map depiction 540. This enhancement is positioned based on the physical environment 100 rather than the map depiction. Specifically, the enhancement is positioned within view 500 and configured to point in a geographic direction (e.g., east) in which user 102 should proceed along the intended route. This type of enhancement (i.e., arrow 568) can move and change as the user changes viewpoint and / or location within the environment. For example, as the user walks along a route depicted by route indicator 566, arrow 568 can remain in the user's field of vision and adjust to always point in the direction the user should proceed from their current location to continue along that route. Additional indicators may be presented to facilitate user navigation along the route, such as showing alternative routes, indicating shortcuts, providing hazard warnings, identifying terrain, hills, steps, and other features, and indicating the location of other users, etc.

[0051] Figure 6 It shows the result of Figure 1The electronic device 105 provides another view 600, in which interactive features are used to enhance the XR experience based on a map. View 600 includes a map depiction 640. In this example, map-identified boundaries are used to identify one or more enhancements for the XR experience. In view 600, the enhancements are positioned relative to the map depiction 640. In this example, enhancement 662 is provided based on user interaction. Specifically, the user's gaze direction 610 is identified and determined to correspond to the Lombard Street location on the map depiction 640 that the user is looking at. Furthermore, a user hand gesture 620 (e.g., a pinch gesture) is detected when the user 102 is looking at that point. This combination of gaze and pinch is used to identify user interaction in which the user intends to identify, select, and / or obtain more information about a particular map location. In this example, in response to the detection of this input, enhancement 660 is provided, which provides information about the identified location.

[0052] Figure 7 It shows the result of Figure 1 The electronic device 105 provides another view 700, in which the XR experience is enhanced based on a map. In this example, map-based boundaries are used to identify enhancements for the XR experience. In view 700, these enhancements are positioned relative to the map depiction 740 and individually within the XR environment. In this example, a first indicator dot and accompanying text 762 identify the location on the map depiction 740 corresponding to the user's location in the world; that is, the user is currently at a building in San Francisco, located at the position of the first indicator dot and accompanying text 762. Furthermore, a second indicator dot and accompanying text 764 identify the location the user intends to travel to (i.e., to Lombard Street in San Francisco). A route indicator 766 is also presented, showing the route from the user's current location to the user's intended location on the map depiction 540. Virtual content (such as overlays and other graphics) is used to provide these enhancements.

[0053] In addition, another enhancement provides an arrow 768 separate from the map depiction 740. In this example, enhancement 768 is provided based on user interaction. Specifically, the user's gaze direction 710 is identified and determined to correspond to the user looking at the route indicator 766. Furthermore, a user hand gesture 720 (e.g., a pinch gesture) is detected when the user 102 is looking at the point. This combination of gaze and pinch is used to identify user interaction where the user intends to identify, select, and / or obtain more information about a particular route. In this example, in response to detecting this input, enhancement 768 is provided, indicating the direction the user should proceed to follow the route.

[0054] In some implementations, the device detects scale. In one example, based on the size of features on the detected entity map, the scale and / or distance to the target can be estimated. This may involve using known information / facts, such as the known size of a lake, the length of a street, the distance between landmarks, the size of land edge contours, etc., to estimate the map scale and / or distances within the map. The device can calculate the distance traveled or to be traveled between the starting point (e.g., the user's current location) and the target.

[0055] Figure 8 This is a flowchart illustrating a method 800 for enhancing an XR experience based on a physical map. In some embodiments, a device such as electronic device 105 performs method 800. In some embodiments, method 800 is performed via an HMD (e.g., an HMD that provides a pass-through or perspective view of the physical environment). Method 800 is performed by processing logic components, including hardware, firmware, software, or combinations thereof. In some embodiments, method 800 is performed on a processor that executes code stored in a non-transitory computer-readable medium (e.g., memory).

[0056] At box 802, method 800 relates to identifying a real-world map in image data, which includes one or more images captured by sensors on a device. For example, an algorithmic classifier or a machine learning-based classifier, or otherwise, as described herein or as will be understood by those skilled in the art based on the description herein, may be used to identify a real-world map in image data and / or other sensor data.

[0057] At box 804, method 800 involves determining the boundaries (e.g., one or more sides of a bounded rectangle or other shaped region) of a real-world area depicted by a real-world map. The boundaries may be identified based on features of the real-world map in the image data. For example, one or more boundaries may be determined based on identifying top / northeast landmarks, right / west landmarks, etc. Boundaries may be based on: identifying which landmarks are easternmost, westernmost, etc.; identifying the southernmost geographical features, such as the tip of a peninsula, buildings visible at the southernmost point on the map, etc.

[0058] In some implementations, determining the boundaries of a real-world region includes identifying the easternmost, westernmost, northernmost, and southernmost elements depicted on a real-world map. Such elements may include, but are not limited to, boundary markers, roads, waterways, bodies of water, land-water boundaries, or geopolitical boundaries. Determining the boundaries of a real-world region may include performing a shape-matching process to identify the easternmost, westernmost, northernmost, and southernmost elements depicted on a real-world map. Boundary determination may be based on optical character recognition (OCR), which identifies the markings on icons depicted on the map and identifies the real-world features associated with those icons.

[0059] If the entity map does not have one or more real-world boundaries (e.g., at the easternmost, westernmost, northernmost, and southernmost edges), the method can estimate the boundaries. For example, this could involve examining the map (or the depicted area) against entity landmarks. The method can examine the map against entity landmarks spanning the entire area of ​​the entity map and associate them with landmarks in the real-world area. It can identify the shape and location of lakes, mountains, rivers, etc., associate city names on the real-world map (if shown) with real-world areas, or otherwise determine features from which boundaries can be estimated.

[0060] At box 806, method 800 also relates to providing an XR experience with supplemental capabilities based on the boundaries of a real-world region depicted by a real-world map. As an example, this could involve providing augmented and / or interactive features on a real-world map corresponding to real-world objects within that region, based on that region. Figures 3 to 7 An example of an XR experience that provides complementary capabilities is shown.

[0061] Providing complementary XR experiences may involve presenting an XR view depicting a real-world map with alterations (e.g., virtual content associated with landmarks or routes on a real-world map, where the virtual content is identified based on boundaries). These alterations may include virtual content that expands a real-world area depicted by the real-world map beyond its boundaries. These alterations may include virtual content corresponding to points of interest or locations previously visited by the user of the electronic device. These alterations may also include virtual content corresponding to routes previously taken by the user of the electronic device or routes the user of the electronic device intends to take. These alterations may involve adding interactive features to the real-world map. As an example, such interactive features may include: recognizing gazes or input gestures toward elements on the real-world map; and initiating a response based on recognizing that input. Providing such responses may involve: identifying real-world features (e.g., landmarks, roads, etc.) corresponding to elements on the real-world map; and providing graphical indications (e.g., arrows, "this way," "this is 4th Street," etc.) on the map based on the location of these real-world features in the real world. Providing such responses may involve providing graphical indications corresponding to the identity and location of elements on the real-world map.

[0062] In some implementations, the supplementary capabilities provided offer information about locations of interest identified based on boundaries and user input. In some implementations, the supplementary capabilities provided offer alternative representations of entity maps in different formats.

[0063] In some implementations, the user's location is determined relative to an area depicted by a map and used to enhance the user experience. For example, method 800 may determine the user's geographic coordinates based on device sensors and / or information systems (e.g., GPS, motion sensors, image sensors, etc.). The device can use this user location to determine where to place a "You are here" sign, user representation, or other indicator corresponding to the user's location on the map. In one example, the device obtains or determines where the location of the area shown on the map is geolocated, and subsequently obtains or determines where the user is geolocated, and positions the user's location within the correct location on the map based on where the user's geolocation is relative to the geolocated area represented by the map.

[0064] Similarly, the location of a target can be determined based on known geographic coordinates that identify the target. In one example, the device obtains or determines where the location of an area shown on a map is geolocated, and then obtains or determines where the target is geolocated, and positions the target within the correct location on the map based on where the target geolocation is relative to the geolocated area represented by the map.

[0065] Route information can be determined similarly, for example, identifying the geolocation of a segment of road a user will travel from their current location to their destination. In one example, the device obtains or determines where the location of an area shown on a map is geolocated, and then obtains or determines where the route is geolocated (e.g., the start and end points), and positions the route location within the correct location on the map based on where the route geolocation is relative to the geolocated area represented by the map.

[0066] Figure 9This is a block diagram of electronic device 900. Device 900 illustrates an exemplary device configuration of electronic device 105. Although certain specific features are illustrated, those skilled in the art will recognize from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure further relevant aspects of the specific embodiments disclosed herein. Therefore, as a non-limiting example, in some specific implementations, device 900 includes one or more processing units 902 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 906, one or more communication interfaces 908 (e.g., USB, Firewire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BlueTooth, ZigBee, SPI, I2C and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 910, one or more output devices 912 (e.g., including displays), one or more internal and / or external image sensor systems 914, memory 920, and one or more communication buses 904 for interconnecting these components and various other components.

[0067] In some embodiments, one or more communication buses 904 include circuitry for interconnecting system components and controlling communication between system components. In some embodiments, one or more I / O devices and sensors 906 include at least one of the following: an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time-of-flight, etc.), etc.

[0068] In some embodiments, one or more output devices 912 include one or more displays configured to present a view of a 3D environment to a user. In some embodiments, the one or more displays correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface-conducting electron emitter display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), microelectromechanical systems (MEMS), and / or similar display types. In some embodiments, the one or more displays correspond to waveguide displays such as diffraction, reflection, polarization, and holography. In one example, device 900 includes a single display. In another example, device 900 includes displays for each of the user's eyes.

[0069] In some embodiments, one or more output devices 912 include one or more audio generating devices. In some embodiments, one or more output devices 912 include one or more speakers, surround sound speakers, speaker arrays, or headphones for generating spatialized sound (e.g., 3D audio effects). Such devices can virtually place sound sources in a 3D environment, including behind, above, or below one or more listeners. Generating spatialized sound may involve transforming sound waves (e.g., using head-related transfer functions (HRTF), reverberation, or cancellation techniques) to simulate natural sound waves (including reflections from walls and floors) emanating from one or more points in the 3D environment. Spatialized sound can induce the listener's brain to interpret sound as if it occurred at one or more points in the 3D environment (e.g., from one or more specific sound sources), even if the actual sound may be generated by speakers in other locations. One or more output devices 912 may additionally or alternatively be configured to generate haptic feedback.

[0070] In some embodiments, one or more image sensor systems 914 are configured to acquire image data corresponding to at least a portion of the physical environment. For example, one or more image sensor systems 914 may include one or more RGB cameras (e.g., having a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), monochrome cameras, IR cameras, depth cameras, event-based cameras, etc. In various embodiments, one or more image sensor systems 914 also include an illumination source emitting light, such as a flash. In various embodiments, one or more image sensor systems 914 also include an on-camera image signal processor (ISP) configured to perform multiple processing operations on the image data.

[0071] Memory 920 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some embodiments, memory 920 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 920 optionally includes one or more storage devices remotely located to one or more processing units 902. Memory 920 includes a non-transitory computer-readable storage medium.

[0072] In some embodiments, memory 920 or a non-transitory computer-readable storage medium of memory 920 stores an optional operating system 930 and one or more instruction sets 940. Operating system 930 includes procedures for handling various basic system services and for performing hardware-related tasks. In some embodiments, instruction set 940 includes executable software defined by binary information stored in charge. In some embodiments, instruction set 940 is software executable by one or more processing units 902 to implement one or more of the techniques described herein.

[0073] Instruction set 940 includes boundary identification instruction set 942, which is configured to identify one or more boundaries associated with the map as described herein at execution time. Instruction set 940 includes XR supplement instruction set 944, which is configured to supplement the XR experience as described herein at execution time. Instruction set 940 may be embodied in a single software executable file or multiple software executable files.

[0074] Although instruction set 940 is shown residing on a single device, it should be understood that in other embodiments, any combination of elements may reside in separate computing devices. Furthermore, the accompanying drawings serve more as a functional description of the various features present in a particular embodiment, and differ from the structural schematics of the embodiments described herein. As will be appreciated by those skilled in the art, individually shown items may be combined, and some items may be separate. The actual number of instruction sets and how features are allocated therein will vary depending on the specific embodiment and may depend in part on the specific combination of hardware, software, and / or firmware chosen for that particular embodiment.

[0075] It should be understood that the specific embodiments described above are cited by way of example, and the invention is not limited to what has been specifically shown and described above. Rather, the scope includes both combinations and sub-combinations of the various features described above, as well as variations and modifications of the various features that would occur to those skilled in the art upon reading the foregoing description and that are not disclosed in the prior art.

[0076] As described above, one aspect of this technology involves collecting and using sensor data, which may include user data, to improve the user experience of electronic devices. This disclosure envisions that, in some cases, the collected data may include personal information data that uniquely identifies a specific person or can be used to identify the interests, characteristics, or preferences of a specific person. Such personal information data may include motion data, physiological data, demographic data, location-based data, telephone numbers, email addresses, home addresses, device characteristics of personal devices, or any other personal information.

[0077] This disclosure recognizes that the use of such personal information data in the present invention can benefit users. For example, personal information data can be used to improve the content viewing experience. Therefore, the use of such personal information data may enable planned control of electronic devices. Furthermore, this disclosure also anticipates other uses of personal information data that benefit users.

[0078] This disclosure further envisions that entities responsible for the collection, analysis, disclosure, transmission, storage, or other use of such personal information and / or physiological data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to privacy policies and measures recognized as meeting or exceeding industry or governmental requirements for maintaining the privacy and security of personal information data. For example, personal information from users should be collected for legitimate and reasonable purposes of the entity and not shared or sold outside of these legitimate purposes. Furthermore, such collection should only be conducted after obtaining informed consent from users. Additionally, such entities should take any necessary steps to safeguard and protect access to such personal information data and ensure that others with access to such personal information data comply with their privacy policies and procedures. Furthermore, such entities may subject themselves to third-party assessments to demonstrate their compliance with widely accepted privacy policies and practices.

[0079] Regardless of the foregoing, this disclosure also envisions specific implementations that allow users to selectively block the use or access to personal information data. That is, this disclosure envisions providing hardware or software components to prevent or block access to such personal information data. For example, with regard to a content delivery service tailored to a user, the technology of this invention can be configured to allow a user to choose to "join" or "opt out" of the collection of personal information data during service registration. In another example, a user may choose not to provide personal information data for a target content delivery service. In yet another example, a user may choose not to provide personal information but allow the transmission of anonymous information for improving device functionality.

[0080] Therefore, while this disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, it is also contemplated that various embodiments can be implemented without access to such personal information data. That is, various embodiments of the present invention will not become inoperable due to the absence of all or part of such personal information data. For example, preferences or settings can be inferred based on non-personal information data or an absolute minimum amount of personal information, such as content requested by a device associated with a user, other non-personal information available to the content delivery service, or publicly available information, thereby selecting content and delivering it to the user.

[0081] In some implementations, data is stored using a public / private key system that only allows the data owner to decrypt the stored data. In other implementations, data may be stored anonymously (e.g., without identification and / or without personal information about the user, such as legal name, username, time, and location data). This prevents other users, hackers, or third parties from identifying the user associated with the stored data. In some implementations, users can access their stored data from a different user device than the one used to upload the stored data. In these cases, users may need to provide login credentials to access their stored data.

[0082] This document sets forth numerous specific details to provide a comprehensive understanding of the claimed subject matter. However, those skilled in the art will understand that the claimed subject matter can be practiced without these specific details. In other instances, methods, apparatus, or systems known to a person of ordinary skill have not been described in detail in order not to obscure the claimed subject matter.

[0083] Unless otherwise specifically stated, it should be understood that throughout this specification, discussions using terms such as “processing,” “calculating,” “calculating,” “determining,” and “identifying” refer to the actions or processes of computing devices, such as one or more computers or similar electronic computing devices, which manipulate or convert data representing physical electronic or magnetic quantities within the memory, registers, or other information storage, transmission, or display devices of a computing platform.

[0084] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. Computing devices may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include computer systems based on multi-purpose microprocessors that access stored software that programs or configures the computing system from a general-purpose computing device to a dedicated computing device that implements one or more specific embodiments of the subject matter of this invention. The teachings contained herein may be implemented in the software used for programming or configuring the computing device using any suitable programming, scripting, or other type of language or combination of languages.

[0085] Specific implementations of the methods disclosed herein can be performed in the operation of such computing devices. The order of the boxes presented in the examples above can be varied; for example, the boxes can be reordered, combined, and / or divided into sub-blocks. Some boxes or processes can be executed in parallel.

[0086] The use of "applies to" or "configured to" in this document implies open and inclusive language, which does not exclude applicability to or configuration to devices performing additional tasks or steps. Furthermore, the use of "based on" implies openness and inclusivity, as processes, steps, calculations, or other actions "based on" one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated. The headings, lists, and numbering included herein are for illustrative purposes only and are not intended to be restrictive.

[0087] It will also be understood that while terms such as "first," "second," etc., may be used in this document to describe various elements, these elements should not be limited by these terms. These terms are merely used to distinguish one element from another. For example, a first node may be called a second node, and similarly, a second node may be called a first node, changing the meaning of the description, provided that all occurrences of "first node" are consistently renamed and all occurrences of "second node" are consistently renamed. First nodes and second nodes are both nodes, but they are not the same node.

[0088] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the claims. As used in the description of these embodiments and in the appended claims, the singular forms “a,” “an,” and “the” are intended to also cover the plural forms unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. It will be further understood that the term “comprising,” as used in this specification, specifies the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0089] As used herein, the term "if" can be interpreted as meaning "when the prerequisite is true" or "when the prerequisite is true" or "in response to determination" or "according to determination" or "in response to detection" that the prerequisite is true, depending on the context. Similarly, the phrase "if it is determined [the prerequisite is true]" or "if [the prerequisite is true]" or "when [the prerequisite is true]" can be interpreted as meaning "when it is determined that the prerequisite is true" or "in response to determination" or "according to determination" that the prerequisite is true or "when it is detected that the prerequisite is true" or "in response to detection" that the prerequisite is true, depending on the context.

[0090] The foregoing description and summary of the present invention should be understood as illustrative and exemplary in every respect, and not restrictive, and the scope of the invention disclosed herein is determined not only by the detailed description of the illustrative specific embodiments, but also by the full extent permitted by patent law. It should be understood that the specific embodiments shown and described herein are merely illustrative of the principles of the invention, and various modifications can be made by those skilled in the art without departing from the scope and spirit of the invention.

Claims

1. A method, the method comprising: In electronic devices that include processors: Identify real-world maps in image data, said image data including one or more images captured by sensors on said device; Determine a region of the real world depicted by the real-world map, the region being identified based on features of the real-world map in the image data; as well as Provides an extended reality (XR) experience with supplementary capabilities based on the real-world region depicted by the real-world map.

2. The method of claim 1, wherein identifying the region includes identifying the boundary of the real-world region, wherein the supplementary capability is based on the boundary of the real-world region.

3. The method of claim 1, wherein providing the XR experience with the supplementary capability includes presenting an XR view depicting the real-world map with alterations.

4. The method of claim 3, wherein the alteration includes virtual content associated with landmarks or routes on the real-world map, wherein the virtual content is identified based on the boundaries.

5. The method of claim 3, wherein the alteration includes virtual content that expands the real-world region depicted by the real-world map beyond the boundary.

6. The method of claim 3, wherein the change includes virtual content corresponding to: Points of interest; or The location previously visited by the user of the electronic device.

7. The method of claim 3, wherein the change includes virtual content corresponding to: The route previously taken by the user of the electronic device; or The route that the user is to take in the electronic device.

8. The method of claim 3, wherein the alteration includes adding interactive features to the real-world map.

9. The method of claim 8, wherein the interactive feature comprises: Recognize gazes or input gestures directed toward elements on the real-world map; as well as Initiate a response based on the identified input.

10. The method of claim 9, wherein the response comprises: Identify the real-world characteristics corresponding to the elements in the real-world map; as well as Provide graphical indications in the map based on the location of the real-world characteristics described therein.

11. The method of claim 9, wherein the response includes providing a graphical indication corresponding to the identity and location of the element in the real-world map.

12. The method of claim 1, wherein determining the boundary of the real-world region comprises identifying the easternmost, westernmost, northernmost, and southernmost elements depicted in the real-world map.

13. The method of claim 12, wherein the element is a boundary marker, road, waterway, body of water, land-water boundary, or geopolitical boundary.

14. The method of claim 12, wherein determining the boundary of the real-world region comprises performing a shape matching process to identify the easternmost element, the westernmost element, the northernmost element, and the southernmost element depicted in the real-world map.

15. The method of claim 12, wherein the boundary is determined based on optical character recognition (OCR), the OCR identifying a marker for an icon depicted in the map and identifying real-world characteristics associated with the icon.

16. The method of claim 1, wherein the real-world map is identified in the image data using an algorithmic classifier or a machine learning-based classifier.

17. The method of claim 1, wherein the electronic device is a head-mounted device.

18. The method of claim 1, wherein the supplementary capability comprises: Provides information about locations of interest identified based on the boundaries and user input, as well as alternative representations of entity maps in different formats.

19. A system comprising: Non-transitory computer-readable storage medium; as well as One or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium includes program instructions that, when executed on the one or more processors, cause the system to perform operations including: Identify real-world maps in image data, which includes one or more images captured by sensors on the device; Determine zones of real-world areas depicted by the real-world map, the zones being identified based on features of the real-world map in the image data; and Provides an extended reality (XR) experience with supplementary capabilities based on the real-world region depicted by the real-world map.

20. A non-transitory computer-readable storage medium storing program instructions executable via one or more processors to perform operations including: Identify real-world maps in image data, which includes one or more images captured by sensors on the device; Determine a region of the real world depicted by the real-world map, the region being identified based on features of the real-world map in the image data; as well as Provides an extended reality (XR) experience with supplementary capabilities based on the real-world region depicted by the real-world map.

Images