Method and device for site-based augmented reality

Abstract

In one general aspect, a method may include receiving a representation of a real-world scene captured by a user using a mobile device, wherein the real-world scene is part of a real-world physical area. The method may include associating a position of the mobile device with an AR anchor based on a comparison of the representation of the real-world scene with the part of a model of the real-world physical area. The method may include triggering display of an AR object associated with the model of the real-world physical area within the mobile device based on the position of the mobile device.

Description

[0001] Related Applications

[0002] This application is a continuation of and claims priority to U.S. non-provisional patent application No. 15 / 929,470, filed on May 4, 2020, entitled “METHODS AND APPARATUS FOR VENUEBASED AUGMENTED REALITY,” which claims priority to U.S. Provisional Patent Application No. 62 / 843,495, filed on May 5, 2019, both of which are incorporated herein by reference in their entirety.

[0003] This application also claims priority to U.S. Provisional Patent Application No. 62 / 843,495, filed on May 5, 2019, the entire contents of which are incorporated herein by reference in their entirety. Background Art

[0004] Placing an augmented reality (AR) object in the proper context of an image of a real-world scene viewed through a user's mobile device can be complex. In particular, placing an AR object in the correct location and / or orientation within a display can be difficult to achieve. The global positioning system (GPS) of the user's mobile device can be used to identify the user's location, and the user's location can then be used to place an AR object associated with the object within the user's display. Summary of the invention

[0005] In one general aspect, a method may include receiving a representation of a real-world scene captured by a user using a mobile device, wherein the real-world scene is part of a real-world physical area. The method may include associating a position of the mobile device with an AR anchor based on a comparison of the representation of the real-world scene with a part of a model of the real-world physical area. The method may include triggering display of an AR object associated with the model of the real-world physical area within the mobile device based on the position of the mobile device. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 is a diagram of the AR objects positioned relative to the AR anchor as viewed by the user within the physical area.

[0007] Figure 2 is a block diagram illustrating a system configured to implement the concepts described herein.

[0008] Figure 3A Illustration of a sample model map.

[0009] Figure 3B Graphics Figure 3BAR objects within the model map shown in .

[0010] Figure 3C Graphics Figure 3A AR anchors in the model map shown in .

[0011] Figure 4A Figure 1 illustrates positioning of a mobile device to one or more AR anchors.

[0012] Figure 4B The diagram illustrates updating the location of the mobile device as an AR anchor based on the movement of the mobile device.

[0013] Figure 4C Another example of positioning to an AR anchor is shown.

[0014] Figure 5A and 5B Real world scenes without and with AR objects are illustrated respectively.

[0015] 6A to 7B Illustration of a real-world scene with AR objects for wayfinding.

[0016] Figure 8 It is shown in Figure 3C A diagram showing how to dynamically add AR anchors to a model map.

[0017] Fig. 9 It is shown in Figure 3B A diagram of dynamically adding AR objects within the model map.

[0018] Fig.10 and 11 Diagrammatically illustrate methods of discovery and / or pathfinding.

[0019] Fig.12 Illustrate a method of creating a model and associated elements for discovery and / or pathfinding.

[0020] Fig.13 Examples of a general purpose computer device and a general purpose mobile computer device are shown. DETAILED DESCRIPTION

[0021] It is difficult to place an augmented reality (AR) object in an appropriate position and / or orientation within an image of a real-world scene viewed through a user's mobile device. The global positioning system (GPS) of a user's mobile device can be used to identify the user's location, and then the user's location can be used to place an AR object associated with the object within the user's display. However, in some cases (e.g., in a multi-story building), GPS may not be available and / or not accurate enough. For example, when a device is indoors, GPS may not usually be used to accurately locate the device orientation (e.g., accurate to a specific floor). In addition, many venues are not open to the public and / or may not be well recorded. Some information about the venue may be unreliable because the owner of the venue may not have the resources to maintain accurate information about the venue. Information about the venue can only be generated using expensive equipment and / or professional technology. After such information is generated, it may be relatively static and difficult to modify or update. Without accurate mapping, location and / or orientation information associated with the venue, the application will not be able to correctly place AR objects in the location and / or event within the device's display.

[0022] The technical solutions described herein involve processing of multiple sensory signals to display augmented reality content (e.g., AR objects) for, for example, pathfinding and / or discovery at a venue (e.g., a location, a physical space, a region, an area). Specifically, accurate location and orientation of a venue and / or event information enables the use of an augmented reality display for use with, for example, pathfinding and / or information discovery. In some embodiments, the context display in AR helps users find their way around unfamiliar locations and / or discover events or places of interest while on location.

[0023] To achieve accurate placement of AR objects (also referred to as points of interest (POIs)), a scaled digital 3D representation of the venue is generated, and the user's position and / or orientation can be localized to the scaled digital 3D representation via an AR anchor (e.g., an AR anchor has a fixed position relative to an origin, where the origin is a predefined fixed position in a real-world physical area). The 3D representation can then be converted into the view space of the user's device, and the AR object can be displayed in an appropriate scene within the real world using augmented reality. In some embodiments, the rendered AR object appears to be anchored to a physical element that the AR object is pointing at, labeled, etc.

[0024] In some embodiments, physical signage can be used to facilitate resolution of location and / or orientation. Signs that exist in a physical space are defined and placed in a digital map representation, and then uniquely identified using perceptual techniques (e.g., image and / or text recognition). In such embodiments, the methods and apparatus described herein may not require an operator to 3D map a space, but may rely on a plan view and information of where the signs are placed and oriented.

[0025] The methods and apparatus described herein have technical advantages over existing mapping applications that use, for example, GPS and Wi-Fi positioning devices. Specifically, the solution described herein is configured to accurately locate a user device when a GPS signal is not available (or cannot be used when multiple floors of an interior space are involved), and does not require additional networking equipment to operate. The methods and apparatus described herein also have advantages over using a magnetometer sensor to orient the device (e.g., a magnetometer sensor may be relatively inaccurate and / or may be impaired by a local magnetic field). The methods and apparatus described herein have advantages over existing augmented reality platform technologies and over existing machine learning technologies for recognizing and reading text. In some embodiments, the methods and apparatus described herein allow the owner of a venue to update information about the venue (e.g., the location of AR objects and physical objects) without requiring an operator to scan the space. The methods and apparatus described herein have advantages over products that rely primarily on GPS, which cannot accurately locate the position and / or orientation of a user device when GPS is unavailable.

[0026] Figure 1 A diagram of a user 100 within a real-world physical area 10 (e.g., a real-world venue) viewing an AR object P through a mobile device 110. The position (e.g., position and / or orientation and / or distance and orientation) of the user 100 is located relative to the position of an AR anchor B, which has a fixed position relative to an origin O. The origin O is a fixed position within the real-world physical area 10, which is modeled in a 1:1 scale model or representation. A 1:1 scale model (e.g., a model map) of the real-world physical area 10 may be referred to as a full-scale model (e.g., a model or representation that is accurate in scale). The AR object P has a fixed position within the full-scale model of the real-world physical area 10. Unless otherwise noted, references to the model are considered to be the same as references to the full-scale model.

[0027] In some embodiments, the position may include a position in X, Y, Z coordinates, and the orientation may include a directional orientation (e.g., the direction or angle, yaw, pitch, and roll, that an object or user is facing). Thus, a user (e.g., user 100) and / or an AR object (e.g., AR object P) may be located at a specific X, Y, Z position and facing a specific direction as the orientation at that X, Y, Z position.

[0028] The AR object P is correctly displayed in the mobile device 110 (e.g., on its display screen) using a combination of the positioning of the user 100's mobile device 110 to the AR anchor B (which may be referred to as the positioning of the user 100 and / or the positioning of the mobile device 110), the origin O, the full-scale model of the real-world physical area 10, and the fixed position of the AR object P in the full-scale model of the real-world physical area 10. In some embodiments, the origin O may be a common origin (e.g., anchor) to which the AR anchor B and the full-scale model of the real-world physical area 10 may be oriented (e.g., fixedly tied, bound). In addition, AR objects such as the AR object P may also be included in the full-scale model of the real-world physical area 10 (at a fixed position and orientation (e.g., X, Y, and Z coordinate orientation)). Therefore, when the mobile device 110 is positioned to the AR anchor B, the origin O may be used to reconcile (e.g., translate, transform) the AR object to the position and / or orientation of the mobile device 110 (of the user 100).

[0029] For example, in some embodiments, the user 100 may use a camera of the mobile device 110 to capture a representation of a real-world scene from the real-world physical area 10. The real-world scene may be a portion of the real-world physical area 10 captured by a camera (e.g., a camera of the mobile device 110). The position (and / or orientation) of the mobile device 110 may be associated with the AR anchor B based on a comparison (e.g., feature matching) of the representation of the real-world scene with a portion of a full-scale model of the real-world physical area 10. In some embodiments, positioning may include determining the position and orientation of the mobile device 110 relative to the AR anchor B. In some embodiments, the position and orientation may include a distance from the AR anchor B and a direction that the mobile device 110 is facing relative to the AR anchor B. Because the AR anchor B has a fixed position relative to the origin O, and because the real-world physical area 10 has a fixed position relative to the origin O, the position and orientation of the mobile device 110 relative to the real-world physical area 10 may be determined. Therefore, the position and orientation of the mobile device 110 relative to the AR object P may be determined in a manner that the AR object P has a fixed position and orientation within the real-world physical area 10. In other words, by utilizing the positioning of the AR anchor B, the full-scale model of the real-world physical area 10 and the orientation of the AR object P around the user 100 can be determined via the origin O. The AR object R can then be displayed to the user 100 at the correct position and orientation within the mobile device 110. Changes in the position and orientation of the mobile device 110 can be determined by sensors (e.g., an inertial measurement unit (IMU), a camera, etc.) and can be used to update the position and / or orientation of the AR object P (and / or other AR objects).

[0030] Figure 2 is an illustration of a system configured to implement the concepts described herein (e.g., Figure 1 2. The system 200 includes a mobile device 110 and an AR server 252. Figure 2 The details of the mobile device 110 and the AR server 252 are shown. Using the system 200, one or more AR objects can be displayed within the display device 208 of the mobile device 110 using a combination of the positioning of the mobile device 110 to the AR anchor, the origin, the full-scale model of the real-world physical area, and the fixed position and orientation of the AR object within the full-scale model of the real-world physical area. Figure 1 The operation of system 200 is described in the context of the accompanying drawings.

[0031] The mobile device 110 may include a processor component 204, a communication module 206, a sensor system 210, and a memory 220. The sensor system 210 may include various sensors, such as a camera component 212, an inertial motion unit (IMU) 214, and a global positioning system (GPS) receiver 216. Embodiments of the sensor system 210 may also include other sensors, including, for example, light sensors, audio sensors, image sensors, distance and / or proximity sensors, contact sensors such as capacitive sensors, timers, and / or other sensors and / or different combinations of sensors. The mobile device 110 includes a device positioning system 242 that may utilize one or more portions of the sensor system 210.

[0032] The mobile device 110 also includes a display device 208 and a memory 220. Applications 222 and other applications 240 are stored in and accessible from the memory 220. The applications 222 include an AR anchor positioning engine 224, a map coordination engine 225, an AR object retrieval engine 226, a map and anchor creation engine 227, an AR anchor presentation engine 228, and a user interface engine 230. In some embodiments, the mobile device 110 is a mobile device such as a smartphone, a tablet, or the like.

[0033] The system illustrates details of an AR server 252, which includes a memory 260, a processor component 254, and a communication module 256. The memory 260 is configured to store a model map 30 (also referred to as a model), an AR anchor A, and an AR object P.

[0034] Although the processing blocks shown in the AR server 252 and the mobile device 110 are illustrated as being included in specific devices, these processing blocks (and the processing associated therewith) may be included in different devices, divided between devices, etc. For example, at least a portion of the map coordination engine 225 may be included in the AR server 252.

[0035] The model map 30 stored in the memory may be a three-dimensional (3D) representation of the real-world physical area 10 (e.g., with depth data). In some embodiments, the model map 30 may be a black and white or color image (e.g., with depth data). In some embodiments, the model map 30 may be a panorama or may include a panorama (e.g., with depth data). As an example, a panorama may include an image or a collection of images (captured at one location) extending at a wide angle, e.g., at least 120 degrees, at least 180 degrees, or even 360 degrees. In some embodiments, the model map 30 may be a point cloud representation, which includes points (e.g., point clouds) in a 3D space representing features (e.g., edges, density, buildings, walls, signs, planes, objects, textures, etc.) within the real-world physical area 10. As described above, the model map 30 may be a 1:1 full-scale map of the real-world physical area 10. The model map 30 (and the real-world physical area 10) may be a venue (e.g., a park, a part of a city, a building (or a part thereof), a museum, a concert hall, etc.).

[0036] Figure 3A A representation of an example model map 30 associated with a real-world physical area 11 is illustrated. Figure 3A The representation shown in is a two-dimensional (2D) top view of a model map 30 including buildings, streets, trees, etc. Figure 3A The origin O of the model map 30 is shown in the figure, and the origin O can be used as the origin of the relative coordinate system of the model map 30. In other words, the model map 30 can have a coordinate system based on the origin O, rather than a GPS coordinate system or other absolute coordinate system that is bound to the actual location of the real-world physical area 11 on the earth. However, the distance represented in the model map 30 can be a real-world distance (e.g., meters). The origin O can be an arbitrary point selected or identified in the model map 30. However, the origin O can be used for coordination with other coordinate systems (e.g., coordinate translation, coordinate transformation).

[0037] In some embodiments, the model map 30 can be created by capturing a video of the real-world physical area 11 using the camera component 212 and the map and anchor creation engine 227. In some embodiments, the model map 30, which is a digital map accurately scaled (e.g., real-world distance (e.g., meters, centimeters) and scale), can be created from a digital map of the location, an architectural diagram, a floor plan (e.g., a technical floor plan) of a venue (e.g., an indoor location, a planned building outside an event space, etc.), etc. In some embodiments, a 2D map can be used (e.g., at least partially used) to generate a 3D model map 30. In some embodiments, the model map 30 can be created quickly (e.g., within an hour) via a mobile device 110 and walking through the area. This is in contrast to methods that require expensive and complex image capture devices with dedicated capture data. The model map 30 can be stored in the AR server 252 after it is captured.

[0038] AR objects P1-P9 (eg, points of interest) are overlaid on Figure 3B . The AR objects P1-P9 (collectively referred to as AR objects P) have fixed positions (e.g., X, Y, Z positions) and orientations (e.g., directions) within the model map 30. The AR objects P have fixed positions and orientations relative to the origin O (as illustrated by the dashed lines). In some embodiments, the model map 30 includes AR objects P that are related to (e.g., associated with, designed for, identified with) places, events, locations, and the like.

[0039] In some embodiments, at least one of the AR objects P may be configured to move as the mobile device 110 moves, the user moves, or may move even if the mobile device 110 does not move. For example, one of the AR objects P, such as a navigation guide (e.g., a wayfinding arrow) for guiding the user, may have a starting point near (e.g., in front of) the location and orientation of the mobile device 110. As the mobile device 110 moves, the navigation guide may also move (e.g., rotate, move in front of the user) to navigate the user to the desired location.

[0040] In some embodiments, each AR object P can be a fixed position and orientation in the coordinate space of the model map 30. Each AR object P can be independent of the coordinate space of the real world (e.g., latitude and longitude, GPS coordinate space). Because the AR object P is in a fixed position and orientation in the coordinate space of the model map 30, the AR object P is in a full-scale position and orientation. In other words, the AR objects P can all be in a fixed position and orientation in the coordinate space of the model map 30. In some embodiments, the AR object P can be in a fixed position and orientation relative to the origin O (at a distance in the real world). In some embodiments, the AR object P can be in a coordinate space independent of the coordinate space of the model map 30 (but with the origin O as a common origin).

[0041] In some embodiments, the AR object P may be a label, a 3D model, an interactive immersive model, etc. In some embodiments, the AR object P may be placed in the model map 30. In some embodiments, the AR object P may be placed in the model map 30 to facilitate discovery and / or pathfinding using the AR object P in the real-world physical area 11.

[0042] AR anchors A1-A3 are covered in Figure 3C 30. An AR object P is also shown. AR anchors A1-A3 (which may be collectively referred to as AR anchors A) have fixed positions (e.g., X, Y, Z positions) and orientations (e.g., directions) within the model map 30. The AR anchor P has a fixed position and orientation relative to an origin O (as illustrated by dashed lines). As previously described, the origin O may be an arbitrarily selected origin.

[0043] The AR anchors A (which may all be unique) may all be fixed positions (and / or orientations) within the coordinate space of the model map 30. Because the AR anchors A are at fixed positions (and / or orientations) within the coordinate space of the model map 30, the AR anchors A are at full-scale positions (and / or orientations). The AR anchors A may all be fixed positions (and / or orientations) within the coordinate space of the model map 30. In some embodiments, the AR anchors P may be at fixed positions (and / or orientations) relative to the origin O. In some embodiments, the AR anchors P may be in a coordinate space independent of the coordinate space of the model map 30. In some embodiments, at least each AR anchor P has a position (without orientation) within the model map 30.

[0044] AR anchor A can be used to locate the user 100 (e.g., the user's mobile device 110) to the model map 30. AR anchors can be considered as AR activation markers. AR anchors A can be created so that the user's mobile device 110 can be located at one or more of the AR anchors A. For example, AR anchor A can be an image and / or representation associated with a location (e.g., point and / or area) of a real-world physical area 11 corresponding to the full-scale model map 30. In some embodiments, AR anchor A (similar to the model map 30) can be a point set (e.g., a point cloud) that represents features (e.g., edges, density, buildings, walls, logos, planes, objects, textures, etc.) at or near a location (e.g., point and / or area) within the model map 30. In some embodiments, AR anchor A can be a spherical image (e.g., a color image) or a panoramic image associated with a location within the model map 30. In some embodiments, one or more of the AR anchors A can be content items. In some embodiments, AR anchor A can be one or more features associated with a location within the model map 30.

[0045] Because the AR anchor A can be, for example, an image or representation associated with a location (e.g., a point and / or an area) within the model map 30, each AR anchor A can be considered to have its own independent coordinate system (rather than a unified coordinate system). In some embodiments, the AR anchor A can be part of a coordinate space that is about the AR anchor A (and independent of other coordinate systems). The AR anchors A can each be independent of a real-world coordinate space (e.g., latitude and longitude, GPS coordinate space). However, the location associated with the AR anchor A can be (in real-world distance) about the origin O. In other words, the AR anchors can be defined using a coordinate space having a common origin with the origin O.

[0046] In some embodiments, one or more of the AR anchors A may be created by capturing features (e.g., images or image sets (e.g., videos), panoramas) while the user 100 (holding the mobile device 110) is physically standing at points and / or areas within the real-world physical area 11. The creation of the AR anchors A may be performed using the map and anchor creation engine 227. The captured features may then be mapped as AR anchors A to locations (e.g., feature sets associated with locations) within the full-scale model map 30. This information may be stored in the AR server 252.

[0047] In some embodiments, one or more of the AR anchors A within the model map 30 may include a uniquely identifiable marker (e.g., a physical marker) to be used as an AR activation marker. Under certain limitations, the marker may include text, QR, a custom-designed visual scan code, etc. In some embodiments, the AR anchor A may be a uniquely identifiable physical marker connected by, for example, a position and / or orientation within the model map 30. Physical markers in real-world physical areas can be used to accurately calibrate the position and / or orientation of the mobile device 110.

[0048] As previously described, in some embodiments, the model map 30, each AR anchor A, and the AR object P are associated with or defined in different (e.g., different and independent) coordinate spaces. Therefore, each of these elements (model map 30, AR anchor A, AR object P) can be dynamically updated without adversely affecting other elements. For example, one or more of the AR anchor A and / or AR object P can be modified (e.g., updated, deleted, changed) in a desired manner. In combination Figure 8 and Fig. 9 More details about dynamic updates are discussed. Due to the independent nature of these coordinate spaces, the position and orientation of the AR object P relative to the mobile device 110 is resolved (e.g., translation, transformation) through a common link to the model map 30 (and origin O) and the AR anchor A to which the mobile device 110 is positioned when in use. This system and method can operate accurately even if the data captured during setup is incomplete, inaccurate, etc. This may be in contrast to other systems that may require complete and very accurate unified data capture during setup.

[0049] Reference again Figure 2 , the AR anchor positioning engine 224 can be configured to determine the position of the mobile device 110 based on a comparison (e.g., matching of features) of the representation of the real-world scene with a portion of the full-scale model map 30 of the real-world physical area. The comparison may include a comparison of features (e.g., edges, density, buildings, walls, signs, planes, objects, textures, etc.) captured by the mobile device 110 with, for example, features included in or represented in the model map 30. In some embodiments, the comparison may include a comparison of a portion of an image captured by the mobile device 110 with a portion of an image associated with the model map 30.

[0050] The camera assembly 212 can be used to capture images or videos of a physical space, such as a real-world scene, from a real-world physical area around the mobile device 110 (and the user 100) for positioning purposes. The camera assembly 212 may include one or more cameras. The camera assembly 212 may also include an infrared camera. In some embodiments, a representation (e.g., an image) of a real-world scene from the real-world physical area 10 can be captured by the user 100 using the camera assembly 212 camera of the mobile device 110. The representation of the real-world scene can be part of the real-world physical area 10. In some embodiments, the features (e.g., images) captured by the camera assembly 212 can be used to locate the mobile device 110 for one of the AR anchors 264 stored in the memory 160 of the AR server 252.

[0051] Based on the comparison of the features, the AR positioning engine 224 may be configured to determine the position and / or orientation of the mobile device 110 relative to one or more of the AR anchors A. By comparing the image as viewed by the mobile device 110, the position (and / or orientation) of the mobile device 110 may be positioned relative to the positions of the AR anchors A. Specifically, for example, images captured by a camera of the mobile device 110 may be used to determine the position and orientation of the mobile device 110 relative to the AR anchors A.

[0052] An example of positioning is Figure 4A As shown in the figure. Figure 4A As shown in FIG. 1 , the user 100 is located at position C1. For the purpose of explanation and by way of example, the position of the user 100 within the model map 30 is Figure 4A However, in reality, the user 100 is located in the real-world physical area 11 associated with the model map 30 and only Figure 4AUsing the mobile phone 110, the user 100 is using the mobile phone 110 to capture an image of an area (e.g., a scene) within the real-world physical area 11. The captured image of the area (e.g., a scene) (as an example) may be compared to the model map 30 to determine the user's position C1 and the user's orientation at the position C1. The position and orientation may include determining the distance D1 of the user 100 from the AR anchor A2 and the direction U that the user is facing, which is toward the building 4 and to the left of the AR anchor A2. The AR anchor A2 may be associated with an image capture that may be compared with a capture of the mobile device 110 along the direction U. Based on the comparison of the capture along the direction U and the capture associated with the AR anchor A2, the AR anchor positioning engine 224 may determine that the mobile device 110 is at a distance D1 (and position C1) with respect to the AR anchor A2 and is facing the direction U. Because the AR anchor A2 has a fixed position with respect to the origin O, and because the real-world physical area 11 represented within the model map 30 has a fixed position with respect to the origin O, the position and orientation of the mobile device 110 relative to the real-world physical area 11 may be determined.

[0053] In some implementations, the location of the mobile device 110 to the AR anchor A can be updated based on the user's movement. For example, if the user moves from Figure 4A Position C1 in Figure 4B 2, the AR positioning engine 224 may be configured to determine the position and / or orientation of the mobile device 110 relative to the AR anchor A1 as the user moves toward the position C2 and away from the AR anchor A2. In this example, when at the position C2, the mobile device 110 is located closer to the AR anchor A1 than the AR anchor A2. The mobile device 110 is at a distance D2 from the AR anchor A1 (and is facing a direction relative to the AR anchor A1).

[0054] The updating of the positioning may facilitate the display accuracy of the AR object P within the display of the mobile device 110 of the user 100. As the mobile device 110 moves within the real-world physical area (corresponding to the model map 30), the user's position may be inaccurate due to drift inherent in the sensor system 210. By dynamically updating the position of the mobile device 110 compared to the AR anchor A, inaccuracies due to drift may be reduced or eliminated.

[0055] exist Figure 4C, where the mobile device 110 captures a portion of a corner of a wall and a portion of a painting 402 (e.g., an interior of a building, an interior of a building on a particular floor (e.g., of multiple floors) of a building). The captured area is shown as area 40. This captured area can be used to locate the mobile device 110 to an AR anchor E1 that was previously captured from a different angle (e.g., by another mobile device) and includes an overlapping area 42, as illustrated by the dotted line. Specifically, features of the captured area 40 can be compared with features of the captured area 42 associated with the AR anchor E1 to locate the mobile device 110 to the AR anchor E1.

[0056] In some embodiments, the AR positioning engine 224 may be configured to determine the position and / or orientation of the mobile device 110 relative to one or more of the AR anchors A by attempting to locate more than one (e.g., all) of the AR anchors A. In some embodiments, when the user is at a specified location (or area) at a given time (or within a time window), only one AR anchor A is selected for positioning. The best matching AR anchor A may be selected for positioning. In some embodiments, the best match may be the AR anchor A closest to the mobile device 110. In some embodiments, the best match may be the AR anchor A with the most features that match the model map 30. In some embodiments, the AR anchor positioning engine 224 determines a confidence score for the identified AR anchor A. A higher confidence score may indicate that a feature from an image (e.g., an image, a representation, extracted text, a barcode, a QR code) is more likely to be associated with the determined AR anchor A than if a lower confidence score is determined.

[0057] Even after positioning at one of the AR anchors A, the precise position and orientation of the mobile device 110 in the physical real world may not be known. Only the relative position and orientation of the mobile device 110 relative to the AR anchor A (and through the AR anchor A within the model map 30) is known. Ad-hoc capture of feature (e.g., image) information by the mobile device 110 is used to determine the relative position of the mobile device 110. Further coordination (e.g., with the mobile map 30) may be required to determine the position and orientation of the mobile device 110 relative to the AR object P.

[0058] In some embodiments, the AR positioning engine 224 may also use images captured by the camera assembly 212 to determine the position and orientation of the mobile device 110 within a physical space, such as an interior space (e.g., the interior space of a building), based on a representation of the physical space received from the memory 260 or an external computing device. In some embodiments, the representation of the physical space may include visual features of the physical space (e.g., features extracted from an image of the physical space). The representation may also include location determination data associated with those features that can be used by the visual positioning system to determine the position and / or orientation within the physical space based on one or more images of the physical space. The representation may also include a three-dimensional model of at least some structures within the physical space. In some embodiments, the representation does not include a three-dimensional model of the physical space.

[0059] In some embodiments, the AR positioning engine 224 can use multiple sensory signals (from one or more of the sensor systems 210) to uniquely identify the identifier. In some embodiments, these include, but are not limited to: image recognition and tracking, text recognition and tracking, AR tracking orientation points, GPS location, Wifi signals, QR codes, custom-designed visual scanning codes, etc. In some embodiments, the AR anchor positioning engine 224 identifies the identifier for positioning. In some embodiments, the uniquely identifiable identifier associated with the model map 30 can correspond to a uniquely identifiable physical marker. In some embodiments, the AR anchor positioning engine 224 identifies one or more codes within the image, such as a barcode, a QR code, or another type of code. The code can then be mapped to an AR anchor A. In some embodiments, the mapping of digital and physical representations allows the mobile device 110 to be accurately positioned.

[0060] After the mobile device 110 of the user 100 has been located at the AR anchor A, the map coordination engine 225 is configured to coordinate the coordinate spaces of the model map 30, the AR object P, and the AR anchor A. In some embodiments, the map coordination engine 225 is configured to coordinate the coordinate spaces of the model map 30, the AR object P, and the AR anchor A based on a common origin O.

[0061] For example, the position and orientation of an AR object P (having a fixed position and orientation within the model map 30) relative to the mobile device 110 may be determined based on the positioning of the mobile device 110 to at least one of the AR anchors A, at least one of which has a fixed relationship to the origin O and a fixed relationship within the model map 30. In other words, the position and orientation of the AR object P relative to the mobile device 110 is resolved (e.g., translated, transformed) through a common link to the model map 30 (and the origin O) and the AR anchor A to which the mobile device 110 is positioned. By doing so, one or more of the AR objects P may be displayed within the mobile device 110 based on the position and orientation of the mobile device 110 even when the mobile device 110 moves within the real-world physical area (and the corresponding model map 30).

[0062] For example, Figure 4C As shown in , the position and orientation of the mobile device 110 relative to the AR object P6 can be determined by the AR object P6 having a fixed position and orientation within the real-world physical area 11 represented by the model map 30. In other words, by positioning using the AR anchor A2, the orientation of the full-scale model of the real-world physical area 11 and the position and orientation of the AR object P6 around the user 100 can be determined via the origin O. Then, the AR object P6 can be displayed to the user 100 within the mobile device 110. For example, Figure 4A As shown in , the mobile device 110 (via the user 100) faces a direction U as determined by a positioning process utilizing AR anchor A2, and an AR object P6 can be displayed within the display device 208 of the mobile device 110 based on the direction U. Accordingly, the position and orientation of the AR object P is displayed within the mobile device 110 as resolved (e.g., transformed, translated) by a common bond to the model map 30 (and origin O) and the AR anchor A to which the mobile device 110 is positioned.

[0063] Changes in the position and orientation of the mobile device 110 may be determined by sensors (e.g., inertial measurement units (IMUs), cameras, etc.) and may be used to update the position and / or orientation of the AR object P6 (and / or other AR objects P1-P5, P7-P9). For example, if the mobile device 110 is moved to a direction different from the direction U, the display of the AR object P6 may be modified accordingly within the display device 208 of the mobile device 110.

[0064] Reference again Figure 2, the AR object retrieval engine 226 may be configured to retrieve one or more AR objects P from the AR server 252. For example, the AR object retrieval engine 226 may retrieve the AR object P within the model map 30 based on the coordination of the coordinate space of the AR object P, the model map 30, and the AR anchor A performed by the map coordination engine 225.

[0065] The AR object presentation engine 228 presents one or more AR objects P on the mobile device 110 or causes one or more AR objects P to be presented on the mobile device 110. For example, the AR object presentation engine 228 may cause the user interface engine 230 to generate a user interface including information or content from the one or more AR objects P for display by the mobile device 110. In some embodiments, the AR object presentation engine 228 is triggered by the AR object retrieval engine 226 that retrieves the one or more AR objects P. The AR object presentation engine 228 may then trigger the display device 208 to display content associated with the one or more AR objects P.

[0066] The user interface engine 230 may be configured to generate a user interface. The user interface engine 230 may also cause the mobile device 110 to display the generated user interface. The generated user interface may, for example, display information or content from one or more of the AR objects P. In some embodiments, the user interface engine 230 generates a user interface including a plurality of user-actuable controls, each of which is associated with one or more of the AR objects P. For example, a user may actuate one of the user-actuable controls (e.g., by touching the control on a touch screen, clicking the control using a mouse or another input device, or otherwise actuating the control).

[0067] exist Figure 5B An example of an AR object 501 displayed within a real world scene 500 is shown in FIG. The AR object 501 may be stored at the AR server 252. Figure 5A A real world scene 500 is shown in FIG. 5 without an AR object 501 .

[0068] exist Fig. 6A and Figure 6B 6 and 602 are shown in FIG. 6 , respectively, as examples of AR objects 601 and 602 displayed within real world scenes 610 and 611. Specifically, the AR objects 601 and 602 are related to pathfinding. Such AR objects 601 and 602 may be stored in the AR server 252 and may be accessed from the AR server 252.

[0069] Fig. 7A and 7BAdditional real-world scenes 710, 711 are illustrated that are associated with AR objects 701 to 703 for wayfinding within a building (e.g., a particular floor of a building). In this example implementation, AR objects 701 (arrows pointing in directions), 702 (destination markers) may be updated as the user moves until the user has reached the location shown by AR object 703. AR objects 701 to 703 may be stored in and accessed from the AR server 252.

[0070] Figure 8 is a diagram illustrating the dynamic addition of AR anchors within the model map 30. Specifically, in this example, AR anchor A4 is added to the set of AR anchors A. AR anchor A4 can be added to a specific location for the model map 30 even after other AR anchors A have been created (e.g., curated) and used for discovery and pathfinding. AR anchor A can be used for further positioning and can be used to identify and display AR object P.

[0071] Being able to dynamically add AR anchors as individual positioning points avoids having to recreate a comprehensive, complete, and final model of the real world, which other systems may need to do. Additional positioning AR anchors can be quickly added and used to improve the system's processing and locate the user 100.

[0072] Fig. 9 is a diagram illustrating the dynamic addition of AR anchors within the model map 30. Specifically, in this example, an AR object P10 is added to a collection of AR objects P. The AR object P10 may be added to a specific location of the model map 30 even after other AR objects P10 have been created (e.g., exhibited) and used for discovery and pathfinding. In some embodiments, one or more AR objects P may also be moved.

[0073] Being able to dynamically add AR objects to the model map 30 facilitates updating the AR world. Additional AR objects can be quickly dynamically added and used to improve the AR experience without having to recreate all aspects of the model.

[0074] Fig.10 Schematic diagram of a method of discovery and / or pathfinding as described herein. Fig.10 As shown in , the method includes (for example, by Figure 2 The AR anchor positioning engine 224 shown in FIG. 1 receives a representation of a real-world scene captured using a mobile device (e.g., by a user) (block 1010). The real-world scene can be part of a real-world physical area (e.g., a venue). The capture of the real-world scene can be performed by at least Figure 2The method includes associating a location of the mobile device with an AR anchor based on a comparison of a representation of the real-world scene with a portion of the model of the real-world physical area (block 1020). The association (e.g., positioning) of the location can be performed by, for example Figure 2 The method also includes triggering the display of an AR object associated with a model of a real-world physical area within the mobile device based on the location of the mobile device (block 1030). The display may be performed by, for example, Figure 2 The AR object retrieval engine 226, the AR presentation engine 228, and the user interface engine 230 shown in FIG. 2 are triggered by one or more of the AR object retrieval engine 226, the AR presentation engine 228, and the user interface engine 230.

[0075] Fig.11 Schematic diagram of a method of discovery and / or pathfinding as described herein. Fig.10 As shown in , the method includes receiving a representation of a real-world scene captured by a mobile device, the real-world scene being part of a real-world physical area (block 1110). The real-world scene may be part of a real-world physical area (e.g., a venue). Capture of the real-world scene may be performed by at least Figure 2 The method includes positioning the mobile device using an AR anchor from a plurality of AR anchors based on a comparison of features of the real-world scene with features of the model map of the real-world physical area (block 1120). The association (e.g., positioning) of the position can be performed by, for example Figure 2 The method also includes identifying a position and orientation of the AR object relative to the mobile device based on the fixed position of the AR object within the model and the fixed position of the AR anchor within the model (block 1130). The identification can be performed by, for example, Figure 2 It may be executed by one or more of the map coordination engine 225 or the AR object retrieval engine 226 shown in .

[0076] Fig.12 Schematic diagram of a method and associated elements for creating a model for discovery and / or pathfinding as described herein. The flowchart elements may be composed of Figure 2 The map and anchor creation engine 227 shown in FIG.

[0077] The method may include capturing a first feature associated with a real-world physical area as a model (block 1210), and associating an AR object with a fixed position within the model (block 1220). In some implementations, the AR object may be associated with a fixed position and a fixed orientation within the model. The method may include capturing a second feature associated with a real-world position corresponding to a portion of the real-world physical area (block 1230), and associating the captured second feature with a position in the model corresponding to the real-world position as an AR anchor (block 1240). In some implementations, one or more features may be captured in an image or a point cloud.

[0078] Reference again Figure 2 , the IMU 214 can be configured to detect motion, movement, and / or acceleration of the mobile device 110. The IMU 214 can include a variety of different types of sensors, such as, for example, accelerometers, gyroscopes, magnetometers, and other such sensors. The orientation of the mobile device 110 can be detected and tracked based on data provided by the IMU 214 or the GPS receiver 216.

[0079] The GPS receiver 216 can receive signals transmitted by GPS satellites. The signals include the time and position of the satellites. Based on receiving signals from several (eg, at least four) satellites, the GPS receiver 216 can determine the global location of the mobile device 110.

[0080] Other applications 240 include any other applications installed on mobile device 110 or otherwise available for execution on mobile device 110. In some implementations, application 222 can cause one of other applications 240 to be launched.

[0081] The device positioning system 242 determines the position of the mobile computing device 202. The device positioning system 242 may use the sensor system 210 to determine the location and orientation of the mobile computing device 202 within a global or physical space.

[0082] The AR anchor positioning engine 224 may include a machine learning module that can recognize at least some types of entities within an image. For example, the machine learning module may include a neural network system. A neural network is a computational model used in machine learning and includes nodes organized in layers with weighted connections. Training a neural network uses training examples, each of which is an input and a desired output, and a series of iterative rounds to determine the weight values ​​of the connections between layers that increase the likelihood that the neural network will provide the desired output for a given input. During each training round, the weights are adjusted to account for erroneous output values. Once trained, the neural network can be used to predict outputs based on the inputs provided.

[0083] In some embodiments, the neural network system includes a convolutional neural network (CNN). A convolutional neural network (CNN) is a neural network in which at least one layer of the neural network is a convolutional layer. A convolutional layer is a layer in which the values ​​of the layer are calculated based on applying a kernel function to a subset of the values ​​of the previous layer. Training a neural network can involve adjusting the weights of the kernel function based on training examples. Typically, the same kernel function is used to calculate each value in the convolutional layer. Therefore, the weights that must be learned when training a convolutional layer are much fewer than in a fully connected layer in a neural network (e.g., a layer that calculates each value in a layer as a weighted combination of the independent adjustments of each value in the previous layer). Because there are typically fewer weights in a convolutional layer, training and using a convolutional layer may require less memory, processor cycles, and time than an equivalent fully connected layer.

[0084] The communication module 206 includes one or more devices for communicating with other computing devices such as the AR server 252. The communication module 206 can communicate via a wireless or wired network such as the network 290. The communication module 256 of the AR server 252 can be similar to the communication module 206. The network 290 can be the Internet, a local area network (LAN), a wireless local area network (WLAN), and / or any other network.

[0085] The display device 208 may, for example, include an LCD (liquid crystal display) screen, an LED (light emitting diode) screen, an OLED (organic light emitting diode) screen, a touch screen, or any other screen or display for displaying images or information to a user. In some embodiments, the display device 208 includes a light projector that is arranged to project light onto a portion of the user's eye.

[0086] The memory 220 may include one or more non-transitory computer-readable storage media. The memory 220 may store instructions and data that the mobile device 110 may use to implement the techniques described herein, such as generating a visual content query based on a captured image, transmitting the visual content query, receiving a response to the visual content query, and presenting a digital supplement identified in the response to the visual content query. The memory 260 of the AR server 252 may be similar to the memory 220 and may store data instructions that may be used to implement the techniques of the AR server 252.

[0087] Processor component 204 and / or processor component 254 include one or more devices capable of executing instructions, such as instructions stored by memory 220, to perform various tasks. For example, one or more of processor components 204, 254 may include a central processing unit (CPU) and / or a graphics processor unit (GPU). For example, if a GPU is present, some image / video rendering tasks, such as generating and displaying portions of a user interface or displaying a digital supplement, may be offloaded from the CPU to the GPU. In some embodiments, some image recognition tasks may also be offloaded from the CPU to the GPU.

[0088] Although Figure 2 Not shown, but some embodiments include a head mounted display device (HMD) (e.g., AR enabled glasses). The HMD can be a separate device from the mobile device 110, or the mobile device 110 can include the HMD. In some embodiments, the mobile device 110 communicates with the HMD via a cable. For example, the mobile device 110 can transmit video signals and / or audio signals to the HMD for display to the user, and the HMD can transmit motion, position, and / or orientation information to the mobile device 110.

[0089] The mobile device 110 may also include various user input components (not shown), such as a controller that communicates with the mobile device 110 using a wireless communication protocol. In some embodiments, the mobile device 110 may communicate with an HMD (not shown) via a wired connection (e.g., a universal serial bus (USB) cable) or via a wireless communication protocol (e.g., any WiFi protocol, any Bluetooth protocol, Zigbee, etc.). In some embodiments, the mobile device 110 is a component of the HMD and may be contained within a housing of the HMD.

[0090] Fig.13 Examples of a general purpose computer device 2000 and a general purpose mobile computer device 2050 that can be used with the techniques described herein are shown. Computing device 2000 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, tablet computers, workstations, personal digital assistants, televisions, servers, blade servers, mainframes, and other appropriate computing devices. Computing device 2050 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit the implementation of the inventions described and / or claimed in this document.

[0091] The computing device 2000 includes a processor 2002, a memory 2004, a storage device 2006, a high-speed interface 2008 connected to the memory 2004 and a high-speed expansion port 2010, and a low-speed interface 2012 connected to a low-speed bus 2014 and the storage device 2006. The processor 2002 can be a semiconductor-based processor. The memory 2004 can be a semiconductor-based memory. Each of the components 2002, 2004, 2006, 2008, 2010, and 2012 is interconnected using various buses and can be installed on a common motherboard or installed in other ways as appropriate. The processor 2002 can process instructions for execution within the computing device 2000, including instructions stored in the memory 2004 or on the storage device 2006 to display graphical information of a GUI on an external input / output device such as a display 2016 coupled to the high-speed interface 2008. In other embodiments, multiple processors and / or multiple buses and appropriately multiple memories and multiple types of memories can be used. Additionally, multiple computing devices 2000 may be connected, with each device providing portions of the necessary operations (eg, as a server bank, a group of blade servers, or a multi-processor system).

[0092] The memory 2004 stores information within the computing device 2000. In one embodiment, the memory 2004 is one or more volatile memory units. In another embodiment, the memory 2004 is one or more non-volatile memory units. The memory 2004 may also be another form of computer-readable medium, such as a magnetic disk or an optical disk.

[0093] Storage device 2006 can provide mass storage for computing device 2000. In one embodiment, storage device 2006 can be or include computer readable media, such as floppy disk device, hard disk device, optical disk device or tape device, flash memory or other similar solid state storage device or array of devices, including devices in storage area network or other configurations. Computer program products can be tangibly implemented in information carriers. Computer program products can also include instructions, which, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine readable medium, such as memory 2004, storage device 2006 or memory on processor 2002.

[0094] The high-speed controller 2008 manages bandwidth-intensive operations of the computing device 2000, while the low-speed controller 2012 manages less bandwidth-intensive operations. This allocation of functions is exemplary only. In one embodiment, the high-speed controller 2008 is coupled to the memory 2004, the display 2016 (e.g., through a graphics processor or accelerator), and is coupled to the high-speed expansion port 2010, which can accept various expansion cards (not shown). In this embodiment, the low-speed controller 2012 is coupled to the storage device 2006 and the low-speed expansion port 2014. The low-speed expansion port, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), is coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or a router, for example, through a network adapter.

[0095] As shown in the figure, computing device 2000 can be implemented in many different forms. For example, it can be implemented as a standard server 2020, or multiple times in a group of such servers. It can also be implemented as part of a rack server system 2024. In addition, it can be implemented in a personal computer such as a laptop computer 2022. Alternatively, components from computing device 2000 can be combined with other components in a mobile device (not shown) such as device 2050. Each of such devices can contain one or more of computing devices 2000, 2050, and the entire system can be composed of multiple computing devices 2000, 2050 that communicate with each other.

[0096] Computing device 2050 includes processor 2052, memory 2064, input / output devices such as display 2054, communication interface 2066 and transceiver 2068, among other components. Device 2050 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of components 2050, 2052, 2064, 2054, 2066 and 2068 are interconnected using various buses, and several components may be mounted on a common motherboard or otherwise appropriately mounted.

[0097] The processor 2052 can execute instructions within the computing device 2050, including instructions stored in the memory 2064. The processor can be implemented as a chipset including separate and multiple analog and digital processor chips. For example, the processor can provide coordination of other components of the device 2050, such as control of a user interface, applications executed by the device 2050, and wireless communications performed by the device 2050.

[0098] The processor 2052 can communicate with the user through a control interface 2058 and a display interface 2056 coupled to the display 2054. The display 2054 can be, for example, a TFT LCD (thin film transistor liquid crystal display) or an OLED (organic light emitting diode) display or other appropriate display technology. The display interface 2056 may include appropriate circuits for driving the display 2054 to present graphics and other information to the user. The control interface 2058 can receive commands from the user and convert them for submission to the processor 2052. In addition, an external interface 2062 in communication with the processor 2052 can be provided to enable near-area communication of the device 2050 with other devices. The external interface 2062 can, for example, be provided for wired communication in some embodiments, or for wireless communication in other embodiments, and multiple interfaces can also be used.

[0099] The memory 2064 stores information within the computing device 2050. The memory 2064 may be implemented as one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. The extended memory 2074 may also be provided and connected to the device 2050 via an extended interface 2072, which may include, for example, a SIMM (single in-line memory module) card interface. Such an extended memory 2074 may provide additional storage space for the device 2050, or may also store applications or other information for the device 2050. Specifically, the extended memory 2074 may include instructions for executing or supplementing the above-mentioned processes, and may also include security information. Therefore, for example, the extended memory 2074 may be provided as a security module for the device 2050, and may be programmed with instructions that allow the device 2050 to be used securely. In addition, secure applications and additional information may be provided via a SIMM card, such as placing identification information on a SIMM card in an unbreakable manner.

[0100] As described below, the memory may include, for example, flash memory and / or NVRAM memory. In one embodiment, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine readable medium, such as memory 2064, expansion memory 2074, or memory on processor 2052, which can be received, for example, via transceiver 2068 or external interface 2062.

[0101] Device 2050 can communicate wirelessly via communication interface 2066, which may include digital signal processing circuitry when necessary. Communication interface 2066 may provide for communication in various modes or protocols, such as GSM voice calls, SMS, EMS or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000 or GPRS, etc. Such communication may occur, for example, via radio frequency transceiver 2068. In addition, short-range communication may occur, such as using Bluetooth, Wi-Fi or other such transceivers (not shown). In addition, GPS (global positioning system) receiver module 2070 may provide additional navigation and location-related wireless data to device 2050, which may be used appropriately by applications running on device 2050.

[0102] Device 2050 may also communicate audibly using audio codec 2060, which may receive spoken information from a user and convert it into usable digital information. Audio codec 2060 may likewise generate audible sounds for the user, such as through a speaker, for example, in an earpiece of device 2050. Such sounds may include sounds from voice phone calls, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications operating on device 2050.

[0103] As shown in the figure, computing device 2050 can be implemented in many different forms. For example, it can be implemented as a cellular phone 2080. It can also be implemented as part of a smart phone 2082, a personal digital assistant, or other similar mobile device.

[0104] Various implementations of the systems and techniques described herein may be implemented in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system comprising at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to send data and instructions to a storage system, at least one input device, and at least one output device.

[0105] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine languages. As used herein, the terms "machine-readable medium," "computer-readable medium," refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0106] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) that the user can use to provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, voice, or tactile input.

[0107] The systems and techniques described herein may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer having a graphical user interface or a web browser that a user can use to interact with implementations of the systems and techniques described herein), or includes any combination of such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0108] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0109] A number of embodiments have been described, however, it will be appreciated that various modifications can be made without departing from the spirit and scope of the invention.

[0110] In addition, the logic flows depicted in the figures do not require the particular order shown or sequential order to achieve the desired results. In addition, other steps can be provided, or steps can be eliminated from the described flows, and other components can be added to the described systems, or other components can be removed from the described systems.

Claims

1. A method for displaying an augmented reality object, comprising: receiving a representation of a real-world scene captured by a user using a mobile device, the real-world scene being a portion of a real-world physical area; comparing the representation of the real-world scene to a portion of a model of the real-world physical area, the model of the real-world physical area being based on a real-world coordinate space; positioning the mobile device using an augmented reality (AR) anchor associated with the representation of the real-world scene, the mobile device at a location separate from the AR anchor, the AR anchor being associated with a coordinate space of the AR anchor that is independent of a coordinate space of the model of the real-world physical area, based on the comparison; as well as Display of an AR object associated with the model of the real-world physical area within a display of the mobile device is triggered based on coordination of the coordinate space of the real-world physical area, the coordinate space of the AR anchor, and the coordinate space of the AR object.

2. The method according to claim 1, further comprising: A position of the AR object relative to a position of the mobile device is determined based on the position of the AR object relative to a position of the AR anchor.

3. The method according to claim 1, wherein: The AR object has a fixed position relative to an origin at a fixed location within the model of the real-world physical area.

4. The method according to claim 1, wherein: The AR object has a fixed position relative to a position of the AR anchor, and the position of the AR anchor is fixed relative to an origin within the model of the real-world physical area.

5. The method according to claim 1, further comprising: A position of the AR object relative to a position of the mobile device is determined based on a position of the AR object relative to an origin within the model of the real-world physical area and a position of the AR anchor relative to the origin.

6. The method according to claim 1, wherein: Positioning the mobile device includes determining a distance of the mobile device from the AR anchor and a direction the mobile device is facing relative to the AR anchor.

7. The method according to claim 1, wherein: The real-world scene is a first real-world scene, the location is a first location, the AR anchor is a first AR anchor, The method further comprises: detecting movement of the mobile device from the first location to a second location; receiving a representation of a second real-world scene captured by the mobile device, the second real-world scene being from the real-world physical area; and The second position of the mobile device is associated with a second AR anchor based on a comparison of the representation of the second real-world scene with the model of the real-world physical area.

8. The method according to claim 1, wherein: The AR object is displayed within an image of a portion of the real-world physical area captured by the mobile device.

9. The method according to claim 1, wherein: The representation of the real-world scene is an image of at least a portion of a real-world physical area.

10. The method according to claim 1, wherein: The model is a 1:1 scale representation of the real-world physical area.

11. The method according to any one of claims 1 to 10, wherein: The AR object is at least one of a text object, an animation, and a model, and the AR object is displayed based on an orientation of the mobile device.

12. A method for displaying an augmented reality object, comprising: receiving a representation of a real-world scene captured by a mobile device, the real-world scene being a portion of a real-world physical area; localizing the mobile device using an AR anchor from a plurality of AR anchors based on a comparison of features of the real-world scene with features of a model map of the real-world physical area; identifying, via coordination of a coordinate space of the real-world physical area, a coordinate space of the AR anchor, and a coordinate space of the AR object, a position and an orientation of the AR object relative to the mobile device based on a fixed position of the AR object within the model map and a fixed position of the AR anchor within the model map, each of the plurality of AR anchors being at a position within the coordinate space of the AR anchor; and Triggering display of the AR object.

13. The method according to claim 12, further comprising: The position of the AR object relative to the position of the mobile device is determined based on a fixed position of the AR object relative to an origin within the model map of the real-world physical area and a fixed position of the AR anchor relative to the origin.

14. The method according to claim 12, wherein: Positioning the mobile device includes determining a distance of the mobile device from the AR anchor and a direction the mobile device is facing relative to the AR anchor.

15. The method according to any one of claims 12 to 14, wherein: The AR object is displayed within an image of a portion of the real-world physical area captured by the mobile device.

16. The method according to claim 12, wherein: A fixed position of the AR anchor is separated from a fixed position of the AR object.

17. A computer readable memory storing instructions which, when executed by one or more processor components, cause the one or more processor components to perform the method of any one of claims 1-16.

18. A system for displaying an augmented reality object, comprising one or more processor components and one or more memories storing instructions, which when executed by the one or more processor components cause the one or more processor components to perform the method of any one of claims 1 to 16.

Images