Mobile device navigation system

Abstract

The invention relates to a mobile device navigation system. More specifically, a location mapping and navigation user interface may be generated and presented via a mobile computing device. A mobile device may use an internal system to detect its position and orientation, and may use a device camera to capture image data. The mobile device may also retrieve map information from a map server corresponding to the current location of the device. Using image data captured at the device, current location data, and corresponding local map information, the mobile device may determine or update a current orientation reading of the device. Location errors and updated location data may also be determined for the device, and a map user interface may be generated and displayed on the mobile device using the updated device orientation and / or location data.

Description

[0001] This application is a divisional application of Chinese patent application 202080064280.6, with the application date of July 28, 2020 and the invention name of “Mobile Device Navigation System”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of U.S. Non-Provisional Patent Application No. 16 / 576,673, filed September 19, 2019, the entire contents of which are hereby incorporated by reference for all purposes. Technical Field

[0004] The present disclosure relates to a mobile device navigation system. Background Art

[0005] With mobile positioning technology and the added capabilities of modern mobile computing devices, these mobile devices are often used as mapping and navigation tools. A mobile device (such as a smart phone, smart watch, or other wearable computing device) can use a global positioning system (GPS) receiver or other mobile positioning technology to identify its current coordinates, and can then view and track its current location via a map application using map data received from a map server. Navigation capabilities can also be provided via a mobile device, where the mobile device and / or server navigation components can determine a route between the current location and a selected destination. The navigation system can use optimization algorithms to select the best route for the user based on road segment data, traffic data, and the user's selected mode of transportation (e.g., walking, biking, driving, etc.).

[0006] For mapping and navigation capabilities, the mobile device can provide a user interface that displays various location and navigation data. The map user interface can include the user's current location, orientation, street address, and nearby landmarks. The navigation user interface can provide turn-by-turn navigation to the destination and can indicate the distance and direction of the next turn. The mapping and navigation user interface can also support a variety of different views, including an overhead (or bird's-eye) view with map data and / or satellite imagery, a turn-by-turn navigation view, a street view, etc.

[0007] However, when a mobile device supports location mapping and navigation capabilities, these capabilities may be vulnerable to any errors in the location or sensor data detected at the device. For example, even relatively small errors in device location data, device orientation data, or device movement data generated by a mobile device may result in significant mapping and navigation errors, as well as user confusion and disorientation. Summary of the invention

[0008] This article describes the technology for generating and presenting user interfaces via mobile computing devices. In various embodiments, the user interface may correspond to a map user interface and / or a navigation user interface, and / or may be configured to provide other augmented reality-based applications for georeferenced enhanced content. A mobile device (such as a smart phone, tablet computer, or smartwatch) may receive data from various sensors or hardware components of the mobile device, including device location and orientation data, and image data captured by a device camera. The mobile device may also, for example, receive map information corresponding to the current location of the device from a geographic service module of a map data server. Using the image data captured at the device, the current location data, and the map information corresponding to the current location, the mobile device may determine the current orientation of the mobile device. For example, the captured image data may be analyzed to detect street or sidewalk boundaries, buildings, landmarks, street perspective vanishing points, horizontal lines, etc., and these detected objects may be compared with corresponding local map data. In some cases, the determined orientation data may be used to update a previous device orientation reading obtained by an internal compass or other device orientation system. The internal compass or orientation system of the mobile device may also be reconfigured or recalibrated based on the determined orientation in some cases. Similar techniques may be used to determine updated location data for a mobile device, detect errors in data provided by a location / positioning system of a mobile device, and reconfigure or recalibrate the location / positioning system of the device as needed.

[0009] Additional aspects described herein relate to generating and displaying a user interface (e.g., a map and / or pedestrian navigation user interface) on a mobile device using updated device orientation and / or location data. In some embodiments, an augmented reality (AR) navigation user interface can be generated and rendered by superimposing a set of navigation user interface components corresponding to the current position and orientation of the device onto the current image data captured by the device camera. An AR-based user interface can be generated based on device orientation and / or location data determined (or updated) based on analysis of captured image data and corresponding local map data. Generating an AR-based map user interface, street view, or turn-by-turn navigation view may include generating and rendering labels (e.g., as AR overlays) onto streets, buildings, landmarks, and the like based on determined location and orientation data. In some embodiments, an AR-based map user interface may be designed as a pedestrian user interface, including turn-by-turn walking navigation instructions, superimposed on the current forward image data captured by the device.

[0010] For example, some embodiments relate to a method of generating a map user interface via a computing device. Such methods may include: receiving, by a computing device, image data corresponding to one or more images captured by a camera of the computing device; receiving location information corresponding to a location of the computing device; receiving map information corresponding to the location of the computing device, determining the orientation of the computing device based at least in part on: (a) image data captured by a camera of the computing device, (b) location information corresponding to the location of the computing device, and (c) map information corresponding to the location; generating a map user interface configured to display a current location of the computing device and one or more user interface elements representing one or more objects within a threshold distance of the current location of the computing device, wherein generating the map user interface includes selecting and positioning one or more objects within the map user interface, and wherein at least one of the selection or positioning of the objects within the map user interface is based at least in part on the determined orientation of the computing device; and presenting the map user interface on a screen of the computing device. In further embodiments, such methods may also include receiving a direction reading from a compass of the computing device, wherein determining the orientation of the computing device is further based at least in part on the direction reading received from the compass. In addition, such methods may include reconfiguring the compass of the computing device based at least in part on the determined orientation of the computing device. In some embodiments, the location information corresponding to the location of the computing device may include at least one of the following: data received from a global positioning system (GPS) receiver of the computing device, data identifying one or more WiFi networks accessible to the computing device, or data identifying one or more additional devices that communicate with the computing device via a short-range wireless communication protocol. In certain embodiments of such methods, the location information may correspond to the location of the computing device and include coordinate data received from a global positioning system (GPS) receiver of the computing device, and the methods may also include: determining updated location information corresponding to the location of the computing device based at least in part on: (a) image data captured by a camera of the computing device, (b) coordinate data received from a GPS receiver of the computing device, and (c) map information corresponding to the location; and reconfiguring the GPS receiver of the computing device based at least in part on the determined updated location information. In other embodiments of such methods, determining the orientation of the computing device may also include: analyzing image data captured by a camera of the computing device; detecting one or more street boundaries within the image data; and comparing one or more trajectories of the one or more street boundaries with street vector data within the received map information. In other embodiments of such methods, determining the orientation of the computing device may also include analyzing image data captured by a camera of the computing device and detecting horizontal lines within the image data.In other embodiments, such methods may include generating a route for the computing device from a current location of the computing device to a second location, wherein the map user interface displays next direction instructions based at least in part on the route.

[0011] These and other embodiments of the present disclosure are described in detail below.For example, other embodiments relate to systems, devices and computer-readable media including computer-executable instructions for executing all methods described herein.

[0012] The nature and advantages of embodiments of the present invention may be better understood by referring to the following detailed description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a diagram showing a mobile device of a pedestrian displaying a navigation user interface according to an embodiment of the present disclosure.

[0014] Figure 2 is a block diagram illustrating components of a mobile device navigation system according to an embodiment of the present disclosure.

[0015] Figure 3 is another block diagram illustrating interfaces and components of a mobile device navigation system according to an embodiment of the present disclosure.

[0016] Figure 4 is a flow chart illustrating a method for generating and displaying a navigation user interface via a mobile computing device according to an embodiment of the present disclosure.

[0017] Figures 5 to 7 is a diagram illustrating an exemplary display screen of a navigation user interface according to an embodiment of the present disclosure.

[0018] Figure 8 is a block diagram illustrating a user device and a service provider computer of a mobile device navigation system according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] Various aspects of the present disclosure provide various techniques (e.g., methods, systems, devices, computer-readable media storing computer-executable instructions for performing computing functions, etc.) for determining the orientation and / or precise location of a mobile device using image data captured via a camera of a mobile device and local map data (e.g., map tiles received from a map server or a geographic service module). In some cases, the determined orientation data and location data can be used to update a previous device orientation reading or device location determination obtained by the device's internal orientation or location / positioning system. In some cases, the internal orientation or location / positioning system of the mobile device can also be reconfigured or recalibrated based on the updated orientation and location readings.

[0020] Additional aspects of the present disclosure provide techniques for generating and displaying maps and navigation user interfaces on a mobile device using updated device orientation and / or location data. In some embodiments, an augmented reality (AR) navigation user interface can be generated and rendered by overlaying a set of navigation user interface components corresponding to the current location and orientation of the device onto current image data captured by a device camera.

[0021] Figure 1 1 is a diagram of a mobile device showing a navigation user interface for a pedestrian according to certain embodiments of the present disclosure. In this example, a pedestrian user is holding a mobile device 110, which is currently displaying a user interface 115 of a mobile navigation application. Although a simplified user interface 115 is shown in this example for clarity, the mobile navigation application can display a map and navigation graphical user interface component, such as a map of a general area near the user, including nearby streets, buildings, and landmarks, and an identification of the user's current location on the map. The embodiments described herein can be applied to both a map view user interface (e.g., showing the user's current location on a map) and a navigation user interface (e.g., showing the user's current location, a selected destination, and / or directions to guide the user to a location). A mobile map application (which may also be referred to as a mobile navigation application when configured with a navigation function) can retrieve map data from a map data repository in a remote map server 120. The mobile map and / or navigation application can display map elements (e.g., streets, buildings, landmarks, etc.) as graphical components and geometric shapes, and / or by using actual image data within the map displayed in the user interface. For example, the mobile navigation application may be configured to receive and display satellite image data or previously captured street view image data for map rendering. In other examples, the mobile navigation application may be configured to display an augmented reality user interface, where real-time image data is captured via a camera on the mobile device 110 and is superimposed with labels and other graphical components to provide navigation functionality to the user. The mobile navigation application may also support different user interface views for the user when being guided to a destination, such as an overhead (or bird's eye) view, a turn-by-turn navigation view, a street view, and an augmented reality view.

[0022] In this example, Figure 1The mobile navigation application shown in can be configured as a pedestrian navigation system, which can differ from a vehicle-based navigation system in several respects. For example, some vehicle-based navigation systems may not detect or use the orientation of the device to generate user interface displays. In contrast, a vehicle-based GPS navigation system or a smartphone executing a "vehicle mode" of the mobile navigation application can assume that the orientation of the display should correspond to the direction of travel of the vehicle. Thus, if the smartphone executing the "vehicle mode" of the mobile navigation application turns while the vehicle is moving forward, there is no need to rotate the user interface to reflect the change in orientation of the device. However, in Figure 1 In the illustrated pedestrian navigation system, a change in the orientation of the mobile device 110 may indicate that the pedestrian has turned or changed direction, and thus the user interface 115 should be changed to reflect the new device orientation.

[0023] Additionally, unlike some vehicle-based navigation systems, the current location of mobile device 110 in a pedestrian navigation system may not be assumed to be on a road or highway, or even on a sidewalk or path. Thus, the pedestrian navigation system may generate different displays for user interface 115 based on even small differences in the location of mobile device 110. For example, the pedestrian navigation system may display different data within user interface 115 depending on which side of the street the user is walking on, or when the user is crossing a parking lot, path, or field, etc.

[0024] The pedestrian navigation system may also not assume that the user is at street level when generating the user interface 115, but can detect when the user is below or above street level (e.g., in a subway tunnel, on a balcony or rooftop). Thus, in some embodiments, the pedestrian navigation system can determine whether the mobile device 110 is at street level, above street level, or below street level, and can generate the user interface 115 accordingly.

[0025] Figure 2is a block diagram showing the components of a mobile device navigation system according to one or more embodiments of the present disclosure. The mobile device navigation system 200 in this example may correspond to a pedestrian navigation system configured to provide location mapping and navigation functions to a pedestrian user via the user's mobile device 110. The navigation system 200 includes a map / navigation mobile application 210, an augmented reality system 220, a media system 230, a map data repository 240, a location system (or mobile positioning system) 205, and a motion system 260. The different components and systems 210-260 of the mobile device navigation system 200 may be implemented within the mobile device 110, within the computer server 120, and / or within one or more third-party systems as described herein. For example, in some embodiments, the mobile application 210, the augmented reality system 220, the media system 230, the location system 250, and the motion system 260 may be implemented within the user's mobile device 110, and the map data repository 240 may be provided by the server 120.

[0026] The map / navigation application 210 may correspond to a mobile application installed on the user's mobile device 110. The application 210 may be configured to display map data and provide various location mapping and navigation functions via the user interface 120. For example, in response to opening or launching the application 210 on the mobile device 110, the application 210 may display a map user interface having elements corresponding to the user's current location and provide directions to a selected destination. In some embodiments, the map / navigation application 210 may be configured to provide an augmented reality (AR) user interface in which a map / navigation component is generated and superimposed on real-time image data captured via a camera on the mobile device 110. Thus, the application 210 may include a turn-by-turn route planning engine, as well as an augmented reality overlay component as shown in this example.

[0027] The augmented reality system 220 may include components configured to generate a map / navigation component to be overlaid within an AR navigation user interface on the mobile device 110. In this example, the augmented reality system 220 includes a semantic image segmentation component, an image-based heading estimation component, a long-baseline visual inertial odometry (VIO) component, and a geo-referenced AR content provider.

[0028] The media system 230 includes one or more device cameras that capture image data for use in the AR navigation user interface. The AR system 220 can send a request to the media system 230 to activate the front camera of the mobile device 110 during the time periods when the map / navigation application 210 is in use and is configured to display the AR navigation interface. During these time periods, the media system 230 can continuously activate the front camera and provide the captured image data to the AR system 220 for use in the AR navigation user interface.

[0029] The map data repository 240, which may be implemented in a map server 120 separate from the mobile device 110, includes a database of map tiles. A map tile contains a collection of object identifiers, attributes, and geographic coordinates representing physical objects (e.g., buildings, streets, natural landmarks, etc.) within a particular area. When using the map / navigation application 210, the AR system 220 may request the appropriate map tile from the map data repository 240 based on the current location of the mobile device 110.

[0030] The location / positioning system 250 can be configured with one or more location subsystems that are configured to determine the location and orientation of the mobile device 110. In this example, the location / positioning system 250 includes a location estimator, a VIO stable position component, a compass / camera heading arbitration component, and a pedestrian map matching component. The location / positioning system 250 can be configured to provide device location data when requested by the map / navigation application 210. In some cases, the device positioning system 250 may include a GPS receiver or other mobile positioning technology configured to detect the current geographic coordinates of the mobile device 110. Additionally or alternatively, the location / positioning system 250 can be configured to detect wireless networks, wireless access points, and / or other nearby devices (e.g., known Bluetooth devices, NFC tags, etc.) from which the location of the mobile device 110 can be determined.

[0031] The location / positioning system 250 may also determine the current device orientation in response to a request from the map / navigation application 210. In some cases, the device orientation may be determined by an internal compass of the mobile device 110. In other cases, triangulation may be used to calculate the device orientation by detecting differences in arrival times of signals at different antennas of the mobile device 110.

[0032] The motion system 260 may include motion sensors, such as accelerometers and gyroscopes. In some embodiments, the motion system 260 may be configured to provide device motion data to the AR system 220 and / or to the position / positioning system 250. For example, motion data may be provided from the motion system 260 to the AR system 220 for use by the AR system 220 in generating AR content. In addition, the motion system 260 may be used to detect device position and / or device orientation based on motion sensors such as accelerometers and gyroscopes within the mobile device 110. For example, assuming that the initial position or orientation of the mobile device 110 is known, monitoring and tracking device acceleration data and / or gyroscope data in three dimensions may allow the mobile device 110 to calculate an updated position or an updated orientation.

[0033] Figure 33 is another block diagram illustrating interfaces and components of a mobile device navigation system according to an embodiment of the present disclosure. The mobile device navigation system 300 shown in this example can be similar or identical to the navigation system 200 discussed above, with additional details illustrating additional components and communication paths between the system and components. In other embodiments, different from the navigation system 200, the mobile device navigation system 300 can correspond to a separate specific implementation of a pedestrian navigation system.

[0034] Figure 4 1 is a flow chart illustrating an exemplary process of generating and displaying a navigation user interface via a mobile computing device according to an embodiment of the present disclosure. As described below, the steps in the process may be performed by a user device 110, such as a smart phone or other mobile device executing a mobile device navigation system. Therefore, reference may be made to the above-mentioned Figures 1 to 3 However, it should be understood that the process of generating and displaying the navigation user interface described herein is not limited to the specific computing environment and system described above, but can be performed in various other computing environments and systems.

[0035] In step 401, the mobile device navigation system 200 receives imaging data captured by one or more cameras 230 of the mobile device 100. In some embodiments, the capturing of image data in step 401 can be performed in response to a user launching or initiating the mobile navigation application 210. For example, the mobile application 210 or the augmented reality system 220 can be configured to activate one or more device cameras in response to the mobile application 210 being opened or launched, or in response to a user action made via the user interface of the mobile application 210 (e.g., the user initiating a pedestrian navigation mode). In other examples, the device camera can be configured to continuously capture data during operation of the mobile device 110, in which case the AR system 220 can receive a continuous stream of image data before the mobile application 210 is opened or launched by the user.

[0036] The image data received in step 401 may correspond to digital images and / or video data captured via one or more cameras of the mobile device. In some cases, it may be advantageous to maximize the amount of image data received in step 401. As described below, maximizing the amount of image data collected in step 401 may allow objects represented in the image data to be more easily matched with corresponding map data, and may also provide more overall image data for analysis when determining device orientation and / or location. Therefore, in some embodiments, receiving image data in step 401 may include activating multiple cameras on the mobile device 110, such as a front camera, a camera facing the user, and / or any other available camera, to maximize the amount of image data collected. In some cases, a wide-angle or panoramic camera mode may also be used to increase the angle at which image data is collected in step 401. In addition, in some embodiments, the image data received in step 401 may be collected by one or more separate computing devices at the same location and then transmitted to the mobile device 110. For example, a navigation system executed on a user's first mobile device 110 (e.g., a smart phone) may activate a camera on one or more other devices of the user's other devices (such as a second smart phone, a smart watch, or other wearable device with a camera, etc.). Thus, the other user devices may collect additional image data and may transmit the image data back to the first mobile device 110 that is executing the navigation system.

[0037] In step 402, the mobile device navigation system 200 can receive and / or determine the current location data of the mobile device 110. In some embodiments, in response to the user opening or activating the map / navigation mobile application 210, the mobile device 110 can activate an internal standard positioning service (SPS) receiver, such as a GPS receiver or other mobile positioning system. One or more SPS receivers on the mobile device 110 may be able to use an SPS antenna to receive signals from one or more SPS satellites. In various embodiments, the SPS receiver on the mobile device 110 can support measurements of signals from spacecraft (SV) of SPS systems such as global navigation satellite systems (GNSS) (e.g., global positioning system (GPS)), Galileo systems, GLONASS systems, quasi-zenith satellite systems (QZSS) over Japan, Indian regional navigation satellite systems (IRNSS) over India, Beidou systems over China, etc. In addition, the SPS receiver can be used with various augmentation systems (e.g., satellite-based augmentation systems (SBAS)), which can be associated with one or more global or regional navigation satellite systems or can be used with one or more global or regional navigation satellite systems in other ways. By way of example and not limitation, SBAS may include augmentation systems that provide integrity information, differential corrections, etc., such as, for example, Wide Area Augmentation System (WAAS), European Geosynchronous Navigation Overlay Service (EGNOS), Multipurpose Satellite Augmentation System (MSAS), GPS-Assisted Geographic Augmentation Navigation or GPS and Geographic Augmentation Navigation System (GAGAN), etc. Thus, as used herein, SPS may include any combination of one or more global or regional navigation satellite systems or augmentation systems, and SPS signals may include SPS, SPS-like, or other signals associated with such one or more SPS.

[0038] Other techniques for receiving the current location of the mobile device 110 in step 402 may include determining the current device location based on the device's connection to one or more additional devices via a short-range wireless communication protocol (including any wireless networks and / or wireless access points to which the mobile device 110 is currently connected). For example, during use of a map / navigation mobile application, the mobile device 110 may be connected to one or more wireless networks through one or more wireless access points or other network components. Wireless access points may include devices such as wireless routers, modems, switches, mobile hotspots, set-top boxes, etc., which allow the mobile device 130 to wirelessly connect and communicate with a local area network ("LAN"), a wide area network ("WAN"). Certain wireless networks and / or wireless access points may be associated with certain geographic locations (e.g., home, workplace, a WiFi network provided by a business, etc.), and thus the mobile device 110 may determine its current geographic location based on the set of wireless networks and wireless access points currently available for connection by the mobile device 110.

[0039] The various techniques used to determine the current location of the mobile device 110 in step 402 may have different levels of accuracy and reliability. For example, the GPS Standard Positioning Service (SPS) and the GPS Precise Positioning Service (PPS) may provide different levels of accuracy, and the accuracy of receivers of either standard may be affected by signal problems and satellite line-of-sight obstructions, measurement biases, and random errors. In addition, in some embodiments, determining the location of the mobile device 110 based on wireless networks and wireless access points detected by the mobile device 110 may not be as accurate as using GPS. For example, signals from known wireless access points may only allow the mobile device 110 to be located within a few hundred feet, depending on the transmission range of the wireless access point. Therefore, it should be understood that the following steps can be customized based on the accuracy and reliability of the location determination in step 402. For example, when the location determination in step 402 is less accurate or more likely to be error-prone, map information for a larger geographic range can be retrieved (step 403).

[0040] In step 403, the mobile device navigation system 200 may receive map information for an area at or near the current location of the mobile device 110. In some embodiments, the location / positioning system 250 of the mobile device 110 may request map information (e.g., one or more map tiles) corresponding to the current location and nearby area of ​​the mobile device 110 from the map server 120. The map server 120 may include one or more map data repositories 240 storing directories of map tiles or other map data files / structures representing different map areas. In some cases, a single map tile may represent an area of ​​one square kilometer of land. In some cases, in response to a user opening or initiating a map / navigation application 210 on the mobile device 110, map information (e.g., one or more map tiles) may be retrieved in step 403. In other cases, the location system 250 may be configured to periodically or continuously track the current location of the mobile device 110 and request new surrounding map information from the map server 120 as needed.

[0041] The map information received in step 403 may include a map tile or other similarly formatted map data file or structure that includes a collection of geographic coordinates identifying physical objects within the area covered by the map tile. For example, a map tile covering a particular area may include coordinate data that defines the precise size, shape, and boundaries of all streets, paths, buildings, natural or man-made landmarks, and any other fixed objects in the area. In addition to geographic coordinates, the map tile may also contain data identifying each unique object and its type (e.g., street, path, building, body of water, other natural landmark, etc.), object name or label data, and other object attributes. The map tile or other map information received in step 403 may include any and all data required for the map / navigation application 210 to render a map graphical user interface for the area.

[0042] Additionally or alternatively, the map information received in step 403 may include image data of the current location of the mobile device 110 or its vicinity. For example, the location system 250 may retrieve satellite image data or previously captured street view image data from one or more external servers 120 corresponding to the current location and surrounding area. As another example, the map information received in step 403 may include a directory of physical object images (e.g., buildings, landmarks, etc.) and known geographic coordinates of the physical objects. As discussed in more detail below, the map information received in step 403 may be analyzed and compared with the image data captured at the mobile device 110. Therefore, when non-image map information (e.g., map tiles) is received in step 403, the comparison with the captured image data can be based on a comparison with the size, shape, etc. of the object. However, when map image data is received in step 403, the comparison with the image data captured at the mobile device 110 may include image matching, etc.

[0043] In some embodiments, map information may be received from a map server 120 without always providing the precise location of the mobile device 110 to the map server 120. For example, the location / positioning system 250 of the mobile device 110 may request and receive one or more map tiles covering a relatively large area without always transmitting the location coordinates of the mobile device 110 to the map server 120. Such embodiments may provide technical advantages with respect to protecting privacy and security, including fully supporting the map / navigation functions described herein (e.g., pedestrian navigation systems, AR-based map / navigation user interfaces, etc.) without requiring the mobile device 110 to transmit its location to any external server 120.

[0044] In step 404, the mobile device navigation system 200 can determine the current orientation of the mobile device 110 based on an analysis of the image data captured at the mobile device 110, the current location of the mobile device 110, and / or the map information received for the current location of the mobile device 110. In some embodiments, the orientation determination can be performed by the image-based heading estimation component 222 within the AR system 220 executed on the mobile device 110. For example, the image-based heading estimation component 222 can initially draw the current location of the mobile device 110 within the map information received in step 403. The heading estimation component 222 can then use the map information to determine one or more physical objects that are currently (or should be) near the mobile device 110. The heading estimation component 222 can then compare the image data captured at the mobile device 110 in step 401 with the characteristics of the physical objects identified in the map data at or near the current location. In some embodiments, such comparisons can include analyzing the image data captured at the mobile device 110 to detect line segments, shapes, and other recognizable objects, and then comparing these objects with nearby physical objects identified in the map data. As described above, image matching comparisons may also be performed between image data captured at the mobile device 110 and map data images within map information received from one or more map servers 120 (e.g., previous street view images, satellite images, image libraries, etc.).

[0045] The analysis performed in step 404 may include detecting any discernible objects within the image data captured at the mobile device 110; determining properties of such objects; and then attempting to match the objects to nearby physical objects identified in the map data. In various examples, object matching may be based on object shape, size, color, relationships between objects, or any other detectable pattern within the image data.

[0046] In some examples, the image-based heading estimation component 222 can analyze the image data captured by the mobile device 110 to detect streets, sidewalks, or walking / biking paths within the captured image data. These objects can be detected by identifying parallel or nearly parallel line segments corresponding to street boundaries, curbs, sidewalk lines or boundaries, and / or drawn lines within the street. Based on the direction (or trajectory) of the line segments, the estimation component 222 can determine the orientation of the mobile device 110 relative to the street, sidewalk, or path (or relative to a particular segment of the street, sidewalk, or path). Based on the angle or distance between the parallel or nearly parallel line segments, the estimation component 222 can also determine the distance between the mobile device 110 and the street, sidewalk, or path, etc. The estimation component 222 can then identify the corresponding street, sidewalk, or path in the map information, for example, by comparing the location, direction, width, curvature, or other distinguishing characteristics of the street, sidewalk, or path. Finally, the estimation component 222 can determine the current orientation of the mobile device 110 based on (1) the direction (or street vector) of the street, sidewalk, or path segment stored in the map information, and (2) the relative orientation of the mobile device 110 to the street, sidewalk, or path segment, which can be determined by analyzing the captured image data. In some embodiments, the heading estimation component 222 can also analyze the image to detect the vanishing point of parallel or nearly parallel street boundaries, sidewalk lines, or railroad tracks, etc., and can compare the direction of the vanishing point from the perspective of the mobile device 110 with the direction of the street, path, railroad track, etc. in the map information to determine the current orientation of the mobile device 110.

[0047] In other examples, image-based heading estimation component 222 can analyze image data captured by mobile device 110 to detect buildings or other landmarks, and can compare the buildings or landmarks to corresponding objects in the map information to determine the current orientation of mobile device 110. For example, estimation component 222 can identify characteristics (e.g., size, shape, location, or orientation) of nearby buildings or landmarks in order to match the buildings or landmarks to corresponding objects within the map information. After matching the buildings or landmarks to objects within the map information, heading estimation component 222 can determine the current orientation of mobile device 110 based on (1) the location and orientation of the object stored in the map information, and (2) the relative orientation of mobile device 110 to the object (or at least one surface of the object) within the captured image data.

[0048] In addition to matching nearby buildings or other landmarks with corresponding objects within the map information, the heading estimation component 222 can also identify distant objects within the image data captured by the device, as long as the location and / or heading of these distant objects is known or can be determined. For example, the heading estimation component 222 can identify the sun, moon, horizon, or other distant landmarks (e.g., mountains, bodies of water, radio towers, aircraft warning beacons, etc.) within the image data captured by the mobile device 210 to determine the current orientation of the mobile device using techniques similar to those described above. Regardless of the type of physical object used to orient the mobile device in 210, the steps of the process can include: matching the physical objects within the image data captured by the mobile device 110 with the corresponding objects in the received map information; determining the absolute orientation of the object from the received map information; and then determining the current orientation of the mobile device based on the absolute orientation of the object and the relative orientation of the mobile device 110 with respect to the object.

[0049] In some embodiments, data from an internal compass of the mobile device 110 may also be used for device orientation determination in step 404. As described above, compass data from the mobile device 110 may be inaccurate or unreliable in some circumstances. However, the heading estimation component 222 can potentially use the compass data in a limited manner, such as to determine the general direction that the device is pointing (e.g., within a range of 90 degrees or 180 degrees, etc.), which can be used as a starting point for the image analysis and object comparison process.

[0050] Although the above examples involve determining the current orientation of the mobile device 110, similar techniques may be used to determine the current location of the mobile device 110. As described above, the location data (or location information) received via the location / positioning system 250 of the mobile device 110 may be inaccurate or imprecise. For example, based on signal interference, erroneous readings, or design limitations of the location / positioning technology, the current device location received in step 402 may be inaccurate, with errors ranging from a few feet to half a mile or more. In these cases, the current location data determined in step 402 may be accurate enough to allow the mobile device navigation system 200 to request the appropriate map tiles / map information, but may not be accurate enough for the map / navigation mobile application 210.

[0051] Thus, step 404 may also include performing similar image processing and object comparison steps to determine the current location of the mobile device 110. For example, a location estimator 252 within the location / positioning system 250 and / or within the AR system 220 may match one or more physical objects within the image data captured by the mobile device 110 with corresponding objects in the received map information. The location estimator 252 may then determine or estimate the distance between the mobile device 110 and the one or more physical objects, for example by calculating the straight-line distance between the mobile device 110 and the object using the size of the object and the angle between the edges of the object. The current location of the mobile device 110 may then be calculated using the absolute location of the object stored in the map information and the distance between the mobile device 110 and the same object. For example, the absolute location of the mobile device 110 may be determined by using triangulation of the distances to two or more objects, or based on the determined distances and orientations to one or more objects.

[0052] In some embodiments, the mobile device navigation system 200 can use techniques similar to those described above for determining device orientation and / or location to determine when the mobile device 110 is at street level, above street level, or below street level. For example, analysis of image data captured by the mobile device 110 can include detecting upper or lower corners relative to nearby physical objects. When the map information confirms that those physical objects are at street level, the pedestrian navigation system 200 can determine the vertical positioning of the mobile device 110 above or below street level.

[0053] In step 405, components of the mobile device navigation system 200 may use the determination of the device orientation and device location made in step 404 to select and position user interface elements to be presented via the map / navigation application 210. In some embodiments, the generation of the user interface for the map / navigation application 210 may be performed entirely by the map / navigation application 210 itself. In other embodiments, for example, where an augmented reality (AR)-based pedestrian navigation system is provided via the map / navigation application 210, a separate AR system 220 may then be used to select and position user interface elements to be presented via the AR-based interface.

[0054] As described above, when the map / navigation application 210 is a pedestrian navigation application, or can be configured to operate in a pedestrian navigation mode (as opposed to when operating in a driving navigation mode), the orientation of the mobile device 110 can determine the display generated for the user interface. For example, if a pedestrian using the navigation application 210 turns the mobile device 110 while remaining in the same location, the user interface can be updated to display a new set of streets, buildings, and other objects that are now in front of the mobile device 110. Pedestrian navigation systems that update displays based on the orientation of the mobile device 110 can include both augmented reality (AR) navigation systems and graphical non-AR navigation systems. The AR pedestrian navigation system can display real-time image data captured from the front camera of the mobile device 110, which is superimposed with labels and other graphical components to provide navigation functionality to the user.

[0055] Both the selection of graphical components (e.g., graphical streets and objects, labels, direction windows, turn indicators, etc.) and the specific positioning of the graphical components can be based on the device orientation and / or position data determined in step 404. Initially, based on the device position and orientation, the app / navigation mobile application 210 and / or the AR system 220 can determine which physical objects are directly in front of and in the field of view of the mobile device 110. For fully graphical (non-AR) user interfaces, these physical objects (e.g., streets, buildings, natural landmarks, etc.), the mobile application 210 can determine the size, position, and screen position of these components to correspond to the user's current viewing angle. For both AR and non-AR user interfaces, labels can be generated for each physical object rendered in the user interface, and the size and arrangement of the labels can be selected to match the size and orientation of the rendered object. For AR-based pedestrian navigation applications, the camera image data displayed in the AR interface can be analyzed to identify the specific physical objects (e.g., streets, buildings, etc.) currently shown in the image, and the AR system 220 can generate object labels to be rendered on the screen at the appropriate size, arrangement position, and orientation on the image of the object.

[0056] In addition to marking physical objects shown in the interface, the map / navigation application 210 can also provide map or navigation-related functions with additional components, including displaying the user's current location, destination location and estimated arrival time, turn-by-turn navigation instructions, etc. In some embodiments, the mobile application 210 and / or the AR system 220 can determine the size and arrangement of the graphical user interface components based on the representation of the physical objects within the user interface. For example, direction instruction windows, arrows, etc. can be positioned on the screen and appropriately resized to avoid obstructing street signs, traffic lights, oncoming moving objects, etc. In addition, the AR system 220 can also select contrasting colors to be used for marking objects and for other user interface components to be covered in the AR-based pedestrian navigation user interface. For example, during the use of the pedestrian navigation system at night, street and building labels, arrows, and direction instructions can be rendered in light colors.

[0057] In step 406, a map / navigation user interface may be generated by the mobile application 110 and rendered on the display screen of the mobile device 110. As described above, the various techniques described herein may be applied to location mapping-specific applications as well as navigation applications, and to augmented reality user interfaces and graphical non-AR user interfaces. Thus, the map / navigation user interface generated and rendered in step 406 may include map-specific and / or navigation-specific user interface elements as appropriate, and may or may not be rendered as an augmented reality user interface. For example, the techniques described herein may also be applied to any augmented reality application to allow the application to provide georeferenced augmented reality content.

[0058] Figures 5 to 7 Three exemplary display screens of a navigation user interface according to an embodiment of the present disclosure are shown. Figure 5 , mobile device 110 is shown rendering an augmented reality user interface 500 generated by a pedestrian navigation system. In this example, augmented reality user interface 500 includes an AR window 510 displaying real-time image data captured via a front-facing camera of the mobile device, and an overlay window 515 having a graphical component that displays current directional instructions 516 for a user 516, as well as an overhead map view 517 of the user's current location and route. As shown in this example, the current directional instruction (i.e., "Turn") has been selected and displayed based on the user's current orientation and location. Figure 6 , the user has turned and the user interface 600 has been updated based on the new orientation detected for the mobile device 110. In this example, the overhead map view 617 has been updated to reflect the orientation, and the turning instructions have been replaced with a separate graphical window 616 that notifies the user of the upcoming turn based on the current route. Figure 7 , the pedestrian is approaching the destination, and both the overhead map view 717 and the direction instruction window 716 have been updated accordingly.

[0059] Figure 8 Components of a pedestrian navigation system 800 are shown in accordance with at least one embodiment. The system 800 can include a user device 802 and / or a service provider computer 804 that can communicate with each other via a network 806 using any suitable communication protocol.

[0060] In some examples, the network 806 may include any one or a combination of many different types of networks, such as a wired network, the Internet, a wireless network, a cellular network, other private networks, and / or a public network. Although the illustrated example shows that the user device 802 communicates with the service provider computer 804 through the network 806, the techniques are equally applicable to instances in which the user device 802 interacts with the service provider computer 804 through a landline, via a kiosk, or in any other suitable manner. It should be understood that the techniques are applicable to other client / server arrangements as well as non-client / server arrangements (e.g., locally stored applications, etc.). For example, in some embodiments, the geographic service module 854 and / or the navigation module 856 discussed in more detail below may operate in whole or in part on the user device 802. In addition, the user device 802 can access the functionality of the geographic service module 854 and / or the navigation module 856 through components of the user device 802 (e.g., the map application module 804), and / or access the service provider computer 804 via a user interface and / or API provided by the geographic service module 854 and / or the navigation module 856.

[0061] As described above, the user device 802 may be configured to execute or otherwise manage a user interface for presenting map data and pedestrian navigation data using an augmented reality overlay (e.g., Figures 5 to 7 The user device 802 may be any type of computing device, such as, but not limited to, a mobile phone (e.g., a smart phone), a tablet computer, a personal digital assistant (PDA), a laptop computer, a desktop computer, a thin client device, a smart watch, a wireless headset, etc.

[0062] In an exemplary configuration, the user device 802 may include at least one memory 820 and one or more processing units (or processors) 822. The processor 822 may be suitably implemented in hardware, computer executable instructions, or a combination thereof. The computer executable instructions or firmware implementation of the processor 822 may include computer executable instructions or machine executable instructions written in any suitable programming language to perform the various functions described.

[0063] The memory 820 can store program instructions that can be loaded and executed on the processor 822 and data generated during the execution of these programs. Depending on the configuration and type of the user device 802, the memory 820 may be a volatile memory (such as a random access memory (RAM)) and / or a non-volatile memory (such as a read-only memory (ROM), a flash memory, etc.). The user device 802 may also include additional removable storage devices and / or non-removable storage devices 824, including but not limited to magnetic storage devices, optical disks and / or tape storage devices. The disk drive and its associated non-transitory computer-readable medium can provide a non-volatile storage device for computer-readable instructions, data structures, program modules and other data for the computing device. In some specific implementations, the memory 820 may include a variety of different types of memory, such as static random access memory (SRAM), dynamic random access memory (DRAM) or ROM. Although the volatile memory described herein may be referred to as RAM, any volatile memory that will not retain the data stored therein after being unplugged from the host and / or power supply is appropriate.

[0064] Removable and non-removable memory 820 and additional storage device 824 are examples of non-transient computer-readable storage media. For example, non-transient computer-readable storage media may include volatile or non-volatile, removable or non-removable media, which are implemented by any method or technology to store information such as computer-readable instructions, data structures, program modules or other data. Memory 820 and additional storage device 824 are examples of non-transient computer storage media. Additional types of computer storage media that may be present in user device 802 may include, but are not limited to: phase change RAM (PRAM), SRAM, DRAM, RAM, ROM, electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital video disk (DVD) or other optical storage device, cassette, tape, disk storage device or other magnetic storage device or any other medium that can be used to store the required information and can be accessed by user device 802. Any combination of the above should also be included in the scope of non-transient computer-readable storage media.

[0065] Alternatively, the computer-readable communication medium may include computer-readable instructions, program modules, or other data transmitted within a data signal such as a carrier wave or other transmission means. However, as used herein, computer-readable storage media does not include computer-readable communication media.

[0066] The user device 802 may also include a communication connector 826 that allows the user device 802 to communicate with a data repository, another computing device or server, a user terminal, and / or other devices via one or more networks. Such networks may include any one or combination of many different types of networks, such as a wired network, the Internet, a wireless network, a cellular network, a satellite network, other private networks, and / or public networks, or any combination thereof. The user device 802 may also include an I / O device 828, such as a touch input device, an image capture device, a video capture device, a keyboard, a mouse, a pen, a voice input device, a display, a speaker, a printer, and the like.

[0067] Turning to the contents of memory 820 in more detail, memory 820 may include an operating system 830 and / or one or more applications or services for implementing the features disclosed herein. Memory 820 may include a data repository 832 that may be configured to store map data (e.g., map tiles) and / or navigation data (e.g., turn-by-turn navigation) for use by a map application module 804 that provides vehicle and / or pedestrian navigation functionality on user device 802.

[0068] In some examples, the map application module 804 can be configured to provide user interfaces 500, 600, and 700 at the user device 802 (e.g., at a display of the I / O device 828). As part of providing the user interfaces 500-700, the map application module 804 can be configured to retrieve map tiles and / or navigation data (e.g., from the data repository 832) and access an augmented reality module (e.g., an AR toolkit) to present an augmented reality overlay incorporating the map and / or navigation data within the user interfaces 500-700.

[0069] In some aspects, the service provider computer 804 can be any suitable type of computing device, such as but not limited to a mobile phone, a smart phone, a personal digital assistant (PDA), a laptop computer, a desktop computer, a server computer, a thin client device, a tablet computer, etc., in addition, it should be noted that in some embodiments, the service provider computer 804 is executed by one or more virtual machines implemented in a hosted computing environment. The hosted computing environment may include one or more rapidly supplied and released computing resources, which may include computing, networking and / or storage devices. The hosted computing environment may also be referred to as a cloud computing environment. In some examples, the service provider computer 804 may communicate with the user device 802 via a network 806. The service provider computer 804 may include one or more servers, which may be arranged in a cluster, as a server farm, or as separate servers that are not associated with each other. These servers may be configured to implement the functions described herein as part of an integrated distributed computing environment.

[0070] In an exemplary configuration, the service provider computer 804 may include at least one memory 840 and one or more processing units (or processors) 842. The processor 842 may be implemented in hardware, computer executable instructions, firmware, or a combination thereof, as appropriate. The computer executable instructions or firmware implementation of the processor 842 may include computer executable instructions or machine executable instructions written in any suitable programming language to perform the various functions described.

[0071] The memory 840 can store program instructions that can be loaded and executed on the processor 842 and data generated during the execution of these programs. Depending on the configuration and type of the service provider computer 804, the memory 840 can be a volatile memory (such as RAM) and / or a non-volatile memory (such as ROM, flash memory, etc.). The service provider computer 804 or server may also include an additional storage device 844, which may include a removable storage device and / or a non-removable storage device. The additional storage device 844 may include, but is not limited to, a magnetic storage device, an optical disk, and / or a tape storage device. The disk drive and its associated computer-readable medium can provide a non-volatile storage device for computer-readable instructions, data structures, program modules, and other data for the computing device. In some specific implementations, the memory 840 may include a variety of different types of memory, such as SRAM, DRAM, or ROM.

[0072] The memory 840 and the additional storage device 844 (whether removable or non-removable) are examples of computer-readable storage media. For example, computer-readable storage media may include volatile or non-volatile, removable or non-removable media, which are implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. The memory 840 and the additional storage device 844 are examples of computer storage media. Other types of computer storage media that may be present in the service provider computer 804 may include, but are not limited to: PRAM, SRAM, DRAM, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, DVD or other optical storage device, cassette, magnetic tape, disk storage device or other magnetic storage device, or any other medium that can be used to store the desired information and can be accessed by the service provider computer 804. Combinations of any of the above should also be included in the scope of computer-readable media.

[0073] Alternatively, the computer-readable communication medium may include computer-readable instructions, program modules, or other data transmitted within a data signal such as a carrier wave or other transmission means. However, as used herein, computer-readable storage media does not include computer-readable communication media.

[0074] The service provider computer 804 may also include a communication connection 846 that allows the service provider computer 804 to communicate with a stored database, another computing device (e.g., user device 802) or a server, a user terminal, and / or other devices over the network 806. The service provider computer 804 may also include I / O devices 848, such as a keyboard, mouse, pen, voice input device, touch input device, display, speaker, printer, etc.

[0075] Turning to the contents of memory 840 in more detail, memory 840 may include an operating system 850, one or more data stores 852, and / or one or more applications, modules, or services for implementing features disclosed herein, such as those provided by geographic service module 854 and navigation model 856. In at least one embodiment, geographic service module 854 may be configured to perform operations in conjunction with the above-described Figures 2 to 3 Any of the operations described in the geographic service model discussed in , including receiving and responding to requests for map data (e.g., map tiles) and / or satellite imagery data corresponding to a geographic location or range. The navigation model 856, which can be implemented within the geographic service module 854 or a separate module, can be configured to receive and respond to requests for point-to-point directions using a route optimization algorithm based on transportation mode (e.g., walking, biking, driving, etc.), map data (e.g., road segments, sidewalks, paths or trail data, etc.), current traffic conditions and historical traffic patterns, etc.

[0076] The user device 802 may be configured with a map application module 804 and an augmented reality module 806 that provides an augmented reality-based pedestrian navigation system for the user. Figure 8 802 may also include additional underlying hardware and software components and modules to support pedestrian navigation functionality, including a location component (e.g., a GPS receiver), an orientation component (e.g., a compass), a media component (e.g., one or more device cameras), and a motion detection component (e.g., an accelerometer, a gyroscope, etc.). As described above, these components of the user device 802 may be used in conjunction with the map application module 804 and the augmented reality module 806 to determine updated orientation data and / or position data to be used by the AR pedestrian navigation system. By way of example, a user may initiate a pedestrian navigation feature on a user device 802 (e.g., a smart phone, a tablet, a wearable device, etc.) having image capture functionality / hardware, location functionality / hardware, and orientation functionality / hardware. The user device 802 may capture image data, location data, and / or orientation data from these components and analyze the data in conjunction with map data received from the service provider computer 804. Based on analysis techniques that may be performed within various modules of the user device 802, updated orientation data and / or position data may be determined for the user device 802. The map application 804 may provide AR pedestrian navigation functionality based on updated orientation data and / or location data.

[0077] In some embodiments, an application executed on a user's device may be used to perform some or all of the operations described herein. Circuits, logic modules, processors, and / or other components may be configured to perform the various operations described herein. It should be understood by those skilled in the art that such configuration may be accomplished by design, setup, interconnection, and / or programming of specific components, depending on the specific implementation, and again depending on the specific implementation, the configured components may be reconfigurable or not reconfigurable for different operations. For example, a programmable processor may be configured by providing appropriate executable code; a dedicated logic circuit may be configured by appropriately connecting logic gates and other circuit elements; and so on.

[0078] Any software component or function described in this patent application can be implemented as a software code executed by a processor, which uses any suitable computer language, such as, for example, Java, C, C++, C#, Objective-C, Swift, or a scripting language such as Perl or Python using, for example, conventional or object-oriented techniques. The software code can be stored on a computer-readable medium as a series of instructions or commands to achieve storage and / or transmission. Suitable non-transient computer-readable media may include random access memory (RAM), read-only memory (ROM), magnetic media (such as a hard drive or floppy disk), or optical media (such as a compact disk (CD) or DVD (digital versatile disk)), flash memory, etc. The computer-readable medium may be any combination of such storage devices or transmission devices.

[0079] Computer programs incorporating various features of the present disclosure may be encoded on various computer-readable storage media; suitable media include disks or tapes, optical storage media such as compact disks (CDs) or DVDs (digital versatile disks), flash memory, etc. Computer-readable storage media encoded with program code may be packaged with compatible devices or provided separately from other devices. In addition, program code may be encoded and transmitted via wired optical and / or wireless networks (including the Internet) that conform to a variety of protocols, thereby allowing distribution, for example, via Internet downloads. Any such computer-readable medium may reside in or be located in a single computer product (e.g., a solid-state drive, a hard drive, a CD, or an entire computer system), and may be present in or located in different computer products within a system or network. The computer system may include a monitor, a printer, or other suitable display for providing any results mentioned herein to a user.

[0080] References to "a," "an," or "the" are intended to mean "one or more," unless expressly indicated to the contrary. Use of "or" is intended to mean an "inclusive or" rather than an "exclusive or," unless expressly indicated to the contrary. Reference to a "first" component does not necessarily require that a second component be provided. Furthermore, reference to a "first" component or a "second" component does not limit the referenced components to specific locations, unless expressly stated. The term "based on" is intended to mean "based, at least in part, on."

[0081] All patents, patent applications, publications, and specifications mentioned herein are incorporated by reference in their entirety for all purposes. No admission that any document is prior art

[0082] In addition, as described above, one aspect of the present technology is to collect and use data available from various sources to improve suggestions for applications and / or people to share content from a host application. The present disclosure contemplates that, in some instances, these collected data may include personal information data that uniquely identifies or can be used to contact or locate a specific person. Such personal information data may include demographic data, location-based data, phone numbers, email addresses, twitter IDs, home addresses, data or records related to a user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other identifying information or personal information.

[0083] The present disclosure recognizes that the use of such personal information data in the present technology can be used to benefit users. For example, personal information data can be used to determine recommendations for applications and people that share content that the user is more interested in. Therefore, the use of such personal information data enables users to more effectively control the sharing of content. In addition, the present disclosure also anticipates other uses of personal information data that benefit users. For example, health and fitness data can be used to provide insights into the user's overall health, or can be used as positive feedback to individuals who use technology to pursue health goals.

[0084] This disclosure envisions that entities responsible for collecting, analyzing, disclosing, transmitting, storing or otherwise using such personal information data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to privacy policies and practices that are recognized as meeting or exceeding industry or government requirements for maintaining the privacy and security of personal information data. Such policies should be easily accessible to users and should be updated as the collection and / or use of data changes. Personal information from users should be collected for legitimate and reasonable purposes of the entity and not shared or sold outside of these legitimate uses. In addition, such collection / sharing should be performed after receiving informed consent from the user. In addition, such entities should consider taking any necessary steps to defend and secure access to such personal information data and ensure that others who have access to personal information data comply with their privacy policies and processes. In addition, such entities may subject themselves to third-party assessments to demonstrate their compliance with widely accepted privacy policies and practices. In addition, policies and practices should be adjusted to specific types of personal information data collected and / or accessed, and to applicable laws and standards including specific considerations of jurisdiction. For example, in the United States, the collection or access of certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); while health data in other countries may be subject to other regulations and policies and should be handled accordingly. Therefore, different privacy practices should be maintained in each country for different types of personal data.

[0085] Regardless of the foregoing, the present disclosure also contemplates implementation schemes in which users selectively block the use or access of personal information data. That is, the present disclosure contemplates that hardware elements and / or software elements may be provided to prevent or block access to such personal information data. For example, in the case of a shared content object, the subject technology may be configured to allow users to choose to "opt in" or "opt out" to participate in the collection of personal information data during registration for a service or at any time thereafter. For another example, a user may choose not to provide data corresponding to their previous interactions with various applications, as well as their sharing preferences and / or historical user interactions. For another example, a user may choose to limit the length of time for maintaining previous application interactions and shared data, or completely prohibit the collection and tracking of such data. In addition to providing "opt-in" and "opt-out" options, the present disclosure contemplates providing notifications related to access or use of personal information. For example, a user may be notified that their personal information data will be accessed when downloading an application, and then the user may be reminded again just before the personal information data is accessed by the application.

[0086] In addition, it is an object of the present disclosure that personal information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use. Risks can be minimized by limiting data collection and deleting data once it is no longer needed. In addition, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. De-identification can be facilitated by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than at the address level), controlling how data is stored (e.g., aggregating data between users), and / or other methods when appropriate.

[0087] Thus, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may also be implemented without access to such personal information data. That is, various embodiments of the present technology will not fail to function properly due to the lack of all or a portion of such personal information data. For example, recommendations for applications and people may be selected to be shared and provided to users based on non-personal information data or an absolute minimum of personal information (such as content requested by a device associated with the user, other non-personal information available to the content sharing system, or publicly available information).

[0088] Item 1 includes a method for generating a user interface via a computing device, the method comprising: receiving, by the computing device, image data corresponding to one or more images captured by a camera of the computing device; receiving, by the computing device, location information corresponding to a location of the computing device; receiving, by the computing device, map information corresponding to the location of the computing device; determining, by the computing device, an orientation of the computing device based at least in part on: (a) the image data captured by the camera of the computing device, (b) the location information corresponding to the location of the computing device, and (c) the map information corresponding to the location; generating, by the computing device, a user interface configured to display a current location of the computing device, and one or more user interface elements representing one or more objects within a threshold distance of the current location of the computing device, wherein generating the user interface includes selecting and positioning the one or more objects within the user interface, and wherein selecting or positioning at least one of the objects within the user interface is based at least in part on the determined orientation of the computing device; and presenting, by the computing device, the user interface on a screen of the computing device.

[0089] Clause 2 includes the method of clause 1, further comprising receiving a direction reading from a compass of the computing device, wherein determining the orientation of the computing device is further based at least in part on the direction reading received from the compass.

[0090] Clause 3 includes the method of Clause 2, further comprising reconfiguring the compass of the computing device based at least in part on the determined orientation of the computing device.

[0091] Clause 4 includes a method according to clause 1, wherein the location information corresponding to the location of the computing device includes at least one of the following items: data received from a global positioning system (GPS) receiver of the computing device, data identifying one or more WiFi networks accessible to the computing device, or data identifying one or more additional devices that communicate with the computing device via a short-range wireless communication protocol.

[0092] Clause 5 includes a method according to clause 1, wherein the location information corresponding to the location of the computing device includes coordinate data received from a global positioning system (GPS) receiver of the computing device, and wherein the method further includes: determining updated location information corresponding to the location of the computing device based at least in part on: (a) image data captured by the camera of the computing device, (b) coordinate data received from the GPS receiver of the computing device, and (c) map information corresponding to the location; and reconfiguring the GPS receiver of the computing device based at least in part on the determined updated location information.

[0093] Clause 6 includes a method according to clause 1, wherein determining the orientation of the computing device comprises: analyzing the image data captured by the camera of the computing device and detecting one or more street boundaries within the image data; and comparing one or more trajectories of the one or more street boundaries with street vector data within the received map information.

[0094] Clause 7 includes the method of clause 6, wherein determining the orientation of the computing device further comprises analyzing the image data captured by the camera of the computing device and detecting a horizontal line within the image data.

[0095] Clause 8 includes the method of clause 1, further comprising generating a route for the computing device from the current location of the computing device to a second location, wherein the user interface displays a next direction instruction based at least in part on the route.

[0096] Item 9 includes a non-transitory computer-readable storage medium storing computer-executable instructions that, when executed by one or more processors of the computing device, cause the one or more processors to: receive image data corresponding to one or more images captured by a camera of the computing device; receive location information corresponding to a location of the computing device; receive map information corresponding to the location of the computing device; determine an orientation of the computing device based at least in part on: (a) the image data captured by the camera of the computing device, (b) the location information corresponding to the location of the computing device, and (c) the map information corresponding to the location; generate a user interface configured to display a current location of the computing device, and one or more user interface elements representing one or more objects within a threshold distance of the current location of the computing device, wherein generating the user interface includes selecting and positioning the one or more objects within the user interface, and wherein selecting or positioning at least one of the objects within the user interface is based at least in part on the determined orientation of the computing device; and presenting the user interface on a screen of the computing device.

[0097] Item 10 includes a non-transitory computer-readable storage medium according to Item 9, wherein the computer-executable instructions further cause the one or more processors to: receive a direction reading from a compass of the computing device, wherein determining the orientation of the computing device is further based at least in part on the direction reading received from the compass.

[0098] Clause 11 includes the non-transitory computer-readable storage medium of clause 10, the computer-executable instructions further causing the one or more processors to: reconfigure the compass of the computing device based at least in part on the determined orientation of the computing device.

[0099] Item 12 includes a non-transitory computer-readable storage medium as described in item 9, wherein the location information corresponding to the location of the computing device includes at least one of the following items: data received from a global positioning system (GPS) receiver of the computing device, data identifying one or more WiFi networks accessible to the computing device, or data identifying one or more additional devices that communicate with the computing device via a short-range wireless communication protocol.

[0100] Item 13 includes a non-transitory computer-readable storage medium as described in item 9, wherein the location information corresponding to the location of the computing device includes coordinate data received from a global positioning system (GPS) receiver of the computing device, and wherein the computer-executable instructions also cause the one or more processors to: determine updated location information corresponding to the location of the computing device based at least in part on: (a) image data captured by the camera of the computing device, (b) coordinate data received from the GPS receiver of the computing device, and (c) map information corresponding to the location; and reconfigure the GPS receiver of the computing device based at least in part on the determined updated location information.

[0101] Item 14 includes a non-transitory computer-readable storage medium according to Item 9, wherein determining the orientation of the computing device includes: analyzing the image data captured by the camera of the computing device and detecting one or more street boundaries within the image data; and comparing one or more trajectories of the one or more street boundaries with street vector data within the received map information.

[0102] Clause 15 includes the non-transitory computer-readable storage medium of clause 14, wherein determining the orientation of the computing device further comprises analyzing the image data captured by the camera of the computing device and detecting a horizontal line within the image data.

[0103] Item 16 includes a non-transitory computer-readable storage medium according to Item 9, wherein the computer-executable instructions further cause the one or more processors to: generate a route for the computing device from the current location of the computing device to a second location, wherein the user interface displays a next direction instruction based at least in part on the route.

[0104] Item 17 includes a computing device, the computing device comprising: one or more processors, the one or more processors communicating with one or more memories, the one or more processors configured to execute computer-executable instructions stored on the one or more memories to at least: receive image data corresponding to one or more images captured by a camera of the computing device; receive location information corresponding to a location of the computing device; receive map information corresponding to the location of the computing device; determine an orientation of the computing device based at least in part on: (a) the image data captured by the camera of the computing device, (b) the location information corresponding to the location of the computing device, and (c) the map information corresponding to the location; generate a user interface configured to display a current location of the computing device, and one or more user interface elements representing one or more objects within a threshold distance of the current location of the computing device, wherein generating the user interface includes selecting and positioning the one or more objects within the user interface, and wherein selecting or positioning at least one of the objects within the user interface is based at least in part on the determined orientation of the computing device; and presenting the user interface on a screen of the computing device.

[0105] Clause 18 includes a computing device according to clause 17, wherein the instructions are further executed to at least: receive a direction reading from a compass of the computing device, wherein determining the orientation of the computing device is further based at least in part on the direction reading received from the compass; and reconfigure the compass of the computing device based at least in part on the determined orientation of the computing device.

[0106] Clause 19 includes the computing device of clause 17, wherein determining the orientation of the computing device comprises: analyzing the image data captured by the camera of the computing device, and detecting one or more street boundaries within the image data; and

[0107] The one or more trajectories of the one or more street boundaries are compared to street vector data within the received map information.

[0108] Clause 20 includes a computing device according to clause 17, wherein the location information corresponding to the location of the computing device includes coordinate data received from a global positioning system (GPS) receiver of the computing device, and wherein the instructions are further executed to at least: determine updated location information corresponding to the location of the computing device based at least in part on: (a) image data captured by the camera of the computing device, (b) coordinate data received from the GPS receiver of the computing device, and (c) map information corresponding to the location;

[0109] The GPS receiver of the computing device is reconfigured based at least in part on the determined updated location information.

Claims

1. A method comprising: determining, by a computing device, a current location of the computing device; receiving, by the computing device, image data corresponding to one or more images captured by one or more cameras of the computing device at the current location; determining, by the computing device, from map information corresponding to the current location of the computing device, one or more physical objects located near the current location; matching, by the computing device, one or more objects captured within the image data with the one or more physical objects located near the current location; as well as A directional orientation of the computing device is determined, by the computing device, based at least in part on matching the one or more objects captured within the image data with the one or more physical objects located proximate to the current location. 2 . The method of claim 1 , wherein the one or more cameras comprise one or more cameras of the computing device.

3. The method according to claim 1 or claim 2, wherein receiving the image data comprises: In response to the mobile application being opened or launched, the one or more images are captured by the one or more cameras.

4. The method according to claim 1 or claim 2, further comprising: At least a portion of the one or more images is captured by the computing device using one or more other devices, and wherein receiving the image data includes receiving the image data from the one or more other devices.

5. The method of claim 1 or claim 2, wherein matching the one or more objects captured within the image data with the map information comprises: identifying one or more line segments within the image data; as well as The one or more objects within the image data are determined based on the one or more line segments.

6. A method according to claim 5, wherein determining the one or more objects includes determining one or more streets, sidewalks, or paths within the image data, and wherein matching the one or more objects captured within the image data with the one or more physical objects includes matching the one or more streets, sidewalks, or paths within the image data with one or more streets, sidewalks, or paths represented within the map information.

7. The method of claim 1 or claim 2, wherein matching the one or more objects captured within the image data with the one or more physical objects comprises: identifying one or more buildings or landmarks within the image data as at least a portion of the one or more objects; as well as The one or more buildings or landmarks within the image data are matched with one or more buildings or landmarks represented within the map information.

8. The method of claim 1 or claim 2, wherein determining the current location of the computing device comprises matching the one or more objects captured within the image data with the one or more physical objects to determine the current location of the computing device.

9. A method according to claim 1 or claim 2, wherein determining the directional orientation includes determining the directional orientation of the computing device by an augmented reality system of the computing device based at least in part on matching the one or more objects captured within the image data with the one or more physical objects.

10. A method as claimed in claim 9, wherein determining the directional orientation of the computing device includes determining the directional orientation of the computing device by an image-based heading estimation component of the augmented reality system based at least in part on matching the one or more objects captured within the image data with the one or more physical objects.

11. One or more computer-readable media having instructions that, when executed, cause a computing device to: determining a current location of the computing device; receiving image data corresponding to one or more images captured by one or more cameras of the computing device at the current location; determining, from map information corresponding to the current location of the computing device, one or more physical objects located near the current location; matching one or more objects captured within the image data with the one or more physical objects located near the current location; as well as A directional orientation of the computing device is determined based at least in part on matching the one or more objects captured within the image data with the one or more physical objects located proximate to the current location.

12. The one or more computer-readable media of claim 11, wherein the one or more cameras comprise one or more cameras of the computing device.

13. The one or more computer-readable media of claim 11 or claim 12, wherein receiving the image data comprises: In response to the mobile application being opened or launched, the one or more images are captured by the one or more cameras.

14. One or more computer-readable media as claimed in claim 11 or claim 12, wherein the instructions, when executed, further cause the computing device to: One or more other devices are caused to capture at least a portion of the one or more images, and wherein receiving the image data comprises receiving the image data from the one or more other devices.

15. The one or more computer-readable media of claim 11 or claim 12, wherein matching the one or more objects captured within the image data with the map information comprises: identifying one or more line segments within the image data; as well as The one or more objects within the image data are determined based on the one or more line segments.

16. One or more computer-readable media according to claim 15, wherein determining the one or more objects comprises determining one or more streets, sidewalks, or paths within the image data, and wherein matching the one or more objects captured within the image data with the one or more physical objects comprises matching the one or more streets, sidewalks, or paths within the image data with one or more streets, sidewalks, or paths represented within the map information.

17. A computing device for: determining a current location of the computing device; receiving image data corresponding to one or more images captured by one or more cameras of the computing device at the current location; determining, from map information corresponding to the current location of the computing device, one or more physical objects located near the current location; matching one or more objects captured within the image data with the one or more physical objects located near the current location; as well as A directional orientation of the computing device is determined based at least in part on matching the one or more objects captured within the image data with the one or more physical objects located proximate to the current location.

18. The computing device of claim 17, wherein matching the one or more objects captured within the image data with the one or more physical objects comprises: identifying one or more buildings or landmarks within the image data as at least a portion of the one or more objects; as well as The one or more buildings or landmarks within the image data are matched with one or more buildings or landmarks represented within the map information.

19. A computing device according to claim 17 or claim 18, wherein determining the current position of the computing device includes matching the one or more objects captured within the image data with the one or more physical objects to determine the current position of the computing device.

20. A computing device according to claim 17 or claim 18, wherein determining the directional orientation includes determining the directional orientation of the computing device by an augmented reality system of the computing device based at least in part on matching the one or more objects captured within the image data with the one or more physical objects.

Images