Mobile device navigation system

Abstract

To provide a technique for generating and presenting user interfaces via mobile computing devices.SOLUTION: Location mapping and navigation user interfaces may be generated and presented via mobile computing devices. A mobile device may detect its location and orientation using internal systems, and may capture image data using a device camera. The mobile device also may retrieve map information from a map server corresponding to the current location of the device. Using the image data captured at the device, the current location data, and the corresponding local map information, the mobile device may determine or update a current orientation reading for the device.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Non-provisional Patent Application No. 16 / 576,673, filed September 19, 2019, the entire contents of which are incorporated herein by reference for all purposes. [Background technology]

[0002] With the added capabilities of mobile positioning technology and modern mobile computing devices, these mobile devices are frequently used as mapping and navigation tools. A mobile device, such as a smartphone, smartwatch, or other wearable computing device, can determine the current coordinates of the mobile device using a Global Positioning System (GPS) receiver or other mobile positioning technology, and can then use map data received from a map server to view and track the current location of the mobile device via a map application. Navigation functionality may also be provided via the mobile device, and the mobile device and / or server navigation components can determine a route between a current location and a selected destination. The navigation system can use optimization algorithms to select an optimal route for a user based on road segment data, traffic data, and the user's selected transportation mode (e.g., walking, biking, driving, etc.).

[0003] For mapping and navigation functions, the mobile device can provide a user interface that displays various location and navigation data. The map user interface can include the user's current location, orientation, street addresses, and nearby landmarks. The navigation user interface can provide turn-by-turn directions to a destination and can indicate the distance and direction of the next turn. The mapping and navigation user interface can also support a variety of different views, including an overhead (or bird's-eye) view with map data and / or satellite imagery, a turn-by-turn navigation view, or a street view, etc.

[0004] However, when a mobile device supports location mapping and navigation functions, these functions can be highly sensitive to errors in location or sensor data detected at the device. For example, even relatively small errors in device location data, device orientation data, or device movement data generated by a mobile device can result in significant mapping and navigation errors, as well as confusion and disorientation for the user. Summary of the Invention

[0005] Described herein are techniques for generating and presenting a user interface via a mobile computing device. In various embodiments, the user interface may correspond to a map user interface and / or a navigation user interface, and / or other augmented reality-based applications that may be configured to provide geo-referenced augmented content. A mobile device, such as a smartphone, tablet computer, or smartwatch, may receive data from various sensors or hardware components of the mobile device, including device position and orientation data, as well as image data captured by a device camera. The mobile device may also receive map information corresponding to the device's current location, for example, from a geo-services module of a map data server. Using the image data captured at the device, the current location data, and the map information corresponding to the current location, the mobile device may determine a current orientation of the mobile device. For example, the captured image data may be analyzed to detect street or sidewalk boundaries, buildings, landmarks, street viewpoint vanishing points, horizon lines, etc., and these detected objects may be compared to corresponding local map data. In some cases, the determined orientation data may be used to update a previous device orientation reading made by an internal compass or other device orientation system. The mobile device's internal compass or orientation system may also, in some cases, be reconfigured or recalibrated based on the determined orientation. Similar techniques may be used to determine updated location data for the mobile device, detect errors in data provided by the mobile device's location / positioning system, and reconfigure or recalibrate the device's location / position system, as necessary.

[0006] Additional aspects described herein relate to generating and displaying a user interface (e.g., a map and / or a pedestrian navigation user interface) on a mobile device using updated device orientation and / or position data. In some embodiments, an Augmented Reality (AR) navigation user interface may be generated and rendered by overlaying a set of navigation user interface components corresponding to the device's current location and orientation on current image data captured by the device camera. The AR-based user interface may be generated based on the determined (or updated) device orientation and / or position data based on an analysis of the captured image data and corresponding local map data. Generating the AR-based map user interface, street view, or turn-by-turn navigation view may include generating and rendering labels (e.g., as an AR overlay) on streets, buildings, landmarks, and the like based on the determined location and orientation data. In an embodiment, the AR-based map user interface may be designed as a pedestrian user interface including turn-by-turn pedestrian navigation instructions overlaid on the current forward-looking image data captured by the device.

[0007] For example, some embodiments relate to a method of generating a map user interface via a computing device. Such a method may include receiving, by the computing device, image data corresponding to one or more images captured by a camera of the computing device, receiving location information corresponding to a location of the computing device, receiving map information corresponding to the location of the computing device, determining an orientation of the computing device based at least in part on (a) the image data captured by the camera of the computing device, (b) the location information corresponding to the location of the computing device, and (c) the map information corresponding to the location, generating a map user interface configured to display a current location of the computing device and one or more user interface elements representing one or more objects within a threshold distance of the current location of the computing device, where generating the map user interface includes selecting and placing one or more objects within the map user interface, where at least one of the selection or placement of the objects within the map user interface is based at least in part on the determined orientation of the computing device, and presenting the map user interface on a screen of the computing device. In additional embodiments, such a method may further include receiving a direction reading from a compass of the computing device, where determining an orientation of the computing device is further based at least in part on the direction reading received from the compass. Additionally, such methods may include reconfiguring a compass of the computing device based at least in part on the determined orientation of the computing device.In some embodiments, the location information corresponding to a location of the computing device may include at least one of data received from a Global Positioning System (GPS) receiver of the computing device, data identifying one or more WiFi networks accessible to the computing device, or data identifying one or more additional devices communicating with the computing device via a short-range wireless communication protocol. In some embodiments of such methods, the location information may correspond to a location of the computing device includes coordinate data received from a Global Positioning System (GPS) receiver of the computing device that may correspond to a location of the computing device, and the method may further include determining updated location information corresponding to the location of the computing device based at least in part on (a) image data captured by a camera of the computing device, (b) the coordinate data received from the GPS receiver of the computing device, and (c) map information corresponding to the location, and reconfiguring the GPS receiver of the computing device based at least in part on the determined updated location information. In further embodiments of such methods, determining an orientation of the computing device may further include analyzing image data captured by a camera of the computing device to detect one or more street boundaries in the image data, and comparing one or more trajectories of the one or more street boundaries to street vector data in the received map information. In further embodiments of such methods, determining an orientation of the computing device may further include analyzing image data captured by a camera of the computing device and detecting a horizon within the image data. In further embodiments, such methods may include generating a route for the computing device from a current location of the computing device to a second location, where the map user interface displays next direction instructions based at least in part on the route.

[0008] These and other embodiments of the present disclosure are described in more detail below. For example, other embodiments relate to systems, devices, and computer-readable media that include computer-executable instructions for performing any of the methods described herein.

[0009] A better understanding of the nature and advantages of embodiments of the present invention may be obtained with reference to the following detailed description and the accompanying drawings. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 illustrates a pedestrian mobile device displaying a navigation user interface according to an embodiment of the present disclosure. [Diagram 2] FIG. 1 is a block diagram illustrating components of a mobile device navigation system according to an embodiment of the present disclosure. [Diagram 3] FIG. 2 is another block diagram illustrating interfaces and components of a mobile device navigation system according to an embodiment of the present disclosure. [Figure 4] 1 is a flowchart illustrating a method for generating and displaying a navigation user interface via a mobile computing device according to an embodiment of the present disclosure. [Diagram 5] 1 illustrates an exemplary display screen of a navigation user interface according to an embodiment of the present disclosure. [Figure 6] 1 illustrates an exemplary display screen of a navigation user interface according to an embodiment of the present disclosure. [Figure 7] 1 illustrates an exemplary display screen of a navigation user interface according to an embodiment of the present disclosure. [Figure 8] FIG. 2 is a block diagram illustrating a user device and service provider computer(s) of a mobile device navigation system according to an embodiment of the disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Aspects of the present disclosure provide various techniques (e.g., methods, systems, devices, computer-readable media storing computer-executable instructions used to perform computer functions, etc.) for determining an orientation and / or precise location for a mobile device using image data captured via a mobile device's camera and local map data (e.g., map tiles received from a map server or geo-services module). In some cases, the determined orientation and position data may be used to update a previous device orientation reading or device position determination made by the device's internal orientation or position / positioning system. The mobile device's internal orientation or position / positioning system may also be reconfigured or recalibrated in some cases based on the updated orientation and position readings.

[0012] Further aspects of the present disclosure provide techniques for generating and displaying maps and navigation user interfaces on a mobile device using updated device orientation and / or position data. In some embodiments, an augmented reality (AR) navigation user interface may be generated and rendered by overlaying a set of navigation user interface components that correspond to the device's current location and orientation onto current image data captured by the device camera.

[0013] FIG. 1 illustrates a pedestrian's mobile device displaying a navigation user interface according to an embodiment of the present disclosure. In this example, the pedestrian user is holding a mobile device 110 currently displaying a user interface 115 of a mobile navigation application. Although a simplified user interface 115 is shown in this example for clarity, the mobile navigation application may display map and navigation graphical user interface components such as a map of the general area near the user, including nearby streets, buildings, and landmarks, and an identification of the user's current location on the map. The embodiments described herein may apply to both map view user interfaces (e.g., showing the user's current location on a map) and navigation user interfaces (e.g., showing the user's current location, selected destination, and / or directions to guide the user to the location). The mobile map application (which may also be referred to as a mobile navigation application when configured with navigation functionality) may retrieve map data from a map data store in a remote map server 120. The mobile map and / or navigation application may display map elements (e.g., streets, buildings, landmarks, etc.) as graphical components and geometric shapes and / or by using actual image data in the map displayed in the user interface. For example, a mobile navigation application may be configured to receive and display satellite image data, or previously captured street view image data for map rendering. In another example, a mobile navigation application may be configured to display an augmented reality user interface, in which real-time image data is captured via a camera on the mobile device 110 and overlaid with labels and other graphical components to provide navigation functionality to a user.A mobile navigation application may also support different user interface views for a user while being navigated to a destination, such as an overhead (or bird's-eye) view, a turn-by-turn navigation view, a street view, and an augmented reality view.

[0014] In this example, the mobile navigation application shown in FIG. 1 may be configured as a pedestrian navigation system, which may differ from vehicle-based navigation systems in some respects. For example, some vehicle-based navigation systems may not detect or use device orientation to generate user interface displays. Instead, a vehicle-based GPS navigation system, or a smartphone running a "vehicle mode" of a mobile navigation application, may assume that the display orientation corresponds to the direction of travel of the vehicle. Thus, if a smartphone running a "vehicle mode" of a mobile navigation application is turned while the vehicle is moving forward, the user interface does not need to rotate to reflect the change in device orientation. However, in the pedestrian navigation system shown in FIG. 1, a change in orientation of the mobile device 110 may indicate that a pedestrian has turned or changed direction, and thus the user interface 115 should be changed to reflect the new device orientation.

[0015] In addition, unlike some vehicle-based navigation systems, the current location of the mobile device 110 in a pedestrian navigation system may not be assumed to be on a road or highway, or even on a sidewalk or path. Thus, the pedestrian navigation system may generate different displays for the user interface 115 based on even small differences in the location of the mobile device 110. For example, the pedestrian navigation system may display different data in the user interface 115 depending on which side of a street the user is walking on, or when the user is crossing a parking lot, a path, a field, or the like.

[0016] The pedestrian navigation system may also not assume that the user is at street level, and instead detect when the user is below or above street level (e.g., in a subway tunnel, on a balcony or roof) when generating the user interface 115. Thus, in an embodiment, the pedestrian navigation system may determine whether the mobile device 110 is at street level, above street level, or below street level, and may generate the user interface 115 accordingly.

[0017] FIG. 2 is a block diagram illustrating components of a mobile device navigation system in accordance with one or more embodiments of the present disclosure. The mobile device navigation system 200 of this example may correspond to a pedestrian navigation system configured to provide location mapping and navigation capabilities to a pedestrian user via the user's mobile device 110. The navigation system 200 includes a map / navigation mobile application 210, an augmented reality system 220, a media system 230, a map data store 240, a position system (or mobile positioning system) 205, and a motion system 260. The different components and systems 210-260 of the mobile device navigation system 200 may be implemented within the mobile device 110, within the computer server 120, and / or within one or more third party systems described herein. For example, in some embodiments, the mobile application 210, the augmented reality system 220, the media system 230, the position system 250, and the motion system 260 may be implemented within the user's mobile device 110, and the map data store 240 may be provided by the server 120.

[0018] The map / navigation application 210 may correspond to a mobile application installed on the user's mobile device 110. The application 210 may be configured to display map data and provide various location mapping and navigation functions via the user interface 120. For example, in response to the application 210 being opened or started on the mobile device 110, the application 210 may display a map user interface having elements corresponding to the user's current location and may provide directions to a selected destination. In some embodiments, the map / navigation application 210 may be configured to provide an augmented reality (AR) user interface in which the map / navigation components are generated and overlaid on real-time image data captured via a camera on the mobile device 110. Thus, the application 210 may include a turn-by-turn routing engine and an augmented reality overlay component, as shown in this example.

[0019] The augmented reality system 220 may include components configured to generate map / navigation components that are overlaid within an AR navigation user interface on the mobile device 110. In this example, the augmented reality system 220 includes a semantic image segmentation component, an image-based heading estimation component, a long-baseline Visual Inertial Odometry (VIO) component, and a geo-referenced AR content provider.

[0020] The media system 230 includes one or more device cameras that capture image data for use in the AR navigation user interface. The AR system 220 may send a request to the media system 230 to activate the front camera of the mobile device 110 during periods when the map / navigation application 210 is in use and configured to display the AR navigation interface. During these periods, the media system 230 may continuously activate the front camera and provide the captured image data to the AR system 220 for use in the AR navigation user interface.

[0021] The map data store 240, which may be implemented in a map server 120 separate from the mobile device 110, contains a database of map tiles. A map tile includes a set of object identifiers, properties, and geographic coordinates that represent physical objects (e.g., buildings, streets, natural landmarks, etc.) in a particular area. The AR system 220 may request an appropriate map tile from the map data store 240 based on the current location of the mobile device 110 when the map / navigation application 210 is being used.

[0022] The position / positioning system 250 may be configured with one or more position subsystems configured to determine the location and orientation of the mobile device 110. In this example, the position / positioning system 250 includes a position estimator, a VIO stabilized position component, a compass / camera heading arbitration component, and a pedestrian map matching component. The position / positioning system 250 may be configured to provide device position data when requested by the map / navigation application 210. In some cases, the device position system 250 may include a GPS receiver or other mobile positioning technology configured to detect current geographic coordinates for the mobile device 110. Additionally or alternatively, the position / positioning system 250 may be configured to detect wireless networks, wireless access points, and / or other nearby devices (e.g., known Bluetooth devices, NFC tags, etc.), and the location of the mobile device 110 may be determined from the wireless networks, wireless access points, and / or other nearby devices.

[0023] The location / positioning system 250 may also determine the current device orientation in response to a request from the map / navigation application 210. In some cases, the device orientation may be determined by an internal compass of the mobile device 110. In other cases, the device orientation may be calculated using triangulation by detecting differences in the arrival times of signals at different antennas of the mobile device 110.

[0024] The motion system 260 may include motion sensors such as an accelerometer and a gyroscope. In some embodiments, the motion system 260 may be configured to provide device motion data to the AR system 220 and / or the position / positioning system 250. For example, the motion data may be provided from the motion system 260 to the AR system 220 for use by the AR system 220 in generating AR content. In addition, the motion system 260 may be used to detect a device position and / or a device orientation based on motion sensors such as an accelerometer and a gyroscope in the mobile device 110. For example, assuming an initial position or orientation of the mobile device 110 is known, monitoring and tracking device acceleration data and / or gyroscope data in three dimensions may enable the mobile device 110 to calculate an updated position or an updated orientation.

[0025] 3 is another block diagram illustrating interfaces and components of a mobile device navigation system according to an embodiment of the present disclosure. The mobile device navigation system 300 shown in this example may be similar or identical to the navigation system 200 described above, with additional details showing additional components and communication paths between the systems and components. In other embodiments, the mobile device navigation system 300 may correspond to a separate implementation of a pedestrian navigation system, different from the navigation system 200.

[0026] 4 is a flow chart illustrating an exemplary process for generating and displaying a navigation user interface via a mobile computing device, according to an embodiment of the present disclosure. As described below, steps in this process may be performed by a user device 110, such as a smartphone or other mobile device running a mobile device navigation system. Accordingly, features and functionality may be described with reference to the devices and systems described in FIGS. 1-3. However, it should be understood that the process for generating and displaying a navigation user interface described herein is not limited to the particular computing environments and systems described above, but may be performed in a variety of other computing environments and systems.

[0027] In step 401, the mobile device navigation system 200 receives image data captured by one or more cameras 230 of the mobile device 100. In some embodiments, the capture of image data in step 401 may be performed in response to a user launching or initiating the mobile navigation application 210. For example, the mobile application 210 or the augmented reality system 220 may be configured to activate one or more device cameras in response to the mobile application 210 being opened or launched, or in response to a user action via the user interface of the mobile application 210 (e.g., a user initiating a pedestrian navigation mode). In other examples, the device camera(s) may be configured to continuously capture data during operation of the mobile device 110, in which case a continuous stream of image data may be received by the AR system 220 before the mobile application 210 is opened or launched by the user.

[0028] The image data received in step 401 may correspond to digital images and / or video data captured via one or more cameras of the mobile device. In some cases, it may be advantageous to maximize the amount of image data received in step 401. As described below, maximizing the amount of image data collected in step 401 may allow objects depicted in the image data to be more easily matched with corresponding map data and may also provide more image data overall to analyze when determining device orientation and / or location. Thus, in some embodiments, receiving image data in step 401 may include activating multiple cameras on the mobile device 110, such as a front-facing camera, a user-facing camera, and / or any other available cameras, to maximize the amount of image data collected. In some cases, a wide-angle or panoramic camera mode may also be used to increase the image data angle collected therefrom in step 401. Additionally, in some embodiments, the image data received in step 401 may be collected by one or more separate computing devices at the same location and then transmitted to the mobile device 110. For example, a navigation system running on a user's first mobile device 110 (e.g., a smartphone) may activate a camera or the like on one or more of the user's other devices, such as a second smartphone, a smartwatch, or other wearable device having a camera. Thus, the other user devices may collect additional image data and may transmit image data back to the first mobile device 110 running the navigation system.

[0029] In step 402, the mobile device navigation system 200 may receive and / or determine current location data for the mobile device 110. In some embodiments, the mobile device 110 may activate an internal Standard Positioning Service (SPS) receiver, such as a GPS receiver or other mobile positioning system, in response to a user opening or activating the map / navigation mobile application 210. One or more SPS receivers on the mobile device 110 may be capable of receiving signals from one or more SPS satellites using an SPS antenna. In various embodiments, the SPS receiver on the mobile device 110 may support measurement of signals from space vehicles (SVs) of an SPS system, such as a Global Navigation Satellite System (GNSS) (e.g., Global Positioning System (GPS)), Galileo, GLONASS, the Quasi-Zenith Satellite System (QZSS) over Japan, the Indian Regional Navigational Satellite System (IRNSS) over India, or Beidou over China. The SPS receiver may also be used in conjunction with various augmentation systems (e.g., Satellite Based Augmentation System (SBAS)) that may be associated with or otherwise enabled for use with one or more global or regional navigation satellite systems.By way of example, SBAS may include an augmentation system or systems that provide integrity information, differential corrections, etc., such as, but not limited to, the Wide Area Augmentation System (WAAS), the European Geostationary Navigation Overlay Service (EGNOS), the Multi-functional Satellite Augmentation System (MSAS), GPS Aided GEO Augmented Navigation or GPS and GEO Augmented Navigation System (GAGAN), etc. Thus, as used herein, SPS may include any combination of one or more global or regional navigation satellite systems or augmentation systems, and SPS signals may include SPS, SPS-like, or other signals associated with one or more such SPS.

[0030] Other techniques for receiving the current location of the mobile device 110 in step 402 may include determining the current device location based on the device's connection to one or more additional devices via short-range wireless communication protocols, including any wireless networks and / or wireless access points to which the mobile device 110 is currently connected. For example, during use of a map / navigation mobile application, the mobile device 110 may be connected to one or more wireless networks through one or more wireless access points or other network components. Wireless access points may include devices such as wireless routers, modems, switches, mobile hotspots, and set-top boxes, which allow the mobile device 130 to wirelessly connect and communicate to a Local Area Network (LAN), Wide Area Network (WAN). Certain wireless networks and / or wireless access points may be associated with certain geographic locations (e.g., a home, a workplace, a WiFi network provided by a merchant, etc.), and thus the mobile device 110 may determine its current geographic location based on the set of wireless networks and wireless access points currently available for connection by the mobile device 110.

[0031] The various techniques used in step 402 to determine the current location of the mobile device 110 may have different levels of accuracy and reliability. For example, GPS Standard Positioning Service (SPS) and GPS Precise Positioning Service (PPS) may provide different levels of accuracy, and the receiver accuracy for either standard may be affected by signal problems and satellite line-of-sight obstructions, measurement bias, and random errors. In addition, in some embodiments, determining the location of the mobile device 110 based on wireless networks and wireless access points detected by the mobile device 110 may be less precise than using GPS. For example, signals from known wireless access points may only allow positioning of the mobile device 110 to within a few hundred feet, depending on the transmission range of the wireless access points. Thus, it should be understood that the various steps below may be customized based on the accuracy and reliability of the location determination in step 402. For example, a larger geographic range of map information may be obtained (step 403) when the location determination in step 402 is less precise or more likely to be error-prone.

[0032] In step 403, the mobile device navigation system 200 may receive map information for an area at or near the current location of the mobile device 110. In some embodiments, the position / location system 250 of the mobile device 110 may request map information (e.g., one or more map tiles) from the map server 120 corresponding to the current location and nearby area of ​​the mobile device 110. The map server 120 may include one or more map data stores 240 that store catalogs or other map data files / structures of map tiles representing different map regions. In some cases, a single map tile may represent a land area of ​​one square kilometer. In some cases, the map information (e.g., one or more map tiles) may be obtained in step 403 in response to a user opening or starting the map / navigation application 210 on the mobile device 110. In other cases, the position system 250 may be configured to periodically or continuously track the current location of the mobile device 110 and request new surrounding map information from the map server 120 as needed.

[0033] The map information received in step 403 may include map tiles or other similar formats of map data files or structures that include sets of geographic coordinates that identify physical objects within the area covered by the map tiles. For example, map tiles covering a particular area may include coordinate data that defines the precise size, shape, and boundaries of all streets, paths, buildings, natural or man-made landmarks, and any other fixed objects within the area. In addition to geographic coordinates, the map tiles may also include data identifying each unique object and the type of object (e.g., street, path, building, body of water, other natural landmark, etc.), object name or labeling data, and other object properties. The map tiles or other map information received in step 403 may include any and all data required by the map / navigation application 210 to render a map graphical user interface for the area.

[0034] Additionally or alternatively, the map information received in step 403 may include image data about or near the current location of the mobile device 110. For example, the location system 250 may obtain satellite image data or previously captured street view image data from one or more external servers 120 corresponding to the current location and surrounding area. In another example, the map information received in step 403 may include a catalog of physical object images (e.g., buildings, landmarks, etc.) along with the known geographic coordinates of the physical objects. As detailed below, the map information received in step 403 may be analyzed and compared to the image data captured at the mobile device 110. Thus, when non-image map information (e.g., map tiles) is received in step 403, the comparison with the captured image data may be based on a comparison of object size, shape, etc. However, when the map image data is received in step 403, the comparison with the image data captured at the mobile device 110 may include image matching, etc.

[0035] In some embodiments, map information may be received from the map server 120 without providing the precise location of the mobile device 110 to the map server 120. For example, the position / location system 250 of the mobile device 110 may request and receive one or more map tiles covering a relatively large area without transmitting the location coordinates of the mobile device 110 to the map server 120. Such embodiments may provide technical advantages for privacy and security protection, including fully supporting the map / navigation features described herein (e.g., pedestrian navigation systems, AR-based map / navigation user interfaces, etc.) without requiring the mobile device 110 to transmit the location of the mobile device 110 to any external server 120.

[0036] In step 404, the mobile device navigation system 200 may determine a current orientation of the mobile device 110 based on an analysis of image data captured at the mobile device 110, the current location of the mobile device 110, and / or map information received for the current location of the mobile device 110. In some embodiments, the orientation determination may be performed by an image-based heading estimation component 222 in the AR system 220 executing on the mobile device 110. For example, the image-based heading estimation component 222 may initially plot the current location of the mobile device 110 in the map information received in step 403. The heading estimation component 222 may then use the map information to determine one or more physical objects that are currently (or should be) in the vicinity of the mobile device 110. The heading estimation component 222 may then compare the image data captured at the mobile device 110 in step 401 to characteristics of physical objects identified in the map data at or near the current location. In some embodiments, such a comparison may include analyzing image data captured at the mobile device 110 to detect lines, shapes, and other recognizable objects, and then comparing these objects to nearby physical objects identified in the map data. As noted above, image matching comparisons may also be performed between image data captured at the mobile device 110 and map data images (e.g., previous street view images, satellite images, image libraries, etc.) in map information received from one or more map servers 120.

[0037] The analysis in step 404 may include detecting identifiable objects in the image data captured at the mobile device 110, determining properties of such objects, and then attempting to match the objects to nearby physical objects identified in the map data. In various examples, object matching may be based on object shape, size, color, relationships between objects, or any other detectable patterns in the image data.

[0038] In one example, the image-based heading estimation component 222 may analyze image data captured by the mobile device 110 to detect streets, sidewalks, or walking / biking paths in the captured image data. These objects may be detected by identifying parallel or near-parallel line segments that correspond to street boundaries, curbs, sidewalk lines or boundaries, and / or painted lines in the street. Based on the direction (or trajectory) of the line segments, the estimation component 222 may determine the orientation of the mobile device 110 relative to the street, sidewalk, or path (or a particular segment of the street, sidewalk, or path). Based on the angle or distance between the parallel or near-parallel line segments, the estimation component 222 may also determine the distance between the mobile device 110 and the street, sidewalk, or path, etc., and the estimation component 222 may then identify the corresponding street, sidewalk, or path in the map information, for example, by comparing the location, direction, width, curve pattern, or other identifying characteristics of the street, sidewalk, or path. Finally, the estimation component 222 may determine a current heading of the mobile device 110 based on (1) the directions (or street vectors) of the streets, sidewalks, or path segments stored in the map information, and (2) the relative orientation of the mobile device 110 to the streets, sidewalks, or path segments, which may be determined by analysis of the captured image data. In some embodiments, the heading estimation component 222 may also analyze the image to detect vanishing points for parallel or near-parallel street boundaries, sidewalk lines, railroad tracks, etc., and use the direction of the vanishing points from the viewpoint of the mobile device 110 to compare the directions of the streets, paths, railroad tracks, etc. in the map information to determine the current heading of the mobile device 110.

[0039] In another example, the image-based heading estimation component 222 may analyze image data captured by the mobile device 110 to detect buildings or other landmarks and may compare the buildings or landmarks to corresponding objects in the map information to determine a current orientation of the mobile device 110. For example, the estimation component 222 may identify characteristics (e.g., size, shape, location, or orientation) of nearby buildings or landmarks to match the buildings or landmarks to corresponding objects in the map information. After matching the buildings or landmarks to objects in the map information, the heading estimation component 222 may determine a current orientation of the mobile device 110 based on (1) the location and orientation of the objects stored in the map information and (2) the relative orientation of the mobile device 110 to the objects (or at least one surface of the objects) in the captured image data.

[0040] In addition to matching nearby buildings or other landmarks to corresponding objects in the map information, the heading direction estimation component 222 may also identify distant objects in image data captured by the device, so long as their location and / or heading direction are known or can be determined. For example, the heading direction estimation component 222 may use techniques similar to those described above to identify the sun, moon, horizon, or other distant landmarks (e.g., mountains, bodies of water, radio towers, aircraft warning towers, etc.) in image data captured by the mobile device 210 to determine the current orientation of the mobile device. Regardless of the type of physical object used to orient the mobile device in 210, the process steps may include matching physical objects in the image data captured by the mobile device 110 to corresponding objects in the received map information, determining an absolute orientation of the object from the received map information, and then determining a current orientation of the mobile device based on the absolute orientation of the object and the relative orientation of the mobile device 110 to the object.

[0041] In some embodiments, data from the internal compass of the mobile device 110 may also be used in the device orientation determination in step 404. As noted above, compass data from the mobile device 110 may be inaccurate or unreliable in some cases. However, the heading estimation component 222 may use the compass data in a limited manner to determine the general direction in which the device is facing (e.g., within a range of 90 degrees, 180 degrees, etc.), which may be used, for example, as a starting point for the image analysis and object comparison process.

[0042] Although the above examples relate to determining a current orientation of the mobile device 110, similar techniques may be used to determine a current location of the mobile device 110. As noted above, the location data (or location information) received via the position / positioning system 250 of the mobile device 110 may be inaccurate or imprecise. For example, based on signal interference, erroneous readings, or design limitations of the position / positioning technology, the current device location received in step 402 may be inaccurate by anywhere between a few feet to a half mile or more. In these cases, the current location data determined in step 402 may be accurate enough to enable the mobile device navigation system 200 to request the appropriate map tiles / map information, but may not be accurate enough for the map / navigation mobile application 210.

[0043] Thus, step 404 may also include performing similar image processing and object comparison steps to determine the current location of the mobile device 110. For example, the position estimator 252 in the position / location system 250 and / or the AR system 220 may match one or more physical objects in the image data captured by the mobile device 110 with corresponding objects in the received map information. The position estimator 252 may then determine or estimate the distance between the mobile device 110 and one or more physical objects, for example, by using the size of the object and the angle between the edges of the object to calculate the straight-line distance between the mobile device 110 and the object. The current location of the mobile device 110 may then be calculated using the absolute position of the object, which is stored in the map information, and the current location of the mobile device 110 may be calculated using the distance between the mobile device 110 and the same object. For example, the absolute location of the mobile device 110 may be determined by triangulation using the distances to two or more objects or based on the determined distances and orientations to one or more objects.

[0044] In some embodiments, the mobile device navigation system 200 may use techniques similar to determining device orientation and / or position described above to determine when the mobile device 110 is at street level, above street level, or below street level. For example, analysis of image data captured by the mobile device 110 may include detecting an up-angle or down-angle relative to nearby physical objects. When map information confirms that these physical objects are at street level, the pedestrian navigation system 200 may determine the vertical positioning of the mobile device 110 above or below street level.

[0045] At step 405, components of the mobile device navigation system 200 may use the device orientation and device position determination at step 404 to select and arrange user interface elements to be presented via the map / navigation application 210. In some embodiments, generation of the user interface for the map / navigation application 210 may be performed entirely by the map / navigation application 210 itself. In other embodiments, for example, an augmented reality (AR) based pedestrian navigation system may be provided via the map / navigation application 210, and then a separate AR system 220 may be used to select and arrange user interface elements to be presented via the AR-based interface.

[0046] As noted above, when the map / navigation application 210 is a pedestrian navigation application or may be configured to operate in a pedestrian navigation mode (as opposed to operating in a driving navigation mode), the orientation of the mobile device 110 may determine the display generated for the user interface. For example, if a pedestrian using the navigation application 210 rotates the mobile device 110 while remaining in the same position, the user interface may be updated to display a new set of streets, buildings, and other objects that are now in front of the mobile device 110. Pedestrian navigation systems that update displays based on the orientation of the mobile device 110 may include both augmented reality (AR) navigation systems and graphical non-AR navigation systems. AR pedestrian navigation systems may display real-time image data captured from the front camera of the mobile device 110 and overlay with labels and other graphical components to provide navigation functionality to the user.

[0047] Both the selection of graphical components (e.g., graphical streets and objects, labels, direction windows, turn indicators, etc.) and the specific placement of the graphical components may be based on the device orientation and / or position data determined in step 404. Initially, based on the device position and orientation, the app / navigation mobile application 210 and / or the AR system 220 may determine which physical objects are directly in front of the mobile device 110 and in the field of view of the mobile device 110. For fully graphical (non-AR) user interfaces, the mobile application 210 may determine the size, location, and screen position of these physical objects (e.g., streets, buildings, natural landmarks, etc.) to correspond to the user's current viewpoint. For AR and non-AR user interfaces, a label may be generated for each physical object rendered in the user interface, and the size and placement of the label may be selected to match the size and orientation of the rendered object. For AR-based pedestrian navigation applications, the camera image data displayed in the AR interface may be analyzed to identify a particular physical object (e.g., a street, a building, etc.) currently shown in the image, and the AR system 220 may generate an object label that is rendered on the screen at the appropriate size, placement, and orientation over the image of the object.

[0048] In addition to labeling the physical objects shown in the interface, the map / navigation application 210 may provide map or navigation related functionality with additional components, including displaying the user's current location, destination location and estimated time of arrival, turn-by-turn navigation instructions, etc. In some embodiments, the mobile application 210 and / or the AR system 220 may determine the size and placement of graphical user interface components based on the representation of the physical objects in the user interface. For example, directional instruction windows, arrows, etc. may be positioned and appropriately sized on the screen so as not to obscure street signs, traffic signals, upcoming moving objects, etc. Additionally, the AR system 220 may also select contrasting colors used to label objects and other user interface components to be overlaid in the AR-based pedestrian navigation user interface. For example, street and building labels, arrows, and directional instructions may be rendered in light colors, such as during nighttime use of the pedestrian navigation system.

[0049] In step 406, a map / navigation user interface may be generated and rendered by the mobile application 110 on the display screen of the mobile device 110. As mentioned above, the various techniques described herein may be applied to location mapping only applications and navigation applications, and may be applied for augmented reality user interfaces and graphical non-AR user interfaces. Thus, the map / navigation user interface generated and rendered in step 406 may include map-specific and / or navigation-specific user interface elements as appropriate, and may or may not be rendered as an augmented reality user interface. For example, the techniques described herein may also be applied to any augmented reality application to enable the application to provide geo-referenced augmented reality content.

[0050] 5-7 show three exemplary display screens of a navigation user interface according to an embodiment of the present disclosure. In FIG. 5, the mobile device 110 is shown rendering an augmented reality user interface 500 generated by a pedestrian navigation system. In this example, the augmented reality user interface 500 includes an AR window 510 that displays real-time image data captured via the mobile device's front camera, and an overlay window with graphical components 515 that displays current direction instructions 516 for a user 516 and an overhead map 517 of the user's current location and route. As shown in this example, the current direction instruction (i.e., "turn") is selected and displayed based on the user's current orientation and location. In FIG. 6, the user has turned and the user interface 600 has been updated based on the new orientation detected for the mobile device 110. In this example, the overhead map 617 has been updated to reflect the orientation, and the turn instruction has been replaced with a separate graphic window 616 that notifies the user of the next turn along the current route. Finally, in FIG. 7, the pedestrian is approaching the destination and both the overhead map 717 and the direction instructions window 716 have been updated accordingly.

[0051] 8 illustrates components of a pedestrian navigation system 800, according to at least one embodiment. The system 800 may include user devices 802 and / or service provider computer(s) 804, which may communicate with each other over network(s) 806 using any suitable communication protocol.

[0052] In some examples, the network(s) 806 may include any one or combination of many different types of networks, such as cable networks, the Internet, wireless networks, cellular networks, and other private and / or public networks. Although the illustrated example represents a user device 802 communicating with a service provider computer(s) 804 via a network(s) 806, the described techniques may equally be applied in cases where the user device 802 interacts with the service provider computer(s) 804 via a landline, via a public phone booth, or in any other suitable manner. It should be understood that the described techniques may be applied in other client / server arrangements and in non-client / server arrangements (e.g., locally stored applications, etc.). For example, in some embodiments, the geo-services module 854 and / or the navigation module 856, described in more detail below, may operate in whole or in part on the user device 802. Additionally, the user device 802 may access functionality of the geo services module 854 and / or navigation module 856 via user interfaces and / or APIs provided by the geo services module 854 and / or navigation module 856 through components of the user device 802 (e.g., map application module 804) and / or service provider computer(s) 804.

[0053] As described above, user device 802 may be configured to execute or otherwise manage applications or instructions to present a user interface (e.g., user interfaces 500, 600, and 700 of FIGS. 5-7) for providing map data and pedestrian navigation data using augmented reality overlays. User device 802 may be any type of computing device, such as, but not limited to, a mobile phone (e.g., a smartphone), a tablet computer, a Personal Digital Assistant (PDA), a laptop computer, a desktop computer, a thin client device, a smart watch, or a wireless headset.

[0054] In one example configuration, the user device 802 may include at least one memory 820 and one or more processing units (or processor(s)) 822. The processor(s) 822 may be implemented in hardware, computer-executable instructions, or a combination thereof, as appropriate. The computer-executable instructions or firmware implementation of the processor(s) 822 may include computer-executable or machine-executable instructions written in any suitable programming language to perform the various functions described.

[0055] The memory 820 may store program instructions that are loadable and executable on the processor(s) 822, and data generated during the execution of these programs. Depending on the configuration and type of user device 802, the memory 820 may be volatile (such as Random Access Memory (RAM)) and / or non-volatile (such as Read-Only Memory (ROM), flash memory, etc.). The user device 802 may also include additional removable and / or non-removable storage 824, including, but not limited to, magnetic storage devices, optical disks, and / or tape storage devices. The disk drives and their associated non-transitory computer-readable media may provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computing device. In some implementations, memory 820 may include multiple different types of memory, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), or ROM. Volatile memory as described herein may be referred to as RAM, although any volatile memory that does not retain data stored therein once unplugged from a host and / or power source may be suitable.

[0056] The removable and non-removable memory 820 and the removable and non-removable additional storage 824 are all examples of non-transitory computer-readable storage media. For example, non-transitory computer-readable storage media may include volatile or non-volatile, removable or non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. Both the memory 820 and the additional storage 824 are examples of non-transitory computer storage media. Additional types of computer storage media that may be present in the user device 802 may include, but are not limited to, Phase-change RAM (PRAM), SRAM, DRAM, RAM, ROM, Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory, or other memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Video Disc (DVD), or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage, or other magnetic storage devices, or any other medium that may be used to store the desired information and that may be accessed by the user device 802. Combinations of any of the above should be included within the scope of non-transitory computer readable storage media.

[0057] Alternatively, computer-readable communication media may include computer-readable instructions, program modules, or other data transmitted within a data signal, such as a carrier wave or other transmission. However, as used herein, computer-readable storage media does not include computer-readable communication media.

[0058] The user device 802 may also include communication connection(s) 826 that allow the user device 802 to communicate with a data store, another computing device or server, a user terminal, and / or other devices over one or more networks. Such networks may include any one or combination of many different types of networks, such as, for example, a cable network, the Internet, a wireless network, a cellular network, a satellite network, other private and / or public networks, or any combination thereof. The user device 802 may also include I / O device(s) 828, such as a touch input device, an image capture device, a video capture device, a keyboard, a mouse, a pen, an audio input device, a display, speakers, a printer, etc.

[0059] Turning in detail to the contents of memory 820, memory 820 may include an operating system 830 and / or one or more application programs or services for implementing the features disclosed herein. Memory 820 may include a data store 832, which may be configured to store map data (e.g., map tiles) and / or navigation data (e.g., turn-by-turn directions) used by map application module 804 to provide vehicular and / or pedestrian navigation features on user device 802.

[0060] In some examples, the map application module 804 may be configured to provide the user interfaces 500, 600, and 700 at the user device 802 (e.g., on a display of the I / O device(s) 828). As part of providing the user interfaces 500-700, the map application module 804 may be configured to retrieve map tiles and / or navigation data (e.g., from the data store 832) and access an augmented reality module (e.g., AR Kit) to present an augmented reality overlay that incorporates the map and / or navigation data within the user interfaces 500-700.

[0061] In some aspects, the service provider computer(s) 804 may be any suitable type of computing device, such as, but not limited to, a mobile phone, a smart phone, a personal digital assistant (PDA), a laptop computer, a desktop computer, a server computer, a thin client device, a tablet PC, etc. Additionally, it should be noted that in some embodiments, the service provider computer(s) 804 is executed by another virtual machine implemented in a hosted computing environment. The hosted computing environment may include one or more rapidly provisioned and released computing resources, which may include computing, networking, and / or storage. The hosted computing environment may also be referred to as a cloud computing environment. In some examples, the service provider computer(s) 804 may communicate with the user device 802 via the network(s) 806. The service provider computer(s) 804 may include one or more servers, perhaps arranged in a cluster, as a server farm, or as individual servers that are not associated with each other. These servers may be configured to implement the functionality described herein as part of an integrated distributed computing environment.

[0062] In one example configuration, the service provider computer(s) 804 may include at least one memory 840 and one or more processing units (or processor(s)) 842. The processor(s) 842 may be implemented in hardware, computer-executable instructions, firmware, or combinations thereof, as appropriate. The computer-executable instructions or firmware implementation of the processor(s) 842 may include computer-executable or machine-executable instructions written in any suitable programming language to perform the various functions described.

[0063] The memory 840 may store program instructions that are loadable and executable on the processor(s) 842, and data generated during the execution of these programs. Depending on the configuration and type of the service provider computer(s) 804, the memory 840 may be volatile (such as RAM) and / or non-volatile (such as ROM, flash memory, etc.). The service provider computer(s) 804 or server may also include additional storage 844, which may include removable and / or non-removable storage. The additional storage 844 may include, but is not limited to, magnetic storage, optical disks, and / or tape storage. The disk drives and the associated computer-readable media of the disk drives may provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computing device. In some implementations, the memory 840 may include multiple different types of memory, such as SRAM, DRAM, or ROM.

[0064] The removable and non-removable memory 840, the removable and non-removable additional storage 844 are all examples of computer readable storage media. For example, computer readable storage media may include volatile or non-volatile removable or non-removable media implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules, or other data. The memory 840 and the additional storage 844 are all examples of computer storage media. Additional types of computer storage media that may be present in the service provider computer(s) 804 may include, but are not limited to, PRAM, SRAM, DRAM, RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, DVD, or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage, or other magnetic storage devices, or any other medium that may be used to store desired information and that may be accessed by the service provider computer(s) 804. Combinations of any of the above should also be included within the scope of computer readable media.

[0065] Alternatively, computer-readable communication media may include computer-readable instructions, program modules, or other data transmitted within a data signal, such as a carrier wave or other transmission. However, as used herein, computer-readable storage media does not include computer-readable communication media.

[0066] The service provider computer(s) 804 may also include communication connection(s) 846 that allow the service provider computer(s) 804 to communicate with stored databases, another computing device (e.g., user device 802) or servers, user terminals, and / or other devices on the network(s) 806. The service provider computer(s) 804 may also include I / O device(s) 848, such as a keyboard, mouse, pen, voice input device, touch input device, display, speakers, printer, etc.

[0067] Referring in detail to the contents of memory 840, memory 840 may include an operating system 850, one or more data stores 852, and / or one or more application programs, modules, or services for implementing features disclosed herein, such as those provided by geo services module 854 and navigation module 856. In at least one embodiment, geo services module 854 may be configured to perform any of the operations described in connection with the above geo services model of FIGS. 2-3, including receiving and responding to requests for map data (e.g., map tiles) and / or satellite image data corresponding to a geographic location or area. Navigation module 856, which may be implemented in geo services module 854 or a separate module, may be configured to receive and respond to requests for point-to-point directions using route optimization algorithms based on transportation modes (e.g., walking, biking, driving, etc.), map data (e.g., road segments, sidewalks, paths, or trail data, etc.), current traffic conditions, historical traffic patterns, and the like.

[0068] The user device 802 may be configured with a map application module 804 and an augmented reality module 806 that provides an augmented reality-based pedestrian navigation system to the user. Although not shown in FIG. 8, the user device 802 may also include additional underlying hardware and software components and modules to support pedestrian navigation functions, including a location component (e.g., a GPS receiver), an orientation component (e.g., a compass), a media component (e.g., one or more device cameras), and a motion detection component (e.g., an accelerometer, a gyroscope, etc.). As described above, these components of the user device 802 may be used in conjunction with the map application module 804 and the augmented reality module 806 to determine updated orientation and / or position data used by the AR pedestrian navigation system. As an example, a user may initiate a pedestrian navigation feature on a user device 802 (e.g., a smartphone, a tablet, a wearable device, etc.) that has image capture capabilities / hardware, location capabilities / hardware, and orientation capabilities / hardware. The user device 802 may capture image data, location data, and / or orientation data from these components and may analyze the data in conjunction with map data received from the service provider computer 804. Based on analysis techniques that may be performed within various modules of the user device 802, updated orientation and / or location data may be determined for the user device 802. The map application 804 may provide AR pedestrian navigation functionality based on the updated orientation and / or location data.

[0069] In some embodiments, some or all of the operations described herein may be performed using an application running on a user's device. Circuits, logic modules, processors, and / or other components may be configured to perform the various operations described herein. Those skilled in the art will appreciate that, depending on the implementation, such configuration may be achieved by design, setup, interconnection, and / or programming of the particular components, and that, depending on the implementation, the configured components may or may not be reconfigurable for different operations. For example, a programmable processor may be configured by providing suitable executable code, and dedicated logic circuits may be configured by suitable connecting of logic gates and other circuit elements.

[0070] Any of the software components or functions described in this application may be implemented as software code executed by a processor, for example using any suitable computer language using existing or object-oriented techniques, such as Java, C, C++, C#, Objective-C, Swift, or scripting languages ​​such as Perl or Python. The software code may be stored as a series of instructions or commands on a computer-readable medium for storage and / or transmission. Suitable non-transitory computer-readable media may include random access memory (RAM), read-only memory (ROM), magnetic media such as hard drives or floppy disks, or optical media such as Compact Disks (CDs) or Digital Versatile Disks (DVDs), flash memories, and the like. The computer-readable medium may be any combination of such storage or transmission devices.

[0071] A computer program incorporating various features of the present disclosure may be encoded on a variety of computer-readable storage media, with suitable media including magnetic disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memory, and the like. A computer-readable storage medium encoded with a program code may be packaged with a compatible device or provided separately from other devices. In addition, the program code may be encoded and transmitted over wired, optical, and / or wireless networks conforming to various protocols, including the Internet, thereby enabling distribution, for example, via Internet download. Any such computer-readable medium may be on or within a single computer product (e.g., a solid-state drive, hard drive, CD, or an entire computer system), or may be present on or within different computer products within a system or network. A computer system may include a monitor, printer, or other suitable display for providing a user with any of the results described herein.

[0072] The terms "a," "an," or "the" are intended to mean "one or more" unless expressly stated to the contrary. The use of "or" is intended to mean "including or" rather than "without including or" unless expressly stated to the contrary. A reference to a "first" element does not necessarily require that a second element is also provided. Additionally, a reference to a "first" element or a "second" element does not limit the referenced element to a particular location unless expressly stated. The term "based on" is intended to mean "based at least in part on."

[0073] All patents, patent applications, publications, and descriptions mentioned herein are incorporated by reference in their entirety for all purposes. None are admitted to be prior art.

[0074] Additionally, as noted above, an aspect of the present technology is the collection and use of available data from various sources to improve applications and / or suggestions by people to share content from the host application. The present disclosure contemplates that in some examples, this collected data may include personal information data that uniquely identifies a particular person or that can be used to contact or locate a particular person. Such personal information data may include demographic data, location-based data, phone numbers, email addresses, Twitter IDs, home addresses, data or records regarding a user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), birth date, or any other identifying or personal information.

[0075] This disclosure recognizes that the use of such personal information data in the present technology can be used to the benefit of the user. For example, the personal information data can be used to determine suggestions by applications and people to share content that is of greater interest to the user. Thus, the use of such personal information data allows the user to more efficiently control the sharing of content. Additionally, other uses of personal information data that benefit the user are also contemplated by this disclosure. For example, health and fitness data can be used to provide insight into the overall wellness of the user, or as proactive feedback to individuals using the technology to pursue wellness goals.

[0076] This disclosure contemplates that entities responsible for the collection, analysis, disclosure, transmission, storage, or other use of such personal information data will adhere to established privacy policies and / or privacy practices. In particular, such entities should implement and consistently use privacy policies and practices that are generally recognized as meeting or exceeding industry or government requirements for keeping personal information data private and secure. Such policies should be easily accessible by users and should be updated when data collection and / or use changes. Personal information from users should be collected for the entity's lawful and legitimate uses and should not be shared or sold outside of these lawful uses. Furthermore, such collection / sharing should occur after receiving the user's informed consent. In addition, such entities should consider taking necessary measures to protect and secure access to such personal information data and ensure that others who access the personal information data adhere to the entity's privacy policies and procedures. Furthermore, such entities may have themselves assessed by a third party to attest that the entity adheres to widely accepted privacy policies and practices. In addition, policies and practices should be adapted to the particular type of personal information data collected and / or accessed and should be adapted to applicable laws and standards, including jurisdiction-specific considerations. For example, in the United States, the collection of or access to certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA), and health data in other countries may be subject to other regulations and policies and should be handled accordingly. Thus, different privacy practices should be maintained for different types of personal data in each country.

[0077] Notwithstanding the above, the present disclosure also contemplates embodiments in which a user selectively blocks use of or access to personal information data. That is, the present disclosure contemplates that hardware and / or software elements may be provided to prevent or block access to such personal information data. For example, in the case of sharing of content objects, the present technology may be configured to allow a user to select "opt-in" or "opt-out" of participating in the collection of personal information data during registration for the service or at any time thereafter. In another example, a user may choose not to provide data corresponding to the user's previous interactions with various applications along with the user's sharing preferences and / or past user interactions. In yet another example, a user may choose to limit the amount of time that previous application interactions and sharing data are maintained, or to prohibit the collection and tracking of such data altogether. In addition to providing "opt-in" and "opt-out" options, the present disclosure contemplates providing notifications regarding access or use of personal information. For example, a user may be notified upon downloading an app that the user's personal information data will be accessed, and then advised again immediately before the personal information data is accessed by the app.

[0078] This disclosure also contemplates that personal information data should be managed and processed to minimize the risk of unintended or unauthorized access or use. Risk can be minimized by limiting the collection of data and deleting data when it is no longer needed. Additionally, and when applicable, including in certain health-related applications, anonymization of data can be used to protect user privacy. Anonymization can be facilitated as needed by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of data stored (e.g., collecting location data at the city level rather than at the address level), controlling how data is stored (e.g., aggregating data across users), and / or other methods.

[0079] Thus, although the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may also be implemented without requiring access to such personal information data. That is, various embodiments of the present technology are not rendered inoperable due to the absence of all or a portion of such personal information data. For example, suggestions about sharing applications and people may be selected and provided to the user based on only a minimal amount of personal information or non-personal information data, such as content requested by a device associated with the user, other non-personal information available to the content sharing system, or publicly available information.

[0080] Item 1 includes a method of generating a user interface via a computing device, the method including: receiving, by the computing device, image data corresponding to one or more images captured by a camera of the computing device; receiving, by the computing device, location information corresponding to a location of the computing device; receiving, by the computing device, map information corresponding to the location of the computing device; determining, by the computing device, an orientation of the computing device based at least in part on (a) the image data captured by the camera of the computing device, (b) the location information corresponding to the location of the computing device, and (c) the map information corresponding to the location; generating, by the computing device, a user interface configured to display a current location of the computing device and one or more user interface elements representing one or more objects within a threshold distance of the current location of the computing device, where generating the user interface includes selection and positioning of one or more objects within the user interface, where at least one of the selection or positioning of the objects within the user interface is based at least in part on the determined orientation of the computing device; and presenting, by the computing device, the user interface on a screen of the computing device.

[0081] Item 2 includes the method of item 1, wherein the method further includes receiving a direction reading from a compass of the computing device, and determining an orientation of the computing device is further based at least in part on the direction reading received from the compass.

[0082] Item 3 includes the method of item 2, further including reconfiguring a compass of the computing device based at least in part on the determined orientation of the computing device.

[0083] Clause 4 includes the method of clause 1, wherein the location information corresponding to the location of the computing device includes at least one of data received from a Global Positioning System (GPS) receiver of the computing device, data identifying one or more WiFi networks accessible to the computing device, or data identifying one or more additional devices that communicate with the computing device via a short-range wireless communication protocol.

[0084] Clause 5 includes the method of clause 1, wherein the location information corresponding to a location of the computing device includes coordinate data received from a Global Positioning System (GPS) receiver of the computing device, and the method further includes determining updated location information corresponding to the location of the computing device based at least in part on (a) image data captured by a camera of the computing device, (b) the coordinate data received from the GPS receiver of the computing device, and (c) map information corresponding to the location, and reconfiguring the GPS receiver of the computing device based at least in part on the determined updated location information.

[0085] Clause 6 includes the method of clause 1, wherein determining an orientation of the computing device includes analyzing image data captured by a camera of the computing device and detecting one or more street boundaries in the image data, and comparing one or more trajectories of the one or more street boundaries to street vector data in the received map information.

[0086] Clause 7 includes the method of clause 6, wherein determining an orientation of the computing device further includes analyzing image data captured by a camera of the computing device and detecting a horizon within the image data.

[0087] Clause 8 includes the method of clause 1, wherein the method further includes generating a route for the computing device from a current location of the computing device to a second location, and wherein the user interface displays next direction instructions based at least in part on the route.

[0088] Clause 9 includes a non-transitory computer-readable storage medium storing computer-executable instructions that, when executed by one or more processors of a computing device, cause the one or more processors to receive image data corresponding to one or more images captured by a camera of the computing device, receive location information corresponding to a location of the computing device, receive map information corresponding to the location of the computing device, determine an orientation of the computing device based at least in part on (a) the image data captured by the camera of the computing device, (b) the location information corresponding to the location of the computing device, and (c) the map information corresponding to the location, generate a user interface configured to display a current location of the computing device and one or more user interface elements representing one or more objects within a threshold distance of the current location of the computing device, wherein generating the user interface includes selecting and placing one or more objects within the user interface, at least one of the selection or placing of the objects within the user interface being based at least in part on the determined orientation of the computing device, and causing the user interface to be presented on a screen of the computing device.

[0089] Clause 10 includes the non-transitory computer-readable storage medium of clause 9, wherein the computer-executable instructions further cause the one or more processors to receive an orientation reading from a compass of the computing device and determine an orientation of the computing device further based at least in part on the orientation reading received from the compass.

[0090] Clause 11 includes the non-transitory computer-readable storage medium of clause 10, wherein the computer-executable instructions further cause the one or more processors to reconfigure a compass of the computing device based at least in part on the determined orientation of the computing device.

[0091] Clause 12 includes the non-transitory computer-readable storage medium of clause 9, wherein the location information corresponding to a location of the computing device includes at least one of data received from a Global Positioning System (GPS) receiver of the computing device, data identifying one or more WiFi networks accessible to the computing device, or data identifying one or more additional devices that communicate with the computing device via a short-range wireless communication protocol.

[0092] Clause 13 includes the non-transitory computer-readable storage medium of clause 9, wherein the location information corresponding to a location of the computing device includes coordinate data received from a Global Positioning System (GPS) receiver of the computing device, and the computer-executable instructions further cause the one or more processors to determine updated location information corresponding to the location of the computing device based at least in part on (a) image data captured by a camera of the computing device, (b) the coordinate data received from the GPS receiver of the computing device, and (c) map information corresponding to the location, and to reconfigure the GPS receiver of the computing device based at least in part on the determined updated location information.

[0093] Clause 14 includes the non-transitory computer-readable storage medium of clause 9, wherein determining an orientation of the computing device includes analyzing image data captured by a camera of the computing device to detect one or more street boundaries in the image data, and comparing one or more trajectories of the one or more street boundaries with street vector data in the received map information.

[0094] Item 15 includes the non-transitory computer-readable storage medium of item 14, wherein determining an orientation of the computing device further includes analyzing image data captured by a camera of the computing device and detecting a horizon within the image data.

[0095] Clause 16 includes the non-transitory computer-readable storage medium of clause 9, wherein the computer-executable instructions further cause the one or more processors to generate a route for the computing device from a current location of the computing device to a second location, and the user interface displays next direction instructions based at least in part on the route.

[0096] Clause 17 includes a computing device having one or more processors in communication with one or more memories, the one or more processors configured to execute computer-executable instructions stored in the one or more memories to at least receive image data corresponding to one or more images captured by a camera of the computing device, receive location information corresponding to a location of the computing device, receive map information corresponding to the location of the computing device, determine an orientation of the computing device based at least in part on (a) the image data captured by the camera of the computing device, (b) the location information corresponding to the location of the computing device, and (c) the map information corresponding to the location, generate a user interface configured to display a current location of the computing device and one or more user interface elements representing one or more objects within a threshold distance of the current location of the computing device, wherein generating the user interface includes selecting and placing one or more objects within the user interface, wherein at least one of the selection or placement of the objects within the user interface is based at least in part on the determined orientation of the computing device, and presenting the user interface on a screen of the computing device.

[0097] Clause 18 includes the computing device of clause 17, wherein the instructions are further executed to at least receive an orientation reading from a compass of the computing device and determine an orientation of the computing device further based at least in part on the orientation reading received from the compass, and reconfigure the compass of the computing device based at least in part on the determined orientation of the computing device.

[0098] Paragraph 19 further describes a method for determining an orientation of a computing device, the method comprising: analyzing image data captured by a camera of the computing device; and detecting one or more street boundaries within the image data. and comparing one or more trajectories of one or more street boundaries with street vector data in the received map information.

[0099] Clause 20 describes a method for determining updated location information corresponding to the location of the computing device based at least in part on: (a) image data captured by a camera of the computing device; (b) the coordinate data received from the GPS receiver of the computing device; and (c) map information corresponding to the location; The computing device of claim 17, further comprising: reconfiguring a GPS receiver of the computing device based at least in part on the determined updated location information.

Claims

1. 1. A method for generating a user interface via a computing device, comprising: receiving, by a computing device, image data corresponding to one or more images captured by a camera at a current location of the computing device while navigating along a route from a first physical location to a second physical location; receiving, by the computing device, location information corresponding to the current location of the computing device; receiving, by the computing device, map information corresponding to the current location of the computing device; determining, by the computing device, an orientation of the computing device based at least in part on (a) the image data captured by the camera of the computing device at the current location, (b) the location information corresponding to the current location of the computing device, and (c) the map information corresponding to the current location; generating, by the computing device, a user interface configured to display the current location of the computing device and one or more user interface elements representing navigation user interface components corresponding to navigation directions for the route based on the current location and the orientation of the computing device, wherein generating the user interface includes overlaying the navigation user interface components on the image data captured by the camera; presenting said user interface by said computing device on a screen of said computing device; A method comprising:

2. 10. The method of claim 1, further comprising receiving an orientation reading from a compass of the computing device, and determining the orientation of the computing device is further based at least in part on the orientation reading received from the compass.

3. The method of claim 2 , further comprising reconfiguring the compass of the computing device based at least in part on the determined orientation of the computing device.

4. 2. The method of claim 1 , wherein the location information corresponding to the location of the computing device includes at least one of: data received from a Global Positioning System (GPS) receiver of the computing device, data identifying one or more WiFi networks accessible to the computing device, or data identifying one or more additional devices communicating with the computing device via a short-range wireless communication protocol.

5. and wherein the location information corresponding to the location of the computing device includes coordinate data received from a Global Positioning System (GPS) receiver of the computing device, the method comprising: determining updated location information corresponding to the location of the computing device based at least in part on (a) the image data captured by the camera of the computing device, (b) the coordinate data received from the GPS receiver of the computing device, and (c) the map information corresponding to the location; reconfiguring the GPS receiver of the computing device based at least in part on the determined updated location information; and The method of claim 1 further comprising:

6. Determining the orientation of the computing device includes: analyzing the image data captured by the camera of the computing device to detect one or more street boundaries within the image data; comparing one or more trajectories of the one or more street boundaries with street vector data in the received map information; The method of claim 1 , comprising:

7. 7. The method of claim 6, wherein determining the orientation of the computing device further comprises: analyzing the image data captured by the camera of the computing device; and detecting a horizon within the image data.

8. the method further comprising generating a route for the computing device from the current location of the computing device to a second location, the user interface displaying next direction instructions based at least in part on the route. The method of claim 1.

9. 1. A non-transitory computer-readable storage medium storing computer-executable instructions, the computer-executable instructions, when executed by one or more processors of a computing device, causing the one or more processors to: receiving image data corresponding to one or more images captured by a camera at a current location of the computing device while navigating along a route from a first physical location to a second physical location; receiving location information corresponding to the current location of the computing device; receiving map information corresponding to the location of the computing device; determining an orientation of the computing device based at least in part on (a) the image data captured by the camera of the computing device at the current location, (b) the location information corresponding to the current location of the computing device, and (c) the map information corresponding to the current location; generating a user interface configured to display the current location of the computing device and one or more user interface elements representing navigation user interface components corresponding to navigation directions for the route based on the current location and the orientation of the computing device, wherein generating the user interface includes overlaying the navigation user interface components on the image data captured by the camera; A non-transitory computer-readable storage medium that causes the user interface to be presented on a screen of the computing device.

10. The computer executable instructions may further cause the one or more processors to:

10. The computer-readable storage medium of claim 9, wherein receiving an orientation reading from a compass of the computing device and determining the orientation of the computing device is further based at least in part on the orientation reading received from the compass.

11. The computer executable instructions may further cause the one or more processors to: The computer-readable storage medium of claim 10 , further comprising: causing the compass of the computing device to be reconfigured based at least in part on the determined orientation of the computing device.

12. 10. The computer-readable storage medium of claim 9, wherein the location information corresponding to the location of the computing device includes at least one of data received from a Global Positioning System (GPS) receiver of the computing device, data identifying one or more WiFi networks accessible to the computing device, or data identifying one or more additional devices in communication with the computing device via a short-range wireless communication protocol.

13. The location information corresponding to the location of the computing device includes coordinate data received from a Global Positioning System (GPS) receiver of the computing device, and the computer-executable instructions further cause the one or more processors to: determining updated location information corresponding to the location of the computing device based at least in part on (a) the image data captured by the camera of the computing device, (b) the coordinate data received from the GPS receiver of the computing device, and (c) the map information corresponding to the location; 10. The computer-readable storage medium of claim 9, further comprising: causing the GPS receiver of the computing device to be reconfigured based at least in part on the determined updated location information.

14. Determining the orientation of the computing device comprises: analyzing the image data captured by the camera of the computing device to detect one or more street boundaries within the image data; comparing one or more trajectories of the one or more street boundaries with street vector data in the received map information; 10. The computer readable storage medium of claim 9, comprising:

15. 15. The computer-readable storage medium of claim 14, wherein determining the orientation of the computing device further comprises: analyzing the image data captured by the camera of the computing device; and detecting a horizon within the image data.

16. The computer executable instructions may further cause the one or more processors to:

10. The computer-readable storage medium of claim 9, further comprising: generating a route for the computing device from the current location of the computing device to a second location; and wherein the user interface displays next direction instructions based at least in part on the route.

17. One or more processors in communication with one or more memories wherein the one or more processors execute computer-executable instructions stored in the one or more memories to perform at least receiving image data corresponding to one or more images captured by a camera of the computing device while navigating along a route from a first physical location to a second physical location; receiving location information corresponding to the current location of the computing device; receiving map information corresponding to the current location of the computing device; (a) determining an orientation of the computing device based at least in part on the image data captured by the camera of the computing device at the current location, (b) the location information corresponding to the current location of the computing device, and (c) the map information corresponding to the current location; generating a user interface configured to display the current location of the computing device and one or more user interface elements representing navigation user interface components corresponding to navigation directions for the route based on the current location and the orientation of the computing device, wherein generating the user interface includes overlaying the navigation user interface components on the image data captured by the camera; A computing device configured to present the user interface on a screen of the computing device.

18. The instructions are further executed to: receiving an orientation reading from a compass of the computing device and determining the orientation of the computing device further based at least in part on the orientation reading received from the compass; The computing device of claim 17 , further comprising: reconfiguring the compass of the computing device based at least in part on the determined orientation of the computing device.

19. Determining the orientation of the computing device includes: analyzing the image data captured by the camera of the computing device to detect one or more street boundaries within the image data; comparing one or more trajectories of the one or more street boundaries with street vector data in the received map information; 20. The computing device of claim 17, comprising:

20. and wherein the location information corresponding to the location of the computing device includes coordinate data received from a Global Positioning System (GPS) receiver of the computing device, the instructions are further executed to at least: determining updated location information corresponding to the location of the computing device based at least in part on (a) the image data captured by the camera of the computing device, (b) the coordinate data received from the GPS receiver of the computing device, and (c) the map information corresponding to the location; 20. The computing device of claim 17, further comprising: reconfiguring the GPS receiver of the computing device based at least in part on the determined updated location information.

Images