Explicit cues of signage visibility in driving navigation

Abstract

To provide navigation instructions with indications of visibility of the visual aid during navigation, the computing device receives a request from a user for navigation directions from a start location to a destination location, and in response to the request, obtains a set of navigation instructions for traveling along a path from the start location to the destination location. For at least one maneuver, the computing device identifies a visual aid to assist the user in identifying a location corresponding to the maneuver, determines a visibility magnitude for the visual aid, and adjusts the navigation instructions corresponding to the maneuver in the set of navigation instructions to include a description of the visibility magnitude for the visual aid. The computing device provides the set of navigation instructions including the adjusted navigation instructions for presentation to the user.

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE This disclosure relates to navigation instructions, and in particular to providing an explanation of the visibility of visual aids when instructing a user. [Background technology]

[0002] The background art description provided herein is intended to provide a general background to the present disclosure. Work by the presently named inventors within the scope of what is described in this background art section, as well as aspects of this description that may not otherwise be considered prior art at the time of filing, are not admitted, expressly or impliedly, as prior art to the present disclosure.

[0003] Systems that automatically route between a driver's geographic locations typically use signs such as distances, street names, and building numbers to generate route-based navigation instructions. For example, these systems may provide the driver with instructions such as, "Go a quarter mile, then turn right onto Maple Street." However, it can be difficult for drivers to accurately judge distances, and it is not always easy for drivers to see street signs. Additionally, there are geographic areas where street and road signage is inadequate.

[0004] To provide the driver with guidance more similar to another person speaking to the driver, the navigation instructions can be augmented with reference to prominent objects along the route, such as visually prominent buildings and billboards. These prominent objects, along with street signs, can be referred to as "visual aids." Thus, the system can generate navigation instructions such as "A quarter mile ahead, you will see a McDonald's® restaurant on your right. Take the next right onto Maple Street." To this end, the operator can input location descriptions and signs (e.g., street addresses, coordinates) for the visual aids so that the system can automatically select the appropriate visual aid when generating the navigation instructions.

[0005] However, not all visual aids are always visible: for example, some street signs may be brightly lit at night but generally not visible during the day, while building signs or billboards may be easily noticeable during the day but may be poorly lit and therefore not visible at night. Summary of the Invention [Means for solving the problem]

[0006] To assist a user in navigation, the sign visibility indicator system obtains a set of navigation instructions for navigating the user from a starting location to a destination location. The sign visibility indicator system then identifies, for each navigation instruction, visual aids included within the navigation instruction to assist the user in identifying a location or intersection for a maneuver. For example, if the navigation instruction is "Turn right on ABC Road," the sign visibility indicator system identifies the street sign for ABC Road as a visual aid. In another example, if the navigation instruction is "Turn right at the intersection near Burger King," the sign visibility indicator system identifies the Burger King sign as a visual aid.

[0007] The sign visibility indicator system may then determine a measure of visibility for the visual aid. For example, the sign visibility indicator system may generate a visibility metric for the visual aid based on objects occluding the visual aid, the size of the visual aid, the amount of illumination for the visual aid, the time of day, the season, weather conditions, a popularity metric for the visual aid, a type of maneuver performed at a location corresponding to the visual aid, etc. The sign visibility indicator system may identify at least some of these factors for determining the visibility metric by obtaining a three-dimensional (3D) panoramic street level image of the geographic area including the visual aid. The sign visibility indicator system may then analyze the street level image to identify objects occluding the visual aid, the size of the visual aid, the amount of illumination for the visual aid, etc.

[0008] In some implementations, the sign visibility indicator system may identify a direction in which a user is traveling when approaching a visual aid based on navigation instructions. The sign visibility indicator system may then generate a visibility metric for the identified direction. For example, the sign visibility indicator system may acquire a 3D panoramic street-level image of the geographic area including the visual aid from a virtual camera perspective that has an orientation that matches the user's orientation when the user approaches the visual aid.

[0009] In either case, the sign visibility indicator system may generate a visibility description of the visual aid for inclusion within the navigation instructions. For example, the description may be "This sign is partially behind trees and is difficult to see," or "There are no street lights around the corner and the sign is small, please slow down early." In some implementations, the sign visibility indicator system generates a visibility description for a subset of the navigation instructions included within the set of navigation instructions. For example, the sign visibility indicator system may generate a visibility description for a visual aid when the visibility metric is below a first threshold or above a second threshold to provide further details to the user when the visual aid is particularly visible or when the visual aid is particularly difficult to see.

[0010] The sign visibility indicator system may then present the user with a set of navigation instructions that includes at least one description of the visibility of the visual aid. In this manner, the navigation instructions are improved by providing additional detail to assist the user in navigating the route. As a result, the sign visibility indicator system reduces the likelihood of navigational errors, such as making a wrong turn.

[0011] An exemplary embodiment of these techniques is a method for providing navigation instructions with indications of visibility of a visual aid during navigation. The method includes receiving a request from a user for navigation instructions from a start location to a destination location, and obtaining a set of navigation instructions for traveling along a path from the start location to the destination location in response to the request. The set of navigation instructions includes one or more maneuvers at corresponding locations along the path. For at least one maneuver, the method includes identifying a visual aid to assist the user in identifying a location corresponding to the maneuver, determining a visibility magnitude for the visual aid, and adjusting navigation instructions corresponding to the maneuver in the set of navigation instructions to include a description of the visibility magnitude for the visual aid. The method further includes providing the set of navigation instructions including the adjusted navigation instructions for presentation to the user.

[0012] Another exemplary embodiment of these techniques is a computing device for providing navigation instructions with indications of visibility of a visual aid during navigation. The computing device includes one or more processors and a computer-readable memory (optionally non-transitory computer-readable memory) coupled to the one or more processors and having instructions stored thereon. The instructions, when executed by the one or more processors, cause the computing device to receive a request from a user for navigation instructions from a starting location to a destination location, and, in response to the request, obtain a set of navigation instructions for moving along a path from the starting location to the destination location. The set of navigation instructions includes one or more maneuvers at corresponding locations along the path. For at least one maneuver, the instructions cause the computing device to identify a visual aid to assist the user in identifying a location corresponding to the maneuver, determine a magnitude of visibility for the visual aid, and adjust the navigation instruction corresponding to the maneuver in the set of navigation instructions to include a description of the magnitude of visibility for the visual aid. The instructions further cause the computing device to provide, for presentation to a user, a set of navigation directions including the tailored navigation instructions.

[0013] Yet another exemplary embodiment of these techniques is a method for presenting navigation instructions with visibility indicators of visual aids during navigation. The method includes receiving a request from a user for navigation instructions from a start location to a destination location, and, in response to the request, obtaining a set of navigation instructions for traveling along a path from the start location to the destination location. The set of navigation instructions includes one or more maneuvers at corresponding locations along the path, and at least one navigation instruction includes a visual aid to assist the user in identifying a location corresponding to the maneuver included in the navigation instruction, and a visibility magnitude description for viewing the visual aid. The method further includes presenting the set of navigation instructions, including presenting at least one navigation instruction having a visibility magnitude description for the visual aid. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a block diagram of an example communication system in which client and server devices can operate to implement a sign visibility indicator system. [Diagram 2] 2 is a diagram of an example visual aid data table contained in a visual aid database that may be implemented in the system of FIG. 1. [Diagram 3] FIG. 2 is an exemplary three-dimensional (3D) panoramic street level image of a geographic area including a visual aid that can be analyzed in the system of FIG. 1 to determine the visibility of the visual aid. [Figure 4] 2 is an example navigation display of a route that can be analyzed in the system of FIG. 1 to determine the direction a user is traveling when approaching a visual aid. [Diagram 5] 1 is a diagram of an example navigation display presented on a client device including navigation instructions with a description of the visibility of visual aids. [Figure 6] FIG. 2 is a flow diagram of an example method for providing navigation instructions with visual aid visibility indicators during navigation that may be implemented in a server device. [Figure 7] 1 is a flow diagram of an example method for presenting navigation instructions with visual aid visibility indicators during navigation that may be implemented on a client device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] overview In general, techniques for providing visual aid visibility indications during navigation can be implemented within a system including one or more client devices, one or more network servers, or a combination of these devices. For ease of understanding, however, the following examples will focus primarily on an embodiment in which a client device sends a request to a server device for navigation instructions from a starting location to a destination location. The server device then retrieves a set of navigation instructions including maneuvers at corresponding locations along the route. For each maneuver, the set of navigation instructions may include visual aids, such as street signs and building signs, to assist the user in identifying intersections for performing that maneuver.

[0016] The server device may then determine a magnitude of visibility of the visual aid, for example, from a direction that a user would be facing when viewing the visual aid in accordance with navigation instructions. For example, the server device may generate a visibility metric for the visual aid based on objects occluding the visual aid, the size of the visual aid, the amount of lighting on the visual aid, the time of day, season, weather conditions, a popularity metric for the visual aid, a type of maneuver being performed at a location corresponding to the visual aid, etc.

[0017] For one or more maneuvers, the server device may generate a description of the visibility of the visual aid for inclusion within the corresponding navigation instructions based on the determined magnitude of visibility. The server device then provides the set of navigation instructions, including the description of the visibility of the visual aid, to the client device for presentation to the user.

[0018] Exemplary Hardware and Software Components 1 illustrates an environment 10 in which at least some of the techniques for providing a description of visibility of a visual aid during navigation may be implemented. The environment 10 includes a client device 12 and a server device 14 interconnected via a communications network 16. The network 16 may be a public network, such as the Internet, or a private network, such as an intranet. The server device 14 may communicate with various databases and with other server devices, such as a navigation server, a map data server, a live traffic server, a weather server, etc.

[0019] The server device 14 may be communicatively coupled to a database 52 that stores visual aids and geographic information for each visual aid, such as the location and height of the visual aid, the orientation the visual aid faces, the size of the visual aid, the appearance of the visual aid, the name of the visual aid, etc., as described in more detail below with reference to Figure 2. The server device 14 may also be communicatively coupled to a map database 50 and a real-world image database 54 that stores real-world images for various geographic areas.

[0020] The client device 12 may include a portable electronic device, such as a smartphone, a wearable device, such as a smart watch or a head-mounted display, or a tablet computer. The client device 12 may include a processing module 22, which may include one or more central processing units (CPUs), one or more graphics processing units (GPUs) for efficiently rendering graphical content, an application specific integrated circuit (ASIC), or any other suitable type of processing hardware. Additionally, the client device 12 may include a memory 24, which may consist of persistent (e.g., hard disk, flash drive) and / or non-persistent (e.g., RAM) components. In the exemplary implementation shown in FIG. 1, the memory 24 stores instructions for implementing a geographic application 26, which may also be referred to as a "mapping application 26."

[0021] In addition, client device 12 includes a camera (not shown), a user interface 28, and a network interface 30. In any case, user interface 28 may include one or more input components, such as a touch screen, a microphone, a keyboard, etc., as well as one or more output components, such as a screen and a speaker.

[0022] Network interface 30 may support short-range and / or long-range communication. For example, network interface 30 may support cellular communication, personal area network protocols such as IEEE 802.11 (e.g., Wi-Fi) and 802.15 (Bluetooth). In some implementations, client device 12 includes multiple network interface modules for interconnecting multiple devices within client device 12 and for connecting client device 12 to network 16.

[0023] Additionally, the network interface 30 may support wireless positioning in some cases. For example, the network interface 30 may support Wi-Fi triangulation. In other cases, the client device 12 may include a dedicated positioning module 32, such as a GPS module. The client device 12 may also include other sensors, such as an accelerometer, a gyroscope, a magnetometer such as a compass, etc.

[0024] Continuing to refer to FIG. 1, client device 12 may communicate with server device 14 over network 16, which may be a wide area network such as the Internet. Server device 14 may be implemented in one or more server devices, including devices distributed across multiple geographic locations. Server device 14 may implement a navigation instruction generator 42 and a sign visibility indicator module 44. These components 42-44 may be implemented using any suitable combination of hardware, firmware, and software. The server device may access databases, such as a map database 50, a visual aid database 52, and a real-world image database 54, which may be implemented using any suitable data storage and access technique.

[0025] In operation, the sign visibility indicator module 44 may receive a request for navigation instructions from the client device 12. The sign visibility indicator module 44 may obtain a set of navigation instructions for proceeding along a route from a start location to a destination location, for example from the navigation generator 42. The sign visibility indicator module 44 may then identify street names, buildings, or other landmarks included in the set of navigation instructions as visual aids. The sign visibility indicator module 44 may then determine visibility magnitudes for viewing the visual aids from the perspective of a user approaching the visual aids from various positions along the route. More specifically, the sign visibility indicator module 44 may retrieve a 3D panoramic street level image of the geographic area containing the visual aids from the real-world database 54. The sign visibility indicator module 44 may analyze the street level image to identify objects occluding the visual aids, the size of the visual aids, the amount of illumination on the visual aids, etc., and store the results of the analysis in the visual aids database 52. The sign visibility indicator module 44 may then generate a description of the visibility of the visual aids.

[0026] Visual aids database 52 may store information about prominent geographic entities that are visible when driving (or biking, walking, or otherwise moving along a navigation path) and thus may serve as visual aids. For example, visual aids database 52 may store geographic information about visual aids, such as the Mila Street sign 70, the Denver Walgreens® sign 72, the Willow Creek Dr. street sign 74, and any other suitable visual aids.

[0027] For each visual aid, visual aid database 52 may store one or more photographs, two-dimensional or three-dimensional geographic coordinates, the height of the visual aid, the orientation the visual aid faces, the size of the visual aid, the appearance of the visual aid, the name of the visual aid, etc. To populate visual aid database 52, server device 14 may receive satellite imagery, photos and videos submitted by various users, street level imagery collected by vehicles equipped with specialized panoramic cameras, imagery of streets and sidewalks collected by pedestrians and cyclists, etc. from real-world database 54. Similarly, visual aid database 52 may receive descriptions of visual aids from various sources, such as the operator of server device 14 or people submitting user-generated content.

[0028] The sign visibility indicator module 44 may then identify objects in the street level image, such as street signs, building signs, poles, street lights, trees, buildings, roads, vehicles, etc. The sign visibility indicator module 44 may then compare the location of the object with the location of the visual aid to identify the object that is occluding the visual aid. The sign visibility indicator module 44 may identify the object using object recognition techniques, semantic segmentation, and / or machine learning techniques. More specifically, the sign visibility indicator module 44 may obtain image data of several template objects from a database as training data for training the machine learning model, where the object types of the objects are known. The template objects may have various object types, such as street signs, building signs, poles, street lights, trees, buildings, roads, vehicles, etc.

[0029] The sign visibility indicator module 44 may identify visual features of each template object. The features of the template object may include the geometry of the edges of the template object, RGB pixel values ​​or colors within the template object, changes in RGB pixel values ​​at different adjacent locations within the template object, etc. These features may be identified by detecting stable regions within the template object that are detectable despite blur, motion, distortion, orientation, lighting, scaling, and / or other changes in camera viewpoint. Stable regions may be extracted from the template object using scale invariant feature transform (SIFT), speeded up robust features (SURF), fast retina keypoint (FREAK), binary robust invariant scalable keypoints (BRISK), or any other suitable computer vision technique. In some implementations, keypoints may be located in high contrast regions of the template object, such as edges within the template object. Bounding boxes may be formed around the keypoints, and the portions of the template object created by the bounding boxes may be features.

[0030] The machine learning models may be generated based on features of template objects corresponding to various object types such as street signs, building signs, poles, street lights, trees, buildings, roads, vehicles, etc. The machine learning models may include a feature set for each object type, such as a first feature set for signs, a second feature set for buildings, a third feature set for street lights, a fourth feature set for trees, a fifth feature set for roads, a sixth feature set for vehicles, etc. For example, the color of an object may be stored as template features along with the width of the object, the color gradation of the object, and / or other visual characteristics of the object.

[0031] The sign visibility indicator module 44 may then obtain image data for a portion of the street level image that corresponds to one or more objects and identify features of the one or more objects using techniques similar to those described above. The sign visibility indicator module 44 may compare features identified for objects in the street level image using image classification and / or machine learning techniques. The machine learning techniques may include linear regression, polynomial regression, logistic regression, random forests, boosting, nearest neighbors, Bayesian networks, neural networks, support vector machines, or any other suitable machine learning techniques. Template features from the machine learning model may each be compared to the features of the object.

[0032] In some embodiments, the template features may be compared to the features of the object using a nearest neighbor algorithm. The nearest neighbor algorithm may identify the template features that are closest to the object features by creating a numerical representation of the features to generate a feature vector, such as the pixel width and height of the object, the RGB pixel values ​​for the object, the color gradation within the object, etc. The numerical representation or feature vector of the object features may be compared to the feature vector of each template object to determine a vector distance between the features of the object and each template object. The sign visibility indicator module 44 may then determine the object type of the object based on the magnitude of similarity, or vector distance in the nearest neighbor algorithm, between the features of the object and the features of each template object that represent a particular object type. For example, for an object in a street level image, the sign visibility indicator module 44 may identify the top three template objects that have template feature vectors that are closest to the feature vector of the object. If a majority of the identified template object or objects are street signs, the sign visibility indicator module 44 may identify the object in the street level image as a street sign. The sign visibility indicator module 44 may repeat this process for multiple objects within the street level imagery.

[0033] The server device 14 may then compare the position of the object with the position of the visual aid to identify objects that are occluding the visual aid. The sign visibility indicator module 44 may also identify street light objects in the street level image and compare the position of the street light objects with the position of the visual aid to determine the amount of illumination for the visual aid. In addition, the sign visibility indicator module 44 may compare the positions of other objects associated with the illumination with the position of the visual aid to determine the amount of illumination for the visual aid.

[0034] In any case, the server device 14 may generate an occlusion metric for the visual aid based on the object occluding the visual aid and / or the object's position relative to the visual aid's position. For example, a tree may partially occlude the visual aid by covering a lower portion of a street sign, which may result in a lower occlusion metric than a building that completely occludes the visual aid. The server device 14 may also generate a visibility metric for the visual aid based on any suitable combination of the occlusion metric for the visual aid, the size of the visual aid, the height of the visual aid, the position of the visual aid, the orientation the visual aid faces, the amount of lighting on the visual aid, the time of day, the season, the weather conditions, a popularity metric for the visual aid, the type of maneuver performed at the location corresponding to the visual aid, and the like.

[0035] For example, the visibility metric may be inversely proportional to the occlusion metric, in that visibility decreases with increasing occlusion metric, which indicates a higher level of occlusion of the visual aid. Further, the visibility metric may be reduced based on foggy, rainy, overcast, or snowy weather conditions, while the visibility metric may be increased during clear weather conditions.

[0036] Additionally, the visibility metric may be adjusted based on the light level metric, the time of day, and the season. During the day, which is determined based on the time of day and / or the season, the visibility metric may be higher than at night. However, the visibility metric may increase at night when the light levels are increased. The light level metric may also vary depending on the time of day and / or the season, such as when a business sign may be turned on during business hours but turned off after the store closes.

[0037] Additionally, the visibility metric may be adjusted based on the location of the visual aid, the orientation the visual aid faces, and the type of maneuver being performed at the location corresponding to the visual aid. For example, if the visual aid is to the left of the user, but the type of maneuver is a right turn, the visibility metric may be reduced since it is difficult to see objects away from the direction of the maneuver.

[0038] A data table 200 including an exemplary geographic information index for each visual aid contained within visual aid database 52 is shown in Figure 2. The geographic information for each visual aid may then be used to generate a visibility metric for the visual aid and / or a description of the visibility of the visual aid.

[0039] For each landmark, the data table 200 includes a visual aid identifier 202, a location of the visual aid 204, including two-dimensional or three-dimensional geographic coordinates and the height of the visual aid, a direction 206 that the visual aid faces, a size or dimension 208 of the visual aid, a textual description of the visual aid's appearance 210, a name 212 of the visual aid, and an occlusion metric 214 for the visual aid indicating whether and to what extent an object occludes the visual aid. For example, the occlusion metric 214 may be a score between 1 and 100, where a score of 1 indicates that the visual aid is visible and that there are no objects occluding the visual aid. A score of 100 may indicate that the visual aid is completely occluded by other objects, such as buildings or trees, and a score between 1 and 100 may indicate that the visual aid is partially occluded.

[0040] The data table 200 may also include a visibility metric (not shown) for the visual aid that indicates a measure of the visibility or popularity of a particular visual aid. For example, McDonald's® may have a high visibility metric based on the number of people who are familiar with and can recognize McDonald's restaurants, while a local business may have a lower visibility metric. A visual aid with a higher visibility metric may have higher visibility because it can be recognized from further away.

[0041] In addition, the data table 200 may include a lighting level metric (not shown) for the visual aid indicating the amount of light present on the visual aid. The lighting level metric may be higher for visual aids that are in proximity to street lights or have brightly lit signs than for visual aids that are not in proximity to street lights or have dimly lit signs. The lighting level metric may change based on the time of day. For example, a business sign may be turned on during business hours but turned off after the store closes. Thus, the lighting level metric for a business sign may be higher during business hours than during hours when the store is closed.

[0042] For example, a first entry in data table 200 includes geographic information for the Walgreens® sign. According to data table 200, the Walgreens® sign is located at a height of 5 m and faces east. The Walgreens® sign is 4 m wide, 2 m long, and includes white cursive lettering and a red background. The occlusion metric for the Walgreens® sign is 95 on a scale of 1 to 100, indicating that the sign is occluded, for example, by other buildings, trees, etc.

[0043] The occlusion metric for a visual aid can be determined, by way of example, by accessing a three-dimensional model associated with the geographic area in which the visual aid is located. The model can include a three-dimensional representation of the visual aid. In one implementation, the three-dimensional model can be a stereo mesh model of the geographic area including a plurality of mesh triangles associated with the visual aid. The model can also include or have access to a synthetic model of geographic features or other landmarks within the geographic area, such as a synthetic model provided in a virtual globe application. The model can also include textures mapped to surfaces within the three-dimensional model. The textures can include photographs or other images associated with the geographic area and can depict geographic objects such as trees (with or without leaves), bridge trestles, and other structures that may occlude the visibility of the visual aid within the geographic area.

[0044] FIG. 3 illustrates an exemplary 3D panoramic street level image 300 of a geographic area including a visual aid 302 (Joe's Restaurant). Server device 14 analyzes street level image 300 using object recognition and / or semantic segmentation techniques to identify objects in street level image 300 and determine an object type for each object. For example, server device 14 analyzes street level image 300 to identify street signs 302, power lines 304, and street lights 306 as visual aids. Server device 14 then determines the position of each object 302-306 in the street level image. For example, server device 14 may determine the position of each object 302-306 in the real world image by identifying a frustum of a virtual camera depicting the street level image and mapping the position within the frustum to pixels in the street level image. Server device 14 may then compare the positions of objects 304, 306 to the position of business sign object 302 to determine the extent to which objects 304, 306 are occluding business sign object 302. In this case, no objects appear to be occluding business sign object 302, and therefore the occlusion metric for business sign object 302 may be a score of 1 on a scale of 1 to 100, or may otherwise indicate that business sign object 302 is not occluded. Additionally, server device 14 may compare the positions of objects 304, 306 to the position of business sign object 302 to determine the lighting level of business sign object 302. In this scenario, street light 306 is not in close proximity to business sign object 302, which may result in a low lighting level metric for business sign object 302. Server device 14 may also determine whether business sign object 302 has its own lighting (e.g., the business sign is a neon sign) and adjust the lighting level metric accordingly.

[0045] Additionally, server device 14 may generate separate occlusion metrics and / or lighting level metrics for the visual aid from various orientations for viewing the visual aid. For example, if the user is traveling east when viewing the visual aid, server device 14 may generate occlusion metrics and / or lighting level metrics for viewing the visual aid from the east. To generate separate occlusion metrics and / or lighting level metrics, server device 14 may obtain 3D panoramic street level images of the visual aid from multiple viewpoints of multiple virtual cameras having multiple orientations or positions. Server device 14 may then analyze the 3D panoramic street level images of the visual aid from the viewpoints of virtual cameras having orientations corresponding to the orientation the user is facing when approaching the visual aid to determine occlusion metrics and / or lighting level metrics for the visual aid.

[0046] To determine the orientation the user is facing when approaching the visual aid, the server device 14 may analyze the set of navigation instructions to determine the direction the user is traveling along the path when the user approaches the visual aid. FIG. 4 shows an example display 400 of a path 406 from a start location 402 to a destination location 412. The server device 14 may analyze positions and / or maneuvers along the path 406 to determine the direction the user is traveling when approaching the visual aid. The server device 14 may then obtain a 3D panoramic street level image of the visual aid from a virtual camera perspective having an orientation corresponding to the orientation the user is facing when approaching the visual aid to determine an occlusion metric and / or a lighting level metric for the visual aid.

[0047] In any case, the server device 14 may use the geographic information for each visual aid, including a photograph of the visual aid, two-dimensional or three-dimensional geographic coordinates of the visual aid, the height of the visual aid, the orientation the visual aid faces, the type of maneuver to be performed at a location corresponding to the visual aid, the size of the visual aid, an occlusion metric for the visual aid, a lighting level metric for the visual aid, a popularity metric for the visual aid, etc., for different weather conditions and / or at different times of day to generate a visibility metric for the visual aid from the orientation from which a user would be approaching the visual aid following navigation instructions, for example.

[0048] The server device 14 may then generate a description of the magnitude of visibility in the visual aid according to a visibility metric. The description of the magnitude of visibility in the visual aid may include a description of the lighting level to see the visual aid according to a lighting level metric, a description of the object occluding the visual aid according to an occlusion metric, a description of the size of the visual aid, a description of the location of the visual aid relative to the user (e.g., "The street sign is on the right side of the street"), or any other suitable description of the magnitude of visibility in the visual aid. Additionally, the description of the magnitude of visibility in the visual aid may include a recommendation based on the magnitude of visibility in the visual aid, such as "Please slow down early."

[0049] More specifically, when the visibility metric for a visual aid is below a first threshold, for example, the server device 14 may generate an explanation indicating that low visibility of the visual aid is occurring. When the visibility metric for a visual aid is above a first threshold but below a second threshold, the server device 14 may generate an explanation indicating that partial visibility of the visual aid is occurring. When the visibility metric for a visual aid is above a second threshold, the server device 14 may generate an explanation indicating that high visibility of the visual aid is occurring. In another example, when the visibility metric for a visual aid is below a threshold visibility metric and the size of the visual aid is below a threshold size, the server device 14 may generate an explanation indicating that low visibility of the visual aid is occurring because the visual aid is small. In yet another example, when the visibility metric for a visual aid is below a threshold visibility metric and the lighting level metric for the visual aid is below a threshold lighting level metric, the server device 14 may generate an explanation indicating that low visibility of the visual aid is occurring because the visual aid is poorly lit. In another example, when the visibility metric for a visual aid is less than a threshold visibility metric and the occlusion metric for that visual aid is less than a threshold occlusion metric, the server device 14 may generate a description indicating that the low visibility of the visual aid is caused because the visual aid is occluded or at least partially occluded, and may also describe the object that is occluding the visual aid.

[0050] In some implementations, the server device 14 may generate a visibility description for a subset of the visual aids in the set of navigation instructions. For example, the server device 14 may generate a visibility description for at least one visual aid that has a visibility metric above a first threshold or below a second threshold. In this way, the navigation instructions may provide the user with further details about visual aids that are particularly noticeable or particularly obscure. In another example, the server device 14 may rank the visual aids in the set of navigation instructions according to their respective corresponding visibility metrics. The server device 14 may then select visual aids ranked above a threshold ranking (e.g., the top three visual aids) and generate a visibility description for the selected visual aid.

[0051] In any case, the server device 14 then provides, for display on the client device 12, a set of navigation instructions including one or more navigation instructions having a description of the magnitude of visibility of the visual aid to assist the user in performing the navigation instruction. FIG. 5 illustrates an exemplary navigation display 500 presented on a user's client device 12. The navigation display 500 includes a navigation instruction that instructs the user to perform a maneuver ("turn left") and references a visual aid (e.g., a sign for State Street) to assist the user in identifying a location (the intersection where the user approaches State Street) for performing the maneuver. The navigation instruction also includes a textual description 502 of the magnitude of visibility of the visual aid (e.g., "no street lights and small signs, slow down early"). In addition to providing the textual description, the client device 12 may also present an audio description of the magnitude of visibility of the visual aid, for example, via a speaker of the client device 12.

[0052] Exemplary methods for providing visual aid visibility indicators during navigation 6 illustrates a flow diagram of an example method 600 for providing navigation instructions with visual aid visibility signs during navigation. The method may be implemented as a set of instructions stored on a computer-readable memory and executable on one or more processors of the server device 14. For example, the method may be implemented by the navigation instruction generator 42 and / or the sign visibility indicator module 44.

[0053] At block 602, a request for navigation instructions from a start location to a destination location is received. In response to receiving the request, the server device 14 may obtain a set of navigation instructions for proceeding along a route to the destination location via the navigation instruction generator 42 (block 604). The set of navigation instructions may include maneuvers at corresponding locations along the route, such as turn left, turn right, continue straight, etc. To assist the user in identifying a location for performing the maneuver, the set of navigation instructions may reference visual aids, such as street signs, building signs, etc., proximate to the location for performing the maneuver.

[0054] In block 606, the server device 14 identifies visual aids referenced in the set of navigation instructions to assist the user in identifying a location for performing the maneuver. The server device 14 then determines a visibility dimension for viewing the visual aid (block 608). More specifically, the server device 14 may retrieve a 3D panoramic street level image of the geographic area including the visual aid from the real-world database 54. In some implementations, the server device 14 may retrieve the 3D panoramic street level image that is from the perspective of a virtual camera having an orientation that corresponds to the orientation the user is facing when approaching the visual aid. To determine the orientation the user is facing when approaching the visual aid, the server device 14 may analyze the set of navigation instructions to determine the direction the user is traveling along the path when the user approaches the visual aid. For example, if a preceding navigation instruction before the navigation instruction that references the visual aid is to turn left onto Main Street, such that the user is heading east, the server device 14 may determine that the user is approaching the visual aid from the west facing east.

[0055] In any case, server device 14 may analyze the 3D panoramic street level imagery to identify objects occluding the visual aid, the size of the visual aid, the amount of lighting on the visual aid, etc., and store the results of the analysis in visual aid database 52. Server device 14 may then determine a magnitude of visibility of the visual aid based on any suitable combination of the height of the visual aid, the location of the visual aid, the orientation the visual aid faces, the size of the visual aid, the type of maneuver being performed at the location corresponding to the visual aid, an occlusion metric for the visual aid, a lighting level metric for the visual aid, a popularity metric for the visual aid, weather conditions at the visual aid, time of day, season, etc.

[0056] At block 610, the server device 14 adjusts the navigation instructions to include a description of the visibility magnitude for the visual aid. The description of the visibility magnitude in the visual aid may include a description of the lighting level to see the visual aid according to a lighting level metric, a description of the object occluding the visual aid according to an occlusion metric, a description of the size of the visual aid, a description of the location of the visual aid relative to the user (e.g., "The street sign is on the right side of the street"), or any other suitable description of the visibility magnitude in the visual aid. Additionally, the description of the visibility magnitude in the visual aid may include a recommendation based on the visibility magnitude in the visual aid, such as "Please slow down early."

[0057] In some implementations, the server device 14 may generate a visibility description for a subset of the visual aids in the set of navigation instructions. For example, the server device 14 may generate a visibility description for at least one visual aid that has a visibility metric above a first threshold or below a second threshold. In this way, the navigation instructions may provide the user with further details about visual aids that are particularly noticeable or particularly obscure. In another example, the server device 14 may rank the visual aids in the set of navigation instructions according to their respective corresponding visibility metrics. The server device 14 may then select visual aids ranked above a threshold ranking (e.g., the top three visual aids) and generate a visibility description for the selected visual aid.

[0058] In any case, the server device 14 then provides, for display on the client device 12 (block 612), a set of navigation instructions including one or more navigation instructions with a description of the magnitude of visibility of visual aids to assist the user in implementing the navigation instructions.

[0059] 7 illustrates a flow diagram of an example method 700 for presenting navigation instructions with visual aid visibility indicators during navigation. The method may be implemented as a set of instructions stored on a computer-readable memory and executable on one or more processors of the client device 12. For example, the method may be implemented by the geographic application 26.

[0060] At block 702, a request for navigational instructions from a starting location to a destination location is received. In response to receiving the request, the client device 12 may obtain a set of navigational instructions for proceeding along a path to the destination location (block 704). For example, the client device 12 may send a request to the server device 14 and may receive a set of navigational instructions from the server device 14. The set of navigational instructions may include maneuvers at corresponding locations along the path, such as turn left, turn right, continue straight, etc. To assist the user in identifying a location for performing the maneuver, the set of navigational instructions may reference visual aids, such as street signs, building signs, etc., proximate to the location for performing the maneuver. In addition, the set of navigational instructions may include one or more navigational instructions with a description of a visibility dimension for viewing the visual aids referenced in the navigational instructions.

[0061] The description of the magnitude of visibility in a visual aid may include a description of the lighting level for viewing the visual aid, a description of objects occluding the visual aid, a description of the size of the visual aid, a description of the location of the visual aid relative to the user (e.g., "The street sign is on the right side of the street"), or any other suitable description of the magnitude of visibility in a visual aid. Additionally, the description of the magnitude of visibility in a visual aid may include a recommendation based on the magnitude of visibility in a visual aid, such as "Please slow down early."

[0062] Geographic application 26 may then present the set of navigation instructions to the user via user interface 28 and / or the speaker of the client device in block 706. Geographic application 26 may present a textual description of the magnitude of visibility of the visual aid and / or may present an audio description of the magnitude of visibility of the visual aid, for example, via the speaker of client device 12.

[0063] Thus, the present technique provides an improved interface between a user and an electronic navigation system by enabling the automatic generation of improved navigation instructions. These navigation instructions may be tailored to a given user's situation, since, as illustrated in the embodiment, a given user's route, planned maneuvers, lighting, weather conditions, etc. may all be taken into account in determining the visibility of visual aids on the user's route. Furthermore, these improved navigation instructions may be automatically generated for any navigation route. For example, when a 3D panoramic street-level image corresponding to the user's position and orientation and a properly trained machine learning model are used, the user can obtain the benefit of improved navigation on any route for which 3D panoramic street-level imagery exists. This is useful in providing easier and safer navigation for all users.

[0064] Additional Considerations Various operations of the example methods described herein may be performed, at least in part, by one or more processors that are temporarily or permanently configured (e.g., by software) to perform such operations. Whether temporarily or permanently configured, such processors may constitute processor-implemented modules that operate to perform one or more operations or functions. Modules referred to herein may, in some example embodiments, comprise processor-implemented modules.

[0065] Similarly, the methods or routines described herein may also be at least partially processor-implemented. For example, at least some of the operations of the methods may be performed by one or more processors, or processor-implemented hardware modules. The performance of some of the operations may be distributed among one or more processors that are not only present in a single machine, but also deployed across several machines. In some exemplary embodiments, one or more processors may be located at a single location (e.g., in a home environment, in an office environment, or as a server farm), while in other embodiments, the processors may be distributed across several locations.

[0066] The one or more processors may also operate to support performance of such operations in a cloud computing environment or as a software as a service (SaaS). For example, at least some of the operations may be performed by a group of computers (examples of machines that include processors), which are accessible over a network (e.g., the Internet) and via one or more suitable interfaces (e.g., application program interfaces (APIs)).

[0067] Those skilled in the art will appreciate, upon reading this disclosure, further and additional alternative structural and functional designs for a system for providing visual aid visibility markers during navigation. Although specific embodiments and applications have been shown and described above, it should be understood that the disclosed embodiments are not limited to the precise structures and components disclosed herein. Various modifications, changes and variations that will be apparent to those skilled in the art may be made in the arrangement, operation and details of the methods and apparatus disclosed herein without departing from the spirit and scope as defined in the appended claims. [Explanation of symbols]

[0068] 10 Environment 12 Client Devices 14 Server Devices 16 Communication Networks 22 Processing Module 24 Memory 26 Geographical and Mapping Applications 28 User Interface 30 Network Interface 32 Positioning module 42 Navigation Instruction Generator, Components, Navigation Generator 44 Sign Visibility Indicator Modules, Components 50 Map Database 52 Visual Aid Database 54 Real World Image Database, Real World Database 70 Mila Street sign 72 Walgreens® Sign Street sign for 74 Willow Creek Dr. 200 Data Table 202 Visual aid identifiers 204 Position of visual aids 206 Direction of visual aid facing 208 Size or dimensions of visual aids 210 Appearance of visual aids 212 Names of visual aids 214 Shielding Metrics 300 3D panoramic street level images 302 Visual aids, street signs, business sign objects 304 Power Lines, Objects 306 Street Light, Object 400 display 402 Starting position 406 Routes 412 Target position 500 Navigation Display 502 A text description of the visibility dimensions of the visual aid 600 ways 700 methods

Claims

1. 1. A method for providing navigation instructions accompanied by visual aid visibility indicators during navigation, comprising: receiving, at one or more processors, a request from a user for navigation directions from a start location to a destination location; obtaining, by the one or more processors, in response to the request, a set of navigation instructions for traveling along a route from the start location to the destination location, the set of navigation instructions including one or more maneuvers at corresponding locations along the route; For at least one maneuver, identifying, by the one or more processors, a visual aid to assist the user in identifying a location corresponding to the maneuver; determining, by the one or more processors, a visibility dimension for the visual aid; adjusting, by the one or more processors, navigation instructions in the set of navigation instructions corresponding to the maneuver to include a description of the magnitude of visibility for the visual aid, the description of the magnitude of visibility for the visual aid including a recommendation based on the magnitude of visibility in the visual aid; and providing, by the one or more processors, the set of navigation instructions including the tailored navigation instructions for presentation to the user; A method comprising:

2. determining a magnitude of visibility for the visual aid, determining, by the one or more processors, an orientation of the user as the user approaches the visual aid based on the set of navigation instructions; determining, by the one or more processors, the magnitude of visibility for the visual aid from the orientation from which the user approaches the visual aid; 2. The method of claim 1, comprising:

3. determining a magnitude of visibility for the visual aid, acquiring, by the one or more processors, a three-dimensional panoramic street level image of a geographic area including the visual aid; determining, by the one or more processors, the magnitude of visibility for the visual aid based on objects contained within the three-dimensional panoramic street level image; 3. The method of claim 1 or 2, comprising:

4. 4. The method of claim 3, wherein acquiring a three-dimensional panoramic street level image of a geographic area includes acquiring the three-dimensional panoramic street level image from the perspective of a virtual camera having a view frustum that corresponds to a position or orientation of the user when the user approaches the visual aid.

5. classifying, by the one or more processors, one or more portions of the 3D panoramic street level image into one or more types of objects; determining, by the one or more processors, whether the one or more types of objects occlude the visual aid; determining, by the one or more processors, an occlusion metric for the visual aid based on whether the one or more types of objects occlude the visual aid; determining, by the one or more processors, the measure of visibility for the visual aid based on the occlusion metric; 5. The method of claim 3 or claim 4, further comprising:

6. classifying one or more portions of the three-dimensional panoramic street level image into one or more types of objects, training, by one or more processors, a machine learning model using (i) image data for a plurality of objects contained within a plurality of three-dimensional panoramic street level images, and (ii) an object type for each of the plurality of objects; acquiring, by the one or more processors, image data for a portion of the three-dimensional panoramic street level image; applying, by the one or more processors, the machine learning model to the image data to classify the portion of the 3D panoramic street level image into a particular type of object; 6. The method of claim 5, comprising:

7. determining a magnitude of visibility for the visual aid, The size of the visual aid; a light level for viewing said visual aid; time, season, the weather conditions at the location of said visual aid; or 7. The method of claim 1, further comprising determining the magnitude of visibility for the visual aid based on at least one of a type of maneuver performed upon approaching the visual aid.

8. adjusting navigation instructions to include a description of the visibility magnitude for the visual aid; adjusting, by the one or more processors, the navigation instructions to include a description of an object occluding the visual aid or a description of a lighting level for viewing the visual aid.

8. The method of claim 1 , comprising:

9. 1. A computing device for providing navigation instructions accompanied by visual aid visibility indicators during navigation, comprising: one or more processors; a non-transitory computer-readable memory coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the computing device to: receiving a request from a user for navigation directions from a start location to a destination location; in response to the request, obtaining a set of navigation instructions for traveling along a route from the start location to the destination location, the set of navigation instructions including one or more maneuvers at corresponding locations along the route; For at least one maneuver, identifying a visual aid to assist the user in identifying a location corresponding to the maneuver; determining a visibility dimension for said visual aid; adjusting navigation instructions in the set of navigation instructions corresponding to the maneuver to include a description of the magnitude of visibility for the visual aid, the description of the magnitude of visibility for the visual aid including a recommendation based on the magnitude of visibility in the visual aid; providing the set of navigation instructions including the tailored navigation instructions for presentation to the user; a non-transitory computer readable memory for causing the A computing device comprising:

10. To determine a visibility magnitude for the visual aid, the instructions cause the computing device to: determining an orientation of the user as the user approaches the visual aid based on the set of navigation instructions; and determining a magnitude of visibility for the visual aid from the orientation from which the user approaches the visual aid.

11. To determine a visibility magnitude for the visual aid, the instructions cause the computing device to: acquiring a three-dimensional panoramic street level image of a geographic area including said visual aid; determining the visibility magnitude for the visual aid based on objects contained within the three-dimensional panoramic street level image; 11. A computing device as claimed in claim 9 or claim 10, adapted to perform the following:

12. 12. The computing device of claim 11, wherein the three-dimensional panoramic street-level image of the geographic area is from the perspective of a virtual camera having a view frustum that corresponds to the user's position or orientation when the user approaches the visual aid.

13. The instructions cause the computing device to: classifying one or more portions of the three-dimensional panoramic street level image into one or more types of objects; determining whether the one or more types of objects occlude the visual aid; and determining an occlusion metric for the visual aid based on whether the one or more types of objects occlude the visual aid; determining the visibility measure for the visual aid based on the occlusion metric; 13. The computing device of claim 11 or claim 12, further configured to:

14. The instructions further include instructions for causing the computing device to: classify one or more portions of the three-dimensional panoramic street level image into one or more types of objects; training a machine learning model using (i) image data for a plurality of objects contained within a plurality of 3D panoramic street level images, and (ii) an object type for each of the plurality of objects; acquiring image data for a portion of the three-dimensional panoramic street level image; applying the machine learning model to the image data to classify the portion of the 3D panoramic street level image into a particular type of object; The computing device of claim 13 , further comprising:

15. The measure of visibility of the visual aid is: The size of the visual aid; a light level for viewing said visual aid; time, season, the weather conditions at the location of said visual aid; or 15. A computing device according to claim 9, wherein the visual aid is approached based on at least one of a type of maneuver to be performed upon approaching the visual aid.

16. 16. A computing device according to claim 9, wherein the navigation instructions are adapted to include a description of an object occluding the visual aid or a description of a lighting level for viewing the visual aid.

17. 1. A method for presenting navigation instructions accompanied by visual aid visibility indicators during navigation, comprising: receiving, at one or more processors, a request from a user for navigation directions from a start location to a destination location; obtaining, by the one or more processors, in response to the request, a set of navigation instructions for traveling along a route from the start location to the destination location, the set of navigation instructions including one or more maneuvers at corresponding locations along the route, at least one navigation instruction including a visual aid to assist the user in identifying a location corresponding to a maneuver included within the navigation instructions and a visibility magnitude description for viewing the visual aid, the visibility magnitude description for the visual aid including a recommendation based on the visibility magnitude in the visual aid; presenting, by the one or more processors, the set of navigation instructions, including presenting the at least one navigation instruction having the description of the visibility magnitude for the visual aid; A method comprising:

18. The method of claim 17 , wherein presenting the set of navigation instructions comprises presenting the at least one navigation instruction with a description of an object that is occluding the visual aid.

19. 19. The method of claim 17 or 18, wherein presenting the set of navigation instructions comprises presenting the at least one navigation instruction having a description of a lighting level for viewing the visual aid.

20. 20. The method of claim 17, wherein the step of presenting the set of navigation instructions comprises the step of presenting the at least one navigation instruction having the description of the magnitude of visibility for viewing the visual aid from an orientation from which the user approaches the visual aid.

Images