Finding Locally Salient Semantic Features for Navigation and Geocoding

A computing system identifies landmarks using entropy criteria to generate navigation instructions that enhance efficiency and safety by prioritizing locally prominent features, addressing inefficiencies in existing navigation systems.

CN112005082BActive Publication Date: 2024-12-03GOOGLE LLC
View PDF 18 Cites 0 Cited by

Patent Information

Application Number
CN201880092242.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2018-10-22
Publication Date
2024-12-03
Estimated Expiration
2038-10-22

AI Technical Summary

Technical Problem

Existing navigation systems often fail to effectively utilize locally salient landmarks for efficient and safe navigation, leading to inefficient resource consumption and potential driver distraction.

Method used

A computing system determines landmarks based on entropy criteria, such as rarity and visibility, using semantic tags associated with images to generate navigation instructions that prioritize locally prominent features.

Benefits of technology

Improves navigation efficiency by reducing network bandwidth usage, enhancing safety through prominent landmark-based instructions, and optimizing resource consumption.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112005082B_ABST
    Figure CN112005082B_ABST
Patent Text Reader

Abstract

Methods, systems, devices, apparatuses, and tangible, non-transitory computer-readable media for navigation and geocoding are provided. The disclosed technology is capable of performing operations including accessing semantic tags associated with images. Each of the semantic tags is capable of being associated with a feature depicted by one of the images. In addition, each of the features is capable of being associated with a geographic location. Based on the semantic tags, landmarks comprising features that satisfy an entropy criterion can be identified. The entropy criterion can measure the localized prominence of each of one or more features. A landmark for navigation at the location can be selected based on contextual data associated with the location on a path comprising a plurality of locations. In addition, at least one navigation instruction can be generated with reference to the landmark.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates generally to navigation and geocoding. More specifically, the present disclosure relates to determining locally salient semantic features using a computing device. Background Art

[0002] Operations associated with the state of a geographic area may be implemented on a variety of computing devices. These operations may include processing data associated with the geographic area for later access and use by a user or computing system. Additionally, operations may include exchanging data with a remote computing system. However, the types of operations and the manner in which the operations are performed may change over time, as may the underlying hardware that implements the operations. Thus, there are different ways to fully utilize computing resources associated with the state of a geographic area. Summary of the invention

[0003] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or may be learned from the description, or may be learned through practice of the embodiments.

[0004] An example aspect of the present disclosure is directed to a computer-implemented method for providing navigation instructions with reference to landmarks. The computer-implemented method may include accessing, by a computing system including one or more processors, a plurality of semantic tags associated with a plurality of images. Each of the plurality of semantic tags may be associated with one or more features depicted by one of the plurality of images. In addition, each of the one or more features may be associated with a geographic location. The computer-implemented method may include identifying, by the computing system at least in part based on the plurality of semantic tags, one or more landmarks including one or more features that satisfy one or more entropy criteria, the entropy criteria measuring the localized prominence of each of the one or more features. The computer-implemented method may also include selecting, by the computing system at least in part based on contextual data associated with a location on a path including a plurality of locations, at least one landmark for navigation at the location. In addition, the computer-implemented method may include generating, by the computing system, at least one navigation instruction with reference to the at least one landmark.

[0005] Another example aspect of the present disclosure is directed to one or more tangible, non-transitory computer-readable media storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform operations. The operations may include accessing image data including multiple images associated with one or more semantic tags. Each of the one or more semantic tags may be associated with one or more features of the multiple images. In addition, each of the one or more features may be associated with a geographic location. The operations may also include determining one or more landmarks including one or more features that satisfy one or more entropy criteria, the entropy criteria being associated with the localized prominence of each of the one or more features. The operations may include determining one or more landmarks associated with a vantage point based at least in part on contextual data associated with a location of the vantage point on a path including multiple locations. In addition, the operations may include generating navigation data including one or more indications associated with one or more landmarks.

[0006] Another example aspect of the present disclosure is directed to a computing system comprising: one or more processors; and one or more non-transitory computer-readable media storing instructions, which, when executed by the one or more processors, cause the one or more processors to perform operations. The operations may include accessing image data including multiple images associated with one or more semantic tags. Each of the one or more semantic tags may be associated with one or more features of the multiple images. In addition, each of the one or more features may be associated with a geographic location. The operations may also include determining one or more landmarks including one or more features that satisfy one or more entropy criteria, the entropy criteria being associated with the localized prominence of each of the one or more features. The operations may also include determining one or more landmarks associated with a vantage point based at least in part on contextual data associated with a location of the vantage point on a path including multiple locations. In addition, the operations may include generating navigation data including one or more indications associated with one or more landmarks.

[0007] Other aspects of the present disclosure are directed to various systems, devices, non-transitory computer-readable media, user interfaces, and electronic devices. These and other features, aspects, and advantages of various embodiments of the present disclosure will be better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and are used together with this description to explain the relevant principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] A detailed discussion of embodiments for those of ordinary skill in the art is set forth in the specification, which refers to the accompanying drawings, in which:

[0009] Figure 1A diagram depicting an example system according to an example embodiment of the present disclosure;

[0010] Figure 2 A diagram depicting an example device according to an example embodiment of the present disclosure;

[0011] Figure 3 depicts an example of landmark detection according to an example embodiment of the present disclosure;

[0012] Figure 4 depicts an example of landmark detection according to an example embodiment of the present disclosure;

[0013] Figure 5 depicts an example of landmark recognition according to an example embodiment of the present disclosure;

[0014] Figure 6 depicts an example of landmark recognition in an environment according to an example embodiment of the present disclosure;

[0015] Figure 7 depicts a flow chart of providing navigation instructions with reference to landmarks according to an example embodiment of the present disclosure;

[0016] Figure 8 depicts a flow chart of providing navigation instructions with reference to landmarks according to an example embodiment of the present disclosure;

[0017] Fig. 9 depicts a flow chart of providing navigation instructions with reference to landmarks according to an example embodiment of the present disclosure;

[0018] Fig.10 depicts a flow chart of providing navigation instructions with reference to landmarks according to an example embodiment of the present disclosure;

[0019] Fig.11 depicts a flow chart of providing navigation instructions with reference to landmarks according to an example embodiment of the present disclosure;

[0020] Fig.12 depicts a flow chart of providing navigation instructions with reference to landmarks according to an example embodiment of the present disclosure;

[0021] Fig.13 A flowchart depicting providing navigation instructions with reference to landmarks according to an example embodiment of the present disclosure; and

[0022] Fig.14 Depicted is a flow diagram for providing landmark-referenced navigation instructions according to an example embodiment of the present disclosure.

[0023] Reference numerals repeated across the various drawings are intended to identify like features in the various embodiments. DETAILED DESCRIPTION

[0024] In general, the present disclosure is directed to determining landmarks for navigation and geocoding. In addition, the present disclosure describes the generation of navigation instructions based on the determined landmarks. In particular, the disclosed technology may include a computing system that receives data including semantic tags associated with features of an image (e.g., a photograph of an area captured from various vantage points). The semantic tags may be based on images that have been processed by a machine learning model or other image content analysis system that detects and identifies features of the image. In addition, the features of the image may be associated with a geographic location (e.g., altitude, latitude, and longitude) and other information (e.g., the time each image was captured). The computing system may then determine landmarks in the image that include features that meet an entropy criterion, the entropy criterion being associated with the localized prominence of the feature (e.g., the visibility, rarity, or familiarity of the feature within the area). The computing system may also determine landmarks associated with a vantage point (e.g., landmarks visible from a vantage point) based on contextual data associated with the vantage point (e.g., the current location of the computing system or a user of the computing system). In addition, the computing system may generate instructions that may be used to facilitate navigation (e.g., navigation instructions with reference to (one or more) landmarks). As such, the disclosed techniques can more effectively determine the locations of landmarks in an environment and select a portion of the landmarks that will facilitate more effective navigation based on a given context and vantage point. Thus, the techniques described herein can enable a user to navigate to a specified location more effectively and / or efficiently. In some examples, landmarks determined in the manner described herein can be referenced in a series of turn-by-turn navigation instructions that can be provided to a driver of a vehicle. The described techniques can therefore assist a driver in more effectively performing the technical task of driving a vehicle to a specified location by means of a continuous and / or guided human-vehicle interaction process.

[0025] As another example, a computing system (e.g., a computing system in a vehicle) may access map data associated with a geographic area (e.g., locally stored or remotely accessible map data). The map data may include an image and a semantic label indicating features of the image that have been previously identified using image recognition (e.g., a machine learning model trained to detect features of an image). The computing system may then determine a set of landmarks in the area that include a set of features that meet an entropy criterion associated with the localized prominence of the landmark. For example, a thirty-meter-tall red granite obelisk may meet the entropy criterion (e.g., large size, unusual shape, and unusual color), while a parking meter (e.g., relatively small size and very ordinary) or a bush may not meet the entropy criterion. In addition, the computing system may determine context including time of day, location, and direction of travel (e.g., the direction of travel of the vehicle), which may be used to select the most effective landmarks for a given vantage point and conditions in the area. The computing device may then use the landmarks in the area to provide a set of instructions to a user (e.g., audible instructions generated through a vehicle speaker and / or text or graphical instructions provided on a display) to assist in navigation. As such, the disclosed technology provides a variety of improvements in navigation and geocoding.

[0026] In some embodiments, the disclosed technology may include a computing system (e.g., a navigation computing system) that may include one or more computing devices (e.g., devices having one or more computer processors and a memory that may store one or more instructions) that may send, receive, process, generate, and / or modify data (e.g., data including semantic tags associated with an image), the data including one or more information patterns or structures that may be stored on one or more memory devices (e.g., one or more random access memory devices) and / or one or more storage devices (e.g., one or more hard disk drives and / or one or more solid-state memory drives); and / or one or more signals (e.g., electronic signals). The data and / or one or more signals may be exchanged by the computing system with various other systems and / or devices including a plurality of service systems (e.g., one or more remote computing systems, one or more remote computing devices, and / or one or more software applications running on one or more computing devices), wherein the service systems may send and / or receive data including data including semantic tags associated with an image (e.g., a digital image associated with data including one or more descriptions of a geographic location, a time of image capture, and / or one or more other features of the image). In some embodiments, the computing system (e.g., a navigation computing system) may include Figure 1 The device 102 and / or Figure 2In addition, the network computing system may be associated with one or more machine learning models, including Figure 1 One or more features of one or more machine learning models 120 depicted in .

[0027] In addition, the computing system may include dedicated hardware (e.g., an application specific integrated circuit) and / or software that enables the computing system to perform one or more operations specific to the disclosed technology, including accessing semantic tags associated with multiple images (e.g., accessing semantic tags stored locally or accessing semantic tags by receiving semantic tags from a remote computing device), determining landmarks that include one or more features that satisfy entropy criteria, using contextual data to determine landmarks associated with vantage points, and generating navigation data.

[0028] More specifically, the navigation computing system may receive a plurality of semantic tags associated with a plurality of images. Each of the plurality of semantic tags may be associated with one or more features depicted by one of the plurality of images. For example, each semantic tag may provide a semantic description of an object included in a scene depicted by one of the plurality of images. In addition, each of the one or more features may be associated with a geographic location. The plurality of images may include one or more digital images (e.g., two-dimensional images) of a portion of an environment (e.g., an image of a location in the environment). The plurality of images may be encoded in any type of image format including a combination of a raster image (e.g., a bitmap including a grid of pixels) and / or a vector image (e.g., a polygonal representation of an image of a location based on coordinates of the x and y axes of a two-dimensional plane). The image may include a still image, an image frame from a movie, and / or other types of images, including LIDAR images, RADAR images, and / or other types of images.

[0029] Examples of digital image formats used by the plurality of images may include JPEG (Joint Photographic Experts Group), BMP (Bitmap), TIFF (Tagged Image File Format), PNG (Portable Network Graphics), and / or GIF (Graphics Interchange Format). The images may be collected from a variety of sources, such as user-submitted imagery, public domain imagery (e.g., obtained via web crawling and suitably aggregated and anonymized), street-level panoramic imagery, and / or other sources of imagery.

[0030] Multiple semantic tags associated with an image can be associated with one or more features, including: physical size (e.g., the physical size of an object in an image); and / or object identity (e.g., the identity of an object depicted in one or more images). Additional information can be associated with the image and / or semantic tags, such as: location (e.g., a street address and / or altitude, latitude, and longitude associated with the image); time of day (e.g., the time of day when the image was captured); date (e.g., the date when the image was captured);

[0031] For example, a navigation computing system may receive data including information associated with multiple semantic tags and multiple images via a communication network (e.g., a wireless and / or wired network including a LAN, WAN, or the Internet) through which one or more signals (e.g., electronic signals) and / or data can be sent and / or received.

[0032] The navigation computing system can identify one or more landmarks including one or more features that satisfy one or more entropy criteria based at least in part on multiple semantic tags, the entropy criteria measuring the localized prominence of each of the one or more features. For example, the navigation computing system can access data associated with multiple semantic tags indicating one or more landmarks in an area. In addition, one or more landmarks may be associated with one or more features (e.g., physical size or shape) that can be compared with one or more entropy criteria that can be used to identify one or more landmarks. One or more entropy criteria can be associated with the frequency with which each of the one or more features appears in a region or the distinctiveness of each feature from other common features in a region.

[0033] For example, satisfaction of one or more entropy criteria may be based on infrequently occurring features (e.g., a unique tree in an area or a unique tall building in an area). Thus, in one example, clustering or other algorithmic techniques may be used to determine the rarity or infrequency associated with each feature, which may then be used to guide the selection of features for use as landmarks. As an example, for each location, the area surrounding the location may be analyzed to identify which features associated with the location are the rarest (e.g., a histogram of semantic labels in an area surrounding a location may reveal that an obelisk appears only once, while a mailbox appears sixty times, thereby indicating that an obelisk would be more suitable for use as a landmark). For another example, satisfaction of one or more entropy criteria may include the uniqueness of various characteristics of a feature relative to other similar features in the area (e.g., while they may all be "buildings," a small house on one corner of a four-sided intersection will contrast with tall buildings on the other three corners).

[0034] Landmarks can therefore be determined from semantic tag statistics aggregated geographically and over time for each location by focusing on high entropy "tags" (i.e., those that appear to persist in time and exhibit high localized salience). The system can thus identify tags that have high confidence at that location but are relatively uncommon or rare in the surrounding area.

[0035] The navigation computing system may select at least one landmark for navigation at the location based at least in part on contextual data associated with the location on the path including the plurality of locations. For example, the navigation computing system may determine context including the time of day, season, and / or the amount of traffic approaching the vantage point. For example, based on a context indicating that night is approaching, the navigation computing system may select a landmark that is illuminated and not obscured by darkness.

[0036] The navigation computing system may generate at least one navigation instruction that references at least one landmark. The at least one navigation instruction may include one or more visual instructions (e.g., text displayed on a display device) and / or one or more audible instructions (e.g., instructions issued from an audio output device). For example, the navigation computing system may generate audible instructions that describe the appearance of a landmark (e.g., "a statue of a horse with a rider") and the location of the landmark (e.g., "the next intersection is on your right").

[0037] In some embodiments, determining a rate at which each of the one or more features occurs within a predetermined area based at least in part on the plurality of semantic tags can be used to identify one or more landmarks including one or more features that satisfy one or more entropy criteria based at least in part on the plurality of semantic tags, the entropy criteria measuring the localized prominence of each of the one or more features. For example, the navigation computing system can determine a rate at which each of the one or more features (e.g., the height of a high-rise building) occurs within an area of ​​two hundred square meters.

[0038] In some embodiments, determining that one or more landmarks include one or more features that occur least frequently or at a rate below a threshold rate can be used to identify one or more landmarks that include one or more features that satisfy one or more entropy criteria based at least in part on multiple semantic tags, the entropy criteria measuring the localized prominence of each of the one or more features. For example, when determining a building to use as a landmark in a group of buildings, the navigation computing system can determine a building whose height occurs less frequently than the average height rate (e.g., a very tall high-rise building whose height is within a height range that occurs once in every thousand high-rise buildings).

[0039] In some embodiments, determining a confidence score for each of one or more features based at least in part on the number of times each respective feature of the one or more features has been associated with a semantic tag in a plurality of semantic tags can be used to identify one or more landmarks including one or more features that satisfy one or more entropy criteria based at least in part on the plurality of semantic tags, the entropy criteria measuring the localized prominence of each of the one or more features. For example, the navigation computing system can access data associated with the number of times each of the one or more features has been tagged with the same tag. Features tagged more times with the same tag can be associated with higher confidence scores.

[0040] In some embodiments, identifying one or more features having confidence scores that satisfy one or more confidence score criteria as landmarks can be used to identify one or more landmarks including one or more features that satisfy one or more entropy criteria based at least in part on a plurality of semantic tags, the entropy criteria measuring the localized prominence of each of the one or more features. For example, the navigation computing system can identify one or more features having confidence scores that exceed a threshold confidence score as landmarks.

[0041] In some embodiments, the confidence score may be based at least in part on the number of different viewing angles from which each of one or more features associated with a semantic label (e.g., semantic labels associated with images of the same object viewed from different angles) has been viewed, and / or the recency with which the one or more features are associated with the semantic label.

[0042] In some embodiments, determining one or more clusters of one or more features that satisfy one or more entropy criteria can be used to identify one or more landmarks that include one or more features that satisfy one or more entropy criteria based at least in part on multiple semantic tags, wherein the entropy criteria measure the localized prominence of each of the one or more features. In addition, each of the one or more clusters can include one or more features of a common semantic type. For example, the navigation computing system can determine a cluster of trees that can satisfy one or more entropy criteria associated with a tree density (e.g., the number of trees within a predetermined area) for a cluster of trees.

[0043] In some embodiments, determining the visibility of each of the one or more landmarks from a vantage point associated with a location can be used to select, by a computing system, at least one landmark for navigation at the location based at least in part on contextual data associated with the location on a path that includes the plurality of locations. Additionally, the contextual data can include a vantage point associated with the location (e.g., a point within the location from which a portion of the surrounding environment can be viewed). For example, the navigation computing system can determine one or more landmarks that are visible from a vantage point based in part on how far away the landmark is and whether the landmark is obscured by other objects.

[0044] In some embodiments, visibility can be based at least in part on the distance at which each of the one or more landmarks is visible from the vantage point, the amount of light projected on each of the one or more landmarks, any obstructions between the vantage point and the one or more landmarks, and / or the physical size of each of the one or more landmarks.

[0045] In some embodiments, determining the direction of travel along the path at a vantage point can be used to determine the visibility of each of the one or more landmarks from the vantage point associated with the location. For example, the visibility of each of the one or more landmarks can be associated with a field of view from the vantage point, the field of view being associated with the direction of travel (e.g., the field of view being equal to a predetermined angle relative to a line corresponding to the direction of travel).

[0046] In some embodiments, determining one or more landmarks facing the direction of travel can be used to determine the visibility of each of the one or more landmarks from a vantage point associated with the location. For example, the navigation computing system can determine that one or more landmarks facing the direction of travel (e.g., one or more landmarks in front of the vantage point) are more visible than one or more landmarks not facing the direction of travel (e.g., one or more landmarks behind the vantage point).

[0047] In some embodiments, determining visibility based at least in part on a mode of transportation associated with a vantage point can be used to determine visibility of each of the one or more landmarks from a vantage point associated with the location. For example, the navigation computing system can determine that visibility from a car is less than visibility from a bicycle in certain circumstances (e.g., on a busy street) and determine visibility accordingly.

[0048] In some embodiments, determining a reaction time based at least in part on a speed at the location and a distance to a nearest landmark of one or more landmarks can be used to select at least one landmark for navigation at the location based at least in part on contextual data associated with the location on a path including the plurality of locations. For example, the navigation computing system can determine the reaction time based on a speed at the location (e.g., a speed in meters per second) and a distance to a nearest landmark (e.g., a distance in meters).

[0049] In some embodiments, determining one or more landmarks that satisfy one or more reaction time criteria associated with a minimum reaction time can be used to select at least one landmark for navigation at the location based at least in part on contextual data associated with the location on a path that includes the plurality of locations. For example, the navigation computing system can determine that the minimum reaction time is two seconds, and select one or more landmarks from the one or more landmarks that will be visible after two seconds.

[0050] In some embodiments, the reaction time can be based at least in part on the mode of transportation associated with the vantage point. In addition, the transportation mode can include motor vehicle, bicycle, and / or walking. For example, the reaction time for a slower mode of transportation (e.g., walking) can be a shorter duration than a faster mode of transportation (e.g., motor vehicle).

[0051] In some embodiments, selecting at least one landmark based at least in part on familiarity with one or more landmarks can be used to select at least one landmark for navigation at the location based at least in part on contextual data associated with a location on a path including multiple locations. In addition, familiarity can be associated with the number of times and / or frequency that a user (e.g., a user associated with a navigation computing device) has previously been to the location (e.g., within a threshold distance of the location). For example, the contextual data can include a record of the frequency with which the user travels past each of the one or more landmarks. The navigation computing system can determine when the frequency with which the user travels past a landmark meets one or more familiarity criteria (e.g., a threshold minimum frequency), and select a landmark that meets the one or more familiarity criteria. For another example, familiarity can be associated with a portion of one or more features that each of the one or more landmarks has in common with another landmark previously viewed by the user (e.g., the number of one or more visual features).

[0052] In some embodiments, adjusting at least one navigation instruction based at least in part on the familiarity can be used to generate at least one navigation instruction that references at least one landmark. For example, the navigation computing system can access information associated with at least one landmark (e.g., the name of the landmark used by local residents) and use the information to modify the navigation instructions used. For another example, the navigation computing system can determine that when the familiarity exceeds a threshold familiarity, the navigation instruction is to be selected from the data including the locally used landmark terms.

[0053] In some embodiments, the contextual data may include information associated with the time of day, season, language (e.g., French, Russian, and / or Chinese), one or more features visible from the location, and / or mode of transportation (e.g., private car, bus, bicycle, and / or walking).

[0054] In some embodiments, the navigation computing system may receive image data including multiple images associated with one or more semantic tags, wherein each of the one or more semantic tags is associated with one or more features of the multiple images. In addition, each of the one or more features may be associated with a geographic location. For example, the navigation computing system may receive image data (e.g., tagged digital photos) including information associated with multiple semantic tags and multiple images via a communication network through which one or more signals (e.g., electronic signals) and / or data may be sent and / or received.

[0055] In some embodiments, the navigation computing system can determine one or more landmarks that include one or more features that satisfy one or more entropy criteria, the entropy criteria being associated with the localized prominence of each of the one or more features. For example, the navigation computing system can access multiple semantic tags to determine one or more landmarks in a region that include one or more features (e.g., size, distance from a vantage point) that satisfy one or more entropy criteria (e.g., size greater than a threshold size or distance within a threshold distance).

[0056] In some embodiments, the navigation computing system can determine one or more landmarks associated with the vantage point based at least in part on context data associated with the location of the vantage point on a path that includes multiple locations. For example, the navigation computing system can access context data that includes the time of day, season, and / or the location of objects between the vantage point and the one or more landmarks. Based on the context, the navigation computing system can determine one or more landmarks that are visible (based on the time of day and season) and are not obscured by other objects between the vantage point and each of the one or more landmarks.

[0057] In some embodiments, the navigation computing system may generate navigation data including one or more indications associated with one or more landmarks. For example, the navigation computing system may generate visual instructions on a display device including a description of the appearance of the landmark (e.g., "tall glass tower") and the location of the landmark (e.g., "one hundred meters ahead on the left").

[0058] In some embodiments, each of the one or more features can be associated with a time of day (e.g., hour and minute of day), a season (e.g., winter, summer, spring, or fall), visual constancy (e.g., the degree to which one or more features appear the same over time), and / or one or more locations from which each of the one or more features is visible (e.g., a geographic location including the latitude, longitude, and / or altitude from which each feature is visible).

[0059] In some embodiments, one or more entropy criteria may include a frequency of occurrence of each of the one or more features within a predetermined area (e.g., how often the feature occurs within the predetermined area) exceeding a predetermined threshold frequency, a temporal persistence of each of the one or more features at a location (e.g., how long the feature has been present) exceeding a predetermined threshold duration, and / or a size (e.g., physical size) of each of the one or more features exceeding a threshold size.

[0060] In some embodiments, the one or more indications may include one or more visual indications (e.g., text displayed on a display device) associated with the relative position of one or more landmarks on the path relative to the vantage point, and / or one or more audible indications (e.g., audio produced by a speaker) associated with the relative position of one or more landmarks on the path relative to the vantage point.

[0061] In some embodiments, determining a travel mode associated with a vantage point can be used to determine one or more landmarks that include one or more features that satisfy one or more entropy criteria associated with a localized prominence of each of the one or more features. For example, a navigation computing system can access data associated with a vantage point (e.g., the middle of a highway, a sidewalk, or a lake), which can be used to determine an associated mode of transportation (e.g., car, walking, or boat).

[0062] Additionally, the navigation computing system may determine one or more landmarks that will be visible from the vantage point within a predetermined time period associated with the travel mode based at least in part on the direction of travel and the speed along the route. For example, the navigation computing system may determine the speed associated with the transportation mode and determine landmarks that will be visible within the next ten seconds.

[0063] In some embodiments, the navigation computing system may generate a navigation instruction query associated with the utility of at least one navigation instruction. The navigation instruction query may include one or more visual indications and / or one or more audible indications. For example, when the navigation computing system has determined that a predetermined amount of time has passed since generating at least one navigation instruction, the navigation computing system may generate a navigation instruction query via an output device (e.g., a display device and / or an audio output device).

[0064] In some embodiments, the navigation computing system may receive one or more responses to the navigation instruction query. The one or more responses to the navigation instruction query may include one or more signals or data received from one or more input devices (e.g., a keyboard, a touch screen display, and / or a microphone), which may receive one or more responses from a user of the navigation computing system.

[0065] In some embodiments, the navigation computing system may adjust one or more entropy criteria based at least in part on one or more responses to the navigation instruction query. When one or more responses indicate that at least one navigation instruction was once useful, one or more entropy criteria may be adjusted by modifying one or more entropy criteria. For example, when one or more entropy criteria include a minimum occurrence rate of a feature (e.g., a certain type of restaurant appears once every 2.5 square kilometers) and one or more responses indicate that a landmark has been confused with another landmark having the same feature, the minimum occurrence rate of the feature may be reduced (e.g., a certain type of restaurant appears once every 3 square kilometers).

[0066] The systems, methods, devices, apparatuses, and tangible non-transitory computer-readable media in the disclosed technology can provide a variety of technical effects and benefits, including improvements in landmark determination for navigation and geocoding. In particular, the disclosed technology can assist a user (e.g., a driver of a vehicle) in performing a technical task (e.g., driving a vehicle to a specified location) by means of a continuous and / or guided human-computer interaction process. It can also bring benefits including improved performance of communication networks, better resource utilization efficiency, and improved safety.

[0067] The disclosed technology can improve the performance of communication network operations by more efficiently determining a set of landmarks to facilitate navigation. The set of landmarks can then be provided to more efficiently route traffic through the transportation network and avoid situations where communication network bandwidth is wasted due to inefficient navigation instructions (e.g., sending additional sets of instructions over the network when the first set of instructions was not correctly followed). As such, more efficiently providing navigation instructions can result in a reduction in the number of navigation instructions sent over the communication network, and a corresponding reduction in bandwidth utilization.

[0068] In addition, the disclosed technology can improve the efficiency of resource consumption (e.g., fuel and electrical energy) by providing more efficient navigation instructions that take advantage of the use of local prominent landmarks. For example, navigation instructions that include landmarks can result in fewer missed turns and / or fewer backtracking, thereby reducing the associated excess use of generated fuel or electrical energy.

[0069] In addition, the use of landmarks for navigation purposes can improve driver safety while traveling by car. For example, navigation instructions that include more easily observable, locally prominent landmarks can reduce driver distraction caused by navigation instructions that do not use prominent landmarks (e.g., street names that may be located on poorly located or poorly illuminated road signs).

[0070] Thus, the disclosed technology can assist the driver of a vehicle to more efficiently / effectively perform the technical task of driving the vehicle to a specified location by means of a continuous and / or guided human-machine interaction process. In addition, the disclosed technology can provide a computing system that facilitates more efficient landmark recognition used in navigation and geocoding. The disclosed technology provides specific benefits of reduced network bandwidth utilization, more efficient fuel and energy use, and greater safety, any of which can be used to improve the effectiveness of a variety of services including navigation services and geocoding services.

[0071] Now refer to Figure 1-14 , further discussing example embodiments of the present disclosure in detail. Figure 1 A diagram of an example system is depicted according to an example embodiment of the present disclosure. System 100 includes computing device 102 , server computing system 130 , training computing system 150 , and one or more remote computing systems 170 communicatively connected and / or coupled via network 180 .

[0072] Computing device 102 may include any type of computing device, including, for example, a personal computing device (e.g., a laptop computing device or a desktop computing device), a mobile computing device (e.g., a smart phone or a tablet computer), a game console, a controller, a wearable computing device (e.g., a smart watch), an embedded computing device, and / or any other type of computing device.

[0073] The computing device 102 includes one or more processors 112 and memory 114. The one or more processors 112 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and can be one processor or multiple processors operatively connected. The memory 114 can include one or more non-transitory computer-readable storage media, including RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 can store data 116 and instructions 118 executed by the processor 112 to cause the computing device 102 to perform operations.

[0074] In some embodiments, computing device 102 may perform one or more operations including accessing image data including multiple images associated with one or more semantic tags. Each of the one or more semantic tags accessed by computing device 102 may be associated with one or more features of multiple images. In addition, each of the one or more features may be associated with a geographic location. The operations performed by computing device 102 may include determining one or more landmarks including one or more features that satisfy one or more entropy criteria, the entropy criteria being associated with the localized prominence of each of the one or more features. The operations performed by computing device 102 may also include determining one or more landmarks associated with a vantage point based at least in part on contextual data associated with a location of a vantage point on a path including multiple locations. In addition, the operations performed by computing device 102 may include generating navigation data including one or more indications associated with one or more landmarks.

[0075] In some implementations, the computing device 102 may store or include one or more machine learning models 120. For example, the one or more machine learning models 120 may include various machine learning models, including neural networks (e.g., deep neural networks) or other types of machine learning models, including nonlinear models and / or linear models. The neural network may include a feedforward neural network, a recurrent neural network (e.g., a long short-term memory recurrent neural network), a convolutional neural network, or other forms of neural networks. Figure 1-14 Examples of one or more machine learning models 120 are discussed.

[0076] In some embodiments, one or more machine learning models 120 may be received from a server computing system 130 via a network 180, stored in a computing device memory 114, and then used or otherwise implemented by one or more processors 112. In some embodiments, the computing device 102 may implement multiple parallel instances of a single machine learning model of the one or more machine learning models 120 (e.g., performing parallel landmark determination on multiple instances of the machine learning model 120). More particularly, the one or more machine learning models 120 may determine and / or identify landmarks and / or locations of landmarks based on various inputs including semantic tags (e.g., semantic tags associated with one or more features of the landmark). In addition, the one or more machine learning models 120 may determine one or more navigation instructions to provide in association with the identified landmark.

[0077] Additionally or alternatively, one or more machine learning models 140 may be included in or otherwise stored and implemented by a server computing system 130 that communicates with computing device 102 according to a client-server relationship. For example, machine learning model 140 may be implemented by server computing system 140 as part of a web service (e.g., a landmark determination service). Thus, one or more machine learning models 120 may be stored and implemented at computing device 102, and / or one or more machine learning models 140 may be stored and implemented at server computing system 130.

[0078] The computing device 102 may also include one or more user input components 122 that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or a touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user may provide user input.

[0079] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 may include any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and may include one processor or multiple processors operatively connected. The memory 134 may include one or more non-transitory computer-readable storage media, including RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 may store data 136 and instructions 138 executed by the processor 132 to cause the server computing system 130 to perform operations.

[0080] In some embodiments, the server computing system 130 may perform one or more operations including accessing image data including multiple images associated with one or more semantic tags. Each of the one or more semantic tags may be associated with one or more features of the multiple images. In addition, each of the one or more features may be associated with a geographic location. The operations performed by the server computing system 130 may include determining one or more landmarks including one or more features that satisfy one or more entropy criteria, the entropy criteria being associated with the localized prominence of each of the one or more features. The operations performed by the server computing system 130 may also include determining one or more landmarks associated with the vantage point based at least in part on contextual data associated with the location of the vantage point on a path including multiple locations. In addition, the operations performed by the server computing system 130 may include generating navigation data including one or more indications associated with one or more landmarks. In addition, the server computing system 130 may perform image content analysis on one or more inputs (e.g., one or more images) provided to the server computing system 130. For example, the server computing system 130 may receive data including one or more images and / or associated semantic tags via the network 180. The server computing system 130 may then perform various operations that may include using one or more machine learning models 140 to detect one or more features of one or more images and / or associating semantic tags with images. For another example, the server computing system 130 may use object recognition techniques to detect one or more objects in an image (e.g., detecting pedestrians, vehicles, various buildings, fountains, street signs, and road markings) and generate semantic information based on the recognition of the objects. In another example, the server computing system 130 may receive data from one or more remote computing systems (e.g., one or more remote computing systems 170), which may include images that have been associated with semantic tags (e.g., images with manually user-associated semantic tags and / or images with semantic tags generated by a user device). The data received by the server computing system 130 may then be stored (e.g., in an image repository) for later use by the computing system 130.

[0081] In some implementations, server computing system 130 includes or is otherwise implemented by one or more server computing devices. Where server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0082] Additionally, server computing system 130 may receive various data.

[0083] As described above, the server computing system 130 may store or otherwise include one or more machine learning models 140. For example, the one or more machine learning models 140 may include various machine learning models. Examples of machine learning models include neural networks or other multi-layer nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Figure 1-14 Examples of one or more machine learning models 140 are discussed.

[0084] Computing device 102 and / or server computing system 130 may train one or more machine learning models 120 and / or 140 via interaction with a training computing system 150 that is communicatively connected and / or coupled via network 180. Training computing system 150 may be separate from server computing system 130, or may be part of server computing system 130.

[0085] The training computing system 150 includes one or more processors 152 and a memory 154. The one or more processors 152 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and may be one processor or multiple processors operatively connected. The memory 154 may include one or more non-transitory computer-readable storage media, including RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 154 may store data 156 and instructions 158 executed by the processor 152 to cause the training computing system 150 to perform operations. In some embodiments, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0086] The training computing system 150 may include a model trainer 160 that trains one or more machine learning models 120 and / or one or more machine learning models 140 of machine learning stored at the computing device 102 and / or the server computing system 130, respectively, using various training or learning techniques, including, for example, back propagation of errors. In some embodiments, performing back propagation of errors may include performing truncated back propagation along time. The model trainer 160 may perform a variety of generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the model being trained.

[0087] In particular, model trainer 160 may train one or more machine learning models 120 and / or one or more machine learning models 140 based on a set of training data 162. Training data 162 may include, for example, semantic data (e.g., semantic labels) describing the location and / or one or more features of landmarks in an area. For example, the training data may include a physical size associated with a landmark, the proximity of a landmark to one or more reference points (e.g., a vantage point), the location of a landmark (e.g., the latitude, longitude, and / or altitude of the landmark), and / or various metadata associated with the landmark (e.g., a nickname for the landmark and / or an old name for the landmark).

[0088] In some implementations, if the user has provided consent, then the training examples may be provided by computing device 102. Thus, in such implementations, one or more machine learning models 120 provided to computing device 102 may be trained by training computing system 150 on user-specific data received from computing device 102. In some cases, this process may be referred to as personalizing the model.

[0089] The model trainer 160 may include computer logic for providing the desired functionality. The model trainer 160 may be implemented in hardware, firmware, and / or software that controls a general purpose processor. For example, in some embodiments, the model trainer 160 includes a program file stored on a storage device, loaded into a memory, and executed by one or more processors. In other embodiments, the model trainer 160 includes one or more sets of computer executable instructions stored in a tangible computer readable storage medium including a RAM hard disk or an optical or magnetic medium.

[0090] In some embodiments, the training computing system 150 may perform one or more operations including accessing image data including multiple images associated with one or more semantic tags. Each of the one or more semantic tags may be associated with one or more features of the multiple images. In addition, each of the one or more features may be associated with a geographic location. The operations performed by the training computing system 150 may include determining one or more landmarks including one or more features that satisfy one or more entropy criteria, the entropy criteria being associated with the localized prominence of each of the one or more features. The operations performed by the training computing system 150 may also include determining one or more landmarks associated with a vantage point based at least in part on contextual data associated with a location of a vantage point on a path including multiple locations. In addition, the operations performed by the training computing system 150 may include generating navigation data including one or more indications associated with one or more landmarks.

[0091] Each of the one or more remote computing systems 170 includes one or more processors 172 and memory 174. The one or more processors 172 may include any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and may include one processor or multiple processors operatively connected. The memory 174 may include one or more non-transitory computer-readable storage media, including RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 174 may store data 176 and instructions 178 executed by the processor 172 to cause the server computing system 170 to perform operations.

[0092] In some embodiments, one or more remote computing systems 170 include or are otherwise implemented by one or more computing devices. Where one or more remote computing systems 170 include multiple computing devices, such computing devices may operate according to a sequential computing architecture, a parallel computing architecture, and / or some combination thereof. In addition, one or more remote computing systems 170 may be used to collect, generate, send, and / or receive one or more signals and / or data including one or more semantic tags that may be associated with one or more locations and / or one or more features of one or more locations. One or more remote computing systems 170 may include a smartphone device, which a user of the smartphone device may use to generate semantic tags associated with one or more images captured by a camera of the smartphone device. For example, a user of a smartphone device may capture an image including a local landmark (e.g., a fountain), the image of the local landmark including the latitude and longitude of the local landmark as geographic location information embedded in the image. The user may then manually annotate the image with a semantic tag including a description of the local landmark, and then send it to a server computing system 170 including an image repository. Manually annotated images can be highly encrypted, anonymized (e.g., any personal information associated with the image is either not collected or deleted after receipt), and maintained in a privacy-enhancing manner for use by server computing system 170, which can provide the image and associated semantic tags for use by the image content analysis portion of server computing system 170 and / or training computing system 150.

[0093] Network 180 may include any type of communication network, including a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications through network 180 may be carried via any type of wired and / or wireless connection using a variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0094] Figure 1 An example computing system that can be used to implement the present disclosure is illustrated. Other computing systems may also be used. For example, in some embodiments, the computing device 102 may include a model trainer 160 and a training data set 162. In such embodiments, one or more machine learning models 120 may be trained and used locally at the computing device 102. In some such embodiments, the computing device 102 may implement the model trainer 160 to personalize one or more machine learning models 120 based on user-specific data.

[0095] Figure 2 A diagram of an example device according to an example embodiment of the present disclosure is depicted. Computing device 200 may include one or more features of computing device 102, server computing system 130, and / or training computing system 150. In addition, computing device 200 may perform one or more actions and / or operations, including Figure 1 One or more actions and / or operations depicted in FIG. 1 are performed by computing device 102 , server computing system 130 , and / or training computing system 150 .

[0096] like Figure 2 As shown in , the computing device 200 may include one or more memory devices 202, landmark data 204, one or more interconnects 232, one or more processors 220, a network interface, one or more mass storage devices 224, one or more output devices 226, a sensor array 228, and one or more input devices 230.

[0097] One or more memory devices 202 may store information and / or data (e.g., landmark data 204) including information associated with processing of one or more instructions for performing one or more actions and / or operations, including accessing semantic tags, identifying landmarks, selecting landmarks for navigation, and / or generating navigation instructions with reference to landmarks.

[0098] Landmark data 204 may include Figure 11, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34,

[0099] The one or more interconnects 210 may include one or more interconnects or buses that may be used to send and / or receive one or more signals (e.g., electronic signals) and / or data (e.g., landmark data) between components of the computing device 200, including one or more memory devices 202, one or more processors 220, a network interface 222, one or more mass storage devices 224, one or more output devices 226, a sensor array 228, and / or one or more input devices 230. The one or more interconnects 210 may be arranged or configured in different ways including parallel or serial connections. In addition, the one or more interconnects 210 may include one or more internal buses to connect the internal components of the computing device 200; and one or more external buses for connecting the internal components of the computing device 200 to one or more external devices. For example, the one or more interconnects 210 may include different interfaces including Industry Standard Architecture (ISA), Extended ISA, Peripheral Component Interconnect (PCI), PCI Express, Serial AT Attachment (SATA), HyperTransport (HT), USB (Universal Serial Bus), Thunderbolt, and / or IEEE 1394 interface (FireWire).

[0100] The one or more processors 220 may include one or more computer processors configured to execute one or more instructions stored in the one or more memory devices 202. For example, the one or more processors 220 may include, for example, one or more general-purpose central processing units (CPUs), application-specific integrated circuits (ASICs), and / or one or more graphics processing units (GPUs). In addition, the one or more processors 220 may perform one or more actions and / or operations, including one or more actions and / or operations associated with the landmark data 204. For example, the one or more processors 220 may include a single or multiple core devices, including a microprocessor, a microcontroller, an integrated circuit, and / or a logic device.

[0101] The network interface 222 can support network communications. For example, the network interface 222 can support communications via a network including a local area network and / or a wide area network (e.g., the Internet). One or more mass storage devices 224 (e.g., a hard drive and / or a solid state drive) can be used to store data including the landmark data 204. One or more output devices 226 can include one or more display devices (e.g., an LCD display, an OLED display, and / or a CRT display), one or more light sources (e.g., LEDs), one or more speakers, and / or one or more tactile output devices.

[0102] The one or more input devices 230 may include one or more keyboards, one or more touch-sensitive devices (e.g., a touch screen display), one or more buttons (e.g., an “on / off” button, a “yes / no” button), one or more microphones, and / or one or more cameras (e.g., a camera that can be used to detect gestures and / or facial expressions).

[0103] The one or more memory devices 202 and the one or more mass storage devices 224 are shown separately, however, the one or more memory devices 202 and the one or more mass storage devices 224 may be regions within the same memory module. The computing device 200 may include one or more additional processors, memory devices, network interfaces, which may be provided separately or on the same chip or board. The one or more memory devices 202 and the one or more mass storage devices 224 may include one or more computer-readable media, including but not limited to non-transitory computer-readable media, RAM, ROM, hard disk drives, flash drives, and / or other memory devices.

[0104] One or more memory devices 202 can store instruction sets for applications including operating systems that can be associated with various software applications or data. One or more memory devices 202 can be used to operate various applications, including mobile operating systems developed specifically for mobile devices. As such, one or more memory devices 202 can store instructions that allow software applications to access data including wireless network parameters (e.g., identity of the wireless network, quality of service), and call various services including phone, location determination (e.g., via global positioning service (GPS) or WLAN) and / or wireless network data call initiation services. In other embodiments, one or more memory devices 202 can be used to operate or execute a general-purpose operating system that works on both mobile and fixed devices (such as smart phones and desktop computers).

[0105] Software applications that may be operated or executed by computing device 200 may include Figure 1In addition, software applications that can be operated or executed by the computing device 200 may include native applications or web-based applications.

[0106] In some embodiments, the computing device 200 may be associated with or include a positioning system (not shown). The positioning system may include one or more devices or circuits for determining the location of the computing device 200. For example, the positioning device may determine the actual or relative location by using a satellite navigation positioning system (e.g., GPS system, Galileo positioning system, Global Navigation Satellite System (GLONASS), Beidou Satellite Navigation and Positioning System), an inertial navigation system, a dead reckoning system, based on an IP address, by using triangulation and / or proximity to a cellular tower or Wi-Fi hotspot, beacon, etc., and / or other suitable techniques for determining location.

[0107] Figure 3 An example of landmark detection according to an example embodiment of the present disclosure is depicted. Figure 3 The output shown in can be replaced by including Figure 1 The computing system or computing device generates and / or determines one or more characteristics of the computing device 102, server computing system 130, and / or training computing system 150 depicted in FIG. Figure 3 As shown in , image 300 includes a pole object 310 , a fountain object 312 , a bus stop object 314 , a pedestrian object 316 , and a vehicle object 318 .

[0108] Image 300 depicts a scene having one or more features, including one or more features associated with various objects that have been identified by a content analysis system (e.g., a content analysis system including one or more machine learning models trained to detect one or more features of input content that may include one or more images). For example, content analysis system may include one or more features of computing device 102, server computing system 130, and / or training computing system 150. Additionally, one or more features of image 300, including pole object 310, fountain object 312, bus stop object 314, pedestrian object 316, and vehicle object 318, may be associated with various semantic tags (e.g., including in the context of a text tag, a description of various aspects of the one or more features). Figure 7 For example, based on the semantic tags including one or more features of the multiple semantic tags described in the method 700 depicted in FIG. Figure 1 In the image content analysis performed by one or more machine learning models of one or more features of one or more machine learning models 120 depicted in FIG, one or more features of image 300 can be associated with one or more of a plurality of semantic tags.

[0109] In this example, the semantic label associated with pole object 310 can be used to indicate that pole object 310 is associated with a type of object that occurs frequently (e.g., there are many poles throughout the city where image 300 is captured, including poles for street signs, telephone cables, power cables, and / or various utility poles), and that pole object 310 has low uniqueness (e.g., various poles found throughout the city where image 300 is captured can have similar sizes, shapes, and appearances). Additionally, with respect to prominence, pole object 310 has a significant size (e.g., is significant relative to a predetermined prominence threshold, which can be associated with one or more physical sizes including the height of the object) and can be seen from various angles and distances.

[0110] Furthermore, pole object 310 is visually constant and its appearance changes very little over time. Changes in the appearance of pole object 310 may include minor changes due to weather conditions (e.g., snow accumulation) and human intervention (e.g., graffiti and / or posters) that do not significantly obscure pole object 310. Furthermore, pole object 310 is attached to the ground and has a constant position over time, which makes pole object 310 easier to select as a landmark than objects that may move away from a certain location (e.g., pedestrians).

[0111] In addition, with respect to the context associated with the rod object 310 (e.g., with respect to the Figure 7 ), pole object 310 is located in the foreground and closer to the vantage point from which the image is captured, which allows pole object 310 to be viewed unobstructed. The image of pole object 310 is captured during a time when daylight allows pole object 310 to be clearly visible without the need for additional light (e.g., street lights). Pole object 310 is also located near street lights (not shown) that can illuminate pole object 310 when night falls. As such, when one or more landmarks are selected (e.g., by computing device 102) at the vantage point from which image 300 is captured, pole object 310 is less likely to be included than other potential landmarks in image 300 that appear less frequently, are more unique, are more prominent, and / or are more visible from the vantage point.

[0112] The semantic tag associated with fountain object 312 may be used to indicate that fountain object 312 is associated with a type of object that does not appear frequently (e.g., there are a small number of fountains throughout the city where image 300 is captured) and that fountain object 312 has high uniqueness (e.g., fountain object 312 has a unique sculpture that distinguishes it from other fountains). Additionally, with respect to prominence, fountain object 312 is large in size and the water ejected into the air from fountain object 312 increases the distance at which fountain object 312 is visible. Fountain object 312 may also be visible from different angles (e.g., fountain object 312 may be visible from all sides except from below fountain object 312) and distances.

[0113] However, since the appearance of the fountain changes substantially depending on the season (e.g., the water of the fountain is frozen or drained in cold seasons) and whether the fountain is in operation (e.g., the appearance of fountain object 312 when it is spraying water is different from the appearance when the fountain is not spraying water), fountain object 312 has low visual constancy. In addition, fountain object 312 is firmly fixed to the ground and has a constant position over time, which makes fountain object 312 easier to be selected as a landmark than a moving object (e.g., a bus or a train).

[0114] In addition, with respect to the context associated with fountain object 312 (e.g., with respect to the context associated with fountain object 312), the following may be included in the context of the fountain object 312: Figure 7 ), fountain object 312 is in the background and further away from the vantage point from which the image is captured, thereby sometimes obstructing the view of fountain object 312 depending on the vantage point from which fountain object 312 is viewed. The image of fountain object 312 is captured during a time when daylight allows fountain object 312 to be clearly visible without the need for additional lights (e.g., street lights). Fountain object 312 also includes its own set of lights that can illuminate fountain object 312 at night. As such, when one or more landmarks are selected (e.g., selected by computing device 102) at the vantage point from which image 300 is captured, fountain object 312 is more likely to be included than other potential landmarks in image 300 that appear more frequently, are less prominent, and / or are less unique.

[0115] The semantic label associated with bus stop object 314 can be used to indicate that bus stop object 314 is associated with a type of object that appears frequently (e.g., there are many bus stops throughout the city where image 300 is captured) and that bus stop object 314 has low uniqueness (e.g., bus stop object 314 lacks unique features that distinguish it from other bus stops). In addition, with respect to prominence, bus stop object 314 has a significant size and can be seen from a variety of angles and distances. In addition, bus stop object 314 is visually constant and has very little change in appearance over time. For example, changes in the appearance of bus stop object 314 may include minor changes due to climatic conditions (e.g., snow accumulation) and human intervention (e.g., graffiti and / or posters) that do not significantly obscure bus stop object 314. In addition, bus stop object 314 is a structure attached to the ground and has a constant position over time, which makes bus stop object 314 more likely to be selected as a landmark.

[0116] Additionally, with respect to the context associated with bus stop object 314 (e.g., with respect to the context associated with the bus stop object 314 that may be included in Figure 7 ), bus stop object 314 is in the background and further away from the vantage point from which the image is captured, which obscures the view of bus stop object 314. For example, from various vantage points, pole object 310 would obscure bus stop object 314. Images of bus stop object 314 are captured during a time when daylight allows bus stop object 314 to be clearly visible without the need for additional light (e.g., street lights). Additionally, bus stop object 314 is also located near street lights (not shown) and has its own light source (e.g., lights within bus stop object 314) that illuminates when night falls. As such, when one or more landmarks are selected (e.g., selected by computing device 102) at the vantage point from which image 300 is captured, bus stop object 314 is less likely to be included than other potential landmarks in image 300 that appear less frequently, are more unique, and / or are more visible from the vantage point.

[0117] The semantic label associated with pedestrian object 316 can be used to indicate that pedestrian object 316 is associated with a type of object that appears frequently (e.g., there are many pedestrians at various locations throughout the city where image 300 is captured) and that pedestrian object 316 has low uniqueness (e.g., pedestrians tend to look similar at a distance, and at a very close distance, most pedestrians are not very unique in appearance). In addition, with respect to prominence, pedestrian object 316 is not particularly large, and depending on the vantage point from which pedestrian object 316 is viewed, pedestrian object 316 is often obscured by other objects (including, for example, pole object 310 and bus stop object 314). In addition, pedestrian object 316 is not visually constant because changes in clothing and other aspects of the physical appearance of the pedestrian are quite frequent. For example, changes in the appearance of pedestrian object 316 can include changes in clothing, glasses, and hats worn by pedestrian object 316. In addition, pedestrian object 316 is mobile and has a position that is highly variable over time. For example, pedestrian object 316 may move from the location of a home to the location of bus stop object 314 to a bus that picks up pedestrian object 316 from the location in image 300 .

[0118] In addition, regarding the context associated with pedestrian object 316 (e.g., Figure 7 ), pedestrian object 316 is in the background and at a mid-range from the vantage point from which the image is captured. The image of pedestrian object 316 is captured during a time when daylight allows pedestrian object 316 to be clearly visible without the need for additional light (e.g., street lights). Pedestrian object 316 is also located near a street light (not shown) that can illuminate pedestrian object 316 when night falls. As such, when one or more landmarks are selected (e.g., selected by computing device 102) at the vantage point from which image 300 is captured, pedestrian object 316 is less likely to be included than other potential landmarks in image 300 that appear less frequently, are more unique, are more prominent, and / or are more visible from the vantage point. In addition, the high mobility associated with pedestrian object 316 also makes pedestrian object 316 less likely to be selected as a landmark.

[0119] The semantic label associated with vehicle object 318 can be used to indicate that vehicle object 318 is associated with a type of object that appears frequently (e.g., there are a large number of vehicles at various locations throughout the city where image 300 is captured) and that vehicle object 318 has low uniqueness (e.g., there are many vehicles that look similar, including vehicles of the same brand, model, and color). In addition, with respect to prominence, vehicle object 318 has a significant size and can be seen from various angles and distances, but, because it is mobile, vehicle object 318 is often obscured by other objects. In addition, vehicle object 318 is visually constant because the shape and color of vehicle object 318 remain constant over time. In addition, vehicle object 318 is highly mobile and has a position that can change quickly in a short period of time. For example, except when parked, vehicle object 318 is used to transport people and goods throughout the city where image 300 is captured, and therefore tends to change position frequently.

[0120] In addition, with respect to the context associated with the vehicle object 318 (e.g., with respect to the Figure 7 318 is located far away in the background relative to the vantage point from which the image 300 is captured (context associated with the information in the context data described in the method 700 depicted in ). The image of the vehicle object 318 is captured during a time when daylight allows the vehicle object 318 to be clearly visible without the need for additional light (e.g., street lights). The vehicle object 318 also has its own light sources (e.g., headlights) that can illuminate the area around the vehicle object 318, but does not provide a large amount of illumination of the vehicle object 318 itself (e.g., when night falls, the headlights and taillights of the vehicle object 318 are highly visible, but the impression of the appearance of the vehicle object 318 is different). As such, when one or more landmarks are selected (e.g., by the computing device 102) at the vantage point from which the image 300 is captured, the vehicle object 318 is less likely to be included than other potential landmarks in the image 300 that appear less frequently, are more unique, are more prominent, and / or are more visible from the vantage point. In addition, the high mobility associated with the vehicle object 318 also makes the vehicle object 318 less likely to be selected as a landmark. For example, Figure 1 The computing device 102 depicted in FIG. 1 may select at least one of the objects 310 / 312 / 314 / 316 / 318 as a landmark to be used as a navigation instruction (e.g., Figure 7 700 described in the navigation instructions described in the method 700 depicted in FIG. 7 ).

[0121] Figure 4 An example of landmark detection according to an example embodiment of the present disclosure is depicted. Figure 4 The output shown in can be replaced by including Figure 1The computing device or computing system described in the figure generates and / or determines one or more characteristics of the computing device 102, server computing system 130 and / or training computing system 150. Figure 4 As shown in , image 400 includes building object 402 , sculpture object 410 , sculpture object 412 , sign object 414 , and address object 416 .

[0122] Image 400 depicts a scene having one or more features, including one or more features associated with various objects that have been identified by a content analysis system (e.g., a content analysis system including one or more machine learning models trained to detect one or more features of input content that may include one or more images). For example, content analysis system may include one or more features of computing device 102, server computing system 130, and / or training computing system 150. Additionally, one or more features of image 400 (including building object 402, sculpture object 410, sculpture object 412, sign object 414, and address object 416) may be associated with various semantic tags (e.g., including descriptions of various aspects of the one or more features). Figure 7 For example, based on the semantic tags including one or more features of the multiple semantic tags described in the method 700 depicted in FIG. Figure 1 In the image content analysis performed by one or more machine learning models of one or more features of the machine learning model 120 depicted in FIG, one or more features of the image 400 can be associated with one or more of a plurality of semantic tags.

[0123] The semantic tags associated with sculptural objects 410 and 412 may be used to indicate that sculptural objects 410 and 412 are associated with a type of object that does not occur frequently (e.g., there are a small number of matching relief pairs on buildings in the city where image 400 is captured), and that sculptural objects 410 and 412 have high uniqueness (e.g., sculptural objects 410 and 412 have unique sculptural features that distinguish them from other sculptures). Additionally, with respect to prominence, sculptural objects 410 and 412 have significant sizes (e.g., are significant relative to a predetermined prominence threshold, which may be associated with one or more physical sizes including the height of the object) and protrude from the surface of building object 402, which allows sculptural objects 410 and 412 to be seen from a variety of angles and distances.

[0124] Furthermore, the location of sculpture object 410 in close proximity to sculpture object 412 further increases the uniqueness of the pair, since the uniqueness of either object may be based not only on the individual uniqueness of each object, but also on the uniqueness of the two objects as part of a pair. Thus, the combination of features may be associated with a semantic label (e.g., a semantic label describing "a pair of reliefs"). Furthermore, since the sculptures are carved from granite (hard stone) and change very little over time, sculpture objects 410 and 412 are visually constant. Additionally, sculpture objects 410 and 412 are part of building object 402 and are firmly attached to building object 402, and remain in the same location over time, which makes sculpture objects 410 and 412 more likely to be selected as landmarks.

[0125] In addition, regarding the context associated with sculpture object 410 and sculpture object 412 (e.g., the context associated with the sculpture object 410 and sculpture object 412 may be included in Figure 7 410 and 412 are located at a height that is easily visible from ground level. The images of sculptural objects 410 and 412 are captured during a time when daylight allows sculptural objects 410 and 412 to be clearly visible without the need for additional light. Neither sculptural objects 410 nor sculptural objects 412 include a light source, and are not located in a position where an external light source illuminates sculptural objects 410 or 412 at night. As such, when one or more landmarks are selected (e.g., by computing device 102) at a vantage point from which image 400 is captured, sculptural objects 410 and 412 are more likely to be included than other potential landmarks in image 400 that appear more frequently, are less prominent, and / or are less unique.

[0126] The semantic label associated with sign object 414 can be used to indicate that sign object 414 is associated with a type of object that appears frequently (e.g., there are many signs of various sizes and shapes on buildings throughout the city in which image 400 is captured) and that sign object 414 has low uniqueness (e.g., from a distance sign object 414 looks similar to many other different signs, and there are other signs that look the same as sign object 414). In addition, with respect to prominence, sign object 416 is not very large, does not occupy a prominent position, and may be difficult to discern from a longer distance or from an angle that is not close to directly in front of the sign object. Sign object 414 itself is visually constant, with very little change in appearance over time. However, exposure to certain elements (e.g., snow accumulation) can significantly change the visibility of sign object 414. In addition, sign object 414 is in a position that is constant over time, which makes sign object 414 easier to be selected as a landmark.

[0127] In addition, with respect to the context associated with the sign object 414 (e.g., with respect to the Figure 7 400 ), the image of sign object 414 is captured during a time when daylight allows sign object 414 to be clearly visible without the need for additional light (e.g., street lights). Sign object 414 includes a light source that can be used to illuminate sign object 414 when night falls. As such, when one or more landmarks are selected (e.g., by computing device 102) at the vantage point from which image 400 is captured, sign object 414 is less likely to be included than other potential landmarks in image 400 that appear less frequently, are more prominent, and are more unique.

[0128] The semantic tag associated with address object 416 can be used to indicate that address object 416 is associated with a type of object that appears frequently (e.g., there are many street address signs of various sizes and shapes on buildings, poles, and other structures throughout the city where image 400 is captured), and that address object 416 has low uniqueness (e.g., address object 416 looks similar to many other signs from a distance, and there are other signs marked with the number "107" that look the same as address object 416). In addition, with respect to prominence, address object 416 is small in size and may be difficult to discern from a distance. Address object 416 itself is visually constant, with very little change in appearance over time. In addition, the heating element in address object 416 prevents address object 416 from being obscured due to environmental conditions including snow or frost. In addition, address object 416 has a constant position over time, which makes address object 416 more likely to be selected as a landmark.

[0129] In addition, with respect to the context associated with address object 416 (e.g., Figure 7 ), the image of address object 416 is captured during a time when daylight allows address object 416 to be clearly visible without the need for additional light (e.g., street lights). However, address object 416 does not include its own light source, nor is address object 416 illuminated by external light sources (e.g., street lights) at night. As such, when one or more landmarks are selected (e.g., selected by computing device 102) at the vantage point from which image 400 is captured, address object 416 is less likely to be included than other potential landmarks that appear less frequently, are more unique, and are more prominent in image 400. For example, Figure 1 The computing device 102 depicted in FIG. 1 may select at least one of the objects 410 / 412 / 414 / 416 as a landmark to be used as a navigation instruction (e.g., in Figure 7 700 described in the navigation instructions).

[0130] Figure 5 An example of landmark recognition according to an example embodiment of the present disclosure is depicted. Figure 5 The output shown in can be replaced by including Figure 1 The computing device or computing system described in the figure generates and / or determines one or more characteristics of the computing device 102, server computing system 130 and / or training computing system 150. Figure 5 As shown in , image 500 includes region 502 , region 504 , object 510 , object 512 , and object 514 .

[0131] The image 500 includes one or more semantic tags (e.g., including Figure 7 A map of an area (Paris and the area surrounding Paris) representing a location associated with one or more features of the plurality of semantic tags described in method 700 depicted in FIG. 5 . In this example, objects 510 / 512 / 514 are locally prominent fountains (e.g., having Figure 3 312). Region 502, which includes object 510, includes fewer fountains than region 504, which has a large number of fountains including object 512 and object 514. In determining the selection of locally prominent landmarks to be used in navigation, the frequency with which objects associated with semantic tags appear can influence the selection of objects to be used as landmarks, with less frequently occurring objects being more likely to be selected as landmarks than more frequently occurring objects. Thus, all other things being equal, object 510 is more likely to be selected as a landmark in region 502 than in region 504. For example, Figure 1 The computing device 102 depicted in FIG. 1 may select one of the objects 502 / 504 / 506 as a landmark to be used as a navigation instruction (e.g., in Figure 7 700 described in the navigation instructions).

[0132] Figure 6 An example of landmark recognition in an environment according to an example embodiment of the present disclosure is depicted. Figure 6 One or more operations in the environment depicted in the example may be performed by including Figure 1 The computing device 102, server computing system 130, and / or training computing system 150 depicted in the figure may be executed by a computing device or computing system. Figure 6 As shown in , environment 600 includes computing system 602 , road 604 , field of view threshold 606 , field of view threshold 608 , vantage point 610 , driving direction 612 , tree object 620 , monument object 622 , lamppost object 624 , street sign object 626 , and park bench object 628 .

[0133] In this example, computing system 602 includes one or more features of computing device 102 and is located within vantage point 610, which is a vehicle traveling on road 604 along a travel path corresponding to travel direction 612. Vantage point 610 includes one or more sensors, including a camera that provides a view of environment 600, which captures a portion of environment 600 before field of view threshold 606 and field of view threshold 608. In addition, field of view 606 / 608 defines a boundary that divides environment 600 into two areas: a first area including street sign object 626, where the object can be selected for use as a landmark, and a second area including park bench object 628, where the object cannot be selected for use as a landmark.

[0134] From the vantage point 610, objects associated with the semantic label include a park bench 628 object, which is a park bench outside the field of view of one or more sensors of the computing system 602, and would not be selected as a potential landmark because the park bench is outside the field of view and the vantage point 610 moves away from the park bench object 628. In some embodiments, objects behind the viewer that move away from the object may have less value for use as navigation instructions. Within the field of view of the vantage point 610 is a tree object 620 that is a maple tree, a monument object 622 that is a monument on a high pedestal (it includes a large bronze statue of a rider wearing Napoleonic-era clothing and riding a horse), a lamp post object 624 that is a small lamp post, and a street sign object 626 that is a stop sign.

[0135] In this example, monument object 622 is selected for use as a landmark due to its rarity (e.g., the monument is unique and similar monuments occur infrequently), prominence (e.g., the monument is large and mounted on a high pedestal), uniqueness (e.g., the monument has a unique and distinctive appearance), and its location within the field of view of vantage point 610.

[0136] Figure 7 A flowchart of providing navigation instructions with reference to landmarks according to an example embodiment of the present disclosure is depicted. One or more portions of method 700 may be performed or implemented on one or more computing devices or computing systems including, for example, computing device 102, server computing system 130, and / or training computing system 150. In addition, one or more portions of method 700 may be performed or implemented as an algorithm on a hardware device or system disclosed herein. For purposes of illustration and discussion, Figure 7 The steps are depicted to be performed in a specific order. Using the disclosure provided herein, one of ordinary skill in the art will appreciate that the various steps of any method disclosed herein may be adjusted, modified, rearranged, omitted, and / or expanded without departing from the scope of the present disclosure.

[0137] At 702, method 700 may include accessing a plurality of semantic tags associated with a plurality of images. Each of the plurality of semantic tags may be associated with one or more features depicted by one of the plurality of images. For example, each semantic tag may include a semantic description of an object included in a scene depicted by one of the plurality of images. Additionally, each of the one or more features may be associated with a geographic location. The plurality of images may include one or more digital images (e.g., two-dimensional images) of a portion of an environment (e.g., an image of a collection of objects at a particular location in the environment).

[0138] The multiple images may be encoded in any type of image format including a combination of raster images (e.g., a bitmap including a grid of pixels) and / or vector images (e.g., a polygonal representation of an image based on positions including coordinates of x and y axes of a two-dimensional plane). The images may include still images, image frames from a movie, and / or other types of imagery, including LIDAR imagery, RADAR imagery, and / or other types of imagery.

[0139] Examples of digital image formats used by the plurality of images may include JPEG (Joint Photographic Experts Group), BMP (Bitmap), TIFF (Tagged Image File Format), PNG (Portable Network Graphics), and / or GIF (Graphics Interchange Format). The images may be collected from a variety of sources, such as user-submitted imagery, public domain imagery (e.g., obtained via web crawling and appropriately aggregated and anonymized), street-level panoramic imagery collected from a variety of sources including user devices (e.g., smartphones with cameras), and / or other image sources.

[0140] The plurality of semantic tags associated with an image may be associated with one or more features including: a physical size (e.g., the physical size of an object in the image, including the height, length, and / or width of the object); a description (e.g., a description including one or more manually created descriptions and / or one or more descriptions generated by a content analysis system, which may include one or more features of computing device 102, server computing system 130, and / or training computing system 150), and / or an object identifier (e.g., the identity of an object depicted in one or more images). For example, an identifier for the Eiffel Tower in Paris, France may include an identifier associated with the name of the Eiffel Tower in English, "Eiffel tower," and the name of the Eiffel Tower in French, "La tour Eiffel." Additionally, the semantic tags associated with the Eiffel Tower may include information associated with the Eiffel Tower, including a physical size (e.g., the height of the Eiffel Tower), a color (e.g., brown), and a semantic tag associated with a structure type (e.g., building, tower, tourist attraction, and monument). The additional information associated with any of the plurality of images and / or the plurality of semantic tags may include: location (e.g., a street address and / or altitude, latitude, and longitude associated with the image); time of day (e.g., the time of day when the image was captured); date (e.g., the date when the image was captured);

[0141] For example, computing device 102 may access locally stored data (e.g., data stored in a storage device of computing device 102) that includes information associated with multiple semantic tags and multiple images. In addition, computing device 102 may access (e.g., via network 180) data (e.g., multiple images and / or multiple semantic tags) stored on a remote storage device (e.g., a storage device of server computing system 130 and / or training computing system 150).

[0142] For another example, computing device 102 may receive data including information associated with a plurality of semantic tags and a plurality of images via a communication network (e.g., a wireless and / or wired network including a LAN, WAN, or the Internet) through which one or more signals (e.g., electronic signals) and / or data may be sent and / or received. Computing device 102 may receive a plurality of images and / or a plurality of semantic tags from server computing system 130 and / or training computing system 150, for example, via network 180.

[0143] In some embodiments, each of the one or more features can be associated with a time of day (e.g., hours, minutes, and seconds of day), a season (e.g., winter, summer, spring, or fall), visual constancy (e.g., the degree to which one or more features appear the same over time), and / or one or more locations from which each of the one or more features is visible (e.g., the geographic location from which each feature is visible, including latitude, longitude, and / or altitude). For example, the visual constancy of an existing building can be associated with the dimensions of the building remaining the same over time (e.g., month to month or year to year). As such, a building that is under construction and initially has only the building foundation, a bare steel frame structure two months later, and a completed facade exterior three months later will have lower visual constancy than a fully constructed building with very minor structural changes over time.

[0144] At 704, method 700 may include identifying one or more landmarks including one or more features that satisfy one or more entropy criteria based at least in part on the plurality of semantic tags, the entropy criteria measuring one or more characteristics of the one or more features, the characteristics including a localized prominence of each of the one or more features. For example, computing device 102 may access data associated with a plurality of semantic tags indicating one or more landmarks in an area. Additionally, one or more landmarks may be associated with one or more characteristics (e.g., physical size or shape) that may be compared to one or more entropy criteria that may be used to identify the one or more landmarks.

[0145] One or more entropy criteria may be associated with the frequency with which each of the one or more features appears in the region (e.g., the ratio of the occurrences of the one or more features), the uniqueness of each feature (e.g., the degree to which each feature differs from other features in the region), the prominence of each feature (e.g., how prominent the feature is in terms of physical size and / or visibility), the visual constancy of the features (e.g., the degree to which the features change visually over time), and / or the persistence of the features in position (e.g., the degree to which the features will remain in the same position over time). Satisfaction of one or more entropy criteria may be based, for example, on infrequent features (e.g., the only flagpole in the region or one of two tall buildings in the region). For example, one or more entropy criteria may be applied to semantic tags associated with an image. For example, one or more entropy criteria may be applied to semantic tags associated with image 300 to determine that fountain object 312 satisfies one or more entropy criteria based in part on the lower frequency, greater uniqueness, and greater prominence of fountain object 312 compared to other objects in image 300. As another example, one or more entropy criteria may be applied to the semantic tags associated with image 400 to determine that sculptural objects 410 and 412 satisfy the one or more entropy criteria based in part on the lower frequency, greater uniqueness, and greater prominence of sculptural objects 410 and 412 compared to other objects in image 400.

[0146] For example, clustering or other algorithmic techniques may be used to determine the rarity or infrequency associated with each feature, which may then be used to guide the selection of features for use as landmarks. As an example, for each location, the area surrounding the location may be analyzed to identify which features associated with the location are the rarest (e.g., a semantic label histogram in the area surrounding a vantage point location may determine that a monument with a rider on a horse appears once, while a group of traffic lights appears twenty times, thereby indicating that the monument has a higher entropy value and is a better choice for use as a landmark). For another example, satisfaction of one or more entropy criteria may include the uniqueness of various characteristics of a feature relative to other similar features in the area (e.g., a small house located at one corner of an intersection with four corners will contrast and be more unique than a tall building located at the other three corners).

[0147] Landmarks can therefore be determined by focusing on high-entropy "tags" (e.g., those tags associated with one or more features that appear persistent in time and can exhibit high localized salience) by aggregating semantic tag statistics geographically and over time for each location. Thus, the system can identify tags that have high confidence at that location but are relatively uncommon or rare in the surrounding area.

[0148] In some embodiments, one or more entropy criteria may include a frequency of occurrence of each of the one or more features within a predetermined area that does not exceed a predetermined threshold frequency (e.g., the number of times the one or more features occur per area unit), a temporal persistence of each of the one or more features at a location that exceeds a predetermined threshold duration (e.g., how long the feature has been present in a particular area), and / or a size of each of the one or more features that exceeds a threshold size (e.g., one or more physical sizes). For example, a feature (e.g., a statue) that occurs more than once in an area of ​​twenty by twenty square meters may exceed a predetermined threshold frequency of a statue that occurs no more than once in an area of ​​fifty by fifty square meters.

[0149] At 706, method 700 may include selecting at least one landmark for navigation at the location based at least in part on contextual data associated with the location on a path including the plurality of locations. For example, computing device 102 may determine a context including a time of day, a season, and / or an amount of traffic approaching the vantage point. Based on the context, for example, indicating that heavy fog has surrounded an area around the location, computing device 102 may select a landmark that is brightly illuminated.

[0150] For another example, computing device 102 may determine that contextual data indicates that the season is winter and the area around the location is covered with snow. Based on a portion of the landmark being covered with snow, computing device 102 may select at least one landmark that is locally prominent when covered with snow, or at least one landmark that is positioned such that it is not covered with snow. For example, a very tall building (the tallest building in the area) will not be covered with snow and may therefore remain locally prominent, while a unique mural on a wall may not be visible due to snow accumulation.

[0151] Additionally, the amount of traffic proximate to the vantage point may be determined by computing device 102 based on one or more outputs from sensor array 228, which may include one or more cameras that may detect the amount of traffic (e.g., pedestrian traffic and / or vehicular traffic) and other activity (e.g., construction) near the vantage point. Computing device 102 may select at least one landmark based on the amount of traffic detected by sensor array 228.

[0152] At 708, method 700 may include generating at least one navigation instruction that references at least one landmark. The at least one navigation instruction may include one or more visual instructions (e.g., text instructions displayed on a display device) and / or one or more audible instructions (e.g., instructions issued from an audio output device).

[0153] For example, computing device 102 may generate one or more visual instructions, including text instructions displayed on a display device (e.g., a flat panel display). The text instructions may, for example, describe the appearance of the landmark (e.g., "a large white building with a row of Ionic columns") and the location of the landmark (e.g., "directly in front"). In addition, in some embodiments, one or more visual instructions may include one or more images associated with the landmark. For example, a text instruction of "a large white building with a row of Ionic columns" may be accompanied by an image of the building. In addition, the image of the large white building with a row of Ionic columns included in the one or more visual instructions may be generated based at least in part on the position of the vantage point relative to the landmark. For example, when the landmark is a four-sided building, the image associated with the one or more visual instructions may be an image of the side of the building visible from the vantage point.

[0154] For another example, computing device 102 may generate one or more audible instructions that describe the appearance of a landmark (e.g., “an ornate gate decorated with lions and dragons”) and the location of the landmark (e.g., “on your left at the next intersection”). Additionally, the one or more audible instructions may be the same as the one or more visual instructions (e.g., when the one or more visual instructions visually indicate “a red building on the right,” the one or more audio instructions may indicate “a red building on the right”). In other embodiments, the one or more audible instructions may be different from the one or more visual instructions (e.g., when the one or more visual instructions visually indicate “a red building on the right,” the one or more audio instructions may indicate “a large red building next to a small white building”).

[0155] In some embodiments, at least one audible instruction may be generated at a volume level associated with the proximity of the at least one landmark relative to the vantage point. For example, one or more audible instructions may be generated at a volume that is inversely proportional to the distance between the vantage point and the at least one landmark (e.g., the volume of the one or more audible instructions increases as the distance to the at least one landmark decreases).

[0156] Figure 8 A flowchart of providing navigation instructions with reference to landmarks according to an example embodiment of the present disclosure is depicted. One or more portions of method 800 may be performed or implemented on one or more computing devices or computing systems including, for example, computing device 102, server computing system 130, and / or training computing system 150. In addition, one or more portions of method 800 may be performed or implemented as an algorithm on a hardware device or system disclosed herein. In some embodiments, one or more portions of method 800 may be implemented as a Figure 7 700 as a portion of the method 700 depicted in FIG. Figure 8The steps are depicted to be performed in a specific order for the purposes of illustration and discussion. Using the disclosure provided herein, one of ordinary skill in the art will understand that the various steps of any method disclosed herein may be adjusted, modified, rearranged, omitted and / or expanded without departing from the scope of the present disclosure.

[0157] At 802, method 800 may include determining a rate at which each of the one or more features occurs within a predetermined area based at least in part on a plurality of semantic tags. Additionally, the one or more features may include: individual features including the size of an object or the brightness of an illuminated object; and a combination of one or more features, including a combination of colors and / or shapes. For example, computing device 102 may determine a rate at which a combination of features (e.g., letters of the alphabet stylized in bright yellow) occurs within a square kilometer area. Additionally, one or more features may be associated with semantic tags for letters of the alphabet (e.g., the letter "N"), particular colors (e.g., bright yellow), and companies associated with the features (e.g., banks, restaurants, or other businesses). Computing device 102 may determine that a combination of features occurs once within a square kilometer area.

[0158] In some embodiments, determining a ratio of each of the one or more features occurring within a predetermined area based at least in part on the plurality of semantic tags can be used to identify, based at least in part on the plurality of semantic tags, a feature that satisfies the following conditions: Figure 7 One or more landmarks of one or more features of one or more entropy criteria measuring localized prominence of each of the one or more features described in 704 of the method 700 depicted in FIG.

[0159] At 804, method 800 may include determining that one or more landmarks include one or more features that occur least frequently or at a rate below a threshold rate. For example, when determining a restaurant in a group of restaurants to be used as a landmark, computing device 102 may determine that the restaurant that occurs least frequently among other restaurants in the area is to be included in the one or more landmarks. For another example, when determining an object to be used as a landmark, computing device 102 may determine that a prominent sculpture that occurs at a rate below a threshold rate (e.g., a threshold rate of once per square kilometer) is to be included in the one or more landmarks.

[0160] In some embodiments, determining that one or more landmarks include one or more features that occur least frequently or at a rate below a threshold rate can be used to identify one or more landmarks including one or more features based at least in part on a plurality of semantic tags, wherein the one or more features satisfy the following conditions: Figure 7 One or more entropy criteria for measuring the localized prominence of each of the one or more features described in 704 of the method 700 depicted in FIG.

[0161] At 806, method 800 may include determining a confidence score for each respective one of the one or more features based at least in part on a number of times each of the one or more features has been associated with a semantic tag in a plurality of semantic tags. For example, computing device 102 may access data associated with a number of times each of the one or more features has been tagged with the same tag or a related set of tags (e.g., a set of tags including a window tag may also include a porthole tag). Features that are tagged more times with the same feature may be associated with a higher confidence score, while features that are tagged less times with a feature may be associated with a lower confidence score.

[0162] In some embodiments, determining a confidence score for each respective one of the one or more features based at least in part on the number of times each of the one or more features has been associated with a semantic tag in a plurality of semantic tags can be used to identify one or more landmarks including the one or more features based at least in part on the plurality of semantic tags, wherein the one or more features satisfy the following conditions: Figure 7 One or more entropy criteria for measuring the localized prominence of each of the one or more features described in 704 of the method 700 depicted in FIG.

[0163] At 808, method 800 may include identifying one or more features having confidence scores that satisfy one or more confidence score criteria as landmarks. For example, computing device 102 may identify one or more features having confidence scores that equal or exceed a threshold confidence score as landmarks.

[0164] In some embodiments, the confidence score may be based at least in part on the number of different perspectives from which each of one or more features associated with a semantic tag has been viewed (e.g., semantic tags associated with images of the same object viewed from various angles, viewpoints, and / or distances), and / or the recency with which one or more features have been associated with the semantic tag (e.g., the amount of time that has passed since one or more features were associated with the semantic tag).

[0165] In some embodiments, identifying one or more features having confidence scores that satisfy one or more confidence score criteria as landmarks can be used to identify one or more landmarks including one or more features based at least in part on a plurality of semantic tags, the features satisfying the criteria as follows: Figure 7 One or more entropy criteria for measuring the localized prominence of each of the one or more features described in 704 of the method 700 depicted in FIG.

[0166] At 810, method 800 may include determining one or more clusters of one or more features that satisfy one or more entropy criteria. Additionally, each of the one or more clusters may include one or more features having a common semantic type. For example, computing device 102 may determine clusters of buildings that may satisfy one or more entropy criteria associated with building density for clusters of buildings (e.g., the number of buildings within a predetermined area, a predetermined radius from a vantage point, or within a predetermined number of blocks).

[0167] In some embodiments, determining one or more clusters of one or more features that satisfy one or more entropy criteria can be used to identify one or more landmarks including one or more features based at least in part on a plurality of semantic tags, the features satisfying the following criteria: Figure 7 One or more entropy criteria for measuring the localized prominence of each of the one or more features described in 704 of the method 700 depicted in FIG.

[0168] Fig. 9 A flowchart of providing navigation instructions with reference to landmarks according to an example embodiment of the present disclosure is depicted. One or more portions of method 900 may be performed or implemented on one or more computing devices or computing systems including, for example, computing device 102, server computing system 130, and / or training computing system 150. In addition, one or more portions of method 900 may be performed or implemented as an algorithm on a hardware device or system disclosed herein. In some embodiments, one or more portions of method 900 may be implemented as a Figure 7 For purposes of illustration and discussion, Fig. 9 The steps are depicted to be performed in a specific order. Using the disclosure provided herein, one of ordinary skill in the art will appreciate that the various steps of any method disclosed herein may be adjusted, modified, rearranged, omitted, and / or expanded without departing from the scope of the present disclosure.

[0169] At 902, method 900 may include determining visibility of each of the one or more landmarks from a vantage point associated with the location. Additionally, the contextual data may include a vantage point associated with the location (e.g., a point within the location from which a portion of the surrounding environment can be viewed). For example, computing device 102 may determine one or more landmarks visible from a vantage point based in part on a distance to each of the one or more landmarks (e.g., visibility may decrease with greater distance from the landmark), whether each of the one or more landmarks is obscured by other objects (e.g., a tree obscures the landmark), and a height of each of the one or more landmarks (e.g., a supertall building that is four hundred meters tall may be more visible than a low-rise building that is 20 meters tall).

[0170] In some embodiments, visibility can be based at least in part on the distance at which each of the one or more landmarks is visible from the vantage point, the amount of light projected on each of the one or more landmarks (e.g., the amount of light emitted by street lights near the one or more landmarks, and / or the amount of light emitted by a light source included in each of the one or more landmarks), any obstructions between the vantage point and the one or more landmarks, and / or the physical size of each of the one or more landmarks (e.g., the height of a building).

[0171] In some embodiments, determining visibility of each of the one or more landmarks from a vantage point associated with a location can be used to select, by a computing system, at least one landmark for navigation at the location based at least in part on contextual data associated with the location on a path that includes the plurality of locations, such as Figure 7 As described in 706 of method 700 depicted in .

[0172] At 904, method 900 may include determining a reaction time based at least in part on the speed at the location and the distance to the closest landmark of the one or more landmarks. For example, computing device 102 may determine the reaction time based on the amount of time it takes to reach the location of the landmark at the current speed. Thus, when the vantage point is fixed (e.g., standing in one place or sitting in a stationary car), the reaction time will be longer than when the vantage point is in a car traveling at a speed of one hundred kilometers per hour.

[0173] In some embodiments, the reaction time can be based at least in part on the mode of transportation associated with the vantage point. In addition, the transportation mode can include motor vehicle, bicycle, and / or walking. For example, the reaction time for a slower mode of transportation (e.g., riding a bicycle) can be longer than the reaction time for a faster mode of transportation (e.g., riding a bus).

[0174] Additionally, in some embodiments, the reaction time may vary based on whether the vantage point is associated with a driver of the vehicle or a passenger of the vehicle. When the vantage point is associated with a driver of the vehicle, the reaction time may be shorter than when the vantage point is associated with a passenger of the vehicle.

[0175] In some embodiments, determining a reaction time based at least in part on a speed at the location and a distance to a closest landmark of one or more landmarks can be used to select at least one landmark for navigation at the location based at least in part on contextual data associated with the location on a path including the plurality of locations, such as Figure 7 As described in 706 of method 700 depicted in .

[0176] At 906, method 900 may include determining one or more landmarks that meet one or more reaction time criteria associated with a minimum reaction time. For example, computing device 102 may determine that the minimum reaction time is five seconds and select one or more landmarks from the one or more landmarks that will be visible more than five seconds after the current time.

[0177] In some embodiments, determining one or more landmarks that satisfy one or more reaction time criteria associated with a minimum reaction time can be used to select at least one landmark for navigation at the location based at least in part on contextual data associated with the location on a path that includes the plurality of locations, such as Figure 7 As described at 706 of method 700 depicted in .

[0178] Fig.10 A flowchart of providing navigation instructions with reference to landmarks according to an example embodiment of the present disclosure is depicted. One or more portions of method 1000 may be executed or implemented on one or more computing devices or computing systems including, for example, computing device 102, server computing system 130, and / or training computing system 150. In addition, one or more portions of method 1000 may be executed or implemented as an algorithm on a hardware device or system disclosed herein. In some embodiments, one or more portions of method 1000 may be implemented as a Fig. 9 Part of the method 900 depicted in and / or as Figure 7 For purposes of illustration and discussion, Fig.10 The steps are depicted to be performed in a specific order. Using the disclosure provided herein, one of ordinary skill in the art will appreciate that the various steps of any method disclosed herein may be adjusted, modified, rearranged, omitted, and / or expanded without departing from the scope of the present disclosure.

[0179] At 1002, method 1000 may include determining a direction of travel along a path at a vantage point. Additionally, visibility of each of the one or more landmarks may be associated with a field of view from the vantage point (e.g., a field of view associated with an area in which the one or more landmarks are visible), the field of view from the vantage point being associated with the direction of travel (e.g., the field of view being equal to a predetermined angle relative to a line corresponding to the direction of travel). For another example, Figure 6 In environment 600 depicted in , field of view threshold 606 and field of view threshold 608 may be associated with visibility from vantage point 610 when traveling along travel direction 612 .

[0180] In some embodiments, determining the direction of travel along the path at the vantage point can be used to determine the visibility of each of the one or more landmarks from the vantage point associated with the location, such as Fig. 9As described in 902 of method 900 depicted in .

[0181] At 1004, method 1000 may include determining one or more landmarks facing the direction of travel. For example, computing device 102 may determine that one or more landmarks facing the direction of travel (e.g., one or more landmarks visible from the vantage point) are more visible than one or more landmarks not facing the direction of travel (e.g., one or more landmarks not visible from the vantage point). Figure 6 In environment 600 depicted in , tree object 624 is more visible than park bench object 628 , which is not facing direction of travel 612 of vantage point 610 .

[0182] In some embodiments, determining one or more landmarks facing the direction of travel can be used to determine the visibility of each of the one or more landmarks from a vantage point associated with the location, such as Fig. 9 As described in 902 of method 900 depicted in .

[0183] At 1006, method 1000 may include determining visibility based at least in part on a mode of transportation associated with the vantage point. For example, computing device 102 may determine that visibility from an open-top vehicle (e.g., the top of a convertible or open-air tour bus) is greater than visibility from inside an enclosed vehicle (e.g., a car with a permanent roof).

[0184] In some embodiments, determining visibility based at least in part on a mode of transportation associated with a vantage point can be used to determine visibility of each of the one or more landmarks from a vantage point associated with a location, such as Fig. 9 As described in 902 of method 900 depicted in .

[0185] Fig.11 A flowchart of providing navigation instructions with reference to landmarks according to an example embodiment of the present disclosure is depicted. One or more portions of method 1100 may be executed or implemented on one or more computing devices or computing systems including, for example, computing device 102, server computing system 130, and / or training computing system 150. In addition, one or more portions of method 1100 may be executed or implemented as an algorithm on a hardware device or system disclosed herein. In some embodiments, one or more portions of method 1100 may be implemented as a Figure 7 For purposes of illustration and discussion, Fig.11 The steps are depicted to be performed in a specific order. Using the disclosure provided herein, one of ordinary skill in the art will appreciate that the various steps of any method disclosed herein may be adjusted, modified, rearranged, omitted, and / or expanded without departing from the scope of the present disclosure.

[0186] At 1102, method 1100 may include selecting at least one landmark based at least in part on familiarity with the one or more landmarks. Additionally, the familiarity may be based at least in part on a user's previous association with the location or one or more features of the one or more landmarks. For example, the familiarity may be greater when the user has previously visited the location (e.g., the user has been to the same neighborhood in the past) and / or has previously seen a different landmark (e.g., a department store with the same unique color logo and physical appearance) that has a collection of one or more features similar to the one or more landmarks (e.g., a department store with a unique color logo and physical appearance).

[0187] In some embodiments, the familiarity can be associated with the number of times and / or frequency that a user (e.g., a user associated with a navigation computing device) has previously been at the location (e.g., within a threshold distance of the location). For example, the context data can include a record of the number of times the user has passed each of one or more landmarks. The computing device 102 can then determine when the number of times the user has passed a landmark meets one or more familiarity criteria (e.g., a threshold minimum number of times the landmark has been passed), and select a landmark that meets the one or more familiarity criteria.

[0188] In some embodiments, the familiarity may be associated with a portion of one or more features (e.g., a number of one or more visual characteristics) that each of the one or more landmarks has in common with another landmark that the user has previously seen (e.g., seen a time before viewing the one or more landmarks from the vantage point). Additionally, the user's familiarity with a landmark viewed from the vantage point may be higher when the landmark viewed from the vantage point shares a predetermined portion of the one or more features (e.g., ninety percent of the one or more features) with a different landmark that the user has previously seen.

[0189] For example, when the user has previously viewed another landmark (e.g., another restaurant) that belongs to a restaurant chain and shares the same features, the familiarity of the currently viewed landmark (which is a restaurant associated with the same restaurant chain that shares one or more features (e.g., similar design features, including the same or similar logos, signage, exterior color schemes, and / or exterior masonry)) will be higher. In addition, the determination of landmarks that the user has previously viewed can be based on information including user travel history (e.g., locations that the user has previously visited), direct user input about the landmark (e.g., the user provides information about landmarks that the user is familiar with), and / or user affiliations based at least in part on past user interactions and / or associations with the landmark (e.g., the user has provided information indicating the user's membership in an organization associated with the landmark) (e.g., the user has agreed to provide information maintained in a privacy-enhancing and secure manner).

[0190] In some embodiments, the contextual data may include information associated with the time of day, season, language (e.g., English, Spanish, and / or Japanese), one or more features visible from the location, and / or mode of transportation (e.g., private car, bus, bicycle, and / or walking).

[0191] Additionally, in some embodiments, the contextual data may include information associated with whether the occupant of the vehicle is a driver or a passenger. For example, when the contextual data indicates that the occupant of the vehicle is a driver, the one or more landmarks may include one or more landmarks that are more easily visible from the driver's side of the vehicle. As another example, when the contextual data indicates that the occupant of the vehicle is a passenger, the one or more landmarks may include one or more landmarks that are more easily visible from various passenger seating positions.

[0192] In some embodiments, selecting at least one landmark based at least in part on familiarity with one or more landmarks can be used to select at least one landmark for navigation at the location based at least in part on contextual data associated with the location on a path that includes a plurality of locations, such as Figure 7 As described in 706 of method 700 depicted in .

[0193] At 1104, method 1100 may include adjusting at least one navigation instruction based at least in part on the familiarity. For example, computing device 102 may access information associated with at least one landmark (e.g., a nickname of the landmark or an old name of the landmark) and use the information to modify the navigation instruction used. For another example, computing device 102 may determine that when the familiarity exceeds a threshold familiarity associated with a user who has passed by the landmark greater than a threshold number of times, the navigation instruction is to be selected from data (e.g., a semantic tag) including the nickname of the landmark.

[0194] In some embodiments, adjusting at least one navigation instruction based at least in part on the familiarity can be used to generate at least one navigation instruction that references at least one landmark, such as Figure 7 As described in 706 of method 700 depicted in .

[0195] Fig.12 A flowchart of providing navigation instructions with reference to landmarks according to an example embodiment of the present disclosure is depicted. One or more portions of method 1200 may be executed or implemented on one or more computing devices or computing systems including, for example, computing device 102, server computing system 130, and / or training computing system 150. In addition, one or more portions of method 1200 may be executed or implemented as an algorithm on a hardware device or system disclosed herein. In some embodiments, one or more portions of method 1200 may be implemented as a Figure 7For purposes of illustration and discussion, Fig.12 The steps are depicted to be performed in a specific order. Using the disclosure provided herein, one of ordinary skill in the art will appreciate that the various steps of any method disclosed herein may be adjusted, modified, rearranged, omitted, and / or expanded without departing from the scope of the present disclosure.

[0196] At 1202, method 1200 may include determining that a travel mode associated with the vantage point may be used to determine one or more landmarks including one or more features that satisfy one or more entropy criteria associated with a localized prominence of each of the one or more features. For example, computing device 102 may access data associated with a mode of transportation associated with the vantage point (e.g., a car traveling on a highway), which may be used to select one or more landmarks (e.g., large billboards and advertisements) having greater localized prominence from the highway.

[0197] In some embodiments, determining a driving pattern associated with a vantage point can be used to determine one or more landmarks that include one or more features that satisfy one or more entropy criteria associated with a localized prominence of each of the one or more features, such as Figure 7 As described in 706 of method 700 depicted in .

[0198] At 1204, method 1200 may include determining one or more landmarks that will be visible from the vantage point within a predetermined time period associated with the travel mode based at least in part on the travel direction and speed along the route. For example, computing device 102 may determine a speed associated with the transportation mode (e.g., a speed in kilometers per hour when riding a bicycle) and determine landmarks that will be visible from the vantage point of the cyclist within the next 30 seconds.

[0199] In some embodiments, determining one or more landmarks that will be visible from a vantage point within a predetermined time period associated with a travel mode based at least in part on a travel direction and speed along a path can be used to determine one or more landmarks that include one or more features that satisfy one or more entropy criteria associated with a localized prominence of each of the one or more features, such as Figure 7 As described in 706 of method 700 depicted in .

[0200] Fig.13A flowchart of providing navigation instructions with reference to landmarks according to an example embodiment of the present disclosure is depicted. One or more portions of method 1300 may be performed or implemented on one or more computing devices or computing systems including, for example, computing device 102, server computing system 130, and / or training computing system 150. In addition, one or more portions of method 1300 may be performed or implemented as an algorithm on a hardware device or system disclosed herein. In some embodiments, one or more portions of method 1300 may be implemented as a Figure 7 For purposes of illustration and discussion, Fig.13 The steps are depicted to be performed in a specific order. Using the disclosure provided herein, one of ordinary skill in the art will appreciate that the various steps of any method disclosed herein may be adjusted, modified, rearranged, omitted, and / or expanded without departing from the scope of the present disclosure.

[0201] At 1302, method 1300 may include accessing image data including a plurality of images associated with one or more semantic tags, wherein each of the one or more semantic tags is associated with one or more features of the plurality of images. Additionally, each of the one or more features may be associated with a geographic location. For example, computing device 102 may receive image data including information associated with a plurality of semantic tags and a plurality of images (e.g., semantically tagged photos from an image repository) via a communication network (e.g., the Internet) through which one or more signals (e.g., electronic signals) and / or data may be sent and / or received.

[0202] At 1304, method 1300 may include determining one or more landmarks that include one or more features that satisfy one or more entropy criteria associated with a localized prominence of each of the one or more features. For example, computing device 102 may access a plurality of semantic tags to determine one or more landmarks (e.g., objects having unique, less common, and / or locally salient features) in a region that include one or more features (e.g., physical size, color, brightness, and / or shape) that satisfy one or more entropy criteria (e.g., a physical size exceeding a physical size threshold).

[0203] At 1306, method 1300 may include determining one or more landmarks associated with the vantage point based at least in part on context data associated with the location of the vantage point on a path including the plurality of locations. For example, computing device 102 may access context data including a time of day, environmental conditions (e.g., rain, snow, fog, hail, and / or sunlight levels), a season (e.g., summer, spring, fall, winter), and / or a location (e.g., latitude, longitude, and / or altitude) of an object between the vantage point and the one or more landmarks. Based on the context data, computing device 102 may determine one or more landmarks that are visible (e.g., determined based on the time of day, pollution levels, and precipitation) and are not partially or fully obscured by other objects between the vantage point and each of the one or more landmarks.

[0204] At 1308, method 1300 may include generating navigation data including one or more indications associated with one or more landmarks. For example, computing device 102 may generate visual instructions on a head-up display device of the vehicle including a description of the appearance of the landmark (e.g., "a restaurant shaped like a castle") and the location of the landmark (e.g., "fifty meters ahead on the right"). In some embodiments, the one or more indications may include information associated with the contextual data, including whether the landmark is obscured (e.g., "the landmark is behind a small tree at the corner"), whether the landmark is illuminated, and / or the location of the landmark relative to other objects (e.g., "a landmark by the river").

[0205] In some embodiments, the one or more indications may include one or more visual indications associated with the relative position of one or more landmarks on the path relative to the vantage point (e.g., a photograph of the one or more landmarks displayed on a display device), and / or one or more audible indications associated with the relative position of one or more landmarks on the path relative to the vantage point (e.g., audible indications from a speaker based on semantic tags associated with the one or more landmarks).

[0206] Fig.14 A flowchart of providing navigation instructions with reference to landmarks according to an example embodiment of the present disclosure is depicted. One or more portions of method 1400 may be executed or implemented on one or more computing devices or computing systems including, for example, computing device 102, server computing system 130, and / or training computing system 150. In addition, one or more portions of method 1400 may be executed or implemented as an algorithm on a hardware device or system disclosed herein. In some embodiments, one or more portions of method 1400 may be implemented as a Figure 7 For purposes of illustration and discussion, Fig.14The steps are depicted to be performed in a specific order. Using the disclosure provided herein, one of ordinary skill in the art will appreciate that the various steps of any method disclosed herein may be adjusted, modified, rearranged, omitted, and / or expanded without departing from the scope of the present disclosure.

[0207] At 1402, method 1400 may include generating a navigation instruction query associated with the utility of at least one navigation instruction. The navigation instruction query may include one or more visual indications (e.g., visual indications generated on a display device associated with computing device 102) and / or one or more audible indications (e.g., one or more audible indications generated via an audio output device associated with computing device 102). For example, after generating at least one navigation instruction (e.g., "Turn right at the big white obelisk") (e.g., five seconds thereafter), computing device 102 may generate a navigation instruction query via a speaker device associated with computing device 102. The navigation instruction query may say "Do you see the landmark?" or "Do you see the big white obelisk?" In addition, the navigation instruction query may, for example, inquire whether the at least one navigation instruction was helpful for navigation (e.g., "Was this navigation instruction helpful?").

[0208] In some embodiments, a navigation instruction query may be generated based at least in part on a distance to at least one landmark referenced in at least one navigation instruction. For example, computing device 102 may be associated with one or more sensors that may be used to determine a distance to at least one landmark, and a navigation instruction query may be generated when a vantage point is within a predetermined distance of at least one landmark. For another example, computing device 102 may use the distance to at least one landmark and the current speed at the vantage point to determine the amount of time before a landmark is within a predetermined distance. Computing device 102 may then generate a navigation instruction query within a predetermined time period of reaching the location of at least one landmark.

[0209] At 1404, method 1400 may include receiving one or more responses to the navigation instruction query. The one or more responses to the navigation instruction query may include one or more signals or data received from one or more devices including a tactile input device (e.g., a keyboard, a touch screen, a pressure-sensitive surface, and / or a button), a microphone, and / or a camera. For example, computing device 102 may be associated with a microphone that receives one or more verbal responses from a passenger (e.g., a passenger in a car) who has been provided with the navigation instruction query "Did you see the landmark?" The response from the passenger may include, for example, a verbal response of "yes" or "no" to indicate whether the passenger saw the landmark.

[0210] In some embodiments, one or more responses may be stored in a highly encrypted and anonymized privacy-enhancing manner so that, for example, aggregation of one or more responses may be used to improve the utility of selecting landmarks and providing navigation instructions.

[0211] At 1406, method 1400 may include adjusting one or more entropy criteria based at least in part on one or more responses to the navigation instruction query. For example, when one or more responses do not confirm the utility of at least one navigation instruction, one or more entropy criteria may be adjusted by changing one or more entropy criteria. For example, when one or more entropy criteria may include a minimum altitude of ten meters. When one or more responses indicate that navigation instructions referencing an eleven-meter-high landmark are not useful (e.g., the landmark does not assist the user in navigating to his intended destination), the minimum altitude may be increased to fifteen meters.

[0212] As another example, one or more responses indicating the degree of effectiveness of navigation instructions from a particular location (e.g., a location associated with a vantage point) can be used to adjust one or more entropy criteria in order to adjust a particular entropy criterion (e.g., the brightness of a landmark) based on the one or more responses (e.g., at night, when a user receiving navigation instructions referenced a landmark does not see the landmark, increasing the minimum brightness of the landmark).

[0213] The techniques discussed herein refer to servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from such systems. The inherent flexibility of computer-based systems allows for a variety of possible configurations, combinations, and division of tasks and functions between components. For example, the processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed on multiple systems. Distributed components can operate sequentially or in parallel.

[0214] Although the subject matter has been described in detail with respect to various specific example embodiments of the present disclosure, each example is provided by way of explanation, rather than limitation of the present disclosure. Those skilled in the art can easily produce changes, variations, and equivalents of such embodiments after understanding the foregoing. Thus, the subject disclosure does not exclude such modifications, variations, and / or additions including those made to the subject matter, as will be apparent to those of ordinary skill in the art. For example, a feature shown or described as part of one embodiment can be used together with another embodiment to produce yet another embodiment. Therefore, the present disclosure is intended to cover such changes, variations, and equivalents.

Claims

1. A computer-implemented method for providing navigation instructions with reference to a landmark, the method comprising: receiving, by a computing system including one or more processors, a plurality of images captured by a computing device, wherein the computing device is located inside a vantage point, the vantage point being a vehicle traveling on a road along a path corresponding to a direction of travel; accessing, by the computing system, a plurality of semantic tags associated with the plurality of images, wherein each semantic tag of the plurality of semantic tags is associated with one or more features depicted by one of the plurality of images, and wherein each of the one or more features is associated with a geographic location; identifying, by the computing system, one or more landmarks including one or more features that satisfy one or more entropy criteria based at least in part on the plurality of semantic tags, wherein the entropy criteria measure a localized prominence of each of the one or more features, wherein the one or more entropy criteria include how long each of the one or more features is present at the geographic location for more than a predetermined threshold duration; Selecting, by the computing system, at least one landmark for navigation at the location based at least in part on contextual data associated with the location on a path that includes a plurality of locations, wherein selecting, by the computing system, at least one landmark for navigation at the location based at least in part on contextual data associated with the location on a path that includes a plurality of locations comprises: determining, by the computing system, a reaction time based at least in part on a speed associated with the vehicle at the location and a distance to a nearest landmark of the one or more landmarks, wherein the reaction time is determined based on an amount of time that would take to reach the nearest landmark at the speed; and determining, by the computing system, at least one landmark that satisfies one or more reaction time criteria associated with a minimum reaction time, wherein the at least one landmark is selected from landmarks that will be visible after a current time for a time that exceeds the minimum reaction time; generating, by the computing system, at least one navigation instruction that references the at least one landmark; and The at least one navigation instruction is provided to a driver of the vehicle.

2. The computer-implemented method of claim 1 , wherein identifying, by the computing system, one or more landmarks comprising one or more features satisfying one or more entropy criteria based at least in part on a plurality of semantic tags, wherein the entropy criteria measures a localized prominence of each of the one or more features, comprises: determining, by the computing system, a rate at which each of the one or more features occurs within a predetermined area based at least in part on the plurality of semantic tags; as well as One or more landmarks are determined by the computing system to include one or more features that occur least frequently or at a rate below a threshold rate.

3. The computer-implemented method of claim 1 or claim 2, wherein identifying, by the computing system, one or more landmarks comprising one or more features satisfying one or more entropy criteria based at least in part on a plurality of semantic tags, wherein the entropy criteria measures a localized prominence of each of the one or more features, comprises: determining, by the computing system, a confidence score for each respective one of the one or more features based at least in part on a number of times each of the one or more features has been associated with a semantic tag in a plurality of semantic tags; as well as One or more features having confidence scores that satisfy one or more confidence score criteria are identified as landmarks by the computing system.

4. A computer-implemented method as described in claim 3, wherein the confidence score is based at least in part on the number of different viewing angles from which each of the one or more features associated with the semantic tag has been viewed, or the recency with which the one or more features have been associated with the semantic tag.

5. The computer-implemented method of claim 1 , wherein identifying, by the computing system, one or more landmarks comprising one or more features satisfying one or more entropy criteria based at least in part on a plurality of semantic tags, wherein the entropy criteria measures a localized prominence of each of the one or more features, comprises: One or more clusters of one or more features that satisfy one or more entropy criteria are determined, by the computing system, wherein each of the one or more clusters includes one or more features having a common semantic type.

6. The computer-implemented method of claim 1 , wherein selecting, by the computing system, at least one landmark for navigation at a location on a path including a plurality of locations based at least in part on contextual data associated with the location comprises: Visibility of each of the one or more landmarks from a vantage point associated with the location is determined by the computing system.

7. A computer-implemented method as described in claim 6, wherein the visibility is based at least in part on a distance at which each of the one or more landmarks is visible from the vantage point, an amount of light projected on each of the one or more landmarks, any obstructions between the vantage point and the one or more landmarks, or a physical size of each of the one or more landmarks.

8. The computer-implemented method of claim 6 or claim 7, wherein determining, by the computing system, visibility of each of the one or more landmarks from a vantage point associated with the location comprises: determining, by the computing system, a direction of travel along a path at the vantage point; as well as One or more landmarks facing the direction of travel are determined by the computing system.

9. The computer-implemented method of claim 6, wherein determining, by the computing system, visibility of each of the one or more landmarks from a vantage point associated with the location comprises: The visibility is determined by the computing system based at least in part on a transportation mode associated with the vantage point.

10. The computer-implemented method of claim 1 , wherein selecting, by the computing system, at least one landmark for navigation at a location on a path including a plurality of locations based at least in part on contextual data associated with the location comprises: At least one landmark is selected by the computing system based at least in part on a familiarity with the one or more landmarks, wherein the familiarity is based at least in part on a previous association of the user with the location or one or more features of the one or more landmarks.

11. The computer-implemented method of claim 10, wherein generating, by the computing system, at least one navigation instruction that references at least one landmark comprises: At least one navigation instruction is adjusted, by the computing system, based at least in part on the familiarity level.

12. The computer-implemented method of claim 1 , further comprising: generating, by the computing system, a navigation instruction utility query associated with a utility of at least one navigation instruction; receiving, by the computing system, one or more responses to the navigation instruction utility query; as well as One or more entropy criteria are adjusted by the computing system based at least in part on one or more responses to the navigation instruction utility query.

13. One or more tangible, non-transitory computer-readable media storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: receiving a plurality of images captured by a computing device, wherein the computing device is located inside a vantage point, the vantage point being a vehicle traveling on a road along a path corresponding to a direction of travel; accessing a plurality of semantic tags associated with the plurality of images, wherein each semantic tag of the plurality of semantic tags is associated with one or more features depicted by one of the plurality of images, and wherein each of the one or more features is associated with a geographic location; identifying one or more landmarks including one or more features that satisfy one or more entropy criteria based at least in part on the plurality of semantic tags, wherein the entropy criteria measure a localized prominence of each of the one or more features, wherein the one or more entropy criteria include how long each of the one or more features is present at the geographic location for more than a predetermined threshold duration; Selecting at least one landmark for navigation at the location based at least in part on contextual data associated with the location on a path that includes a plurality of locations, wherein selecting at least one landmark for navigation at the location based at least in part on contextual data associated with the location on a path that includes a plurality of locations comprises: determining a reaction time based at least in part on a speed associated with the vehicle at the location and a distance to a nearest landmark of the one or more landmarks, wherein the reaction time is determined based on an amount of time that would take to reach the nearest landmark at the speed; and selecting at least one landmark that satisfies one or more reaction time criteria associated with a minimum reaction time, wherein the at least one landmark is selected from landmarks that will be visible after a current time for a time that exceeds the minimum reaction time; generating at least one navigation instruction that references the at least one landmark; and The at least one navigation instruction is provided to a driver of the vehicle.

14. One or more tangible, non-transitory computer-readable media as described in claim 13, wherein each of the one or more features is associated with a time of day, a season, visual constancy, or one or more locations from which each of the one or more features is visible.

15. One or more tangible, non-transitory computer-readable media as described in claim 13 or claim 14, wherein the one or more entropy criteria include the frequency of occurrence of each of the one or more features within a predetermined area not exceeding a predetermined threshold frequency, the time persistence of each of the one or more features at a location exceeding a predetermined threshold duration, or the size of each of the one or more features exceeding a threshold size.

16. A computing system comprising: one or more processors; One or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: receiving a plurality of images captured by a computing device, wherein the computing device is located inside a vantage point, the vantage point being a vehicle traveling on a road along a path corresponding to a direction of travel; accessing a plurality of semantic tags associated with the plurality of images, wherein each semantic tag of the plurality of semantic tags is associated with one or more features depicted by one of the plurality of images, and wherein each of the one or more features is associated with a geographic location; identifying one or more landmarks including one or more features that satisfy one or more entropy criteria based at least in part on the plurality of semantic tags, wherein the entropy criteria measure a localized prominence of each of the one or more features, wherein the one or more entropy criteria include how long each of the one or more features is present at the geographic location for more than a predetermined threshold duration; Selecting at least one landmark for navigation at the location based at least in part on contextual data associated with the location on a path that includes a plurality of locations, wherein selecting at least one landmark for navigation at the location based at least in part on contextual data associated with the location on a path that includes a plurality of locations comprises: determining a reaction time based at least in part on a speed associated with the vehicle at the location and a distance to a nearest landmark of the one or more landmarks, wherein the reaction time is determined based on an amount of time that would take to reach the nearest landmark at the speed; and selecting at least one landmark that satisfies one or more reaction time criteria associated with a minimum reaction time, wherein the at least one landmark is selected from landmarks that will be visible after a current time for a time that exceeds the minimum reaction time; generating at least one navigation instruction that references the at least one landmark; and The at least one navigation instruction is provided to a driver of the vehicle.

17. A computing system as described in claim 16, wherein at least one navigation instruction includes at least one visual indication associated with the relative position of the at least one landmark on the path relative to the vehicle, or at least one audible indication associated with the relative position of the at least one landmark on the path relative to the vehicle.

Citation Information

Patent Citations

  • Navigation system and navigation system voice prompt method

    CN101339044A

  • Natural language image search

    CN106255968A

  • Sparse map for autonomous vehicle navigation

    CN107438754A

  • Contextually disambiguating queries

    CN108628919A

  • Landmark display method, navigation device, on-vehicle equipment, and navigation system

    JP2008070557A