Unified destination input method based on tiles

Abstract

The invention relates to a tile-based unified destination input method. In an infotainment system of a vehicle, a mass storage unit stores map data and environmental data. The map data includes geographic information about a geographic area around the vehicle, and the environmental data includes environmental information about a location of the geographic area for use in voice interaction with a vehicle occupant. The environmental data includes subsets of environmental data, each subset associated with a different geographic area. A memory manager copies a subset of these environmental data into dynamic memory based on motion of the vehicle.

Description

Technical Field

[0001] The present invention relates to an electronic device for providing location related information. Background Art

[0002] The mapping system receives location information from the GPS and displays it on a dynamically changing map on the vehicle screen. As your location changes, the map changes too.

[0003] The mapping system's ability to draw on-screen graphics stems from its access to an integral data structure that stores "map data." This data structure is typically very large. Therefore, typically only the portion of the map data that the user might be interested in is loaded into memory.

[0004] In many cases, users wish to interact with map data (e.g., by specifying a location or by asking questions about what is displayed on the map). For users on foot, this is often accomplished by typing. In situations where the user is driving, it is most convenient to use a voice interface.

[0005] Map data includes only a limited amount of information that a voice interface can utilize to support such interaction. Therefore, it is useful to provide additional data rendered by a voice interface when interacting with a user who is also interacting with map data. Summary of the invention

[0006] The present invention addresses the use of context data that can be mapped by a voice interface to provide additional information about map data. Such context data is sometimes referred to as "Uniform Destination Entry Data" or "UDE data". The present invention is applicable to digital assistants, such as automotive assistants running in vehicles or mobile assistants executing on mobile devices.

[0007] It has been found that the user's location when performing a voice interaction provides context for how the speech is interpreted. This is particularly true when the speech relates to features in map data. According to the present invention, the user's location when asking a question provides context for disambiguating speech.

[0008] For example, if a user asks, "Tell me about the Statue of Liberty," the answer will depend on whether the user is parked near New York Harbor or near the Luxembourg Gardens in Paris, where a smaller version of the statue can be found. Thus, the voice interface's response will be informed by the location where the question is asked.

[0009] In some cases, location affects the pronunciation of local names. For example, the name of the Thames River is pronounced one way in London and another way in New London, Connecticut. In this case, the voice interface's response will have the pronunciation appropriate to the location.

[0010] It is also found that in most cases, the probability that a user query involving geographic features involves features within about 500 kilometers from their current location decreases with the distance from that location.

[0011] Therefore, the present invention contemplates dividing the environmental data into units or "tiles" that are loaded into memory as needed. In this case, the assistant loads a tile containing the user's location (hereinafter referred to as a "local tile") and a set of "neighbor tiles". Each neighbor tile has at least one point that is closer to the user's location than a specified threshold. These tiles are dynamically loaded at runtime to provide the most likely required environmental data based on the user's location.

[0012] For each tile, there is a set of addresses and points of interest. Using these sets of addresses and points of interest, an information module used by the voice interface can be created. This information includes finite state sensor environment information, a language model for the automatic speech recognition unit, a set of place names used by the natural language understanding unit, and database content for navigation or other services, including reasoning services. The above information is associated with the addresses and points of interest of the area covered by the relevant tile.

[0013] In operation, the user's known location is used to identify those tiles that have at least one point within a certain distance from the user's location and dynamically load those tiles into memory at runtime. The distance is set by the user, the maximum memory available to load tiles, or the number of tiles required to cover a specified area of ​​interest.

[0014] In some embodiments, there are tiles that cover an area farther from the user than the first loading distance. Such tiles are referred to herein as "remote tiles". Unless these remote tiles are also loaded, the voice interface will not be able to use the information contained therein. However, because the area to be covered increases with the square of the distance from the user, loading these remote tiles at runtime will require considerable memory.

[0015] The present invention exploits these characteristics by decomposing a remote tile into multiple "overlapping tiles" that include some but not all of the information of the original remote tile. This is equivalent to decomposing the original remote tile into multiple components, where selected components can be reassembled to provide different amounts of information and require different amounts of memory.

[0016] Tiles and overlapping tiles provide a way to inject information from regions of interest based on its relevance, where relevance comes from the user's location, environmental information, user history, and user profile. This flexibility allows digital assistants to adapt to the footprint constraints present on embedded platforms and allows for optimized use of information relevant to the task at hand.

[0017] In one aspect, the present invention features an apparatus for optimizing memory usage. The apparatus includes a digital assistant, a voice interface, a dynamic memory, a memory manager, and a mass storage unit storing map data and environmental data. The map data includes geographic information about geographic areas near a user. The geographic areas include first and second geographic areas. The environmental data includes location-related environmental information about the geographic areas around the user for use by the voice interface when voice interacting with the user. The environmental data includes a first subset of environmental data and a second subset of environmental data. The dynamic memory stores a copy of the first subset of environmental data. The memory manager replaces the copy of the first subset of environmental data with a copy of information from the second subset of environmental data to respond to the user's speed.

[0018] Embodiments include embodiments where the user is in a vehicle and the digital assistant is a car assistant running in the vehicle's infotainment system. In such embodiments, the apparatus attempts to optimize memory usage in the vehicle's infotainment system.

[0019] Embodiments include: embodiments in which the first and second subsets of environmental data contain information related to the first and second geographic areas, respectively, embodiments in which the first and second subsets of environmental data contain information related to the first geographic area and the first and second subsets contain different amounts of information, embodiments in which the second subset of environmental data omits information in the first subset of environmental data, and embodiments in which the first subset of environmental data omits information about the first geographic area in the second subset of environmental data.

[0020] In some embodiments, the mass storage unit stores first and second tiles comprising first and second subsets of environmental data, respectively. In such embodiments, the first tile has environmental information about a first geographic area and the second tile has environmental information about a second geographic area.

[0021] In some embodiments, the apparatus further comprises an overlap operator. In these embodiments, the overlap operator receives an environment tile comprising a first subset of the environment data from the mass storage unit and returns a plurality of overlap tiles, each overlap tile comprising a respective proper subset of the first subset of the environment data.

[0022] In the embodiment that comprises overlapping operator, overlapping operator receives the environment tile that comprises first subset environment data from mass storage unit.And two or more overlapping tiles that are constructed by environment tile.Each of these overlapping tiles comprises some information from environment tile.In such embodiment, the intersection of overlapping tiles in information space produces empty set.Therefore, these overlapping tiles can not overlap.

[0023] Other embodiments including the overlap operator include such embodiments, wherein the overlap operator receives the environment tile and the first and second overlap tiles derived from the environment tile from a large capacity storage unit. In such embodiments, the environment tile includes first information and second information. The first information is information about a first feature in a first geographic area, and the second information is information about a second feature in the first geographic area. These first features have a popularity score within a first popularity score range, and the second feature has a popularity score within a second popularity score range. The first information is included in the first overlap tile, and the second information is omitted from the first overlap tile. On the contrary, the second information is included in the second overlap tile, and the first information is omitted from the second overlap tile.

[0024] Other embodiments that include an overlap operator are embodiments in which the overlap operator receives environment tiles and overlapping tiles from a large storage unit. Overlapping tiles that include first and second overlapping tiles correspond to different overlapping layers. In such an embodiment, the environment tile includes first information about a first feature and second information about a second feature, the first feature being in a first management area, and the second feature being in a second management area. The first overlapping tile includes the first information and omits the second information. On the contrary, the second overlapping tile includes the second information, but omits the first information.

[0025] In other embodiments, the mass storage unit stores overlapping tiles, and these overlapping tiles comprise all information in the second subset of environmental data jointly.Each of these overlapping tiles has less than all information in the second subset of environmental data.The copy from the information of the second subset is made up of all overlapping tiles that are less than the second subset.In these embodiments, the overlapping operator loads some but not all overlapping tiles.This allows to build the superposition of some but not all overlapping tiles.

[0026] In other embodiments, the mass storage unit stores a plurality of overlapping tiles corresponding to a first subset of the environmental data. Each overlapping tile comprises a true subset of the first subset of the environmental data. In these embodiments, the memory manager loads into the dynamic memory tiles that are less than all overlapping tiles. This can result in a superposition of overlapping tiles, wherein the number of superimposed tiles depends on the distance between the user and the first area. Likewise, in these embodiments, the memory manager loads into the dynamic memory a superposition of less than all overlapping tiles, wherein the number of tiles in the superposition increases as the distance between the user and the first area decreases and / or decreases as the distance decreases, for example as a result of a vehicle carrying the user moving toward or away from the first area, or as a result of a user walking toward or away from the first area.

[0027] In another aspect, the invention features a method that includes causing a digital assistant to run in a mobile device, causing a voice interface to execute in the device and communicate with the digital assistant, storing map data and environmental data in a mass storage unit, storing a copy of a first subset of the environmental data in a dynamic memory, and optimizing memory usage in an infotainment system. The act of optimizing memory usage includes responding to information about a speed of the device so that a memory manager installed therein replaces a copy of the first subset of environmental data with a copy of information from a second subset of environmental data, the second subset having been stored in the mass storage unit. The map data includes geographic information about a geographic area surrounding the device. The geographic areas include first and second geographic areas. The environmental data includes location-related environmental information about the surrounding geographic area for use by the digital assistant and the voice interface in voice interaction with a user of the device.

[0028] The method is practiced including the user being in a vehicle and the digital assistant being a car assistant running in an infotainment system of the vehicle. The practice includes attempting to optimize memory usage in the infotainment system of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] These and other features of the present invention will become apparent from the following detailed description and accompanying drawings, in which:

[0030] Figure 1 shows overlapping operators in a vehicle infotainment system; and

[0031] Figure 2 Showing the different kinds of tiles on the map. DETAILED DESCRIPTION

[0032] Figure 1 A vehicle 10 is shown having an infotainment system 12 in which a voice interface 14 receives voice input via a microphone 16 and provides voice output via a speaker 18. The voice interface 14 has an automatic speech recognition system 20 and a natural language understanding unit 22.

[0033] When the vehicle 10 is moving, it is not uncommon for occupants to ask questions, particularly questions about the vehicle environment, using the microphone 16. The voice interface 14 provides verbal responses to these questions through the speaker 18.

[0034] As the vehicle 10 moves, the local characteristics of the questions that may be asked will continually change. Therefore, the information required to answer these questions will also change.

[0035] The car assistant 24 running in the infotainment system 12 answers questions from the vehicle occupants. In the process, it may need information about local features. Such information is stored in the dynamic memory 26.

[0036] The dynamic memory 26 is a limited resource. Therefore, it is best to avoid storing more information in it than is needed. Instead, the dynamic memory 26 obtains a subset of information from the mass storage unit 28 as needed. There are two types of information in the mass storage unit 28: map data 30 and environmental data 32.

[0037] The map data 30 shows geographical information about a geographical area. The map data 30 is divided into map tiles 34 showing geographical information of a limited geographical area. The geographical information of the area includes information such as street names and locations of named entities within the geographical area.

[0038] The environmental data 32 contains location-specific information used by the voice interface 14 and the automotive assistant 24 to interact with the user of the vehicle 10. Such information may include facts that are not present in the map data 30 or that are not accessible to the voice interface 14. Examples include historical information, dimensions (e.g., the height of a building or bridge body), hours of operation, pronunciation of place names, entry restrictions (e.g., restrictions on entering a gated community), etc. Like the information in the map data 30, this information is location-specific.

[0039] Like the map data 30, the environment data 32 is divided into environment tiles 36. Each environment tile 36 includes environment information for a limited geographical area. In some embodiments, it is convenient to have one environment tile 36 for each map tile 34.

[0040] The vehicle 10 includes a GPS 38 that provides the real-time location of the vehicle to the car assistant 24. This enables the car assistant 24 to use the location of the vehicle as context to answer questions posed by the user via the voice interface 14. This enables the car assistant to answer the same question in different ways depending on the location of the vehicle.

[0041] The GPS 38 also provides the vehicle position in real time to the memory manager 40. The memory manager 40 replaces the map tiles 34 and environment tiles 36 in the dynamic memory 26 with new map tiles 34 and new environment tiles 36 from the mass storage unit 28 in response to changes in the vehicle's position. The map tiles 34 and environment tiles 36 contain appropriate subsets of the map data 30 and environment data 32, respectively.

[0042] Figure 2 Three environment tiles 36 are shown: a local tile 42 , a neighbor tile 44 , and a remote tile 46 .

[0043] The local tile 42 contains the location of the vehicle. The neighbor tile 44 contains at least one point within a certain distance from the vehicle's location. Given the entity, the local tile 42 is also a neighbor tile 44. The remote tile 46 is an environment tile 36 that is not a neighbor tile 44.

[0044] These different types of environment tiles 36 are identical in structure. These terms are only used in relation to the vehicle's location. Thus, as the vehicle approaches a remote tile 46, it will eventually become a neighbor tile 44, and then eventually become a local tile 42 when the vehicle arrives.

[0045] However, environment tiles 36 are different from map tiles 34. In many cases, environment tiles 36 contain a considerable amount of information, in many cases much more than map tiles 34 that overlap similar areas. Therefore, environment tiles 36 consume a considerable amount of dynamic memory 26.

[0046] It has been found that the likelihood of needing information from a particular environment tile 36 decreases as the environment tile is farther from the local tile 42. To avoid consuming an excessive amount of dynamic memory 26 while still providing at least some information about areas far from the vehicle 10, some embodiments include an overlap operator 48, such as Figure 1 shown.

[0047] The overlay operator 48 receives the environment tile 36 from the mass storage unit 28. It also receives two or more overlay tiles 50 that are already stored in the mass storage unit 28. To avoid delays and save local computing resources, the environment tile 36 and overlay tiles 50 are typically created first on a tile system remote from the vehicle and then stored in the mass storage unit 28.

[0048] Therefore, each environment tile 36 produces a set of overlapping tiles 50. Each overlapping tile 50 contains some but not all information in the environment tile 36 from which it originates. In addition, overlapping tiles 50 contain information different from each other. Therefore, overlapping tiles 50 are orthogonal to each other in information space. However, since all overlapping tiles 50 correspond to the same environment tile 36, they overlap each other to cover the same geographic area. Therefore, overlapping tiles 50 overlap each other in physical space, but are orthogonal to each other in information space.

[0049] The memory manager 40 creates an overlay of overlapping tiles 50 using the vehicle's position. For environment tiles 36 near the vehicle, the memory manager 40 uses most or all of the overlapping tiles 50 and thus imparts considerable information to the environment tiles 36. On the other hand, for environment tiles 36 that are farther from the vehicle's position, the memory manager 40 forms an overlay with fewer overlapping tiles 50. For those environment tiles 36 that are farthest from the vehicle, i.e., for the remote tiles 46, the overlay consists of two overlapping tiles 50, one of which is a zero overlap tile. It is apparent from basic set theory that the overlay of overlapping tiles 50 with zero overlap tiles is structurally and functionally equivalent to using only one overlapping tile 50, and that zero overlap tiles are introduced primarily to allow consistency in the use of the term "overlay."

[0050] The information content in the resulting overlay of the overlapped tile 50 consists of a subset of the information in the original environment tile 36 from which the overlapped tile 50 was derived. Since the overlay is smaller than the size of the environment tile 36 to begin with, this reduces consumption of the dynamic memory 26 while retaining at least some information about areas away from the vehicle 10.

[0051] The memory manager 40 builds overlays of varying "densities." The "density" of an overlay depends on how many overlay tiles 50 are used in its construction. In the extreme case, when the overlay uses all available overlay tiles 50, the result is identical to the original environment tile 36 that produced the overlay tile 50.

[0052] In some embodiments, the memory manager 40 constructs an overlay whose density decreases with increasing distance from the local tile 42. This is advantageous because it has been found that the likelihood that a user will need information about an area decreases as the distance of the area from the user's vehicle 10 increases.

[0053] In some embodiments, the metric used to measure distance is derived from the latitude and longitude of the vehicle 10 and some reference point on the environment tile 36. For the case where the latitude and longitude of the vehicle position and the reference point differ by Δφ and Δλ, respectively, and the relevant latitudes are φ1 and φ2, respectively, a suitable formula is that the distance between them is the product of twice the inverse tangent of the radius of the earth and the square root of (1-a) / a, where "a" is given by is given. For cases where the angle difference is not too large, a simpler formula can be used, relying on the small angle approximation to eliminate the sine function. In other cases, the distance is the Manhattan distance.

[0054] The construction of overlay tiles 50 mainly depends on three kinds of information: management information 52 , popularity information 54 , and overlay layers 56 .

[0055] Administrative information 52 appears because the addresses on the map are organized into a hierarchy of administrative levels. Examples of these administrative levels are "state," "province," "region," or "state" at a high level, and "county," "city," "district," or "street" at a more local level. Although the names of these administrative levels change from one location to another, the end result is a hierarchical tree of administrative levels.

[0056] Popularity information 54 comes from popularity scores for various points of interest. These popularity scores, obtained by collecting data about user activity and making inferences based on that activity, indicate the importance of a point of interest or another location.

[0057] Each overlapping tile 50 corresponds to an overlapping layer 56. Therefore, the number of overlapping layers 56 defines the dimensionality of the information space.

[0058] Each overlay layer 56 has an associated management layer from management information 52 and an associated popularity range from popularity information 54. The information in overlay tiles 50 corresponding to a particular overlay layer 56 is controlled by the popularity information 54 and management information 52 assigned to that overlay layer 56.

[0059] In operation, the overlap operator 48 receives a set of overlap layers 56 and environment tiles 36. For each overlap layer 56, the overlap operator 48 constructs an overlap tile 50. This is achieved by removing all information from the environment tile 36 except for the information assigned to the overlap layer 56 corresponding to the overlap tile 50 in question. This results in as many overlap tiles 50 as overlap layers 56, each overlap tile 50 containing only the information assigned to that overlap layer 56. The result is an effective decomposition of the original environment tile 36 into components that are orthogonal in information space, each component being represented by an overlap tile 50. In other words, a set of overlap tiles 50 is given, wherein a first and a second overlap tile 50 from the set comprises a first and a second information, respectively, and the intersection of the first and the second information is the empty set.

[0060] Overlay tiles 50 of a particular overlay layer 56 include only address information for a particular administrative layer, and only information about points of interest having popularity scores within a particular popularity score range.

[0061] The ability to selectively combine different overlay tiles 50 means that, as the distance between an environment tile 36 and the vehicle 10 increases, an overlay of selected overlay tiles 50 of that environment tile can be used in place of the environment tile 36 to obtain the correct amount and type of information required. Because there are as many overlays as the factorial of the number of overlay layers 56, great flexibility can be had in constructing overlays of overlay tiles 56 to use as replacements for environment tiles 36.

[0062] Furthermore, different distances from the vehicle 10 may be defined and an overlay layer 56 may be assigned to each distance, thereby gradually reducing the amount of information in the remote tile 46 depending on the distance of the environment tile 36 from the vehicle 10 .

[0063] The present invention is described herein as a situation where the user's actions occur in a vehicle (e.g., an automobile). However, the subject matter described herein is agnostic to the user's locomotion source. Thus, it is equally useful for a digital assistant executed on a portable mobile device carried by a user whose locomotion source is not a motor vehicle.

[0064] Having described the invention and its preferred embodiments, what is claimed as new by Letters Patent is referred to the claims.

Claims

1. An apparatus for optimizing memory usage in a vehicle infotainment system, the apparatus comprising: a car assistant, operating in said infotainment system of said vehicle; a voice interface, executed in the infotainment system, the voice interface communicating with the automotive assistant; A large-capacity storage unit for storing map data and environmental data; wherein the map data includes geographic information about a geographic area around the vehicle, the geographic area including a first geographic area and a second geographic area; wherein the environmental data includes location-related environmental information about the geographic area surrounding the vehicle for use by the automotive assistant and the voice interface in voice interaction with a passenger of the vehicle; Wherein, the environmental data includes a first subset of the environmental data and a second subset of the environmental data; a dynamic memory storing a copy of said first subset of said environmental data; and A memory manager, responsive to a velocity of the vehicle, replaces a copy of the first subset of the environmental data with a copy of information from the second subset of the environmental data.

2. The device according to claim 1, wherein: The first and second subsets of the environmental data contain information related to the first and second geographic areas, respectively.

3. The device according to claim 1, wherein: The first and second subsets of the environmental data contain information related to the first geographic area, and the first and second subsets have different amounts of the information.

4. The device according to claim 1, wherein: The second subset of the environmental data omits information in the first subset of the environmental data.

5. The device according to claim 1, wherein: The first subset of the environmental data omits information about the first geographic area within the second subset of the environmental data.

6. The device according to claim 1, wherein: The first tile and the second tile stored in the mass storage unit include the first subset and the second subset of the environmental data, respectively, whereby the first tile has environmental information about the first geographical area and the second tile has environmental information about the second geographical area.

7. The apparatus of claim 1 further comprising an overlap operator that receives an environment tile comprising the first subset of the environment data from the mass storage unit and returns a plurality of overlap tiles, each overlap tile comprising a corresponding proper subset of the first subset of the environment data.

8. The apparatus of claim 1 , further comprising an overlay operator that receives from the mass storage unit an environment tile comprising the first subset of the environment data and a plurality of overlay tiles constructed from the environment tile, each of the overlay tiles comprising some information from the environment tile, wherein The intersection of the overlapping tiles in information space produces an empty set.

9. The apparatus of claim 1, further comprising an overlay operator that receives an environment tile, a first overlay tile, and a second overlay tile from the mass storage unit, wherein: The environment tile includes first information and second information, wherein the first information is information about a first feature within the first geographic area, wherein the second information is information about a second feature within the first geographic area, wherein the first feature has a popularity score within a first range of popularity scores, wherein the second feature has a popularity score within a second range of popularity scores, wherein the first overlapping tile includes the first information and omits the second information, and wherein the second overlapping tile includes the second information and omits the first information.

10. The apparatus according to claim 1, further comprising an overlay operator that receives an environment tile unit and overlay tiles from the large capacity storage unit, the overlay tiles corresponding to different overlay layers and comprising a first overlay tile and a second overlay tile, wherein The environment tile includes first information about a first feature and second information about a second feature, the first feature is within a first management area, and the second feature is within a second management area, wherein the first overlapping tile includes the first information, the second overlapping tile includes the second information, the first overlapping tile omits the second information, and the second overlapping tile omits the first information.

11. The device according to claim 1, wherein: The mass storage unit stores a plurality of overlapping tiles that collectively contain all of the information within the second subset of the environmental data, wherein each of the overlapping tiles has less than all of the information within the second subset of the environmental data, and wherein the copy of the information from the second subset consists of less than all of the overlapping tiles of the second subset.

12. The device according to claim 1, wherein: The mass storage unit stores a plurality of overlapping tiles corresponding to the first subset of the environmental data, wherein each of the overlapping tiles comprises a true subset of the first subset of the environmental data, wherein the memory manager is further configured to load an overlap of less than all of the overlapping tiles into the dynamic memory, wherein the number of tiles in the overlap depends on a distance between the vehicle and the first area.

13. The apparatus according to claim 1, wherein: The mass storage unit stores a plurality of overlapping tiles corresponding to the first subset of the environmental data, wherein each of the overlapping tiles comprises a true subset of the first subset of the environmental data, wherein the memory manager is further configured to load into the dynamic memory a stack of less than all of the overlapping tiles, wherein the number of tiles in the stack increases as the distance between the vehicle and the first area decreases.

14. A method comprising: Enables the car assistant to run in the vehicle's infotainment system; enabling a voice interface to operate in the infotainment system and communicate with the automotive assistant; storing map data and environmental data in a mass storage unit; storing a copy of the first subset of the environmental data in a dynamic memory; as well as optimizing memory usage in the infotainment system; wherein optimizing the memory usage comprises: in response to information about the speed of the vehicle, causing a memory manager installed in the vehicle to replace a copy of a first subset of the environmental data with a copy of information from a second subset of the environmental data, the second subset already stored in the mass storage unit; wherein the map data includes geographic information about a geographic area surrounding the vehicle; and The environmental data includes location-related environmental information about the geographic area for use by the car assistant and the voice interface when performing voice interaction with passengers of the vehicle.

Images