Automated assistant suggestions for third-party vehicle computing devices with restricted architecture

Abstract

The invention relates to automated assistant suggestions for a third-party vehicle computing device with a restricted architecture. An automated assistant provides suggestion elements at an OEM application and / or a third party application accessible via an OEM vehicle computing device that employs restrictions on communications between native applications. The automated assistant may render the suggestions via the OEM vehicle application and / or the third-party application and initialize actions for execution via the OEM vehicle application and / or the third-party application despite the limitation. The suggestions may be rendered at a graphical user interface of the vehicle and include content that, when spoken by the user, causes the automated assistant to initialize such actions. The action suggested by the automated assistant may be performed by a third-party application that is different than the suggested third-party application being rendered. In this manner, although there is a difference in communication restrictions between vehicle applications, a common automated assistant may be available without difference via various vehicle computing devices.

Description

Description of the case This application is a divisional application of Chinese invention patent application No. 201980091586.8, with a filing date of May 9, 2019. Technical Field The present disclosure relates to automated assistant suggestions for third-party vehicle computing devices with constrained architecture. Background Art

[0001] Computing devices and their corresponding applications typically provide robust functionality to assist users with a variety of tasks. The provider of a particular application or device may not expect the user to adopt all of the functionality during initial use, but rather to learn the functionality of the application or device over time. When a particular device provides access to an automated assistant, the automated assistant can be used to familiarize the user with the various functions using suggestions that guide the user in controlling one or more actions of the particular device and / or application. However, in some instances, such as at a vehicle computing device or other OEM computing device, the automated assistant may be accessed through the vehicle computing device but is limited in the type of data that can be collected from the operation of the vehicle computing device. As a result, any suggestions made by the automated assistant may not be appropriately tailored to accommodate the vehicle computing device and / or functionality of other applications that may enable more effective and / or efficient use of the vehicle and / or vehicle computing device.

[0002] Furthermore, such limitations may prevent users from learning the safety features of the vehicle computing device more quickly. In other words, because the vehicle computing device may offer a plethora of safety features that the user cannot reasonably be expected to immediately acknowledge, any delay in learning these features may limit the safety and / or usefulness of the vehicle. Additionally, because the user may spend a significant amount of their time controlling the vehicle in order to navigate rather than learning the features, the user will not gain benefits such as reduced power consumption and increased efficiency. This may be particularly evident when the vehicle computing device is not specifically and proactively suggesting features to the user, as such benefits are often achieved when the user outsources various tasks to the computing device. Summary of the invention

[0003] The implementation described herein relates to a hierarchy of vehicle applications provided at a vehicle computing device, the vehicle computing device also including an automated assistant that can render suggestions via the vehicle application and initialize actions via the vehicle application. The vehicle application can be loaded onto the vehicle computing device of the vehicle and can provide various functions related to navigation, communication, media, and / or any other type of service that can be provided by the computing device. The vehicle computing device can operate according to an operating system loaded onto the vehicle computing device and including one or more original equipment manufacturer (OEM) applications. The OEM application can be provided by an entity or source that also provides the vehicle. In addition, the vehicle computing device can also include one or more vehicle applications provided by a third-party source (i.e., a third-party application rather than a native vehicle application), and an automated assistant provided by a source separated from the third-party source. Communication between the automated assistant and other applications can be limited to communication occurring via the vehicle computing device, which can improve the reliability of the vehicle system. The automated assistant can be away from the vehicle computing device. One or more actions suggested by the automated assistant can be performed by a local vehicle application (whether a local OEM application or a third-party application), which can improve the reliability of vehicle performance, especially in the case where the vehicle is expected to maintain full control of its system, such as in an autonomous vehicle environment. One or more actions may be performed by a remote automated assistant, which may protect user data and information associated with the automated assistant (assistant data), and thereby improve user privacy and data security by preventing local vehicle applications from accessing user data.

[0004] In some implementations, while a user is riding in a vehicle, an OEM vehicle application and / or an automated assistant may render a graphical user interface that includes a variety of different content based on data from one or more other vehicle applications. For example, the graphical user interface may include a first area for rendering navigation-related content, a second area for rendering communication-related content, and / or a third area for rendering automated assistant content. In order for the user to learn the various functions of the vehicle computing device and / or vehicle applications, the automated assistant may provide suggestions and / or cause suggestions to be rendered at each area of ​​the graphical user interface. As a result, although the automated assistant is accessible via a variety of different vehicles using data from one or more other vehicle applications provided at the vehicle computing device, the automated assistant will still be able to provide suggestions to the user regardless of the vehicle in which the automated assistant operates.

[0005] In some implementations, when the graphical user interface is displaying at least a first area and a second area, the OEM vehicle application can generate data representing what is provided at the user interface and provide such generated data to the automated assistant. As used herein, data can represent or indicate some or all of the content provided at the graphical user interface so as to provide a context that the automated assistant can use to generate suggestions. For example, when the first area is displaying navigation directions to a destination and the second area is displaying a notification about an incoming phone call, the OEM vehicle application can generate data representing (or indicating) the status of the incoming phone call and the status of the navigation to the destination. In some implementations, the automated assistant can use the data to generate one or more suggestions or actions to be performed by the automated assistant and / or any other application accessible via the vehicle computing device.

[0006] For example, an automated assistant may provide a suggestion element at a graphical user interface to invoke a messaging application to provide an estimated time of arrival to a person who is the source of an incoming call. The automated assistant may cause the suggestion element to be provided at a second area of ​​the graphical user interface because the suggestion is related to the communication. The suggestion element may include natural language content that characterizes (or indicates) any action that the assistant can take, such as "Send Megan my estimated time of arrival". The natural language content may be text provided at the suggestion element, may be an example utterance for invoking the automated assistant to perform an action associated with the suggestion element, and / or may otherwise characterize a spoken utterance for invoking the automated assistant. For example, to invoke the automated assistant so that a message is sent, a user may tap a display panel at a location where the suggestion element is being rendered, and / or provide a verbal input that includes at least a portion of the verbal utterance rendered at the graphical user interface (e.g., "Assistant, send a message to Meganindicating my ETA". In another example, the verbal input may have the same meaning or intent as the natural language content of the suggestion element.

[0007] In some implementations, the OEM vehicle application may provide various data to the automated assistant over time, at least in response to changes in content rendered at a display panel of the vehicle computing device. As the automated assistant receives subsequent data from the OEM vehicle application, the automated assistant may generate further suggestions, and / or rank the suggestions according to various different attributes. In some cases, suggestion elements provided by the automated assistant may be ranked according to the strength of the correlation between the assistant data and the vehicle application data provided by the OEM vehicle application. For example, vehicle application data characterizing a destination displayed at a display panel may have a stronger correlation with assistant data that also identifies the destination, and this "stronger" correlation may be relative to the correlation when the vehicle application data does not identify a destination that is also identified by the assistant data.

[0008] A set of suggestion elements can be generated for each area of ​​the graphical user interface that is associated with a particular type of application and / or activity. For each set of suggestion elements, a ranking can be assigned to each suggestion element in the set of suggestion elements. In this manner, when a suggestion element is to be rendered at a particular area of ​​the graphical user interface, the suggestion element selected to be rendered can be the highest ranked suggestion element from the set of suggestion elements corresponding to the particular area. In some implementations, the ranking of a particular suggestion element can be based on the content of the entire graphical user interface, previous interactions between the user and the automated assistant, third-party application data available to the automated assistant with permission from the user, vehicle data characterizing one or more operations of the vehicle, and / or any other information on which the ranking of the suggestions can be based.

[0009] In some implementations, the automated assistant may provide suggestions for rendering by third-party vehicle applications, and these suggestions may be generated based on information from the OEM vehicle application. For example, multiple different third-party applications may be accessible via the vehicle computing device and may communicate with the OEM vehicle application to send and / or receive data. In some implementations, the OEM vehicle application may restrict communication between other applications on the vehicle computing device. Thus, the third-party application may not be able to communicate with the automated assistant locally, but rather only via the vehicle computing device, and the automated assistant may therefore know that operations are performed by the third-party application using data provided by the OEM vehicle application rather than through direct communication with the third-party application.

[0010] For example, when a third-party application is rendering content at a display panel of a vehicle computing device, the OEM vehicle application can generate data representing the content and provide the data to the automated assistant. Alternatively or additionally, the third-party application can communicate with a remote computing device such as a server to provide data to the automated assistant (e.g., using an automated assistant application programming interface (API)). Suggestion data can then be generated via the automated assistant and then shared with the OEM vehicle application and / or a remote device associated with the third-party application. One or more suggestion elements can then be presented at a graphical user interface of the third-party application based on the suggestion data. In this way, although the OEM vehicle application limits communication between local applications to communication that occurs only via the vehicle computing device, the automated assistant can still provide the user with suggestions for various actions that the user may not be aware of, at least with respect to the vehicle computing device and / or the vehicle.

[0011] In some implementations, the automated assistant may bypass providing suggestions related to actions that have recently been performed at the user's direction via the third-party application and / or the OEM vehicle application. For example, when the third-party application corresponds to a vehicle maintenance application and the vehicle maintenance application provides a notification indicating that a part of the vehicle requires attention, the automated assistant may be notified of the notification via the OEM vehicle application. For example, when the vehicle maintenance application indicates that the vehicle requires gasoline and / or charging, the vehicle maintenance application may provide notifications about nearby locations to refuel the vehicle. The OEM vehicle application may generate data based on notifications and / or content rendered at a graphical user interface of the third-party application, and provide the generated data to the automated assistant. The automated assistant may use the data to generate suggestions, which may be ranked and / or filtered based on content that has already been presented to the user.

[0012] For example, when a generated suggestion corresponds to a nearby location to refuel a vehicle, the generated suggestion may be ranked (ie, prioritized) lower than separate suggestions that do not correspond to nearby locations to refuel a vehicle. As a result, other suggestions regarding, for example, a spoken utterance for obtaining additional information about the vehicle (e.g., “Assistant, what is my highway miles per gallon?”) may be prioritized over suggestions related to nearby locations for refueling. Alternatively or additionally, other suggestions may be generated based on a comparison between data based on content from a third-party application and assistant data, where the assistant data is data associated with interactions between a user and an automated assistant. For example, the assistant data may include content of previous interactions between the user and the automated assistant, contact information or calendar information linked to or associated with the user's account and interacted with by the user through the automated assistant, and / or the assistant data may include time and / or location information associated with the time and / or location of an interaction between the user and the automated assistant. In one example, data from an OEM vehicle application may be compared to the assistant data to determine that a notification from a third-party application is associated with an operating feature of the vehicle. Based on this determination, the automated assistant may determine when the user has previously engaged in a conversation session with the automated assistant regarding an operating feature of the vehicle.

[0013] For example, a user may have asked an automated assistant to find out what the appropriate tire pressure is for their vehicle. Thus, in response to receiving data from an OEM vehicle application, the automated assistant may generate a suggestion representative of a spoken utterance such as "Assistant, what is the tire pressure of my vehicle?". Suggestion data corresponding to the suggestion may be transmitted from the automated assistant to the OEM vehicle application and / or a third-party application, and the third-party application may then present the suggestion at a graphical user interface with content representative of the fuel status of the vehicle. In this way, while the user is being notified to refuel their vehicle, the user may learn to conduct similar conversational sessions to check other things related to the vehicle so that these things can be resolved more quickly, thereby promoting a healthy routine of vehicle maintenance.

[0014] Other implementations may include a computer program or non-transitory computer-readable storage medium storing instructions that are executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) to perform methods such as one or more of the methods described above and / or elsewhere herein. Still other implementations may include a system of one or more computers and / or one or more robots, including one or more processors operable to execute stored instructions to perform methods such as one or more of the methods described above and / or elsewhere herein.

[0015] It should be understood that all combinations of the foregoing concepts and additional concepts described in more detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered to be part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1A , Figure 1B and Figure 1C A diagram of a vehicle including an OEM vehicle computing device providing access to an automated assistant is shown.

[0017] Figure 2A , Figure 2B and Figure 2C One or more implementations of an automated assistant that provides suggestion elements at OEM applications and / or third-party applications accessible via an OEM vehicle computing device that employs restrictions on communications between native applications are shown.

[0018] Figure 3 A system is shown that provides an OEM vehicle computing device that includes a vehicle automated assistant that can render suggestions via OEM vehicle applications and / or third-party applications and initiate actions via OEM vehicle applications and / or third-party applications.

[0019] Figure 4 A method for providing one or more suggestion elements at a graphical user interface of a third-party vehicle computing device and / or at a third-party application operating at the third-party vehicle computing device is shown.

[0020] Figure 5 is a block diagram of an example computer system. DETAILED DESCRIPTION

[0021] Figure 1A , Figure 1B and Figure 1CViews 100 , 130 , and 140 are shown of a vehicle 108 including an OEM vehicle computing device 104 that provides access to an automated assistant 128 that can cause suggestions to be rendered at a graphical user interface 106 of an OEM vehicle application 124 and / or a third-party application 126 . Specifically, Figure 1A The view 100 shows an OEM vehicle computing device 104 providing a graphical user interface 106, one or more of which correspond at different areas to different applications and / or content that may be presented at the graphical user interface 106. The OEM vehicle computing device 104 may include one or more OEM vehicle applications 124, one or more third-party applications 126, and / or an automated assistant 128. When the user 102 is riding in the vehicle 108, one or more applications accessible via the OEM vehicle computing device 104 may communicate with a particular OEM vehicle application 124 to cause certain content to be rendered at the graphical user interface 106.

[0022] For example, the OEM vehicle computing device 104 may include an OEM vehicle application 124, such as a navigation application, which may cause content to be rendered at the first area 110 of the graphical user interface 106. Alternatively or additionally, the OEM vehicle computing device 104 may include a third-party application 126, such as a messaging application, which may cause other content to be rendered at the second area 114 of the graphical user interface 106. Each application that provides data for rendering content at the graphical user interface 106 may communicate with a specific OEM vehicle application 124. The specific OEM vehicle application may communicate with other OEM vehicle applications 124, third-party applications 126, and automated assistants 128. For example, a navigation application may provide data representative of upcoming navigation instructions, which may be rendered as a first graphical element 112 in the first area 110 of the graphical user interface 106. Additionally, the messaging application may provide other data representative of an incoming message (or indicative of the content of the incoming message), as well as other dates that may be rendered as a second graphical element 116 at the graphical user interface 106 by the specific OEM vehicle application 124.

[0023] In some implementations, the specific OEM vehicle application 124 may mediate communication between applications by generating data representing content provided by one or more applications of the OEM vehicle computing device 104 and providing the generated data to the automated assistant 128. For example, the generated data may represent natural language content provided in the first graphical element 112 and other natural language content provided in the second graphical element 116. The specific OEM vehicle application 124 may provide such generated data to the automated assistant 128, and the automated assistant 128 may use the generated data to provide additional suggestions for each corresponding area of ​​the graphical user interface 106.

[0024] Figure 1B A view 130 of a graphical user interface 106 is shown that provides a make-allowance for a suggestion element for each region of the graphical user interface 106 based on data provided by an automated assistant 128, and the suggestion element is generated based on data from a specific OEM vehicle application 124 and based on data associated with assistant data from the OEM vehicle application 124. If the vehicle application data corresponds at least in part to data associated with an interaction between a user and an automated assistant, the vehicle application data and the assistant data may be associated or related. For example, the vehicle application data may correspond to the content of a previous interaction or conversation session between the user and the automated assistant, relate to a previous action performed by the automated assistant, indicate an action associated with a location or contact linked to the user, or otherwise correspond to assistant data. Specifically, the specific OEM vehicle application 124 may provide data characterizing the content of the first graphical element 112 to the automated assistant 128. The automated assistant 128 may receive data from the OEM vehicle application 124 and generate suggestion data based on the received data. In some implementations, the received data may be compared to one or more actions that can be performed by the automated assistant 128, and based on the comparison, the suggestion data may be generated. Additionally or alternatively, the OEM vehicle application 124 may provide other data characterizing the content of the second graphical element 116 to the automated assistant 128. The automated assistant 128 may additionally receive data to be processed for determining a correlation between the other received data and one or more actions that can be performed by the automated assistant 128. Based on the comparison, other suggested data may be generated and provided to the OEM vehicle application 124.

[0025] In response to the OEM vehicle application 124 receiving the suggestion data from the automated assistant 128, the OEM vehicle application 124 may cause a suggestion element to be rendered at the graphical user interface 106. For example, the OEM vehicle application 124 may include a launcher application that renders a contact representing the operating state of one or more applications, and also initializes one or more actions of the one or more applications based on the data received from the automated assistant 128. For example, the automated assistant 128 may provide suggestion data representing a first suggestion element 132 that may represent a spoken utterance that, when spoken by the user and / or another user, causes the automated assistant to initialize or perform a specific action. For example, the vehicle application may provide an indication of the selection to the automated assistant, and the automated assistant may cause another application of the vehicle computing device to perform an action. The other application may be a third-party application (i.e., from a different source than the automated assistant and the vehicle OEM application) or one of the OEM vehicle applications. In addition, the automated assistant 128 may provide another suggestion data representing a second suggestion element 134 that may represent a different spoken utterance that, when spoken by the user and / or another user, causes the automated assistant 128 to initialize or perform a different specific action. For example, the first suggestion element 132 may represent a spoken utterance, such as “Assistant, what is my ETA?”

[0026] In response to the user tapping the suggestion element at display panel 120 or providing verbal input that includes at least a certain amount of the content of suggestion element 132, the automated assistant can convey to the user an estimated time to arrive at the destination. In addition, second suggestion element 134 can represent a verbal utterance, such as "Assistant, reply to Jane". In response to the user tapping second suggestion element 134 at display panel 120 and / or providing verbal input that includes at least a certain amount of the content of second suggestion element 134, automated assistant 128 can prompt the user to provide additional content for a message to be transmitted to a contact named Jane. In this way, the user can be notified that they can employ the automated assistant to initialize actions of OEM vehicle applications 124 and / or third-party applications 126, and / or otherwise perform actions based on information provided by vehicle applications 124 and / or third-party applications 126.

[0027] Figure 1CThe automated assistant 128 is shown causing the launcher OEM vehicle application 124 to provide a view 140 of suggested elements based on content presented at the graphical user interface 106 for a plurality of different applications. Specifically, a third suggested element 142 can be generated by the automated assistant 128 using vehicle application data provided by the launcher OEM vehicle application 124. The launcher OEM vehicle application 124 can receive data from a plurality of different third-party applications and / or other OEM vehicle applications 124 for rendering at the graphical user interface 106. The launcher OEM vehicle application 124 can generate data representing content presented at the graphical user interface 106 and provide the generated data to the automated assistant 128. The automated assistant 128 can use the data received from the launcher OEM vehicle application 124 to generate suggestion data, which can be rendered in the third area 118 of the graphical user interface 106.

[0028] The suggestion data may be generated via the automated assistant 128 to provide the user with suggestions related to multiple different sources of content being presented at the graphical user interface 106. In some implementations, the data generated from the launcher OEM vehicle application 124 may characterize ongoing actions being performed by a separate OEM vehicle application, as well as recently performed actions or recently presented notifications provided by a separate third-party application 126. In order to generate appropriate suggestions for the user 102, the automated assistant 128 may determine whether any of the generated data (which characterizes or indicates content presented at the graphical user interface 106) carries any relationship with assistant data accessible to the automated assistant 128. For example, since the automated assistant 128 may be linked to the account of the user 102, the automated assistant 128 may be notified of an incoming message from a contact of the user referred to as "Jane" to be a third-party application. Based on this association or relationship between the content of the second graphical element 116 and the assistant data that also indicates the incoming message, the automated assistant 128 may generate a blank message to reply to Jane. In addition, the automated assistant 128 may also determine the relationship between the content of the first graphical element 112 and the assistant data. For example, assistant data may represent one or more actions that can be performed or initiated by automated assistant 128 .

[0029] The suggestion data may be generated via the automated assistant 128 to provide the user with suggestions related to multiple different sources of content being presented at the graphical user interface 106. In some implementations, the data generated from the launcher OEM vehicle application 124 may characterize ongoing actions being performed by a separate OEM vehicle application, as well as recently performed actions or recently presented notifications provided by a separate third-party application 126. In order to generate appropriate suggestions for the user 102, the automated assistant 128 may determine whether any of the generated data carries any relationship with assistant data accessible to the automated assistant 128. For example, since the automated assistant 128 may be linked to the account of the user 102, the automated assistant 128 may be notified of an incoming message from a known contact "Jane" of the user to be a third-party application. Based on this relationship between the content of the second graphical element 116 and the assistant data that also indicates the incoming message, the automated assistant 128 may generate a blank message to reply to Jane. In addition, the automated assistant 128 may also determine the relationship between the content of the first graphical element 112 and the assistant data. For example, the assistant data may characterize one or more actions that can be performed or initialized by the automated assistant 128.

[0030] Figure 2A , Figure 2B and Figure 2C One or more implementations of an automated assistant 228 that provides suggestion elements at OEM applications and / or third-party applications accessible via an OEM vehicle computing device 204 that employs restrictions on communication between local applications. Figure 2A As provided in the view 200 of FIG. 2 , the OEM vehicle computing device 204 may include OEM vehicle applications 224, third-party applications 226, and / or an automated assistant 228. The OEM vehicle computing device 204 may implement access restrictions on the automated assistant 228, thereby limiting the automated assistant 228's access to the locally stored data of the OEM vehicle applications 224 and the locally stored data of the third-party applications 226. However, one or more of the OEM vehicle applications 224 may provide the automated assistant 228 with a certain amount of data regarding the operation of the vehicle 208, the OEM vehicle computing device 204, the OEM vehicle applications 224, the third-party applications 226, and / or any other information that may be associated with the vehicle 208.

[0031] For example, a launcher OEM vehicle application may generate content data representing content of a graphical user interface 206 rendered at a display panel 220 of an OEM vehicle computing device 204. The content data may represent natural language content presented in a first area 210 of the graphical user interface 206. The first area 210 may include a first graphical element 212 associated with a third-party application, such as a navigation application. The content data may also represent natural language content presented in a second area 214 of the graphical user interface 206. The second area 214 may include a second graphical element 216 associated with a third-party application, such as a messaging application.

[0032] In order to familiarize the user 202 with the functionality of allowing the automated assistant at 228 to initialize the OEM vehicle application 224 and / or the third-party application 226, the automated assistant 228 can provide suggestion elements related to such functionality. In other words, the automated assistant 228 can use the content data from the OEM vehicle application 224 to generate suggestions for spoken utterances that can be provided by the user 202 for invoking the automated assistant 228 to initiate the performance of an action by another OEM vehicle application 224 and / or the third-party application 226.

[0033] For example, the automated assistant 228 may use the content data to generate suggestion data corresponding to a suggestion that the user provide a command related to both the content of the first graphical element 212 and the content of the second graphical element 216. In this way, the automated assistant 228 acts within the boundaries of the restrictions implemented by the OEM vehicle computing device 204 while still providing useful suggestions to facilitate the use of all functions of the OEM vehicle computing device 204. For example, the suggestion data may characterize a spoken utterance, such as "Assistant, where is the nearest grocery store?" This is a spoken utterance that may be suggested based on the ongoing action of the navigation application to guide the user to a particular destination and the user 202 receiving a message about picking up coffee during their driving of the vehicle 208. The suggestion data may be provided to the launcher OEM vehicle application, which may process the suggestion data and cause the third graphical element 214 to be presented within the third area 222 of the graphical user interface 206. As a result, the suggestion is presented to the user 202 without the automated assistant 228 violating any access restrictions set forth by the OEM vehicle computing device 204, thereby allowing other instances of the automated assistant to operate on other vehicles with similar restrictions. Additionally or alternatively, automated assistant 228 can use content data to generate suggestion data corresponding to a suggestion for the user to provide a command related only to one of the content of first graphical element 212 or the content of second graphical element 216 .

[0034] In some implementations, in response to the user tapping the third graphical element 216 and / or providing the identified spoken utterance to the OEM vehicle computing device 204, the automated assistant 228 can be invoked. In addition, in response to the spoken utterance, the automated assistant at 228 can initiate the execution of one or more corresponding actions by the navigation application. For example, in response to the automated assistant 228 receiving the spoken utterance, "Assistant where is the nearest grocery store?" The automated assistant 228 can generate command data to be provided to a launcher OEM vehicle application and / or a server device associated with the navigation application. The command data can represent one or more commands generated according to an API corresponding to the navigation application. When the launcher OEM application and / or the server device receives the command data, the launcher OEM application and / or the server device can initiate the execution of one or more commands via the navigation application and / or any other third-party application 226.

[0035] In response to execution of the one or more commands being initiated, the navigation application may cause the third party application interface 242 to be provided at the graphical user interface 206, such as Figure 2B 240. Thus, despite the limitations set forth by the launcher OEM vehicle application, the user may provide spoken utterances to the automated assistant 228 to invoke a third-party application to perform one or more actions at the OEM vehicle computing device 204. To further the foregoing example, the third-party application 226 may provide content to the launcher OEM vehicle application for providing the third-party application interface 242. In addition, the content may characterize the location of a nearby grocery store, details about the route currently being taken by the user, and details about when the user 202 will arrive at the destination.

[0036] In some implementations, the launcher OEM vehicle application may generate data representative of content provided at the graphical user interface 206. Such generated data may be provided to the automated assistant 228, which may generate further suggestions based on recent changes to the graphical user interface 206. Specifically, the generated data may represent content provided at the third-party application interface 242, and the automated assistant 228 may use the generated data to generate other suggestion data for causing the automated assistant 228 to initiate execution of one or more actions by the OEM vehicle application 224 and / or the third-party application 226. Furthermore, the suggestion data may represent one or more actions related to the content provided at the graphical user interface 206, although the one or more actions are performed by an application different from the navigation application and / or other application that provides the content presented at the graphical user interface 206.

[0037] like Figure 2C , the automated assistant can cause the launcher OEM vehicle application to provide one or more suggestions 262 for invoking a separate application to perform one or more actions using a different application. The automated assistant 228 can use data representing content previously rendered at the graphical user interface 206 and content currently rendered at the graphical user interface 206 to generate appropriate suggestions for the user 202. For example, because Figure 2A The messaging application is indicated as having a new message from Jane in Figure 2C The automated assistant 228 may provide an ETA from the navigation application, so the automated assistant may suggest to the user to invoke the automated assistant to send the new ETA to Jane. Specifically, the suggestion data generated by the automated assistant 228 may represent a spoken utterance, such as "Assistant, send my new ETA to Jane". The suggestion data may be provided from the automated assistant 228 to a launcher OEM vehicle application and / or a server device associated with the navigation application, so that one or more suggestions 262 are presented at the graphical user interface 206 according to the suggestion data.

[0038] Figure 3 A system 300 is shown that provides an OEM vehicle computing device 318 that includes a vehicle automated assistant 322 that can render suggestions via OEM vehicle applications 320 and / or third-party applications 332 and initiate actions via OEM vehicle applications 320 and / or third-party applications 332. The system 300 may include an automated assistant 304 that may operate as part of an assistant application provided at one or more computing devices, such as the OEM vehicle computing device 318 and / or the server device 302. A user may interact with the automated assistant 304 via an assistant interface, which may be a microphone, a camera, a touch screen display, a user interface, and / or any other device capable of providing an interface between a user and an application.

[0039] For example, a user may initialize the automated assistant 304 by providing verbal, textual, and / or graphical input to an assistant interface (e.g., the assistant interface of the OEM vehicle computing device 318 and / or the assistant interface of any other client device) to cause the automated assistant 304 to perform a function (e.g., provide data, control peripherals, access agents, generate input and / or output, etc.). The OEM vehicle computing device 318 may include a display device, which may be a display panel including a touch interface for receiving touch input and / or gestures to allow the user to control applications of the OEM vehicle computing device 318 via the touch interface. In some implementations, the OEM vehicle computing device 318 may lack a display device, thereby providing audible user interface outputs but not graphical user interface outputs; in such an arrangement, the vehicle computing device 318 may be coupled or connected to an external display device to render the graphical user interface described herein. In addition, the OEM vehicle computing device 318 may provide a user interface, such as a microphone, for receiving verbal natural language input from a user. In some implementations, the OEM vehicle computing device 318 may include a touch interface and may be devoid of a camera or include a camera, but may optionally include one or more other sensors.

[0040] OEM vehicle computing device 318 and / or other third-party server devices 350 may communicate with server device 302 via a network 340, such as the Internet. Additionally, OEM vehicle computing device 318 and other client devices may communicate with each other via a local area network (LAN), such as a Wi-Fi network. Automated assistant 322 may offload computing tasks to server device 302 in order to conserve computing resources at OEM vehicle computing device 318. For example, server device 302 may host automated assistant 304, and OEM vehicle computing device 318 may transmit input received at one or more assistant interfaces to server device 302. However, in some implementations, automated assistant 304 may be hosted at OEM vehicle computing device 318 as a client automated assistant 322.

[0041] In various implementations, all or less than all aspects of automated assistant 304 may be implemented on OEM vehicle computing device 318. In some of those implementations, aspects of automated assistant 304 are implemented via a client automated assistant 322 of OEM vehicle computing device 318 and may interface with server device 302, which may implement other aspects of automated assistant 304. Server device 302 may optionally serve multiple users and their associated assistant applications via multiple threads. In implementations in which all or less than all aspects of automated assistant 304 are implemented via a client automated assistant 322 at OEM vehicle computing device 318, client automated assistant 322 may be an application separate from the operating system of OEM vehicle computing device 318 (e.g., installed “on top” of the operating system) or may alternatively be implemented directly by the operating system of OEM vehicle computing device 318 (e.g., considered an application of the operating system, but integrated with the operating system).

[0042] In some implementations, automated assistant 304 and / or client automated assistant 322 may include an input processing engine 306 that may employ a number of different modules to process input and / or output of OEM vehicle computing device 318 and / or server device 302. For example, input processing engine 306 may include speech processing engine 308 that may process audio data received at the assistant interface to identify text contained in the audio data. Audio data may be transmitted from, for example, OEM vehicle computing device 318 to server device 302 in order to preserve computing resources at OEM vehicle computing device 318.

[0043] The process for converting audio data into text may include a speech recognition algorithm, which may use a neural network and / or a statistical model to identify an audio data group corresponding to a word or phrase. The text converted from the audio data may be parsed by a data parsing engine 310 and available to an automated assistant as text data, which may be used to generate and / or identify command phrases, intentions, actions, slot values, and / or any other content specified by the user. In some implementations, the output data provided by the data parsing engine 310 may be provided to a parameter module 312 to determine whether the user has provided input corresponding to a specific intention, action, and / or routine that can be performed by an automated assistant 304 and / or an application or agent that can be accessed via an automated assistant 304. For example, assistant data 316 may be stored as vehicle data 324 at a server device 302 and / or an OEM vehicle computing device 318, and may include data defining one or more actions that can be performed by an automated assistant 304 and / or a client automated assistant 322, and parameters necessary for performing the action.

[0044] In some implementations, automated assistant 304 may be accessible via OEM vehicle computing device 318, which may limit local communications between applications accessible at OEM vehicle computing device 318. Automated assistant 304 may be operable to provide suggestions for invoking automated assistant 304 to initialize OEM vehicle application 320, third-party application 332, and / or client device to perform one or more actions. Automated assistant 304 may be accessible via OEM vehicle computing device 318, which may provide access to an instance of automated assistant 322 in communication with server device 302. Additionally or alternatively, automated assistant 304 may be provided at OEM vehicle computing device 318.

[0045] Vehicle computing device 318 may operate application interaction engine 334 to limit data transfer between automated assistant 322 and third-party applications 332 and / or between automated assistant 322 and OEM vehicle applications 320. Application interaction engine 334 may send and / or receive data from various applications at OEM vehicle computing device 318, and determine when and / or how much data to share with other applications at OEM vehicle computing device 318. For example, third-party applications 332 may be available at OEM vehicle computing device 318, and third-party applications 332 may communicate with third-party server device 350. Third-party server device 350 may use input request engine 354 to process requests provided via OEM vehicle computing device 318, and may use command engine 356 to respond to the requests.

[0046] In some implementations, application state engine 352 of third-party server device 350 can generate server data 358, which can characterize the state of one or more third-party applications 332. In addition, because third-party applications 332 can be associated with a user's account, and the account can also be associated with automated assistant 304 (with the user's prior permission), data can be shared between third-party server device 350 and server device 302. For example, changes in application state determined by application state engine 352 and / or operations performed by one or more applications can be communicated to automated assistant 304 to bring automated assistant 304 to notice one or more actions performed by third-party applications 332. Third-party interaction engine 326 can receive data from third-party server device 350, OEM vehicle computing device 318, client device, and / or any other device or apparatus to provide contextual data to automated assistant 304.

[0047] Suggestion engine 328 of automated assistant 304 can use contextual data to generate suggestions about relevant actions that can be initiated by automated assistant 304. In addition, suggestion data characterizing the suggestions can be stored as assistant data 316 and processed by suggestion ranking engine 336. Suggestion ranking engine 336 can rank the suggestions based at least in part on the contextual data. In addition, in some implementations, suggestion ranking engine 336 can be ranked using contextual data so as to prevent suggesting actions that have recently been performed by the application and / or are currently being performed by the application.

[0048] In some implementations, the suggestions may be ranked and / or presented based on contextual data, which may include details of the user's route. For example, the contextual data may indicate whether the user is currently driving on a portion of a route that lasts X minutes and / or Y miles. In some implementations, when the portion of the route meets the time and / or distance thresholds, the suggestion engine 328 may cause one or more suggestion elements to be presented at the graphical user interface of the OEM vehicle computing device 318. However, when the portion of the route does not meet the time and / or distance thresholds, the suggestion engine 328 may bypass causing the graphical user interface to present one or more suggestion elements.

[0049] Figure 4 A method 400 for providing one or more suggestion elements at a graphical user interface of a third-party vehicle computing device and / or at a third-party application operating at the third-party vehicle computing device is shown. The method 400 may be performed by one or more computing devices, applications, and / or any other means or modules capable of providing suggestions at a user interface. The method 400 may include an operation 402, which includes receiving vehicle application data representing content rendered at a display panel of a vehicle computing device. The vehicle computing device may operate in accordance with restrictions that restrict local communications between applications accessible via the vehicle computing device.

[0050] For example, a vehicle computing device may include an OEM third-party vehicle application, one or more other third-party applications, and / or an automated assistant. The automated assistant may be remote from the vehicle computing device. Therefore, the vehicle computing device may limit communication between the automated assistant and one or more other third-party applications. For example, when the vehicle computing device includes a third-party navigation application and / or a third-party messaging application, the above-mentioned application may be restricted from communicating directly with the automated assistant, and conversely restricted to communicating only through the vehicle application via the vehicle computing device, optionally. However, the OEM vehicle application may generate vehicle application data, which may characterize or indicate content being rendered at a display panel of the vehicle computing device. Alternatively or additionally, the OEM vehicle application may generate vehicle application data that characterizes one or more operations performed by one or more other third-party applications, the vehicle, and / or any other device or module that may be associated with the vehicle and / or the vehicle computing device. Limiting direct communication access between the automated assistant and other third-party applications can help maintain the integrity of the vehicle system, which may be important for vehicle safety and reliability.

[0051] Method 400 may also include an operation 404 of determining a relationship between the received vehicle application data and assistant data associated with the automated assistant. For example, operation 404 may include determining whether the vehicle application data represents any information that has been the subject of one or more conversation sessions between the user and the automated assistant. Alternatively or in addition, operation 404 may include determining whether the vehicle application data represents any information that has been accessed by the user using one or more computing devices that provide access to the automated assistant. Access to such information may be provided with permission from the user in order to protect information that the user does not want other devices and / or people to access. Alternatively or in addition, operation 404 may also include determining whether the vehicle application data and the assistant data represent information that may be associated with a data type and / or subjective information. For example, when the vehicle application data indicates that an incoming call is received from a particular person at a vehicle computing device and the automated assistant data indicates that the user has previously called the automated assistant to answer a call from the same particular person, a relationship between the vehicle application data and the assistant data may be identified.

[0052] Method 400 may also include an optional operation 406 of determining whether the (vehicle) application data represents a past action. A past action may be an action that has been performed within a period of time before the vehicle application data is received (e.g., providing a notification, sending data, accessing data, generating data, etc.). In some implementations, the time period may be a threshold time period that is static or dynamic according to the characteristics of the interaction between the user and the vehicle computing device and / or the user and the automated assistant. As an example, a third-party application available at the vehicle computing device may provide a notification indicating that the fluid sensor of the vehicle is in an alarm state. The notification may be considered a past action because the notification is provided at a graphical user interface rendered at a display panel within a threshold time period of receiving the vehicle application data. In some implementations, the time to present the notification may be determined based on time information represented by the vehicle application data. The automated assistant may access an application for tracking certain attributes of the vehicle, and may therefore also know the information represented by the notification about the fluid sensor. However, depending on when the notification is presented at the display panel of the vehicle computing device, the automated assistant may or may not bypass causing the notification about the fluid sensor to appear at the display panel.

[0053] When the application data characterizes a past action at operation 406, method 400 may proceed to operation 408, which may include generating another suggestion data that avoids suggesting the past action. As an example, when a third-party application provides a notification related to a fluid sensor, the another suggestion data may avoid suggesting obtaining information about the state of the fluid sensor by providing one or more suggestions that do not include a request for information detailing the state of the fluid sensor. In contrast, the automated assistant may generate suggestion data characterizing a spoken utterance for requesting information about purchasing a fluid corresponding to the fluid sensor, sending a message that includes information detailing the state of the fluid sensor, calling a business that performs maintenance associated with the fluid sensor, and / or any other request other than a request for information detailing the state of the fluid sensor.

[0054] When it is determined at operation 406 that the application data does not represent a past action, method 400 may proceed to operation 410. Operation 410 may include generating suggestion data based on a relationship between the vehicle application data and the assistant data. For example, when a graphical user interface at a third-party application includes content representing a route to a particular destination and the automated assistant has not provided a notification related to a fluid sensor within a threshold time period, the automated assistant may provide suggestion data corresponding to the notification for the fluid sensor. The suggestion data may represent natural language content, such as a spoken utterance, which, when spoken by a user, causes the automated assistant to provide information related to the status of the fluid sensor. For example, the spoken utterance may be “Assistant, what is the status of the fluid sensor?”

[0055] Method 400 may proceed from operation 410 and / or operation 408 to operation 412, which may include prioritizing suggested data and / or another suggested data based on a relationship between the vehicle application data and the assistant data. In some implementations, operation 412 may be an optional operation. In some implementations, the strength of the correlation between the beautiful vehicle application data and the assistant data may be used to determine the priority of the suggested data. For example, the correlation of vehicle application data representing natural language content explicitly included in a conversation session between an automated assistant and a user may be characterized as having a stronger correlation than another correlation between the vehicle application data and the assistant data when a past conversation session between the user and the automated assistant resulted in an action represented by the natural language content.

[0056] Method 400 may proceed from operation 412 to operation 414, which may include causing a congestion suggestion element to be rendered by a third-party application based on suggestion data. Optionally, the suggestion element may be rendered by a vehicle application. Causing the suggestion element rendered by the vehicle application to include providing action data from an automated assistant to the vehicle application. The action data may indicate an action to be performed by the vehicle application when the suggestion element is selected, and may include, for example, parameters for the execution of an action associated with the suggestion element. The suggestion element may be an optional element that may be selected by a user tapping a display panel at a location where the suggestion element is presented and / or by a user providing a verbal utterance including at least a certain amount of natural language content included in the suggestion element. When rendering the suggestion element at the display panel, method 400 may proceed to optional operation 416 to determine whether the rendered suggestion element has been selected within a specific time period. In some embodiments, the action associated with the suggestion element may be performed by one of the OEM vehicle application or the third-party application based on the action data. In the case where a third-party application is not expected to access a critical vehicle system, for example, in an autonomous vehicle, access to a sensor system or a navigation system may be limited to the vehicle's native application, so that the native OEM vehicle application performs an action that can improve safety. In other embodiments, the action data may include a unique ID number provided by the automated assistant; the user's selection of the suggestion element may cause the vehicle application to cause the automated assistant to perform an action associated with the unique ID number of the action data of the suggestion element. Causing the automated assistant to perform the application may improve the user's privacy because third-party applications or vehicle-native OEM applications cannot access information about the user or about the action performed by the automated assistant based on the correlation or association between the vehicle application data and the assistant data. Therefore, security may be improved for the user.

[0057] When the rendered suggestion element is selected within a specific time period, the method may proceed from operation 416 to optional operation 418. Optional operation 418 may include assigning a priority to the suggestion data according to the selection of the suggestion. However, if the user does not select the suggestion within a specific time period, the method may proceed from operation 416 to operation 420. Operation 420 may include assigning a priority to the suggestion data according to the non-selection of the suggestion element. In some implementations, the priority of the suggestion data may be increased in response to the user selecting the suggestion element, and / or the priority of the suggestion data may be reduced in response to the user not selecting the suggestion element within a specific time period. In this way, when the user interacts with the vehicle computing device, the suggestion may be ranked according to its relevance to the user and whether the user has confirmed the content corresponding to the suggestion data. Method 400 may be based on the new vehicle application data rendered by the third-party vehicle application cycle, so as to promote various functions at the vehicle and / or vehicle computing device, thereby also promoting more efficient use of the vehicle and / or vehicle computing device.

[0058] Figure 5 5 is a block diagram of an example computer system 510. The computer system 510 generally includes at least one processor 514 that communicates with a number of peripheral devices via a bus subsystem 512. These peripheral devices may include a storage subsystem 524 (e.g., including a memory 525 and a file storage subsystem 526), ​​a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input and output devices allow a user to interact with the computer system 510. The network interface subsystem 516 provides an interface to an external network and is coupled to corresponding interface devices in other computer systems.

[0059] The user interface input devices 522 may include a keyboard, a pointing device such as a mouse, a trackball, a touch pad or a graphics tablet, a scanner, a touch screen incorporated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways to input information into the computer system 510 or onto a communication network.

[0060] The user interface output device 520 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display such as via an audio output device. In general, the use of the term "output device" is intended to include all possible types of devices and ways to output information from the computer system 510 to a user or another machine or computer system.

[0061] The storage subsystem 524 stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 524 may include logic to perform selected aspects of the method 400 and / or implement one or more of the OEM vehicle computing device 104 and / or 204, the OEM vehicle application 124 and / or 224, the third-party application 126 and / or 226, the automated assistant 128 and / or 228, the vehicle 108 and / or 208, the server device 302, the OEM vehicle computing device 318, the third-party server device 350, and / or any other application, device, apparatus, and / or module discussed herein.

[0062] These software modules are typically executed by the processor 514 alone or in conjunction with other processors. The memory 525 used in the storage subsystem 524 may include multiple memories, including a main random access memory (RAM) 530 for storing instructions and data during program execution and a read-only memory (ROM) 532 in which fixed instructions are stored. The file storage subsystem 526 may provide permanent storage for program and data files, and may include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain implementations may be stored in the storage subsystem 524 by the file storage subsystem 526, or in other machines accessible by the processor 514.

[0063] The bus subsystem 512 provides a mechanism for the various components and subsystems of the computer system 510 to communicate with each other as desired. Although the bus subsystem 512 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0064] Computer system 510 can be of various types, including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Figure 5 The description of the computer system 510 depicted in FIG. 5 is intended only as a specific example for the purpose of illustrating some implementations. Many other configurations of the computer system 510 may have more Figure 5 The computer system depicted in FIG.

[0065] In cases where the systems described herein collect or make available personal information about users (or, as they are often referred to herein, "participants"), users may be provided with the opportunity to control whether a program or feature collects user information (e.g., information about the user's social network, social actions or activities, occupation, the user's preferences, or the user's current geographic location) or to control whether and / or how content that may be more relevant to the user is received from a content server. Moreover, certain data may be processed in one or more ways before being stored or used to remove personally identifiable information. For example, the user's identity may be processed so that personally identifiable information cannot be determined for the user, or the user's geographic location may be summarized (such as to a city, zip code, or state level) where geographic location information is available so that the user's specific geographic location cannot be determined. Thus, users may have control over how information is collected and / or used about them.

[0066] Although several implementations have been described and shown herein, various other devices and / or structures for performing the functions described herein and / or obtaining the results and / or one or more advantages described herein may be utilized, and each of such changes and / or modifications is considered to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are meant to be exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications using this teaching. Those skilled in the art will recognize or be able to determine many equivalents of the specific implementations described herein using no more than routine experiments. Therefore, it should be understood that the aforementioned implementations are presented only by way of example, and within the scope of the appended claims and their equivalents, implementations may be practiced in a manner different from that specifically described and claimed. The implementations of the present disclosure relate to each individual feature, system, article, material, kit, and / or method described herein. In addition, if these features, systems, articles, materials, kits, and / or methods are not mutually contradictory, any combination of two or more of these features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure.

[0067] In some implementations, a method implemented by one or more processors is described as including operations such as receiving vehicle application data corresponding to content rendered at a graphical user interface of the vehicle application via a vehicle application operated by a user while riding in a vehicle, wherein the graphical user interface is displayed at a display panel that communicates with a vehicle computing device of the vehicle. The method may also include, in response to receiving the vehicle application data, determining that the vehicle application data is associated with assistant data available via an automated assistant, wherein the automated assistant is accessible via the vehicle computing device. The method may also generate suggestion data in response to determining that the vehicle application data is associated with the assistant data, wherein the suggestion data identifies an action associated with the assistant data and the vehicle application data, and the suggestion data represents natural language content that is different from the content being rendered at the display panel. The method may also include determining, based on the suggestion data, a priority of the suggestion data relative to priority data associated with previously generated suggestion data, wherein the previously generated suggestion data corresponds to one or more suggestion elements previously rendered at the graphical user interface of the vehicle application. The method may also include, when the priority of the suggestion data relative to the priority data indicates that the suggestion data takes precedence over other suggestion data: based on the suggestion data and when the user rides in the vehicle, causing a specific suggestion element to be rendered on the graphical user interface of the vehicle application, wherein the specific suggestion element includes the natural language content.

[0068] In some implementations, the priority data is based on one or more previous instances in which the user and / or one or more other users selected the one or more suggestion elements while the user and / or the one or more other users were riding in the vehicle. In some implementations, the method may also include determining an estimated arrival time of the user at a destination location, wherein determining the priority of the suggestion data relative to priority data associated with other suggestion data is based at least in part on the estimated arrival time of the user at the destination location. In some implementations, the method may also include, when the priority of the suggestion data relative to the priority data indicates that the suggestion data takes precedence over other suggestion data: after causing the particular suggestion element to be rendered at a graphical user interface of the vehicle application, determining that the particular suggestion element is selected, and in response to determining that the particular suggestion element is selected, causing the priority of the suggestion data to be modified.

[0069] In some implementations, the graphical user interface of the vehicle application includes a first area including the content and a second area including other content, and wherein causing the specific suggestion element to be rendered at the graphical user interface of the vehicle application includes causing the specific suggestion element to be rendered at the first area of ​​the graphical user interface and other suggestion elements to be rendered at the second area of ​​the graphical user interface, wherein the other suggestion elements are based at least in part on the other content. In some implementations, the vehicle application data also corresponds to the other content, and generating the suggestion data includes identifying other actions associated with the other content. In some implementations, the method may further include generating additional suggestion data based on the content provided in the first area of ​​the graphical user interface and based on the other content provided in the second area of ​​the graphical user interface, wherein the additional suggestion data identifies additional actions associated with the assistant data and the vehicle application data, and the suggestion data represents other natural language content that is different from the content and the other content.

[0070] In some implementations, the graphical user interface of the vehicle application includes a third area corresponding to the automated assistant, and the method further includes: based on the additional suggestion data and when the user is riding in the vehicle, causing a separate suggestion element to be rendered at the third area of ​​the graphical user interface of the vehicle application, wherein the separate suggestion element includes the other natural language content. In some implementations, causing the specific suggestion element to be rendered at the graphical user interface of the vehicle application based on the suggestion data includes providing action data from the automated assistant to the vehicle application. In some implementations, the specific suggestion element is rendered by the vehicle application, and the natural language content of the suggestion element represents a verbal utterance, which, when spoken by the user to the vehicle computing device, causes the action to be initiated via the automated assistant. In some implementations, the method may also include, when the priority of the suggestion data relative to the priority data indicates that the suggestion data takes precedence over other suggestion data: determining that the user has provided a verbal input corresponding to the natural language content of the suggestion element, and in response to determining that the user has provided the verbal input, causing the automated assistant to initiate the action, wherein the action is performed by the separate vehicle application in response to the automated assistant initiating the action.

[0071] In some implementations, the automated assistant is provided by a source that generates the priority data, the vehicle application data is generated by a third-party source different from the source, and the separate vehicle application is provided by a source separate from the source and the third-party source. In some implementations, the vehicle application is a navigation application provided by the source, and the separate vehicle application is a communication and / or media application provided by the separate source. In some implementations, the automated assistant is provided by a source that generates the priority data, and the vehicle application data is generated by a third-party source different from the source. In some implementations, the automated assistant is remote from a vehicle computing device of the vehicle.

[0072] In other implementations, a method implemented by one or more processors is described as including operations such as determining, by a vehicle application of a vehicle computing device, that a third-party application is providing content via a graphical user interface rendered at a display panel of the vehicle computing device, wherein the vehicle application is accessible by a user via the vehicle computing device and the vehicle computing device is part of a vehicle that navigates the user to a destination. In some implementations, the method may also include generating vehicle application data based on determining that the graphical user interface includes the content, the vehicle application data representing at least a portion of the content provided by the third-party application. In some implementations, the method may also include providing, by the vehicle application of the vehicle computing device, the vehicle application data to an automated assistant, wherein the automated assistant is also accessible by the user via the vehicle computing device. In some implementations, the method may also include providing the vehicle application data to the automated assistant so that the automated assistant provides suggestion data generated based on a correspondence between the assistant data and the vehicle application data, wherein the assistant data is associated with an account of the user. In some implementations, the method may further include receiving, by the vehicle application, the suggestion data from the automated assistant, wherein the suggestion data represents natural language content that, when spoken by the user to the vehicle computing device, causes the automated assistant to initiate performance of an action by the third-party application. In some implementations, the method may further include causing the third-party application to render the natural language content at a graphical user interface of the third-party application in response to receiving the suggestion data from the automated assistant, wherein the natural language content is rendered at the graphical user interface while the content is rendered at the graphical user interface and the vehicle navigates the user to the destination.

[0073] In some implementations, the method may further include, after the natural language content is rendered at the graphical user interface: determining that the user and / or another user has provided a spoken utterance that includes at least some of the natural language content rendered at the graphical user interface, and in response to determining that the user and / or the other user has provided the spoken utterance, causing the automated assistant to initiate execution of an action by the third-party application. In some implementations, the method may further include, in response to receiving the suggestion data, determining an amount of time remaining and / or a remaining distance for the vehicle to navigate the user to the destination; and determining whether the amount of time remaining and / or the remaining distance for the vehicle to navigate the user to the destination satisfies a threshold, wherein when the amount of time remaining and / or the remaining distance satisfies the threshold, causing the third-party application to render the natural language content at the graphical user interface of the third-party application. In some implementations, the method may further include providing updated content data representing additional data being presented at the graphical user interface.

[0074] In other implementations, a method implemented by one or more processors is set forth as including operations such as providing content data representing content to be rendered at a display panel of a vehicle computing device from a third-party application to a vehicle application, wherein the vehicle computing device is included in a vehicle that a user is driving to a destination. The method may also include, based on providing the content data to the vehicle application, causing the vehicle application to provide at least a portion of the content data to an automated assistant, wherein the automated assistant is accessible by the user via one or more interfaces of the vehicle computing device, and wherein the vehicle computing device limits accessibility of the content data to the automated assistant. The method may also include receiving suggestion data from the vehicle application, the suggestion data being generated at least in part based on the automated assistant receiving at least the portion of the content data, wherein the suggestion data represents natural language content that, when included in a spoken utterance from the user to the automated assistant, causes the automated assistant to initiate execution of one or more actions by the third-party application, the vehicle application, and / or another third-party application. The method may, in response to receiving the suggestion data from the vehicle application, cause a display panel of the vehicle to compute to render the natural language content at a graphical user interface. The method may also include, when the user provides the spoken utterance to the automated assistant via an interface of the vehicle computing device: causing the automated assistant to initiate execution of the one or more actions by the third-party application, the vehicle application, and / or another third-party application.

[0075] In some implementations, at least the portion of the content data is selected for providing to the automated assistant based on rendered data displayed at a display panel of the vehicle computing device. In some implementations, the method may also include, when the user provides the spoken utterance to the automated assistant via an interface of the vehicle computing device: providing, via a server device in communication with the third-party application, an operational state of the one or more actions to another computing device separate from the vehicle computing device.

Claims

1. A method implemented by one or more processors, the method comprising: receiving vehicle sensor data via vehicle sensors operated while the user is riding in the vehicle; rendering content corresponding to the vehicle sensor data at a graphical user interface of a display panel in communication with a vehicle computing device of the vehicle; determining that the vehicle sensor data is associated with assistant data available via an automated assistant, wherein the automated assistant is accessible via the vehicle computing device; generating suggestion data in response to determining that the vehicle sensor data is associated with the assistant data, wherein the suggestion data identifies an action associated with the assistant data and the vehicle sensor data; and A suggestion element is caused to be rendered at the graphical user interface of the display panel based on the suggestion data and while the user is riding in the vehicle, wherein the suggestion element includes natural language content.

2. The method according to claim 1, wherein: The suggestion element comprises a selectable element that, when selected, initiates the action.

3. The method according to claim 1, wherein: The graphical user interface is executed via a vehicle application.

4. The method according to claim 3, wherein: The suggestion element is rendered based on the third-party application.

5. The method according to claim 4, wherein: The suggestion element is rendered by the third-party application or another third-party application.

6. The method according to claim 1, further comprising: An estimated arrival time of the user at a destination location is determined, wherein generating the suggestion data is based on the estimated arrival time of the user at the destination location.

7. The method according to claim 1, in, The graphical user interface of the display panel includes a first area containing the content and a second area containing other content, and Wherein, causing the suggestion element to be rendered at the graphical user interface of the display panel comprises: The suggestion element is caused to be rendered at the first area of ​​the graphical user interface, and another suggestion element is caused to be rendered at the second area of ​​the graphical user interface, wherein the another suggestion element is based on other content.

8. The method according to claim 7, in, The vehicle sensor data also corresponds to the other content, and Wherein, generating the suggestion data comprises identifying another action associated with the other content.

9. The method according to claim 8, further comprising: generating additional suggestion data based on the content provided in the first area of ​​the graphical user interface and the other content provided in the second area of ​​the graphical user interface, The additional suggestion data identifies additional actions associated with the assistant data and the vehicle sensor data, and the suggestion data represents other natural language content that is different from the content and the other content.

10. The method according to claim 1, wherein: The suggestion element is rendered by a vehicle application, and the natural language content of the suggestion element represents a spoken utterance that, when spoken by the user into the vehicle computing device, causes the action to be initiated via the automated assistant.

11. A method implemented by one or more processors, the method comprising: receiving vehicle sensor data via vehicle sensors operated while a user is riding in a vehicle, the vehicle sensor data corresponding to content rendered at a graphical user interface of a display panel in communication with a vehicle computing device of the vehicle; determining that the vehicle sensor data is associated with assistant data available via an automated assistant, wherein the automated assistant is accessible via the vehicle computing device; generating suggestion data in response to determining that the vehicle sensor data is associated with the assistant data, wherein the suggestion data identifies an action associated with the assistant data and the vehicle sensor data; and A suggestion element is caused to be rendered at the graphical user interface of the display panel based on the suggestion data and while the user is riding in the vehicle, wherein the suggestion element includes natural language content.

12. The method according to claim 11, wherein: The suggestion element comprises a selectable element that, when selected, initiates the action.

13. The method according to claim 11, wherein: The graphical user interface is executed via a vehicle application.

14. The method according to claim 13, wherein: The suggestion element is rendered based on the third-party application.

15. The method according to claim 14, wherein: The suggestion element is rendered by the third-party application or another third-party application.

16. The method according to claim 11, further comprising: An estimated arrival time of the user at a destination location is determined, wherein generating the suggestion data is based on the estimated arrival time of the user at the destination location.

17. The method according to claim 11, in, The graphical user interface of the display panel includes a first area containing the content and a second area containing other content, and Wherein, causing the suggestion element to be rendered at the graphical user interface of the display panel comprises: The suggestion element is caused to be rendered at the first area of ​​the graphical user interface, and another suggestion element is caused to be rendered at the second area of ​​the graphical user interface, wherein the another suggestion element is based on other content.

18. The method according to claim 17, in, The vehicle sensor data also corresponds to the other content, and Wherein, generating the suggestion data comprises identifying another action associated with the other content.

19. The method according to claim 18, further comprising: generating additional suggestion data based on the content provided in the first area of ​​the graphical user interface and the other content provided in the second area of ​​the graphical user interface, The additional suggestion data identifies additional actions associated with the assistant data and the vehicle sensor data, and the suggestion data represents other natural language content that is different from the content and the other content.

20. The method according to claim 11, wherein: The suggestion element is rendered by a vehicle application, and the natural language content of the suggestion element represents a spoken utterance that, when spoken by the user into the vehicle computing device, causes the action to be initiated via the automated assistant.

21. A method implemented by one or more processors, the method comprising: receiving, via a vehicle application operated by a user while riding in a vehicle, vehicle application data indicating a destination location of the user, the vehicle application data corresponding to content rendered at a graphical user interface of the vehicle application, wherein the graphical user interface is displayed at a display panel in communication with a vehicle computing device of the vehicle; receiving assistant data, the assistant data comprising communications from additional users not riding in the vehicle available via an automated assistant, wherein the automated assistant is accessible via the vehicle computing device; determining an estimated arrival time of the user at the destination location in response to receiving the assistant data and receiving the vehicle application data indicating the destination location; responsive to determining the estimated time of arrival, generating suggestion data, the suggestion data identifying an action to transmit the estimated time of arrival to the additional user; and causing a particular suggestion element to be rendered at a first area of ​​the graphical user interface of the vehicle application based on the suggestion data, and causing a separate suggestion element to be rendered at a separate area of ​​the graphical user interface based on the suggestion data, while the user is riding in the vehicle and before the user arrives at the destination location, wherein the particular suggestion element, when selected via a touch input by the user, causes the estimated time of arrival to be transmitted to the additional user by the application, and wherein the separate suggestion element, when selected via the user's touch input, causes the action to be performed by a separate application; and causing a particular additional suggestion element to be rendered at a second area of ​​the graphical user interface of the vehicle application and to be rendered in the second area separately from but concomitantly with the rendering of the particular suggestion element in the first area and the separate suggestion element in the separate area, wherein the particular additional suggestion element, when selected via a touch input by the user, enables the user to provide an utterance and causes content determined from the utterance to be transmitted to the additional user, and wherein the particular suggestion element is presented and the particular additional suggestion element is presented based on a measure that satisfies a threshold, wherein the measure reflects an amount of travel remaining before the user reaches the destination.

22. The method according to claim 21, wherein: Generating the suggestion data identifying the action to transmit the estimated time of arrival to the additional user comprises: Natural language content is generated that identifies the action of transmitting the estimated time of arrival to the additional user.

23. The method according to claim 22, wherein: The natural language content of the particular suggestion element represents a spoken utterance that, when spoken by the user into the vehicle computing device, causes the action to be initiated via the automated assistant.

24. The method of claim 23, further comprising, after the natural language content is rendered at the graphical user interface: determining that the user and / or another user has provided the spoken utterance, and In response to determining that the user and / or another user has provided the spoken utterance, causing the automated assistant agent to initiate performance of the action of transmitting the estimated arrival time to the additional user.

25. The method according to claim 21, wherein: Determining the estimated time of arrival of the user to the destination location includes determining an amount of time remaining for the vehicle to navigate the user to the destination location.

26. The method according to claim 21, wherein: Determining the estimated time of arrival of the user to the destination location includes determining a remaining distance for the vehicle to navigate the user to the destination location.

27. The method of claim 21, wherein: The measurement reflects the amount of time remaining to reach the destination.

28. The method of claim 21, wherein: The measurement reflects the remaining distance to reach the destination.

29. A system comprising: one or more processors; as well as a memory configured to store instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: receiving, via a vehicle application operated by a user while riding in a vehicle, vehicle application data indicating a destination location of the user, the vehicle application data corresponding to content rendered at a graphical user interface of the vehicle application, wherein the graphical user interface is displayed at a display panel in communication with a vehicle computing device of the vehicle; receiving assistant data, the assistant data comprising communications from additional users not riding in the vehicle available via an automated assistant, wherein the automated assistant is accessible via the vehicle computing device; determining an estimated arrival time of the user at the destination location in response to receiving the assistant data; responsive to determining the estimated time of arrival, generating suggestion data identifying an action to transmit the estimated time of arrival to the additional user; and causing a particular suggestion element to be rendered at a first area of ​​the graphical user interface of the vehicle application based on the suggestion data, and causing a separate suggestion element to be rendered at a separate area of ​​the graphical user interface based on the suggestion data, while the user is riding in the vehicle and before the user arrives at the destination location, wherein the particular suggestion element, when selected via a touch input by the user, causes the estimated time of arrival to be transmitted to the additional user by the application, and wherein the separate suggestion element, when selected via the user's touch input, causes the action to be performed by a separate application; and causing a particular additional suggestion element to be rendered at a second area of ​​the graphical user interface of the vehicle application, separate from but concomitantly with the rendering of the particular suggestion element in the first area, wherein the particular additional suggestion element, when selected via a touch input by the user, enables the user to provide an utterance and causes content determined from the utterance to be transmitted to the additional user, and wherein the particular suggestion element is presented and the particular additional suggestion element is presented based on a measure that satisfies a threshold, wherein the measure reflects an amount of travel remaining before the user reaches the destination.

30. The system of claim 29, wherein: Generating the suggestion data identifying the action to transmit the estimated time of arrival to the additional user comprises: Natural language content is generated that identifies the action of transmitting the estimated time of arrival to the additional user.

31. The system of claim 30, wherein: The natural language content of the particular suggestion element represents a spoken utterance that, when spoken by the user into the vehicle computing device, causes the action to be initiated via the automated assistant.

32. The system of claim 31, wherein: The operations further include, after the natural language content is rendered at the graphical user interface: determining that the user and / or another user has provided the spoken utterance, and In response to determining that the user and / or another user has provided the spoken utterance, causing the automated assistant agent to initiate performance of the action of transmitting the estimated arrival time to the additional user.

33. The system of claim 29, wherein: Determining the estimated time of arrival of the user to the destination location includes determining an amount of time remaining for the vehicle to navigate the user to the destination location.

34. The system of claim 29, wherein: Determining the estimated time of arrival of the user to the destination location includes determining a remaining distance for the vehicle to navigate the user to the destination location.

35. A non-transitory computer-readable medium configured to store instructions that, when executed by one or more processes, cause the one or more processors to perform operations comprising: receiving, via a vehicle application operated by a user while riding in a vehicle, vehicle application data indicating a destination location of the user, the vehicle application data corresponding to content rendered at a graphical user interface of the vehicle application, wherein the graphical user interface is displayed at a display panel in communication with a vehicle computing device of the vehicle; receiving assistant data, the assistant data comprising communications from additional users not riding in the vehicle available via an automated assistant, wherein the automated assistant is accessible via the vehicle computing device; determining an estimated arrival time of the user at the destination location in response to receiving the assistant data; responsive to determining the estimated time of arrival, generating suggestion data, the suggestion data identifying an action to transmit the estimated time of arrival to the additional user; and causing a particular suggestion element to be rendered at a first area of ​​the graphical user interface of the vehicle application based on the suggestion data, and causing a separate suggestion element to be rendered at a separate area of ​​the graphical user interface based on the suggestion data, while the user is riding in the vehicle and before the user arrives at the destination location, wherein the particular suggestion element, when selected via a touch input by the user, causes the estimated time of arrival to be transmitted to the additional user by the application, and wherein the separate suggestion element, when selected via the user's touch input, causes the action to be performed by a separate application; and causing a particular additional suggestion element to be rendered at a second area of ​​the graphical user interface of the vehicle application, separate from but concomitantly with the rendering of the particular suggestion element in the first area, wherein the particular additional suggestion element, when selected via a touch input by the user, enables the user to provide an utterance and causes content determined from the utterance to be transmitted to the additional user, and wherein the particular suggestion element is presented and the particular additional suggestion element is presented based on a measure that satisfies a threshold, wherein the measure reflects an amount of travel remaining before the user reaches the destination.

Images