Providing context-based automated assistant action suggestion(s) via a vehicle computing device

Abstract

The embodiments described herein relate to an automated assistant that may provide suggestions for a user to interact with the automated assistant to control an application while in a vehicle. Suggestions may be provided to encourage hands-free interaction with the application by suggesting assistant inputs that invoke the automated assistant to act as an interface between the user and the application. The assistant suggestions may be based on the user's context and / or the vehicle's context, such as the content of a display interface of a device that the user is accessing while in the vehicle. For example, the automated assistant may identify that an action performed by the user using the application can be initiated more safely and / or in less time by utilizing a particular assistant input. The particular assistant input may then be rendered in an interface of the vehicle computing device.

Description

[Technical field]

[0001] The present invention relates to providing context-based automated assistant action suggestion(s) via a vehicle computing device. [Background technology]

[0002] Humans may engage in human-computer interactions using interactive software applications referred to herein as "automated assistants" (also referred to as "digital agents," "chatbots," "interactive personal assistants," "intelligent personal assistants," "assistant applications," "conversational agents," etc.). For example, a human (who may be referred to as a "user" when interacting with an automated assistant) may provide commands and / or requests to the automated assistant using natural language input via voice (i.e., speaking) and / or by providing natural language input via text (e.g., typed), which in some cases may be converted to text and then processed.

[0003] In some cases, the automated assistant can be accessed in the vehicle via an integrated vehicle computing device, which can also provide access to other applications. Although the automated assistant can provide many benefits to the user, the user may not be fully aware of all of the automated assistant's functionality. In particular, the automated assistant can provide functionality that can assist the user with hands-free control of other applications available via the vehicle computing device. However, the user may only invoke and utilize the automated assistant when attempting to control functionality that the user believes is limited to the automated assistant, and may manually control the functionality of other applications without invoking and utilizing the automated assistant. Such manual control of the functionality of these applications may increase the amount of user input received via the vehicle computing device, which wastes the computational resources of the vehicle computing device. Furthermore, such manual control of the functionality of these applications may also increase the user's distraction while driving the vehicle, since manually interacting with the vehicle computing device while driving may distract the user from driving.

[0004] For example, a user making a call via a vehicle phone application may need to enter each digit of a phone number or each letter of a contact's name by staring and tapping at a graphical keypad rendered on the display interface of the vehicle computing device. Similarly, a user attempting to stream media from the Internet may navigate the media application by staring and tapping at GUI elements of the media application to begin playing some desired media. Thus, this manual control of the functionality of these applications not only unnecessarily consumes the computational resources of the vehicle computing device, but also increases user distraction. Summary of the Invention [Means for solving the problem]

[0005] The embodiments described herein relate to an automated assistant that may provide suggestions of assistant inputs to provide to the automated assistant to control certain other applications when the user is in the vehicle and currently controls or is expected to control certain other applications. The vehicle may include a vehicle computing device that may provide access to a variety of different applications, including an automated assistant application associated with the automated assistant. Additionally, an automated assistant accessible via the vehicle computing device may correspond to another automated assistant accessible via one or more other devices, such as a mobile computing device (e.g., a mobile phone). In some embodiments, the automated assistant may act as an interface between a user and a given vehicle application of the vehicle computing device and / or as an interface between a user and a given mobile application of the mobile computing device. In certain cases, the automated assistant may provide suggestions of assistant inputs via the vehicle computing device that the user may submit to the automated assistant, which, when submitted, may cause the automated assistant to control one or more functions of these applications. This may streamline certain interactions between the user and the applications by reducing the amount of inputs that may be used to control functions of certain applications through manual interaction with the applications. Additionally, this may reduce user distraction while driving or riding in a vehicle by allowing the user to rely on hands-free interaction with the automated assistant to control various applications while driving or riding in a vehicle.

[0006] As an example, a user may ride while interacting with the application. The application may be, for example, a medical application, and the user may interact with the medical application to schedule an appointment with his / her primary care physician. The vehicle may include a vehicle computing device, which may provide access to the automated assistant application and the medical application, and may include a display interface. The automated assistant may also be accessible via the user's mobile phone, and the mobile phone may include an automated assistant application corresponding to the automated assistant accessible via the vehicle computing device. Furthermore, the medical application may also be accessible via the user's mobile phone. Thus, with prior permission from the user, the automated assistant may identify, via the vehicle computing device and / or the user's mobile phone, that the user is interacting with the medical application during the ride, and may process the context data to provide a context-based automated assistant suggested action to the user via a display interface of the vehicle computing device. The context-based automated assistant suggested action may, for example, educate the user on how to interact with the medical application hands-free via the automated assistant.

[0007] In this example, the context data may include, for example, context data associated with the user, the user's mobile phone, and / or the vehicle computing device, such as screenshots of a medical application with which the user is interacting via the user's mobile phone and / or the vehicle computing device, a destination of the vehicle, a current location of the vehicle, an identifier of the user (with prior permission from the user), data characterizing features of recent interactions between the user and one or more applications, and / or any other relevant data that may be processed by the automated assistant. Additionally, the context data may be processed to identify one or more actions that the user is currently engaged in and / or is expected to initiate via the medical application. For example, a screenshot of the medical application may include graphics and / or text characterizing a graphical user interface (GUI) for scheduling an appointment with a doctor. The data characterizing the screenshot may be processed using one or more heuristic processes and / or one or more trained machine learning models to identify one or more actions that may be controlled via the GUI (e.g., creating an appointment, canceling an appointment, rescheduling an appointment, etc.). Based on this identification, one or more specific actions may be selected as a basis for generating suggested data, and the suggested data may characterize assistant inputs that the user may submit to the automated assistant in order to control one or more specific actions of the medical application, in lieu of the user manually interacting with the medical application.

[0008] In this example, the suggestion data generated by the automated assistant may characterize the utterance, such as, "Assistant, try saying, 'Assistant, schedule an appointment with Dr. Chow for Tuesday at 3 PM.'" The automated assistant may cause the text of the utterance to be visually rendered as a graphical element on a display interface of the vehicle computing device contemporaneously with and / or after the user interacts with the medical application. In response to the user providing the rendered utterance, the graphical element may be displayed with respect to a current interaction with the medical application and / or on behalf of future interactions with the medical application before, during, and / or after the user interacts with the medical application. In response to a request for assistance from the user, the automated assistant may notify the user that the automated assistant can perform a particular action (e.g., schedule an appointment). For example, the user may provide an utterance to the automated assistant when the user turns to view the suggestions rendered on the display interface of the vehicle computing device. Voice embodying the utterance may be received by an audio interface (e.g., one or more microphones) of the vehicle computing device and / or the mobile phone, which may cause an instance of the automated assistant to interact with the medical application. This may cause the automated assistant to have the medical application schedule an appointment for the user without the user having to manually interact with the mobile phone and / or the vehicle computing device (e.g., by directly tapping on the mobile phone's touch interface) while riding in the car.

[0009] In some implementations, assistant input suggestions may be rendered after a user interacts with the application. For example, in the above example, it is assumed that the user manually interacts with the medical application to schedule an appointment. It is further assumed that through this manual interaction, the user successfully schedules the appointment and a confirmation of the appointment is displayed via the mobile phone and / or vehicle computing device. In this example, the user may receive a suggestion from the automated assistant during the ride when the confirmation is displayed. A suggestion such as "Note: next time, you can say, 'Assistant, schedule an appointment with Dr. Chow for Tuesday at 3:00 PM.'" may characterize the utterance such that when the utterance is provided to the automated assistant in the future, it will cause the automated assistant to control a function of the medical application that was previously available to the user during the ride.

[0010] In some implementations, suggestions for assistant inputs may be rendered before the user interacts with another application. For example, a user who interacts with a particular application during a ride one day may receive suggestions from an automated assistant during a ride on another day. The suggestions may characterize an utterance such that, when the utterance is provided to the automated assistant, it causes the automated assistant to control a function of the application that the user previously had access to during a ride. For example, if the user previously accessed a podcast application and played "medical podcast" during a ride, the automated assistant may make a suggestion such as "Assistant, play medical podcast" during the ride. Alternatively or additionally, applications accessed directly via the vehicle computing device may be controlled by the automated assistant with prior permission from the user and thus may be subject to assistant suggestions. For example, a user who is expected to interact with a vehicle maintenance application of the vehicle computing device may receive suggestions for controlling a function of the vehicle maintenance application via an interface of the vehicle computing device. For example, a user may typically access a vehicle maintenance application a few minutes into a drive that exceeds a threshold distance (e.g., 100 miles) to see if there is a charging station near the destination. Based on this contextual data, the automated assistant can cause the next time the user chooses to navigate to a destination that is more than a threshold distance away (e.g., more than 100 miles away) to be rendered on the vehicle computing device with a suggestion such as, "Assistant, show me charging stations near my destination" before the user accesses the vehicle maintenance application.

[0011] In some implementations, a given suggestion for a given assistant input is rendered for presentation to a given user only a threshold number of times, which may reduce the amount of computational resources expended in generating the suggestions and may reduce user annoyance. For example, the suggestion "Note: next time you can say, 'Assistant, schedule an appointment with Dr. Chow for Tuesday at 3:00 PM'" in the above example may be presented to the user only once to train the user on automated assistant functionality. Thus, if the user subsequently initiates an interaction with the medical application via the vehicle computing device to schedule that next appointment, no suggestion may be provided. However, if the user subsequently initiates an interaction with the medical application via the mobile phone and / or vehicle computing device to cancel a previously scheduled medical appointment, an additional suggestion "Note: next time you can say, 'Assistant, cancel an appointment with Dr. Chow'" may be generated and presented to the user in the same or similar manner as described above.

[0012] In additional or alternative embodiments, a given suggestion for a given application may be rendered for presentation to a given user only a threshold number of times, which may reduce the amount of computational resources expended in generating the suggestion and may reduce user annoyance. For example, the suggestion "Note: next time you can say, 'Assistant, schedule an appointment with Dr. Chow for Tuesday at 3 PM'" in the above example may be presented to the user only once to train the user on automated assistant functionality. Furthermore, the suggestion may additionally include other suggestions related to the medical application. For example, the suggestion may additionally include the suggestion "Assistant, cancel my appointment" or "Assistant, reschedule my appointment," or any further functionality that the automated assistant may perform with respect to the medical application. In this instance, the automated assistant actively trains the user on multiple assistant inputs that may be provided to allow the automated assistant to control various functions of the medical application.

[0013] By using the techniques described herein, various technical advantages may be achieved. As one non-limiting example, the techniques described herein enable a system to provide context-relevant automated assistant action suggestion(s) in a vehicle environment to reduce computational resource consumption and / or reduce driver distraction. For example, the techniques described herein may detect an interaction with a vehicle computing device of a user's vehicle and / or a mobile computing device of the user while the user is in the vehicle. Additionally, the techniques described herein may identify context information associated with the interaction. The system may generate and provide suggestion(s) for presentation to the user when the user completes an interaction and / or when predicted to initiate an interaction based on the interaction and / or the context information associated with the interaction, enabling hands-free initiation and completion of the interaction. As a result, computational resources may be conserved by reducing at least the amount of user input to accomplish an interaction, and user distraction may be reduced by eliminating the need for user input.

[0014] The above description is provided as a summary of some embodiments of the present disclosure. Further descriptions of these and other embodiments are provided in more detail below. [Brief description of the drawings]

[0015] [Figure 1A] FIG. 1 illustrates a diagram of a user receiving suggestions for assistant inputs that can be used to instruct an automated assistant to safely interact with an application while the user is in a vehicle, according to various embodiments. [Figure 1B] FIG. 1 illustrates a diagram of a user receiving suggestions for assistant inputs that can be used to instruct an automated assistant to safely interact with an application while the user is in a vehicle, according to various embodiments. [Figure 1C]FIG. 1 illustrates a diagram of a user receiving suggestions for assistant inputs that can be used to instruct an automated assistant to safely interact with an application while the user is in a vehicle, according to various embodiments. [Figure 1D] FIG. 1 illustrates a diagram of a user receiving suggestions for assistant inputs that can be used to instruct an automated assistant to safely interact with an application while the user is in a vehicle, according to various embodiments. [Diagram 2] FIG. 1 illustrates a system for providing an automated assistant, according to various embodiments, that can generate suggestions to invoke an automated assistant to control applications that may distract a user from operating a vehicle. [Diagram 3] Various embodiments illustrate a method of providing an automated assistant that can generate suggestions for assistant inputs that a user can provide to facilitate having the automated assistant control a separate application while the user is in the vehicle. [Figure 4] FIG. 1 is a block diagram of an exemplary computer system, in accordance with various embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] 1A, 1B, 1C, and 1D respectively show scenes 100, 120, 140, and 160 in which a user 102 receives suggestions of assistant inputs that can be used to instruct an automated assistant to safely interact with an application while the user 102 is in a vehicle 108. In some implementations, the user 102 can interact with the application via a mobile or portable computing device 104 (e.g., the user's 102 mobile phone) and / or a vehicle computing device 106 in the vehicle 108. The application can be separate from the automated assistant, and the automated assistant can be accessible via the portable computing device 104 and / or the vehicle computing device 106 by respective instances of the automated assistant application running thereon. The user 102 may access each instance of the application via the portable computing device 104 while the user 102 is driving and / or riding in the vehicle 108 and / or via the vehicle computing device 106 while the user 102 is driving and / or riding in the vehicle 108. The automated assistant may act as an interface between the user 102 and the instances of the application to facilitate safe driving practices and also reduce the number of inputs that may be required to control each instance of the application.

[0017] For example, as shown in FIG. 1A, while the user 102 is driving the vehicle 108, the user 102 may access an instance of an Internet of Things (IoT) application via the portable computing device 104. In this example, the user 102 may interact with the instance of the IoT application, such as to change a temperature setting on a home thermostat, via an application interface of the IoT application. Prior to and / or during the user 102's interaction with the portable computing device 104, the automated assistant may determine that the user 102 is in the vehicle 108 and interacting with the instance of the IoT application. The automated assistant may process data from a variety of different sources to make this determination. For example, with prior permission from the user 102, data based on one or more sensors in the vehicle 108 (e.g., using facial recognition, voice signatures, touch input, etc.) may be utilized to determine that the user is in the vehicle 108. Alternatively or additionally, with prior permission from the user 102, data characterizing the screen content 110 of the portable computing device 104 may be processed to identify one or more functions of an IoT application and / or one or more application instances running in the background of the portable computing device 104 that the user 102 may control and / or attempt to control.

[0018] For example, the screen content 110 may include a GUI element for controlling the temperature in the living room of the user's 102 residence. Using one or more heuristic processes and / or one or more trained machine learning models, the screen content 110 may be processed to identify one or more different controllable features of the screen content 110. Using the identified feature(s), suggestions of assistant inputs may be generated that may be provided by the user 102 to the automated assistant to invoke the automated assistant to control the identified feature(s) of the instance of the IoT application. In some implementations, the suggestions may be generated using an application programming interface (API) for data communicated between the instance of the IoT application and the automated assistant. Alternatively or additionally, the suggestion data may be generated using content accessible through one or more interfaces of the instance of the IoT application. For example, the features of the screen content 110 may be selectable GUI elements, and the selectable GUI elements may be rendered in association with natural language content (e.g., "72 degrees"). In some implementations, a GUI element may be identified as selectable based on data available to the automated assistant, such as HTML code, XML code, Document Object Model (DOM) data, Application Programming Interface (API) data, and / or any other information that may indicate whether a particular feature of an application interface of an instance of an IoT application is selectable.

[0019] The portion of the screen content 110 occupied by the selectable GUI element may be identified along with the natural language content and utilized in generating the suggested assistant input to be presented to the user 102. In this manner, when the user 102 subsequently provides the assistant input included in the suggestion back to the automated assistant, the automated assistant may interact with the application interface of the IoT application to control the suggested functionality and / or communicate API data to the IoT application to control the suggested functionality. For example, as shown in FIG. 1B, the automated assistant may cause the display interface 122 of the vehicle computing device 106 to render assistant suggestions 124 before the user 102 accesses an instance of the IoT application, while the user 102 accesses the instance of the IoT application, and / or after the user 102 causes an action to be performed via the instance of the IoT application. For example, assume that the user 102 manually interacts with the instance of the IoT application to change the temperature in the user 102's home. After the user 102 manually interacts with the instance of the IoT application, the automated assistant may render the assistant suggestions 124 including natural language content corresponding to the suggested assistant input. A suggested assistant input (i.e., a suggested command phrase) may be, "Next time, try saying, 'Assistant, turn down the temperature in the home control application,'" which may refer to a request to the automated assistant to automatically change a temperature value managed by an instance of an IoT application (i.e., the "home control application"). For example, an assistant suggestion 124 may be stored in association with a command submitted by the automated assistant to the IoT application, such as "lowerTemp(HomeControlApp, Assistant, [0, -3 degrees])."The automated assistant may additionally or alternatively provide other suggestions associated with the instance of the IoT application, such as, for example, “(1) ‘Assistant, open the garage,’ and (2) ‘Assistant, disarm the alarm system,’” and / or any other action that is not directly related to the user's 102 manual interaction with the instance of the IoT application but that the automated assistant can perform by interfacing with the instance of the IoT application on behalf of the user 102.

[0020] In some implementations, the suggestions 124 may be presented to the user 102 on the display interface 122 of the vehicle computing device 106, as shown in FIG. 2. In additional or alternative implementations, the suggestions 124 may additionally or alternatively be presented on the display interface of the portable computing device 104. Notably, in some implementations, the user 102 may not immediately utilize the suggestions 124. For example, when the user 102 gets into the vehicle 108 and is looking at the portable computing device 104, the user 102 may see the assistant suggestions 124 provided on the display interface 122 of the vehicle computing device 106, recognize the automated assistant's ability to control IoT applications, and then put his or her phone down. In this example, the next time the user 102 wants to utilize an automated assistant to control one or more IoT devices, the user 102 may utilize the suggestions 124. Alternatively or additionally, the user 102 may see the assistant suggestions 124 during a first trip (e.g., a road trip) in the vehicle 108, but may not utilize the suggestions 124 until a second or subsequent trip in the vehicle 108. In other words, the user 102 may drive to a first destination during a first trip and be presented with the assistant suggestions 124, but may not utilize the suggestions 124 until the second trip. Thus, the user 102 may then utilize the natural language content of the suggestions 124 to cause the automated assistant to perform an action on behalf of the user 102, rather than manually interacting with various applications described herein.

[0021] For example, as shown in FIG. 1C, user 102 may provide utterance 142 corresponding to assistant suggestion 124 during the same journey as shown in FIG. 1A and FIG. 1B or during a subsequent journey (e.g., during a journey on the day following the day of FIG. 1A and FIG. 1B). Utterance 142 may be "Assistant, turn down the temperature in the living room," which may embody an invocation phrase (e.g., "Assistant") and a request or command phrase (e.g., "Turn down the temperature in the living room"). In particular, utterance 142 provided by user 102 is similar to assistant suggestion 124, which included an invocation phrase (e.g., "Assistant") and a request or command phrase (e.g., "Turn down the temperature in the home control application"), as shown in FIG. 1B. In some implementations, user 102 may provide utterance that is similar, but not exactly the same, to the natural language content of assistant suggestion 124, but may cause an action corresponding to assistant suggestion 124 to be performed. For example, context data associated with an instance when user 102 provides utterance 142 may be processed to generate a similarity value that may quantify the similarity of the context data to a previous instance (i.e., a previous context) when assistant suggestion 124 was previously rendered.

[0022] When the similarity value meets the similarity threshold, the automated assistant may respond to the utterance 142 by initiating execution of one or more actions corresponding to the assistant suggestion 124 (e.g., lowering the temperature in the living room of the user 102 associated with the home control application). For example, as shown in FIG. 1D, the automated assistant may respond to the utterance 142 by initiating execution of one or more actions that facilitate the fulfillment of the request or command phrase, and, optionally, by rendering an assistant output 162 indicating that one or more of the actions have been performed. The assistant output 162 may be an audible and / or textual output that communicates to the user that the automated assistant has fulfilled the request or command phrase. For example, the assistant output 162 may be text and / or audio that embodies the message "The living room temperature has been changed via the home control application," as shown in FIG. 1C. In some implementations, training data may be generated based on this interaction between the user 102, the automated assistant, and / or an instance of the IoT application to drive the training of one or more trained machine learning models. The one or more trained machine learning models may then be utilized in subsequent contexts to process the contextual data and provide suggestions to the user 102 that safely streamline interactions with automated assistants and / or other applications.

[0023] 1A, 1B, 1C, and 1D are described with respect to the user 102 interacting with an IoT application via the portable computing device 104, it should be understood that this is for purposes of example and not limitation. For example, manual interaction with an instance of an IoT application may occur via the vehicle computing device 106 rather than the described portable computing device 104. In this instance, suggestions 124 may be provided via the vehicle computing device 106 in the same or similar manner as described above. Also, for example, the user 102 may additionally or alternatively interact with other applications via the portable computing device 104 and / or the vehicle computing device 106, and suggestions may be generated for these other applications in the same or similar manner. For example, the user 102 may interact with an instance of a media application via the vehicle computing device 106 to cause a song to be played. In some of these embodiments, when the user 102 opens an instance of a music application, a suggestion may be audibly and / or visually presented to the user 102, for example, "Tell me a song or artist you'd like to hear." Thus, rather than navigating through instances of the media application to play music, the user 102 may simply provide an indication of the song or artist he or she would like to hear. In other embodiments, the suggestion may not be provided until the user manually selects the song or artist he or she would like to hear and the song or artist is played.

[0024] FIG. 2 illustrates a system 200 that provides an automated assistant 204 that may provide suggestions to invoke the automated assistant 204 to control applications that a user may access from a vehicle (e.g., via the portable computing device 104 and / or via the vehicle computing device 106 of FIGS. 1A-1D). The automated assistant 204 may operate as part of an automated assistant application provided on one or more computing devices, such as the computing device 202 and / or a server device. A user may interact with the automated assistant 204 via the automated assistant interface(s) 220, which may be a microphone, a camera, a touchscreen display, a user interface, and / or any other device capable of providing an interface between a user and an application. For example, a user may initiate the automated assistant 204 by providing verbal, textual, gestural, and / or graphical input to the assistant interface 220 to cause the automated assistant 204 to initiate one or more actions (e.g., providing data, controlling a peripheral device, accessing an agent, generating input and / or output, and / or other actions). Alternatively, the automated assistant 204 may be initiated based on processing of the context data 236 using one or more trained machine learning models and / or heuristic processes. The context data 236 may characterize one or more features of an environment in which the automated assistant 204 has access (e.g., one or more features of the computing device 202 and / or one or more features of the vehicle in which the user is located) and / or one or more features where the user is predicted to intend to interact with the automated assistant 204. The computing device 202 may include a display device, which may be a display panel including a touch interface for receiving touch inputs and / or gestures that enable a user to control the applications 234 of the computing device 202 via a touch interface.In some implementations, the computing device 202 may not have a display device, and thus may not provide graphical user interface output, but may provide audible user interface output. Additionally, the computing device 202 may provide a user interface, such as a microphone, to receive spoken natural language input from a user. In some implementations, the computing device 202 may include a touch interface and may not have a camera, but may optionally include one or more other sensors.

[0025] The computing device 202 and / or other third party client devices (e.g., third party client devices provided by an entity in addition to the computing device 202 and / or the entity providing the automated assistant 204) may communicate with a server device over a network such as the Internet. Additionally, the computing device 202 and any other computing devices may communicate with each other over a local area network (LAN), such as a Wi-Fi network or a Bluetooth network. The computing device 202 may offload computational tasks to a server device to conserve computational resources at the computing device 202. For example, the server device may host the automated assistant 204 and / or the computing device 202 may send inputs received at one or more assistant interfaces 220 to the server device. However, in some implementations, the automated assistant 204 may be hosted at the computing device 202 and various processes that may be associated with automated assistant operations may be executed at the computing device 202.

[0026] In various implementations, all or less than aspects of the automated assistant 204 may be implemented on the computing device 202. In some of these implementations, aspects of the automated assistant 204 may be implemented via the computing device 202 and interface with a server device, which may implement other aspects of the automated assistant 204. The server device may optionally provide services to multiple users and their associated automated assistant applications via multiple threads. In implementations in which all or less than aspects of the automated assistant 204 are implemented via the computing device 202, the automated assistant 204 may be an application that is separate from the operating system of the computing device 202 (e.g., installed "on top of" the operating system) or may be implemented directly by the operating system of the computing device 202 (e.g., may be considered an operating system application but integral to the operating system).

[0027] In some implementations, the automated assistant 204 may include an input processing engine 206, which may process input and / or output of the computing device 202 and / or the server device using a number of different modules. For example, the input processing engine 206 may include a speech processing engine 208, which may process audio data received at the assistant interface 220 to identify text corresponding to speech embodied in the audio data. To conserve computational resources at the computing device 202, the audio data may be transmitted from the computing device 202 to a server device, for example. Additionally or alternatively, the audio data may be processed only at the computing device 202.

[0028] The process for converting audio data to text may include automatic speech recognition (ASR) algorithms, which may use neural networks and / or statistical models to identify groups of audio data that correspond to words or phrases. The text converted from the audio data (or text received as text input) may be analyzed by the data analysis engine 210 and made available to the automated assistant 204 as text data that may be used to generate and / or identify command phrase(s), intent(s), action(s), slot value(s), and / or any other content specified by the user. In some implementations, the output data provided by the data analysis engine 210 may be provided to the parameter engine 212 to determine whether the user has provided input that corresponds to a particular intent, action, and / or routine that may be executed by the automated assistant 204 and / or an application or agent accessible via the automated assistant 204. For example, the assistant data 238 may be stored on the server device and / or the computing device 202 and may include data defining one or more actions that the automated assistant 204 may execute, as well as parameters necessary to execute the action. The parameter engine 212 may generate one or more parameters for the intent, action, and / or slot value and provide the one or more parameters to the output generation engine 214. The output generation engine 214 may use the one or more parameters to communicate with the assistant interface 220 to provide output to the user and / or to communicate with one or more applications 234 to provide output to the one or more applications 234.

[0029] The automated assistant application includes and / or has access to on-device ASR, on-device natural language understanding (NLU), and on-device fulfillment. For example, the on-device ASR may be performed using an on-device ASR module that processes audio data (detected by microphone(s)) using, for example, an end-to-end speech recognition machine learning model stored locally on the computing device 202. The on-device ASR module generates recognized text for utterances (if any) present in the audio data. Additionally, the on-device NLU may be performed using an on-device NLU module that processes the recognized text generated using the on-device speech recognition and, optionally, the context data to generate NLU data. The NLU data may include intent(s) corresponding to the utterance and, optionally, parameter(s) of the intent(s) (e.g., slot values). Further, on-device fulfillment may be performed using an on-device fulfillment module that utilizes the NLU data (from the on-device NLU module) and optionally other local data to identify action(s) to take (and optionally parameter(s) of the intent) to resolve the intent(s) of the utterance. This may include identifying local and / or remote responses (e.g., answers) to the utterance, interaction(s) to perform with locally installed application(s) based on the utterance, command(s) to send (directly or via corresponding remote system(s)) to Internet of Things (IoT) device(s) based on the utterance, and / or other resolution action(s) to perform based on the utterance. The on-device fulfillment module may then initiate local and / or remote performance / execution of the identified action(s) to resolve the utterance.

[0030] In various implementations, remote ASR, remote NLU, and / or remote fulfillment may be utilized, at least selectively. For example, recognized text generated by an on-device ASR module may be at least selectively sent to a remote automated assistant component(s) for remote NLU and / or remote fulfillment. For example, recognized text generated by an on-device ASR module may be optionally sent for remote performance in parallel with on-device performance or in response to a failure of on-device NLU and / or on-device fulfillment. However, on-device ASR, on-device NLU, on-device fulfillment, and / or on-device execution may be preferred, at least due to reduced latency in resolving utterances (since no client-server roundtrip(s) are required to resolve the utterance). Additionally, on-device functionality may be the only functionality available in situations where there is no or limited network connectivity.

[0031] In some implementations, the computing device 202 may include one or more applications 234, which may be provided by the same first party entity as the entity that provided the computing device 202 and / or the automated assistant 204, and / or one or more applications 234, which may be provided by a third party entity different from the entity that provided the computing device 202 and / or the automated assistant 204. An application state engine (not shown) of the automated assistant 204 and / or the computing device 202 may access the application data 230 to identify one or more actions that may be performed by the one or more applications 234, as well as the state of each of the one or more applications 234, and / or the state of each device associated with the one or more applications 234. A device state engine (not shown) of the automated assistant 204 and / or the computing device 202 may access the device data 232 to identify one or more actions that may be performed by the computing device 202 and / or one or more devices associated with the computing device 202 and / or the one or more applications 234. Additionally, the application data 230 and / or any other data (e.g., device data 232) may be accessed by the automated assistant 204 to generate context data 236, which may characterize the context in which a particular application of the one or more applications 234 and / or device is being executed and / or the context in which a particular user is accessing the computing device 202, the particular application of the one or more applications 234 and / or any other device or module.

[0032] While one or more applications 234 are executing on the computing device 202, the device data 232 may characterize a current operational state of each of the one or more applications 234 executing on the computing device 202. Additionally, the application data 230 may characterize one or more features of the executing one or more applications 234, such as the content of one or more graphical user interfaces rendered at the direction of the one or more applications 234. Alternatively or additionally, the application data 230 may characterize action schemas, which may be updated by each of the one or more applications 234 and / or by the automated assistant 204 based on the current operational state of the respective applications. Alternatively or additionally, the one or more action schemas of the one or more applications 234 may be static, but may be accessed by the application state engine to identify actions suitable for initiation via the automated assistant 204.

[0033] Computing device 202 may further include assistant invocation engine 222, which may use one or more trained machine learning models to process input received via assistant interface 220, application data 230, device data 232, context data 236, and / or any other data accessible by computing device 202. Assistant invocation engine 222 may process this data to determine whether to wait for the user to explicitly say an invocation phrase to invoke automated assistant 204 via assistant interface 220, or whether to consider the data to indicate a user's intent to invoke an automated assistant instead of requiring the user to explicitly say an invocation phrase to invoke automated assistant 204. For example, one or more trained machine learning models may be trained using instances of training data based on a scenario in which a user is in an environment in which multiple devices and / or applications exhibit various operating states. Instances of training data may be generated to capture training data that characterize contexts in which a user invokes an automated assistant and other contexts in which the user does not invoke automated assistant 204. When one or more trained machine learning models are trained according to these training data instances, the assistant invocation engine 222 allows the automated assistant 204 to detect or limit detection of invocation phrases uttered by the user based on features of the context and / or environment.

[0034] In some implementations, the system 200 may further include an interaction analysis engine 216, which may process various data to determine whether an interaction between a user and an application should be the subject of an assistant suggestion. For example, the interaction analysis engine 216 may process data indicating the number of user inputs that a user has provided to the application 234 to cause the application 234 to perform a particular action while the user is in his or her vehicle. Based on this processing, the interaction analysis engine 216 may determine whether the automated assistant 204 may perform a particular action even with less input from the user. For example, a user who switches a corresponding instance of a given application on a portable computing device to render a particular media on the vehicle computing device may be able to cause the automated assistant 204 to render the particular media by uttering a particular utterance (e.g., one input).

[0035] Further, based on the interaction analytics engine 216 determining that a particular action can be initiated via the automated assistant 204 with less input, the vehicle context engine 218 can identify when to render a corresponding assistant suggestion to the user. For example, the vehicle context engine 218 can process data from one or more different sources to identify an appropriate time, location, and / or computing device for rendering an assistant suggestion to the user. For example, the application data 230 can be processed to determine whether the user switched applications on the vehicle computing device and / or portable computing device to prompt for rendering a particular media. In response to this determination, the vehicle context engine 218 may determine that this instance of the switching application is suitable for rendering an assistant suggestion, which relates to invoking an automated assistant to interact with a particular application (e.g., an application that renders media via a suggestion such as "next time, just say 'open my media application'") to perform a particular action. Alternatively or additionally, data from one or more different sources may be processed using one or more heuristic processes and / or one or more trained machine learning models to identify times, locations, and / or computing devices suitable for rendering a particular assistant suggestion. The data may be periodically and / or responsively processed to generate an embedding or other low-dimensional representation, which may be mapped into a latent space to which other existing embeddings were previously mapped. When a generated embedding is identified as being a threshold distance away from an existing embedding in the latent space, an assistant suggestion corresponding to the existing embedding may be rendered to the user.

[0036] In some implementations, the system 200 may further include an assistant suggestion engine 226, which may generate assistant suggestions based on interactions between a user and one or more applications and / or devices. For example, the application data 230 may be processed by the assistant suggestion engine 226 to identify one or more actions that may be initiated by a particular application of the one or more applications 234. Alternatively or additionally, the interaction data may be processed to determine whether a user previously initiated execution of one or more actions of a particular application of the one or more applications 234. The assistant suggestion engine 234 may determine whether the automated assistant 204 may initiate one or more actions in response to an assistant input sent from a user to the automated assistant 204. For example, an API of a particular application of the one or more applications 234 may be accessed by the automated assistant 204 to determine whether the automated assistant 204 may utilize API commands to control a particular action of a particular application of the one or more applications 234. When the automated assistant 204 identifies an API command for controlling a particular action that may be of interest to a user, the assistant suggestion engine 226 may generate a request or command phrase and / or other assistant input that may be rendered as an assistant suggestion to the user. For example, based on identifying that an API call (e.g., playMedia(Podcast Application, 1, resume_playback()) is available to the automated assistant to control a particular application feature of interest to the user, the automated assistant 204 may generate the suggestion "Assistant, play my recent podcasts," where "Assistant" corresponds to the invocation phrase of the automated assistant 204 and "Play my recent podcasts" corresponds to the request or command phrase that, when detected, causes the automated assistant 204 to perform an action on behalf of the user.

[0037] In some implementations, the automated assistant 204 may render assistant suggestions while the user interacts with one or more applications 234, before the user interacts with one or more applications 234, in response to the completion of a particular application action via one or more applications 234, and / or before the completion of a particular application action via one or more applications 234. In some implementations, the assistant suggestions may be generated and / or rendered based on whether the user is the owner of the vehicle, the driver of the vehicle, a passenger in the vehicle, a renter of the vehicle, and / or any other person that may be associated with the vehicle. The automated assistant 204 may determine whether a user in the vehicle belongs to one or more of these categories using any known technology (e.g., speaker identification, face identification, password identification, fingerprint identification, and / or other technology). For example, the automated assistant 204 may render more personalized assistant suggestions to the owner when the owner is driving the vehicle and other assistant suggestions to passengers and / or renters of the vehicle that may be useful to a broader audience. The automated assistant 204 may operate such that a user's identified familiarity with various automated assistant features may be indirectly proportional to the frequency with which assistant suggestions are rendered to that user. For example, a given suggestion may not be subsequently presented to a user of a vehicle if the suggestion has already been presented to the user, but may be subsequently presented to another user of the vehicle (e.g., a renter of the vehicle).

[0038] In some implementations, system 200 may further include a suggestion training engine 224 that may be utilized to generate training data that may be used to train one or more different machine learning models that may be used to provide assistant suggestions to different users associated with the vehicle. Alternatively or additionally, suggestion training engine 224 may generate training data based on whether a particular user interacted with rendered assistant suggestions. In this manner, one or more models may be further trained to provide assistant suggestions that are deemed relevant to the user and to avoid distracting the user with an excess of suggestions that the user may not be interested in and / or that the user may already be aware of.

[0039] In some implementations, the assistant suggestions may embody command phrases that, when detected by the automated assistant 204, cause the automated assistant 204 to interact with one or more applications 234 to perform one or more different actions and / or routines. Alternatively or additionally, multiple different assistant suggestions may be rendered simultaneously for a particular one of the one or more applications 234 and / or for multiple applications of the one or more applications 234, thereby allowing a user to learn multiple different command phrases when controlling one or more applications 234. In some implementations, assistant suggestions may be rendered more for renters of the vehicle compared to the owner of the vehicle. The decision to render more assistant suggestions for renters and / or other guests may be based on how frequently the owner of the vehicle utilizes the automated assistant 204 via the vehicle computing device. In other words, the assistant suggestion engine 226 may display fewer assistant suggestions for users who frequently use the features of the automated assistant 204 and display more assistant suggestions for users who infrequently use the features of the automated assistant 204 (users who have received prior permission from the user). As noted above, vehicle owners may be distinguished from other users using a variety of techniques.

[0040] FIG. 3 illustrates a method 300 for providing an automated assistant that may generate suggestions for assistant inputs that a user may provide to facilitate having an automated assistant control an application while the user is in the vehicle. The method 300 may be performed by one or more computing devices, applications, and / or any other devices or modules that may be associated with an automated assistant. The method 300 may include an operation 302 of determining whether a user is in the vehicle. The vehicle may be, for example, a vehicle that includes a vehicle computing device that may provide access to an automated assistant. The automated assistant may identify that a user is in the vehicle based on data from one or more different sources, such as the vehicle computing device, a portable computing device (e.g., a computer that a user may bring in the vehicle), and / or any other device that may communicate with the automated assistant. For example, the device data may indicate that a user has entered the vehicle based at least on the device data indicating that a portable computing device (e.g., a mobile phone) owned by the user has been paired (e.g., via a Bluetooth protocol) with the vehicle computing device. Alternatively or additionally, the automated assistant may identify that a user is in the vehicle based on context data that may characterize the user's context at a particular time. For example, when the automated assistant determines whether the user is in the vehicle, the calendar entry may indicate that the user is driving to class. Based on this indication, the automated assistant may use one or more heuristic processes and / or one or more trained machine learning models to determine whether the user is likely to be in the vehicle (e.g., with a threshold probability). Alternatively or additionally, one or more sensors of the vehicle connected to the vehicle computing device of the vehicle may generate sensor data indicating that the user is in the vehicle (e.g., via an occupancy sensor).

[0041] Method 300 may proceed from operation 302 to operation 304, which may include processing contextual data associated with the user and / or the vehicle. In some implementations, the contextual data may include an identifier for the user (e.g., username, user account, and / or any other identifier), an identifier for the vehicle (e.g., vehicle type, vehicle name, vehicle original equipment manufacturer (OEM), etc.), and / or a location of the vehicle. Alternatively or additionally, the contextual data may include device data, application data, and / or any other data that may be indicative of devices and / or applications with which the user has recently interacted, is currently interacting, and / or is expected to interact. Alternatively or additionally, the application data may indicate that the user has recently established a habit of accessing a real estate application each morning to view the “recent listings” application page.

[0042] From operation 304, the method 300 may proceed to operation 306, which determines whether the user is interacting or is predicted to interact with a particular application and / or application functionality. If the user is not identified or predicted to interact with a particular application, the method 300 may proceed from operation 306 to optional operation 308, which updates the training data. The training data may be updated to reflect a state in which the user is not interacting with the automated assistant and / or the particular application functionality while in the vehicle. In this manner, subsequent processing of the contextual data using one or more trained machine learning models trained with the updated training data may provide more relevant and / or more useful suggestions to the user. Alternatively, if the user is identified or predicted to interact with a particular application functionality, the method 300 may proceed from operation 306 to operation 310.

[0043] Operation 310 may include generating suggestion data that characterizes an assistant input for controlling a particular application feature. For example, the suggestion data may be generated using an application programming interface (API), which may enable the automated assistant to submit executable requests to and / or receive executable requests from a particular application. Alternatively or additionally, the automated assistant may process HTML code, XML code, document object model (DOM) data, application programming interface (API) data, and / or any other information that may indicate whether a particular feature of an application is controllable. Based on this determination, the automated assistant may generate one or more suggestions for an utterance that, when subsequently submitted by a user, causes the automated assistant to control one or more features of the particular application. For example, the automated assistant may determine that accessing a "recent listings" page of a real estate application includes opening the real estate application and selecting a selectable GUI element labeled "recent listings." Based on this identification, the automated assistant may generate one or more executable requests to the real estate application and text content corresponding to the utterance, which, when provided by the user, causes the automated assistant to submit the executable requests to the real estate application. The one or more requests and / or text content may then be stored as suggestion data, and the suggestion data may be used to render suggestions to one or more users. When the user provides the utterance, the automated assistant provides the one or more requests to the real estate application. The real estate application may respond by generating a "recent listings" page, and the automated assistant may cause the "recent listings" page to be rendered on a display interface of the vehicle computing device. Alternatively or additionally, the automated assistant may render an audible output characterizing the content of the "recent listings" page.

[0044] Method 300 may proceed from operation 310 to operation 312, which may include causing the text of the utterance to be rendered on a display interface of the vehicle computing device and / or another computing device associated with the user (e.g., a portable computing device associated with the user). For example, according to the above-described embodiment, the automated assistant may cause a suggested GUI element (optionally selectable) to be rendered on a display interface of the vehicle computing device while the user is in the vehicle. The suggested GUI element may be rendered with natural language content characterizing the utterance, such as "Assistant, show me my recent list." In some implementations, the automated assistant may render the request suggestion without a specific identifier of the application to be controlled. Rather, the suggestion may be generated such that the user can necessarily correlate the suggestion with the context of the suggestion to infer the application to be controlled. In some implementations, the user may initiate communication of a request from the automated assistant to the real estate application by providing an utterance and / or tapping a touch interface of the vehicle computing device. For example, a display interface of the vehicle computing device may respond to the touch input, and / or one or more buttons (e.g., buttons on a steering wheel), switches, and / or other interfaces of the vehicle computing device may respond to the touch input. One or more of these types of touch inputs may be utilized to initiate the execution of one or more actions corresponding to one or more suggestions from the automated assistant.

[0045] Method 300 may proceed from operation 312 to operation 314, which determines whether the user has provided an assistant input to prompt for controlling an application function. For example, a user may provide an utterance such as "Assistant, show me recent listings" to prompt for having an automated assistant control a real estate application. If the automated assistant receives the assistant input, method 300 may proceed from operation 314 to operation 316, which may include having the automated assistant control a particular application function. Alternatively, if the user does not provide an assistant input corresponding to a suggestion from the automated assistant, method 300 may proceed from operation 314 to optional operation 308.

[0046] In some implementations, when the user provides an assistant input corresponding to the suggestion, the method 300 may proceed from operation 316 to optional operation 318. Optional operation 318 may include updating training data based on the user utilizing the automated assistant to control a particular application function. For example, training data may be generated based on the user providing the assistant input, and the training data may be utilized to train one or more trained machine learning models. The one or more trained machine learning models may then be used to process the contextual data when the user or another user is identified as being in the vehicle. This further training of the one or more trained machine learning models may enable the automated assistant to provide more relevant and / or effective suggestions for the user to invoke the automated assistant to control one or more separate applications while promoting safe driving habits.

[0047] 4 is a block diagram 400 of an exemplary computer system 410. The computer system 410 typically includes at least one processor 414 that communicates with a number of peripheral devices via a bus subsystem 412. These peripheral devices may include a storage subsystem 424, including, for example, memory 425 and a file storage subsystem 426, a user interface output device 420, a user interface input device 422, and a network interface subsystem 416. The input and output devices enable user interaction with the computer system 410. The network interface subsystem 416 provides an interface to external networks and is connected to corresponding interface devices in other computer systems.

[0048] The user interface input devices 422 may include a keyboard, a pointing device (e.g., a mouse, trackball, touchpad, or graphics tablet), a scanner, a touch screen integrated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computer system 410 or communications network.

[0049] The user interface output devices 420 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computer system 410 to a user or to another machine or computer system.

[0050] Storage subsystem 424 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 424 may include logic for performing selected aspects of method 300 and / or logic for implementing one or more of system 200, vehicle computing device 106, portable computing device 104, and / or any other applications, assistants, devices, apparatus, and / or modules discussed herein.

[0051] These software modules are typically executed by the processor 414 alone or in combination with other processors. The memory 425 used by the storage subsystem 424 may include a number of memories, including a main random access memory (RAM) 430 for storing instructions and data during program execution, and a read-only memory (ROM) 432 in which fixed instructions are stored. The file storage subsystem 426 may provide persistent storage of program files and data files, and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of a particular embodiment may be stored by the file storage subsystem 426 in the storage subsystem 424, or on another machine accessible to the processor(s) 414.

[0052] Bus subsystem 412 provides a mechanism that allows the various components and subsystems of computer system 410 to communicate with each other as intended. Although bus subsystem 412 is shown generally as a single bus, in alternative implementations of the bus subsystem, multiple buses may be used.

[0053] Computer system 410 can be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Because computers and networks are constantly changing, the description of computer system 410 shown in Figure 4 is intended only as a specific example to illustrate some implementations. Many other configurations of computer system 410 can have more or fewer components than the computer system shown in Figure 4.

[0054] In situations where the systems described herein collect or may use personal information about users (or "participants" as they are often referred to herein), the user may be provided with an opportunity to control whether a program or function collects user information (e.g., information about the user's social networks, social actions or activities, occupation, user preferences, or the user's current geographic location) or whether and / or how content that may be more relevant to the user is received from a content server. Also, certain data may be processed in one or more ways such that personally identifiable information is removed before it is stored or used. For example, the user's identity may be processed such that personally identifiable information is not identified about the user, or if geographic location information (such as to the city, zip code, or state level) is obtained, the user's geographic location may be generalized such that the user's specific geographic location cannot be identified. Thus, the user may control how information about the user is collected and / or used.

[0055] Although several embodiments have been described and illustrated herein, various other means and / or structures may be utilized to perform the functions and / or obtain the results and / or one or more of the advantages described herein, and each such variation and / or modification is deemed to be within the scope of the embodiments described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be illustrative, and the actual parameters, dimensions, materials, and / or configurations will depend on the particular application(s) in which the teachings are used. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific embodiments described herein. Thus, it should be understood that the foregoing embodiments are presented by way of example only, and that within the scope of the appended claims and their equivalents, embodiments may be practiced other than as specifically described and claimed. The embodiments of the present disclosure are directed to each individual function, system, product, material, kit, and / or method described herein. Furthermore, any combination of two or more such features, systems, articles of manufacture, materials, kits, and / or methods is within the scope of the disclosure, unless such features, systems, articles of manufacture, materials, kits, and / or methods are mutually inconsistent.

[0056] In some embodiments, a method implemented by one or more processors is provided, the method including identifying, via a mobile computing device present in the vehicle with the user, that a user is engaged in an interaction with a given mobile application. The given mobile application is separate from an automated assistant application, the automated assistant application being accessible via the mobile computing device and a vehicle computing device of the vehicle. The method further includes generating suggestion data based on the interaction with the given mobile application, the suggestion data characterizing command phrases, the command phrases, when submitted by the user to the automated assistant application, causing the automated assistant application to control a particular operation of the given vehicle application. The given vehicle application corresponds to the given mobile application. The method further includes causing the command phrases to be visually rendered in the foreground of a display interface of the vehicle computing device based on the suggestion data.

[0057] These and other implementations of the technology disclosed herein can optionally include one or more of the following features.

[0058] In some implementations, generating the suggestion data may be performed on a mobile computing device, and the method may further include providing the suggestion data to the vehicle computing device in response to an interaction with the given mobile application.

[0059] In some implementations, generating the suggestion data may be performed at a vehicle computing device and the method may further include receiving, from the given mobile application, interaction data characterizing an interaction between a user and the given mobile application. The suggestion data may be generated further based on the interaction data.

[0060] In some implementations, the command phrases may be visually rendered as selectable graphical user interface (GUI) elements on a display interface, and the selectable GUI elements may be selectable via touch input received at an area of ​​the display interface corresponding to the selectable GUI elements.

[0061] In some implementations, generating suggestion data characterizing command phrases may include generating command phrases based on one or more application actions that may be initiated through direct interaction between a user and a GUI interface of a given mobile application, the GUI interface may be rendered on a mobile computing device, and the one or more application actions may include a particular action.

[0062] In some implementations, generating the suggestion data characterizing the command phrases may include generating the command phrases based on one or more application actions initiated by the user via a given mobile application of the mobile computing device during one or more previous instances when the user was in the vehicle. The one or more application actions may include a particular action.

[0063] In some embodiments, a method implemented by one or more processors is provided, the method including generating prediction data, the prediction data indicating that a user is predicted to interact with an application interface of a given application to control a function of the given application via a display interface of a vehicle computing device, the given application being separate from an automated assistant application accessible via a vehicle computing device of the vehicle. The method further includes generating suggestion data characterizing command phrases based on the prediction data, the command phrases, when submitted by the user to the automated assistant application, causing the automated assistant application to control a function of the given application; causing at least the command phrases to be rendered at a display interface of the vehicle computing device before the user interacts with the function of the application; and receiving an assistant input from the user to the automated assistant application including at least the command phrase in response to at least the command phrases being rendered at the display interface of the vehicle computing device, and causing the automated assistant application to control the function of the given application based on the assistant input based on the receiving the assistant input.

[0064] These and other implementations of the technology disclosed herein can optionally include one or more of the following features.

[0065] In some implementations, the command phrases may be rendered as selectable graphical user interface (GUI) elements on a display interface, and the assistant input may be touch input received at an area of ​​the display interface corresponding to the selectable GUI element. In some versions of these implementations, generating the prediction data may include identifying that the vehicle is being driven toward a location where a user is predicted to interact with an application interface to control a function of the application. The suggestion data may further be based on a location to which the vehicle is headed.

[0066] In some implementations, the method may further include determining that an application interface of the given application is being rendered on a display interface of the vehicle computing device. In response to determining that the application interface is being rendered on the display interface of the vehicle computing device, generating the predictive data may be performed.

[0067] In some implementations, the method may further include determining that the user is currently in the vehicle and determining that the user accessed an application interface of the given application during a previous instance when the user was in the vehicle. Generating the prediction data may be performed based on determining that the user is currently in the vehicle and that the user previously accessed the application interface of the given application.

[0068] In some implementations, the method may further include processing the context data using one or more trained machine learning models. The context data may characterize one or more features of the user's context, and generating the predictive data may be performed based at least on processing the context data. In some versions of these implementations, the one or more trained machine learning models may be trained using data generated during one or more previous instances in which one or more other users accessed functionality of the given application while in their respective vehicles.

[0069] In some embodiments, a method implemented by one or more processors is provided, the method including identifying, by a vehicle computing device of a vehicle, while the user is in the vehicle, that the user is engaged in an interaction with a given application of the vehicle computing device. The given application is separate from an automated assistant application accessible via the vehicle computing device. The method further includes generating, by the vehicle computing device, suggestion data based on the interaction with the given application, the suggestion data characterizing a command phrase that, when submitted by the user to the automated assistant application, causes the given application to perform a particular action associated with the interaction between the user and the given application, and causing the vehicle computing device to visually render, on a display interface of the vehicle computing device, the suggestion data. The command phrase is rendered in the foreground of the display interface of the vehicle computing device.

[0070] These and other implementations of the technology disclosed herein can optionally include one or more of the following features.

[0071] In some implementations, the command phrase may include an invocation phrase for invoking the automated assistant application and natural language content characterizing a request to perform a particular action. In some versions of these implementations, the method may further include receiving, by the vehicle computing device, an utterance embodying a request to the automated assistant application, identifying, by the vehicle computing device, that the utterance includes a request rendered on a display interface of the vehicle computing device, and, in response to receiving the utterance, causing the given application to perform a particular action associated with an interaction between the user and the given application. In some further versions of these implementations, the command phrase may be rendered when the user is in the vehicle during the first outing, and the utterance may be received when the user is in the vehicle during the second outing.

[0072] In some implementations, the method may further include identifying certain actions as controllable via selectable content rendered at a display interface of the vehicle computing device. Generating suggestion data characterizing command phrases may be based on identifying certain actions as controllable via selectable content rendered at a display interface of the vehicle computing device. In some versions of these implementations, the application may be a communications application and the selectable content includes one or more selectable elements for specifying a phone number and / or contact to call. In additional or alternative versions of these implementations, the application may be a media application and the selectable content includes one or more selectable elements for specifying media to be visually rendered via a display interface of the vehicle computing device and / or audibly rendered via one or more speakers of the vehicle computing device.

[0073] In some implementations, causing the suggested data to be rendered on a display interface of the vehicle computing device may be in response to a user identifying that they have completed interaction with a given application of the vehicle computing device.

[0074] In some implementations, generating the suggestion data characterizing the command phrases may be further based on a context of a user engaged in an interaction with a given application of the vehicle computing device. In some versions of these implementations, generating the suggestion data characterizing the command phrases may be further based on a display context of a display interface of the vehicle computing device.

[0075] In some implementations, the method may further include, after the command phrase is rendered at the display interface of the vehicle computing device, identifying, by the vehicle computing device, while the user is in the vehicle, via the display interface of the vehicle computing device, that the user is engaged in a separate interaction with an additional application of the vehicle. The additional application may be separate from the automated assistant application and may be separate from the given application. The method may further include generating, by the vehicle computing device, additional suggestion data characterizing additional command phrases based on the separate interaction with the additional application, the additional command phrases, which when submitted by the user to the automated assistant application, cause the automated assistant application to control additional operations of the additional application, and causing, by the vehicle computing device, the additional suggestion data to be visually rendered at the display interface of the vehicle computing device. In some versions of these implementations, causing the additional suggestion data to be rendered at the display interface of the vehicle computing device may be performed in response to identifying that the user has completed a separate interaction with the additional application of the vehicle computing device.

[0076] Other embodiments may include a non-transitory computer-readable storage medium storing instructions executable by one or more processors (e.g., central processing unit (CPU)(s), graphics processing unit (GPU)(s), and / or tensor processing unit (TPU)(s)) to perform methods, such as one or more of the methods described above and / or elsewhere herein. Still other embodiments may include one or more computer systems including one or more processors operable to execute the stored instructions to perform methods, such as one or more of the methods described above and / or elsewhere herein.

[0077] It should be understood that all combinations of the foregoing concepts and additional concepts described in more detail herein are contemplated as being part of the subject matter disclosed herein, for example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein.

Claims

1. 1. A method implemented by one or more processors, comprising: Identifying that a user is engaged in an interaction with a given mobile application via a mobile computing device present in a vehicle with the user, comprising: the given mobile application is separate from an automated assistant application; the automated assistant application is accessible via the mobile computing device and a vehicle computing device of the vehicle; The identifying, generating suggestion data characterizing command phrases based on the interaction with the given mobile application, the command phrases, when submitted by the user to the automated assistant application, cause the automated assistant application to control a particular operation of a given vehicle application; the given vehicle application corresponds to the given mobile application; The generating. causing the command phrase to be visually rendered in the foreground of a display interface of the vehicle computing device based on the suggestion data; A method comprising:

2. The generating of the suggestion data is performed on the mobile computing device, the method further comprising: providing the recommendation data to the vehicle computing device in response to the interaction with the given mobile application; The method of claim 1 , comprising:

3. The generating of the suggestion data is performed at the vehicle computing device, the method further comprising: receiving, from the given mobile application, interaction data characterizing the interaction between the user and the given mobile application; The suggestion data is generated further based on the interaction data. said receiving; The method of claim 1 , comprising:

4. 4. The method of claim 1, wherein the command phrases are visually rendered as selectable graphical user interface (GUI) elements on the display interface, the selectable GUI elements being selectable via touch input received at areas of the display interface corresponding to the selectable GUI elements.

5. Generating the suggestion data characterizing the command phrases includes: generating the command phrases based on one or more application actions that may be initiated via direct interaction between the user and a GUI interface of the given mobile application, the GUI interface is rendered on the mobile computing device, and the one or more application actions include the particular action. The generating, The method of any one of claims 1 to 4, comprising:

6. Generating the suggestion data characterizing the command phrases includes: generating the command phrase based on one or more application actions initiated by the user via the given mobile application on the mobile computing device during one or more previous instances when the user was within the vehicle; The one or more application actions include the specific action. The generating, The method of any one of claims 1 to 4, comprising:

7. 1. A method implemented by one or more processors, comprising: generating prediction data indicative of a user being predicted to interact with an application interface of a given application to control functionality of the given application via a display interface of the vehicle computing device; The given application is separate from an automated assistant application accessible via the vehicle computing device of the vehicle. The generating. generating suggestion data characterizing command phrases based on the prediction data, the command phrase, when submitted by the user to the automated assistant application, causes the automated assistant application to control the functionality of the given application; The generating. causing at least the command phrase to be rendered on the display interface of the vehicle computing device before the user interacts with the functionality of the application; in response to at least the command phrase being rendered on the display interface of the vehicle computing device; receiving an assistant input from the user to the automated assistant application, the assistant input including at least the command phrase; Based on receiving the assistant input, causing the automated assistant application to control the function of the given application based on the assistant input; A method comprising:

8. the command phrases are rendered as selectable graphical user interface (GUI) elements on the display interface; the assistant input is a touch input received at an area of ​​the display interface corresponding to the selectable GUI element; The method of claim 7.

9. Generating the prediction data includes: determining that the vehicle is being driven to a location where the user is expected to interact with the application interface to control the functionality of the application; The suggestion data is further based on the location to which the vehicle is heading. The identifying, The method of claim 8 , comprising:

10. determining that the application interface of the given application is being rendered on the display interface of the vehicle computing device, generating the predictive data in response to identifying the application interface being rendered on the display interface of the vehicle computing device. The identifying, The method of claim 7 or 8, further comprising:

11. determining that the user is currently within the vehicle; determining that the user accessed the application interface of the given application during a previous instance while the user was in the vehicle; generating the predictive data based on identifying that the user is currently in the vehicle and that the user has previously accessed the application interface of the given application; The identifying, The method of any of claims 7 to 10, further comprising:

12. Processing the context data using one or more trained machine learning models, the context data characterizing one or more features of the user's context; generating the predictive data based at least on processing the context data; said processing, The method of any of claims 7 to 11, further comprising:

13. 13. The method of claim 12, wherein the one or more trained machine learning models are trained using data generated during one or more previous instances when one or more other users accessed the functionality of the given application while in their respective vehicles.

14. 1. A method implemented by one or more processors, comprising: determining, by a vehicle computing device of a vehicle, that a user is engaged in an interaction with a given application of the vehicle computing device while the user is within the vehicle, comprising: the given application is separate from an automated assistant application accessible via the vehicle computing device; The identifying, generating, by the vehicle computing device, suggestion data characterizing command phrases based on the interaction with the given application; the command phrase, when submitted by the user to the automated assistant application, causes the given application to perform a particular action associated with the interaction between the user and the given application; The generating. causing the vehicle computing device to visually render the suggestion data on a display interface of the vehicle computing device; the command phrase is rendered in the foreground of the display interface of the vehicle computing device. causing said rendering; A method comprising:

15. The method of claim 14 , wherein the command phrase comprises natural language content characterizing an invocation phrase for invoking the automated assistant application and a request to perform the particular action.

16. receiving, by the vehicle computing device, an utterance embodying the request to the automated assistant application; determining, by the vehicle computing device, that the utterance includes the request rendered on the display interface of the vehicle computing device; in response to receiving the utterance, causing the given application to perform the particular action associated with the interaction between the user and the given application; The method of claim 14 or 15, further comprising:

17. The command phrase is rendered when the user is in the vehicle during a first trip; the utterance is received while the user is in the vehicle during a second trip; 17. The method according to any one of claims 14 to 16.

18. Identifying the particular action as controllable via selectable content being rendered on the display interface of the vehicle computing device; generating the suggestion data characterizing the command phrase is based on identifying the particular action as controllable via the selectable content being rendered on the display interface of the vehicle computing device. The identifying, 18. The method of any of claims 14 to 17, further comprising:

19. 20. The method of claim 18, wherein the application is a communications application and the selectable content includes one or more selectable elements for specifying a phone number and / or a contact to call.

20. 20. The method of claim 18, wherein the application is a media application and the selectable content includes one or more selectable elements for specifying media to be visually rendered via the display interface of the vehicle computing device and / or audibly rendered via one or more speakers of the vehicle computing device.

21. 21. The method of claim 14, wherein causing the suggested data to be rendered on the display interface of the vehicle computing device is in response to the user identifying that he or she has completed the interaction with the given application of the vehicle computing device.

22. 22. The method of claim 14, wherein generating the suggestion data characterizing the command phrase is further based on a context of the user engaged in the interaction with the given application of the vehicle computing device.

23. The method of claim 22 , wherein generating the suggestion data characterizing the command phrase is further based on a display context of the display interface of the vehicle computing device.

24. After the command phrase is rendered on the display interface of the vehicle computing device, determining, by the vehicle computing device, while the user is within the vehicle, via the display interface of the vehicle computing device, that the user is engaged in a separate interaction with a further application of the vehicle; The further application is also separate from the automated assistant application and separate from the given application. The identifying, generating, by the vehicle computing device, further suggestion data characterizing further command phrases based on the separate interaction with the further application; the further command phrase, when submitted by the user to the automated assistant application, causes the automated assistant application to control further operation of the further application; The generating. causing the vehicle computing device to visually render the further suggestion data on the display interface of the vehicle computing device; 24. The method of any of claims 14 to 23, further comprising:

25. 25. The method of claim 24, wherein causing the further suggestion data to be rendered on the display interface of the vehicle computing device is performed in response to the user identifying that the user has completed the separate interaction with the further application of the vehicle computing device.

26. one or more processors; A memory for storing instructions; A system comprising: The instructions, when executed, cause the one or more processors to perform the method of any one of claims 1 to 25. system.

27. 26. A non-transitory computer readable storage medium storing instructions that, when executed, cause one or more processors to perform the method of any one of claims 1 to 25.

Images