Automated assistant performance of non-assistant application operation in response to user input that can be limited to parameter

Abstract

To provide a selectable GUI element when a user interacts with an application that can be controlled via an automated assistant.SOLUTION: A method includes the steps of: when an assistant operation is compatible with an application that is executed at a computing device, rendering a selectable element on a display; detecting, by an automated assistant, touch selection of the selectable element; after the touch selection, performing voice recognition of spoken utterance; and in response to the spoken utterance, causing the automates assistant to control the application on the basis of the assistant operation and a particular value of a parameter.SELECTED DRAWING: Figure 1B

Description

[Background technology]

[0001] Humans can engage in human-computer interactions using interactive software applications referred to herein as "automated assistants" (also referred to as "digital agents," "chatbots," "interactive personal assistants," "intelligent personal assistants," "conversational agents," etc.). For example, a human (who may be referred to as a "user" when interacting with an automated assistant) can provide commands and / or requests using verbal natural language input (i.e., speech) and / or by providing textual (e.g., typed) natural language input, which may in some cases be converted to text and then processed.

[0002] In some cases, the automated assistant can provide a variety of features that can be initiated even while the user is interacting with a separate application in the foreground of the user's computing device. For example, the user can utilize the automated assistant to perform a search within the separate foreground application. As an example, in response to a spoken utterance directed to the automated assistant while a separate restaurant review application is in the foreground, "Search for Asian Fusion," the automated assistant can interact with the foreground application (e.g., directly and / or via emulating user input) to submit a search for "Asian Fusion" using the search interface of the restaurant review application. Also, as an example, in response to a spoken utterance directed to the automated assistant while a separate calendar application is in the foreground, "Add a calendar entry for patent meeting at 2:00 tomorrow," the automated assistant can interact with the foreground application to create a calendar entry titled "Patent Meeting" for 2:00 "tomorrow."

[0003] However, when interacting with a foreground application using an automated assistant, the user must specify both the intent (e.g., "search" or "add calendar entry" in the previous example) and the parameters for that intent (e.g., "Asian Fusion" or "Patent meeting tomorrow at 2:00") in a spoken utterance. In some circumstances, the user may also need to provide a spoken invocation phrase or other automated assistant invocation input for the automated assistant before providing the spoken utterance.

[0004] Further, in some situations, a user may not be aware of the automated assistant's ability to interact with a foreground application as the user desires, in response to the user's verbal utterances. Thus, the user may instead utilize more inputs and / or more duration of inputs directed to the application when interacting directly with the application. For example, as described above, assume that the user is unaware that the automated assistant interacts with a calendar application by saying to the automated assistant, "Add a calendar entry for a patent meeting tomorrow at 2:00." In such a situation, to add a corresponding calendar entry in the calendar application, the user may instead have to locate and tap an "add calendar entry" interface element of the calendar application that causes the calendar application's entry interface to be presented, and then click (e.g., using a virtual keyboard and / or selection menus) on the date, time, and title fields of the entry interface to fill those fields.

[0005] Further, in some situations, the automated assistant may not be able to properly ascertain whether a spoken utterance is attempting to control a foreground app or instead asking for a general response from the automated assistant that is generated independently of and without any control of the foreground app. For example, again assume a spoken utterance of "search Asian fusion" directed to the automated assistant while a separate restaurant review application is in the foreground. In such an example, it may not be clear whether the user is asking the assistant to conduct a search for "Asian fusion" restaurants within the restaurant review application or instead wants the automated assistant to conduct a general search (independent of the restaurant review application) and return a general description of what "Asian fusion" cuisine is. Summary of the Invention [Means for solving the problem]

[0006] Implementations described herein relate to an automated assistant that provides selectable GUI elements when a user is interacting with an application that can be controlled via the automated assistant. The selectable GUI elements can be rendered when the automated assistant determines that the application interface identifies an action (e.g., a search function) that can be initiated or otherwise controlled via the automated assistant. The selectable GUI elements can include content, such as textual and / or graphical content, that identifies the action and / or asks the user to provide one or more parameters for the action. When the selectable GUI elements are rendered, a microphone and / or camera can be activated with prior permission from the user to allow the user to specify one or more action parameters without explicitly identifying the automated assistant or the intent / action. When one or more action parameters are provided by the user, the automated assistant can control the application to perform an action (e.g., a search function) using the one or more action parameters (e.g., a search term).

[0007] In these and other ways, the interaction between the application and the automated assistant can be performed with reduced and / or more concise user input. For example, the user's verbal utterance can specify only parameters for the intent or action, without specifying the intent or action. This results in a more concise verbal utterance and a corresponding reduction in the processing of the verbal utterance by the automatic speech recognition component and / or other components. Also, the user does not need to provide an explicit invocation phrase (e.g., "Assistant..."), thereby further reducing the duration of the verbal utterance and the overall duration of the human-assistant interaction. Furthermore, the user's intent is made clear through the user's selection of GUI elements, so that the automated assistant does not misinterpret the verbal utterance as a general assistant request rather than a request for the assistant to control the foreground app. Furthermore, through the presentation of the GUI element, the user may become aware of the ability to control the foreground application through verbal utterances directed to the automated assistant rather than through more complex direct interactions with the foreground application, and / or the user may control the foreground application more frequently through verbal utterances (e.g., provided after selection of a GUI element).

[0008] In some implementations, the automated assistant can determine whether the application interface includes features corresponding to respective actions that are compatible with the automated assistant and / or the assistant's actions. In some cases, multiple different compatible actions may be identified for an application interface, causing the automated assistant to render one or more selectable GUI elements for each action. The type of selectable GUI element rendered by the automated assistant may depend on the corresponding action identified by the automated assistant. For example, when a user is accessing a home control application that includes a dial GUI element for controlling the temperature of a home, the automated assistant may render a selectable GUI element that identifies a command phrase for adjusting the temperature. In some implementations, the selectable GUI element may include text such as "set temperature to _____," which may indicate that the selectable GUI element corresponds to an action for setting the temperature of the home.

[0009] A blank or placeholder area (e.g., "____") of a selectable GUI element may prompt the user for and / or otherwise provide an indication that the user may provide a verbal utterance or other input that identifies parameters for completing a command phrase and / or for initializing performance of a corresponding action. For example, a user may tap on a selectable GUI element and / or subsequently provide a verbal utterance such as "65 degrees" to complete a command phrase recited in the text of the selectable GUI element. In response to receiving the verbal utterance, the automated assistant may control the application to adjust the application's temperature setting to "65" degrees. In some implementations, when a selectable GUI element is rendered by the automated assistant, the automated assistant may also activate an audio interface (e.g., one or more microphones) of the computing device. Thus, instead of tapping on a selectable GUI element, a user can provide a spoken utterance that identifies a parameter value (e.g., "65 degrees") without specifying an action to be performed (e.g., "change the temperature") and without specifying an assistant (e.g., "Assistant").

[0010] In some implementations, a selectable GUI element may be rendered by an automated assistant in the foreground of a display interface of a computing device for a threshold duration. The duration may be selected according to one or more characteristics associated with an interaction between a user and an application. For example, when a home screen of an application is rendered in a display interface and a user is not otherwise providing input to the application, the selectable GUI element may be rendered for a static duration (e.g., 3 seconds). However, when a user is interacting with an application (e.g., scrolling through the application interface) at the same time that the selectable GUI element is rendered on the application interface, the selectable GUI element may be rendered for a duration based on how often the user provides input to the application. Alternatively or additionally, the duration for which the selectable GUI element is rendered may be based on an amount of time that the corresponding application interface element is rendered or is expected to be rendered in the application interface. For example, if a user typically provides an application input that transitions an application from a home screen to a login screen within a time t of viewing the home screen, the selectable GUI element may be rendered on the home screen for a duration based on the time t.

[0011] In some implementations, the selection of the type of selectable GUI element to render can be based on a heuristic process and / or one or more trained machine learning models. For example, an automated assistant and / or an operating system of a computing device can process data related to an application interface to identify one or more actions that can be initiated via user interaction with the application interface of the application. This data can include, with prior permission from the user, screenshots of the application interface, links corresponding to graphical elements of the interface, library data and / or other functional data related to the application and / or interface, and / or any other information that can indicate actions that can be initiated via the application interface.

[0012] Depending on one or more actions identified for the application interface, the automated assistant can select and / or generate a selectable GUI element corresponding to the respective action. The selectable GUI element can be selected to provide an indication that the respective action can be controlled via the automated assistant. For example, the automated assistant can determine that a magnifying glass icon (e.g., a search icon) disposed on or adjacent to a blank text field (e.g., a search field) of the application interface can indicate that the application interface can control the search action of the application. Based on this determination, the automated assistant can render a selectable GUI element that includes the same or a different magnifying glass icon and / or includes one or more natural language terms synonymous with the word "search" (e.g., "search for ______"). In some implementations, when a selectable GUI element, a user can select the selectable GUI element by providing a spoken utterance that identifies a search parameter (e.g., "restaurants near me") or by tapping the selectable GUI element and then providing a spoken utterance that identifies the search parameter.

[0013] In some implementations, a microphone of a computing device rendering a selectable GUI element may remain active after a user selects the selectable GUI element with prior permission from the user. Alternatively or additionally, when an application interface changes in response to a selection of a selectable GUI element and / or a verbal utterance, the automated assistant may select another selectable GUI element to render. The automated assistant may select another selectable GUI element to render based on the next application interface to which the application transitions. For example, when a user issues a search parameter directed to a selectable GUI element, the application may render a search result list. A search result from the search result list may be selectable by the user to cause the application to perform a particular action. The automated assistant may determine that the particular action matches an assistant action (e.g., an action that can be performed by an automated assistant) and cause another selectable GUI element (e.g., a hand with an index finger extended toward the corresponding search result) to be rendered on or adjacent to the corresponding search result. Alternatively or additionally, the another selectable GUI element may include, for example, a text string identifying a word corresponding to the search result (e.g., "Time Four Thai Restaurant"). When a user provides a spoken utterance that includes one or more words that identify a corresponding search result (e.g., "Thai restaurants"), the automated assistant can cause the corresponding search result to be selected by the user without the user having to perform a "select" action or explicitly identify the automated assistant. In this manner, as the automated assistant continues to identify matching actions in each interface of the application, the user can navigate the interfaces by providing parameter values ​​(e.g., "Restaurants Near Me...Thai Restaurants...Menu...").In some implementations, a user can begin an interaction by first directing the automated assistant to open a particular application via a first spoken utterance (e.g., "Assistant, open my recipes application..."). Thereafter, when the automated assistant identifies a matching application action, the user can provide another command via a second spoken utterance to enable the automated assistant to control this particular application according to parameters (e.g., the user can say "Pad Thai" to have the automated assistant search for "Pad Thai" within his recipes application). The user can then continue to navigate this particular application using these brief spoken utterances, at least until the automated assistant recognizes one or more application actions as matching and / or controllable via one or more respective actions of the automated assistant.

[0014] The above description has been provided as a summary of some implementations of the disclosure. Further descriptions of these and other implementations are set forth in more detail below.

[0015] Other implementations may include a non-transitory computer-readable storage medium storing instructions executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) for performing a method, such as one or more of the methods described above and / or elsewhere herein. Still other implementations may include a system of one or more computers including one or more processors operable to execute stored instructions for performing a method, such as one or more of the methods described above and / or elsewhere herein.

[0016] It should be understood that any combination of the foregoing concepts, and additional concepts described in more detail herein, is contemplated as being part of the present subject matter disclosed herein, for example, any combination of claimed subject matter appearing at the end of this disclosure is contemplated as being part of the present subject matter disclosed herein. [Brief description of the drawings]

[0017] [Figure 1A] FIG. 1 illustrates a view in which a user interacts with an automated assistant that recognizes and suggests assistant-compatible actions in a third-party application interface. [Figure 1B] FIG. 1 illustrates a view in which a user interacts with an automated assistant that recognizes and suggests assistant-compatible actions in a third-party application interface. [Figure 1C] FIG. 1 illustrates a view in which a user interacts with an automated assistant that recognizes and suggests assistant-compatible actions in a third-party application interface. [Figure 2A] FIG. 1 illustrates a view in which a user interacts with an application to perform one or more actions that can be controlled by an automated assistant. [Figure 2B] FIG. 1 illustrates a view in which a user interacts with an application to perform one or more actions that can be controlled by an automated assistant. [Figure 2C] FIG. 1 illustrates a view in which a user interacts with an application to perform one or more actions that can be controlled by an automated assistant. [Diagram 3]FIG. 1 illustrates a system with an automated assistant that can provide selectable action intent suggestions that include indicators indicating that one or more parameters should be provided by the user to control the action when the user is accessing a third-party application controllable via the automated assistant. [Figure 4] FIG. 1 illustrates a method for providing selectable GUI elements in a computing device interface when an application action compatible with an automated assistant is executable through that interface. [Diagram 5] FIG. 1 is a block diagram of an exemplary computer system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] 1A, 1B, and 1C respectively show views 100, 120, and 140 in which a user 102 interacts with an automated assistant that recognizes and suggests assistant adaptation actions in a third-party application interface. When an assistant adaptation action is detected in the application interface, the automated assistant can provide an indication that one or more parameters can be provided by the user to initialize performance of the adaptation action. In this manner, the user can avoid providing an assistant invocation phrase and / or other inputs that may otherwise be required to control the automated assistant and / or the third-party application.

[0019] For example, a user 102 may be interacting with an assistant-enabled device, such as a computing device 104, to change settings of a home control application 108. Initially, the user 102 may initialize 106 the home control application 108 in response to user input, which may be touch input provided by a hand 116 of the user 102. When the home control application 108 is launched, the user 102 may select to control a particular device in the user's 102 home, such as the home's heating, ventilation, and air conditioning (HVAC) system. To control the HVAC system, the home control application 108 may render an application interface 110, which may include a thermostat GUI 112.

[0020] When the application interface 110 is being rendered on the display interface 118 of the computing device 104, the automated assistant can identify 114 one or more assistant-adapted actions. When an assistant-adapted action is identified as being associated with a selectable GUI element, the automated assistant can cause one or more graphical elements to be rendered on the display interface 118. For example, as presented in view 120 of FIG. 1B, the automated assistant can cause a selectable GUI element 122 and / or a suggestion element 126 to be rendered on the display interface 118. In some implementations, the selectable GUI element 122 can be rendered in the foreground and / or on top of the home control application 108. The selectable GUI element 122 can provide an indication to the user that the automated assistant is currently initialized and does not necessarily require an invocation phrase when the selectable GUI element 122 is being rendered. Alternatively or in addition, the suggestion element 126 can provide natural language content characterizing a spoken utterance that can be provided to the automated assistant to control the home control application 108. Additionally, the presence of the suggestion element 126 can indicate that the automated assistant has been initialized and does not necessarily require an invocation phrase before recognizing spoken input. Alternatively or additionally, the natural language content of the suggestion element 126 can include at least a portion of a suggested spoken utterance and also a blank space that can be a placeholder for one or more parameters for the assistant adaptation operation.

[0021] For example, as presented in FIG. 1B , the automated assistant can cause a selectable GUI element 122 to be rendered on the home control application 108 to indicate that the automated assistant can receive a temperature input. In some implementations, an icon corresponding to an assistant-compatible action can be selected to be rendered with the selectable GUI element 122. The selection of the icon can be based on parameters that can be provided by the user 102 to control the thermostat GUI 112. For example, a thermometer icon can be selected to indicate that the user can specify a temperature value to control the application interface 110 and adjust the thermostat GUI 112. The user 102 can then provide a spoken utterance 124, such as “65 degrees,” to indicate to the automated assistant that the user would like the automated assistant to modify the temperature setting of the home control application 108 from 72 degrees to 65 degrees.

[0022] In response to the verbal utterance 124, the automated assistant can initiate 144 an assistant-adapted action. For example, the automated assistant can generate a request to the home control application 108 to modify the thermostat's current setting from 72 degrees to 65 degrees. In some implementations, an application programming interface (API) can be used to interface between the automated assistant and the home control application 108. The home control application 108 can process the request from the automated assistant and modify the thermostat's setting accordingly. Additionally, an updated thermostat GUI 142 can be rendered 146 as an update in the application interface 110 to indicate to the user 102 that the automated assistant and home control application 108 have successfully performed the action. In some implementations, the selectable GUI element 122 and / or the suggestion element 126 can be removed from the display interface 118 after a threshold duration has occurred and / or regardless of whether the user 102 has interacted with the selectable GUI element 122 or the suggestion element 126. For example, the selectable GUI element 122 may be rendered for a threshold duration from when the home control application 108 was first rendered in the display interface 118. If the user 102 has not interacted with the selectable GUI element 122 for the threshold duration, the automated assistant may cause the selectable GUI element 122 to no longer be rendered in the display interface 118 and / or provide a notification that the selectable GUI element 122 will be removed after a certain period of time.

[0023] 2A, 2B, and 2C respectively show views 200, 220, and 240 in which a user 202 interacts with an application to perform one or more actions that can be controlled by an automated assistant. For example, the user 202 can access a messaging application 208 using a computing device 204. In response to the user 202 launching the messaging application 208, an automated assistant accessible via the computing device 204 can determine whether features of the messaging application 208 can be controlled through an automated assistant. In some implementations, the automated assistant can determine that one or more actions can be performed using specific parameters specified by the user 202. For example, the automated assistant can determine that to reply to a message, a checkbox 212 must be selected for a particular message, followed by a reply icon 218. In some implementations, the automated assistant can make this determination based on a heuristic process and / or one or more trained machine learning models. For example, training data for the one or more trained machine learning models may characterize instances in which the reply icon 218 is selected without the checkbox 212 being selected, and other instances in which the reply icon 218 is selected when the checkbox 212 is selected. In some implementations, screenshots and / or other images of application interfaces may be processed, with prior permission from a user, using the one or more trained machine learning models to cause particular selectable GUI elements and / or selectable suggestions to be rendered at the computing device 204.

[0024] When the computing device 204 initializes 206 the messaging application 208 in response to user input, the automated assistant can identify assistant-compatible actions available via the application interface 210 of the messaging application 208. When one or more assistant-compatible actions are identified, the automated assistant can cause 214 one or more selectable GUI elements 222 and / or one or more selectable suggestions 224 to be rendered at the computing device 204. The selectable GUI elements 222 can be rendered to indicate that the automated assistant and / or audio interface have been initialized and that the user 202 can specify parameters to have the automated assistant use the parameters to control the messaging application 208.

[0025] For example, the selectable GUI element 222 can include a graphical representation of a person or contact, thereby indicating that the user 202 should specify the name of the person to whom the user 202 would like to send a message. Alternatively or additionally, the selectable suggestions 224 can include a textual identifier and / or graphical content that identifies a command that can be issued to the automated assistant, but that also lacks one or more parameters. For example, the selectable suggestions 224 can include the phrase "Reply to a message from _____," which can indicate that the automated assistant can reply to a message identified in the application interface 210 if the user 202 specifies a contact associated with the particular message. The user 202 can then provide a spoken utterance 226 that specifies the parameters for this assistant-adapted operation. In response to the spoken utterance 226, the automated assistant can make a selection of the checkbox 212 that corresponds to the parameter (e.g., "Linus") specified by the user 202. Additionally, in response to the verbal utterance 226, the automated assistant can select the reply icon 218 to have the messaging application 208 reply to the message from the contact identified by the user 202. Alternatively or additionally, as a backend process, the automated assistant can communicate an API call to the messaging application 208 to initiate a reply to the message from the contact identified by the user 202.

[0026] In response to the verbal utterance 226 from the user 202, the automated assistant can have the messaging application 208 process a request to reply to a message from a contact (e.g., Linus) identified by the user 202. The messaging application 208 can render an updated application interface 248 when it receives a request from the automated assistant. The application interface 248 can correspond to a draft reply message that can be modified by the user 202. The automated assistant can process the content of the application interface 248 and / or other data stored in association with the application interface 248 to determine whether to provide additional suggestions to the user 202. For example, the automated assistant can cause one or more additional selectable GUI elements 242 to be rendered in the foreground of the application interface 248. The selectable GUI elements 242 can be rendered in step 244 to indicate to the user 202 that the automated assistant is active and that the user 202 can provide a verbal utterance that elaborates on the text of the reply message. For example, when the selectable GUI element 242 is being rendered, the user 202 may provide another spoken utterance 246 as a sentence of the message, such as, "Yes, I'll see you then," without explicitly identifying an action and / or without explicitly identifying an automated assistant.

[0027] In response, the automated assistant can communicate another request to the messaging application 208 to have the messaging application 208 perform one or more actions to enter the text "Yes, see you then" into the body of the message. The user 202 can then provide another verbal utterance (e.g., "send") directed to the automated assistant and the separate selectable GUI element 250. In this manner, the user 202 can have the messaging application 208 send a message without explicitly identifying the automated assistant and without providing touch input to the computing device 204 and the messaging application 208. This can reduce the number of inputs that need to be provided directly by the user 202 to the third-party application. Additionally, the user 202 can utilize the automated assistant when interacting with a majority of other applications that may not use trained machine learning models trained based on actual interactions with the user 202.

[0028] FIG. 3 illustrates a system 300 with an automated assistant 304 that can provide selectable action intent suggestions that include an indicator indicating that one or more parameters should be provided by the user to initiate an action when the user is accessing a third-party application controllable via the automated assistant. One or more actions associated with the action intent can be initiated in response to the user providing an input (e.g., a verbal utterance) that identifies one or more parameters without necessarily identifying the automated assistant 304 or the third-party application. The automated assistant 304 can operate as part of an assistant application provided on one or more computing devices, such as the computing device 302 and / or a server device. A user can interact with the automated assistant 304 via an assistant interface 320, which can be a microphone, a camera, a touch screen display, a user interface, and / or any other device capable of providing an interface between a user and an application. As an example, a user may initialize the automated assistant 304 by providing verbal, textual, and / or graphical input to the assistant interface 320 to cause the automated assistant 304 to initialize one or more actions (e.g., to provide data, control a peripheral device, access an agent, generate input and / or output, etc.). Alternatively, the automated assistant 304 may be initialized based on processing context data 336 with one or more trained machine learning models. The context data 336 may characterize one or more characteristics of an environment in which the automated assistant 304 is accessible and / or one or more characteristics of a user who is expected to be interacting with the automated assistant 304.The computing device 302 can include a display device, which can be a display panel including a touch interface for receiving touch input and / or gestures that enable a user to control applications 334 of the computing device 302 via the touch interface. In some implementations, the computing device 302 can lack a display device and thus provide an audible user interface output without providing a graphical user interface output. Additionally, the computing device 302 can provide a user interface such as a microphone for receiving verbal natural language input from a user. In some implementations, the computing device 302 can include a touch interface and the computing device 302 may lack a camera, although the computing device 302 can optionally include one or more other sensors.

[0029] The computing device 302 and / or other third party client devices can be in communication with a server device via a network such as the Internet. In addition, the computing device 302 and any other computing devices can be in communication with each other via a local area network (LAN) such as a Wi-Fi network. The computing device 302 can offload computational tasks to the server device to conserve computational resources at the computing device 302. As an example, the server device can host the automated assistant 304 and / or the computing device 302 can send inputs received at one or more assistant interfaces 320 to the server device. However, in some implementations, the automated assistant 304 can be hosted at the computing device 302 and various processes that can be associated with the operation of the automated assistant can be implemented at the computing device 302.

[0030] In various implementations, all or some aspects of the automated assistant 304 can be implemented on the computing device 302. In some of those implementations, aspects of the automated assistant 304 can be implemented by the computing device 302 and interface with a server device that can implement other aspects of the automated assistant 304. The server device can optionally serve multiple users and their associated assistant applications via multiple threads. In implementations in which all or some aspects of the automated assistant 304 are implemented by the computing device 302, the automated assistant 304 can be an application separate from (e.g., installed "on") the operating system of the computing device 302, or alternatively, can be implemented directly by (e.g., integral with but considered an operating system application of) the operating system of the computing device 302.

[0031] In some implementations, the automated assistant 304 can include an input processing engine 306 that can process input and / or output of the computing device 302 and / or the server device using a number of different modules. As an example, the input processing engine 306 can include a speech processing engine 308 that can process audio data received at the assistant interface 320 to perform speech recognition and / or identify text embodied in the audio data. To conserve computational resources at the computing device 302, the audio data can be transmitted from the computing device 302 to a server device, for example. Additionally or alternatively, the audio data can be processed solely at the computing device 302.

[0032] The process for converting the audio data to text can include a speech recognition algorithm, which can use neural networks and / or statistical models to identify groups of audio data that correspond to words or phrases. The text converted from the audio data can be parsed by the data parsing engine 310 and made available to the automated assistant 304 as text data that can be used to generate and / or identify command phrases, intents, actions, slot values, and / or any other content specified by the user. In some implementations, the output data provided by the data parsing engine 310 can be provided to a parameter engine 312 to determine whether the user has provided input that corresponds to a particular intent, action, and / or routine that can be performed by the automated assistant 304 and / or an application or agent that can be accessed via the automated assistant 304. For example, assistant data 338 can be stored at the server device and / or computing device 302, and can include data defining one or more actions that can be performed by the automated assistant 304, as well as parameters necessary to perform the action. The parameter engine 312 can generate one or more parameters for the intent, action, and / or slot value and provide the one or more parameters to the output generation engine 314. The output generation engine 314 can use the one or more parameters to communicate with the assistant interface 320 to provide output to the user and / or to communicate with one or more applications 334 to provide output to the one or more applications 334.

[0033] In some implementations, the automated assistant 304 can be an application that can be installed “on top of” the operating system of the computing device 302 and / or can itself form part (or the entirety) of the operating system of the computing device 302. The automated assistant application can include and / or access on-device speech recognition, on-device natural language understanding, and on-device fulfillment. For example, on-device speech recognition can be implemented using an on-device speech recognition module that processes audio data (detected by a microphone) using an end-to-end speech recognition machine learning model stored locally in the computing device 302. On-device speech recognition generates recognized text for the spoken utterances (if any) present in the audio data. Also, for example, on-device natural language understanding (NLU) can be implemented using an on-device NLU module that processes the recognized text generated using on-device speech recognition and, optionally, contextual data to generate NLU data.

[0034] The NLU data can include an intent corresponding to the verbal utterance and, optionally, parameters for the intent (e.g., slot values). On-device fulfillment can be implemented using an on-device fulfillment module that utilizes the NLU data (from the on-device NLU) and, optionally, other local data to determine actions to take to resolve the intent (and, optionally, parameters for the intent) of the verbal utterance. This can include determining local and / or remote responses (e.g., answers) to the verbal utterance, interactions with locally installed applications to be implemented based on the verbal utterance, commands to be sent (directly or via a corresponding remote system) to an Internet of Things (IoT) device based on the verbal utterance, and / or other resolution actions to be implemented based on the verbal utterance. The on-device fulfillment can then initiate local and / or remote implementation / execution of the determined actions to resolve the verbal utterance.

[0035] In various implementations, remote speech processing, remote NLU, and / or remote fulfillment can be utilized at least selectively. For example, recognized text can be at least selectively sent to a remote automated assistant component for remote NLU and / or remote fulfillment. By way of example, recognized text can be optionally sent for remote fulfillment in parallel with on-device fulfillment or in response to failure of on-device NLU and / or on-device fulfillment. However, on-device speech processing, on-device NLU, on-device fulfillment, and / or on-device execution can be prioritized at least due to the reduced latency they provide when resolving spoken utterances (since no client-server round trips are required to resolve spoken utterances). Additionally, on-device functionality can be the only functionality available in situations without network connectivity or with limited network connectivity.

[0036] In some implementations, the computing device 302 may include one or more applications 334 that may be provided by a third party entity different from the entity that provided the computing device 302 and / or the automated assistant 304. The application state engine of the automated assistant 304 and / or the computing device 302 may access the application data 330 to determine one or more actions that may be performed by the one or more applications 334, as well as the state of each application of the one or more applications 334 and / or the state of each device associated with the computing device 302. The device state engine of the automated assistant 304 and / or the computing device 302 may access the device data 332 to determine one or more actions that may be performed by the computing device 302 and / or one or more devices associated with the computing device 302. Additionally, the application data 330 and / or any other data (e.g., device data 332) can be accessed by the automated assistant 304 to generate context data 336, which can characterize the context in which a particular application 334 and / or device is running and / or the context in which a particular user is accessing the computing device 302, the application 334, and / or any other device or module.

[0037] While one or more applications 334 are executing on the computing device 302, the device data 332 can characterize a current operational state of each application 334 executing on the computing device 302. Additionally, the application data 330 can characterize one or more features of the executing applications 334, such as the content of one or more graphical user interfaces being rendered at the direction of the one or more applications 334. Alternatively or in addition, the application data 330 can characterize action schemas that can be updated by the respective applications and / or by the automated assistant 304 based on the respective applications' current operational state. Alternatively or in addition, one or more action schemas for one or more applications 334 can remain fixed but can be accessed by the application state engine to determine appropriate actions to be initiated via the automated assistant 304.

[0038] The computing device 302 may further include an assistant invocation engine 322, which may use one or more trained machine learning models to process the application data 330, the device data 332, the context data 336, and / or any other data accessible to the computing device 302. The assistant invocation engine 322 may process this data to determine whether the data should be considered indicative of the user's intent to invoke an automated assistant, instead of waiting for the user to explicitly speak an invocation phrase to invoke the automated assistant 304 or requesting the user to explicitly speak an invocation phrase. For example, the one or more trained machine learning models may be trained using training data instances based on a scenario in which a user is present in an environment in which multiple devices and / or applications exhibit various operating conditions. The training data instances may be generated to capture training data that characterizes contexts in which a user invokes an automated assistant and other contexts in which a user does not invoke an automated assistant.

[0039] When one or more trained machine learning models are trained according to these training data instances, the assistant invocation engine 322 can cause the automated assistant 304 to detect or limit the detection of a spoken invocation phrase from a user based on context and / or environmental features. Additionally or alternatively, the assistant invocation engine 322 can also cause the automated assistant 304 to detect or limit the detection of one or more assistant commands from a user based on context and / or environmental features. In some implementations, the assistant invocation engine 322 can be disabled or limited based on the computing device 302 detecting an assistant suppression output from another computing device. In this way, when the computing device 302 detects an assistant suppression output, the automated assistant 304 will not be invoked based on the context data 336 that would otherwise cause the automated assistant 304 to be invoked if the assistant suppression output had not been detected.

[0040] In some implementations, the system 300 can include an action detection engine 316 that can identify one or more actions that can be performed by the application 334 and that can be controlled via the automated assistant 304. For example, the action detection engine 316 can process the application data 330 and / or the device data 332 to determine whether the application is running on the computing device 302. The automated assistant 304 can determine whether the application can be controlled via the automated assistant 304 and can identify one or more application actions that can be controlled via the automated assistant 304. For example, the application data 330 identifies one or more application GUI elements that are rendered in the interface of the computing device 302, and the application data 330 can be processed to identify one or more actions that can be controlled via the application GUI elements. When an action is identified that is compatible with the automated assistant 304, the action detection engine 316 can communicate with the GUI element content engine 318 to generate a selectable GUI element that corresponds to the action.

[0041] The GUI element content engine 318 can identify one or more actions determined by the automated assistant 304 that are compatible with the automated assistant 304 and generate one or more respective selectable GUI elements based on the one or more actions. For example, when a search icon and / or a search text field are determined to be available by the application and an application search action is compatible with the automated assistant 304, the GUI element content engine 318 can generate content for rendering in a display interface of the computing device 302. The content can include textual content (e.g., natural language content) and / or graphical content that can be based on the compatible action (e.g., application search action). In some implementations, to inform the user of the identified compatible action, a command phrase can be generated and rendered to instruct the automated assistant 304 to initialize the action. Alternatively or additionally, the command phrase can be a partial command phrase that omits one or more parameters for the action, thereby indicating to the user that the user can provide one or more parameters to the automated assistant 304 to initialize the action. The textual and / or graphical content can be rendered in the display interface of the computing device 302 at the same time that the application renders one or more additional GUI elements. A user can initiate the performance of a matching operation by tapping the display interface to select a selectable GUI element and / or by providing a verbal utterance to the automated assistant 304 that specifies one or more parameters.

[0042] In some implementations, the system 300 can include a GUI element duration engine 326 that can control the duration for which a selectable GUI element is rendered in a display interface by the automated assistant 304. In some implementations, the amount of time for which a selectable GUI element is rendered can be based on the amount of interaction between the user and the automated assistant 304 and / or the amount of interaction between the user and an application associated with the selectable GUI element. For example, the GUI element duration engine 326 can establish a longer duration of display time for a selectable GUI element when the selectable GUI element has been rendered and the user has not yet provided input to the application. This longer duration can be longer than the duration of display time for a selectable GUI element that is being rendered when the user has provided input to the application (but not to the automated assistant 304). Alternatively or additionally, the duration of display time for a selectable GUI element can be longer than for a selectable GUI element that the user has previously interacted with in the past. This longer duration may be in comparison to durations for other selectable GUI elements that were previously presented to the user but that the user did not previously interact with or otherwise show interest in.

[0043] In some implementations, the system 300 can include an action execution engine 324 that can initialize one or more actions of an identified application in response to a user identifying one or more parameters for the action. For example, when a selectable GUI element is being rendered on an application interface by the automated assistant 304, the user can select a selectable GUI element and / or provide a verbal utterance identifying a parameter. The action execution engine 324 can then process the selection and / or verbal utterance and generate one or more requests to the application based on the one or more parameters identified by the user. For example, the verbal utterance processed by the input processing engine 306 can identify one or more specific parameter values. The parameter values ​​can be used by the action execution engine 324 to generate one or more requests to the application that correspond to the selectable GUI element identified by the user. For example, the request generated by the automated assistant 304 can identify an action to be performed, one or more parameters identified by the automated assistant 304, and / or one or more parameters specified by the user. In some implementations, the automated assistant 304 can select one or more parameters for the operation, and the user can specify one or more additional parameters, in order for the operation to be initialized. As an example, when the application is a travel booking application, the automated assistant 304 can assume a date parameter (e.g., the month "January"), and the user can specify the destination city (e.g., "Nairobi") via a spoken utterance.Based on this data and the corresponding selectable GUI elements rendered in the display interface, the action execution engine 324 can generate a request to the travel booking application to initiate the performance of the action (e.g., Application.Travel.com[search.setCity("Nairobi"), search.setTime("January")]). This request can be received by the travel booking application from the automated assistant 304, and in response, the travel booking application can render another application interface that includes the results of the action (e.g., a list of results for hotels with availability in Nairobi in January).

[0044] FIG. 4 illustrates a method 400 for providing selectable GUI elements in an interface of a computing device when an application operation compatible with the automated assistant is executable through the interface. The method 400 can be implemented by one or more applications, devices, and / or any other devices or modules capable of interacting with an automated assistant. In some implementations, the method 400 can include step 402 of determining whether a non-assistant application is running in the interface of the computing device. The computing device can enable access to an automated assistant, and the automated assistant can respond to natural language input from a user to control a number of different applications and / or devices. The automated assistant can process data indicating whether a particular application is running in the computing device. For example, data based on content rendered in the interface can be processed by the automated assistant to identify application operations that can be initiated through the interface. When it is determined that a non-assistant application is running in an interface, such as a display interface of the computing device, the method 400 can move from step 402 to step 404. Otherwise, the automated assistant can continue to determine whether an application is running in the interface of the computing device.

[0045] Step 404 may include determining whether the application operation is compatible with the automated assistant. In other words, the automated assistant may determine whether an operation that can be performed by the application can be controlled or otherwise initiated by the automated assistant. For example, when the application is a home control application and the application interface includes a control dial GUI, the automated assistant may determine that the operation controlled by the control dial GUI is compatible with one or more functions of the automated assistant. Thus, the automated assistant may operate to control the control dial GUI and / or the corresponding application operation. When it is determined that the application operation is compatible with the automated assistant, the method 400 may move from step 404 to step 406. Otherwise, the automated assistant may continue to determine whether any other application operation is compatible with the automated assistant or whether any other non-assistant applications are running on the computing device or a separate computing device.

[0046] Step 406 may include causing a selectable GUI element to be rendered in the interface and also activating an audio interface at the computing device. The selectable GUI element may provide an indication that the automated assistant is active to receive one or more input parameters. In some implementations, the selectable GUI element may include textual and / or graphical content based on the application operation identified in step 404. In this manner, the user may be informed that the automated assistant is able to receive input specifying one or more parameters for a particular application operation at least while the selectable GUI is being rendered in the interface. In some implementations, the graphical and / or textual content of the selectable GUI element may indicate that a microphone is active to receive user input from a user. For example, the selectable GUI element may have a dynamic characteristic that indicates that one or more sensors associated with the computing device are active. Alternatively or additionally, the textual content of the selectable GUI element may identify one or more partial assistant command phrases that lack one or more respective parameters to be specified for the one or more respective application operations to be performed.

[0047] When the selectable GUI element is rendered in the interface, the method 400 can move from step 406 to optional step 408, which includes determining whether the user has provided a touch input or another input directed to the selectable GUI element. When it is determined that the user has provided an input directed to the selectable GUI element, the method 400 can move from step 408 to step 410. Step 410 can include initiating detection of audio data corresponding to parameters for the application action. For example, the automated assistant can identify one or more speech processing models for identifying one or more parameters related to the application action. In some cases, when the application action includes one or more numbers as possible parameters, a speech processing model for identifying numbers of various magnitudes may be used. Alternatively or additionally, when the application action includes one or more proper names as possible parameters, a speech processing model for identifying proper names in speech may be used.

[0048] Method 400 can move from step 410 or step 408 to step 412, which can determine whether a user has provided an input parameter associated with an application operation to the automated assistant. For example, a user can provide input associated with an application operation by specifying a value for a control dial GUI. Alternatively or in addition, a user can provide input associated with an application operation by specifying one or more other values ​​that can be used as one or more parameters for the application operation. For example, when an application operation is controllable via a control dial GUI of the application, a user can provide a verbal utterance, such as "10 percent," to the automated assistant. The verbal utterance can indicate that the user has specified "10 percent" as a parameter for the application operation and that the automated assistant should initialize the application operation based on the specified parameter. When an application operation corresponds, for example, to the brightness of a light in a user's home, the user can specify a value for the parameter to cause the automated assistant to adjust the brightness of the light via an application (e.g., an IoT application that controls a Wi-Fi enabled light bulb).

[0049] When it is determined that the user has provided an input specifying one or more parameters for the application operation, the method 400 can move from step 412 to step 414. Step 414 can include having the automated assistant control the non-assistant application according to the input parameters specified by the user. For example, when the user provides a verbal utterance, such as "ten percent," the automated assistant can control the non-assistant application to adjust one or more lights associated with the non-assistant application to a brightness level of 10%. This can be performed without the user explicitly identifying the assistant or non-assistant application in the verbal utterance. This can conserve computational resources and limit the possibility that certain interferences (e.g., background noise) can affect the audio data captured by the automated assistant. When the user does not provide an input specifying the parameters within a threshold duration, the method 400 can move from step 412 to step 416, which can include having the selectable GUI element removed from the interface after the threshold duration. The method 400 may proceed from step 414 to step 416, after which the method 400 may return to step 402 or another step.

[0050] 5 is a block diagram 500 of an exemplary computer system 510. The computer system 510 typically includes at least one processor 514 that communicates with several peripheral devices via a bus subsystem 512. These peripheral devices may include, for example, a storage subsystem 524 including a memory 525 and a file storage subsystem 526, a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input and output devices enable user interaction with the computer system 510. The network interface subsystem 516 provides an interface to external networks and is coupled to corresponding interface devices in other computer systems.

[0051] The user interface input devices 522 may include a keyboard, a pointing device such as a mouse, a trackball, a touchpad, or a graphics tablet, a scanner, a touch screen integrated into a display, a voice recognition system, an audio input device such as a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and ways for inputting information into the computer system 510 or onto a communications network.

[0052] The user interface output devices 520 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or some other mechanism for producing a visible image. The display subsystem may also provide a non-visual display, such as through an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and ways to output information from the computer system 510 to a user or to another machine or computer system.

[0053] Storage subsystem 524 stores programming and data structures that provide functionality for some or all of the modules described herein. For example, storage subsystem 524 may include logic for performing selected aspects of method 400 and / or for implementing one or more of system 300, computing device 104, computing device 204, an automated assistant, and / or any other applications, devices, apparatus, and / or modules discussed herein.

[0054] These software modules are generally executed by the processor 514 alone or in combination with other processors. The memory 525 used within the storage subsystem 524 may include several memories, including a main random access memory (RAM) 530 for storing instructions and data during program execution, and a read-only memory (ROM) 532 in which fixed instructions are stored. The file storage subsystem 526 may provide persistent storage for program files and data files, and may include hard disk drives, floppy disk drives and associated removable media, CD-ROM drives, optical drives, or removable media cartridges. Modules implementing the functionality of a particular implementation may be stored within the storage subsystem 524 or within other machines accessible by the processor 514 via the file storage subsystem 526.

[0055] Bus subsystem 512 provides a mechanism for allowing the various components and subsystems of computer system 510 to communicate with each other as intended. Although bus subsystem 512 is shown generally as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0056] The computer system 510 can be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of the computer system 510 depicted in Figure 5 is intended only as a specific example intended to illustrate some implementations. Many other configurations of the computer system 510 are possible, having more or fewer components than the computer system depicted in Figure 5.

[0057] In situations where the systems described herein may collect or use personal information about users (or, as is often referred to herein, "participants"), the user may be given the opportunity to control whether a program or function collects user information (e.g., information about the user's social network, social actions or activities, occupation, user preferences, or the user's current geographic location) or whether and / or how to receive content from a content server that may be more relevant to the user. Also, certain data may be handled in one or more ways such that personally identifiable information is removed before it is stored or used. For example, a user's identification information may be handled such that personally identifiable information cannot be identified for that user, or if geographic location information is obtained, the user's geographic location may be generalized (such as to the city level, zip code level, or state level) such that the user's specific geographic location cannot be identified. Thus, the user may be able to control how information is collected and / or used about the user.

[0058] Although several implementations have been described and illustrated herein, various other means and / or structures for performing the functions and / or obtaining the results and / or one or more of the advantages described herein can be utilized, and each such variation and / or modification is considered to be within the scope of the implementations described herein. More generally, any parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on the particular application or applications in which the teachings are used. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific implementations described herein. Thus, it should be understood that the implementations described above are presented by way of example only, and that within the scope of the appended claims and their equivalents, implementations may be practiced otherwise than as specifically described and claimed. Implementations of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent.

[0059] In some implementations, a method implemented by one or more processors is described as including steps such as determining that the assistant's operation is compatible with an application running on a computing device, where the application is separate from an automated assistant accessible via the computing device. The method may further include rendering a selectable graphical user interface (GUI) element in a display interface of the computing device based on the assistant's operation being compatible with the application, where the selectable GUI element identifies the assistant's operation and is rendered in the foreground of the display interface of the computing device. The method may further include detecting, by the automated assistant, a selection of the selectable GUI element by a user via the display interface of the computing device. The method may further include performing speech recognition on audio data capturing a verbal utterance provided by the user after selection of the selectable GUI element and received at an audio interface of the computing device, where the verbal utterance specifies a particular value of a parameter of the assistant's operation without explicitly identifying the assistant's operation. The method can further include the step of, in response to a verbal utterance from the user, causing the automated assistant to control the application based on the assistant's behavior and particular values ​​of the parameters.

[0060] In some implementations, causing the selectable GUI element to be rendered in the display interface of the computing device includes generating content to be rendered with the selectable GUI element, the content including a text identifier or graphical representation of the assistant's action and a placeholder area indicating that a user can specify a value for the parameter. In some implementations, causing the selectable GUI element to be rendered in the display interface of the computing device includes rendering the selectable GUI element on an application interface of the application for a threshold duration, the threshold duration being based on an amount of interaction between the user and the application. In some implementations, when a spoken utterance is provided by the user after the threshold duration and after the selectable GUI element is no longer rendered in the display interface of the computing device, the automated assistant does not respond to the spoken utterance.

[0061] In some implementations, determining that the assistant's operation is compatible with the application running on the computing device includes determining that additional selectable GUI elements rendered in an application interface of the application correspond to an application operation that can be performed in response to initializing the assistant's operation. In some implementations, determining that the assistant's operation is compatible with the application running on the computing device includes determining that the additional selectable GUI elements include a search icon or a search field and that the application operation corresponds to a search operation. In some implementations, having the automated assistant control the application based on the assistant's operation and the particular value of the parameter includes having the automated assistant cause the application to provide search results based on the particular value of the parameter specified in a verbal utterance from the user to the automated assistant.

[0062] In another implementation, a method implemented by one or more processors is described as including steps such as determining that a user has provided a first spoken utterance to an automated assistant accessible via a computing device, the first spoken utterance including a request to initialize an application that is separate from the automated assistant. The method may further include, in response to the first spoken utterance, initializing the application and causing the application to render an application interface in the foreground of a display interface of the computing device, the application interface including content identifying an operation that can be controlled via the automated assistant. The method may further include rendering a selectable GUI element on the application interface of the application based on the operation being controllable via the automated assistant, the selectable GUI element including a text identifier or a graphical representation of the operation that can be controlled by the automated assistant. The method may further include determining that a user has provided a second spoken utterance to the automated assistant, the second spoken utterance identifying parameters that may be utilized by the application in performing the operation, and determining that the second spoken utterance does not specifically identify the operation. The method may further include, in response to the second spoken utterance, causing the automated assistant to initialize performance of the operation via the application using the parameters identified in the second spoken utterance.

[0063] In some implementations, rendering the selectable GUI element on the application interface of the application includes rendering the text identifier with a command phrase, the command phrase including a word identifying the action and a blank space indicating that a user-specifiable parameter is omitted from the command phrase. In some implementations, the method can further include causing an audio interface of the computing device to be initialized to receive a specific spoken utterance from a user based on the action being controllable via the automated assistant, where when the audio interface is initialized, the user can provide the specific spoken utterance to control the automated assistant without explicitly identifying the automated assistant. In some implementations, rendering the selectable GUI element on the application interface of the application includes generating content rendered with the selectable GUI element, the content including a graphical representation of an action of the assistant that is selectable via touch input to a display interface of the computing device.

[0064] In some implementations, causing the selectable GUI element to be rendered on the application interface of the application includes causing the selectable GUI element to be rendered on the application interface of the application for a threshold duration, the threshold duration being based on an amount of interaction between the user and the automated assistant since the selectable GUI element was rendered on the application interface. In some implementations, when an additional spoken utterance is provided by the user after the selectable GUI element is no longer rendered on the application interface, the automated assistant does not respond to the additional spoken utterance. In some implementations, the method may further include initializing an audio interface of the computing device to detect another spoken utterance that identifies one or more parameters for the operation based on the operation being controllable via the automated assistant.

[0065] In yet another implementation, a method implemented by one or more processors is described as including steps such as determining that the assistant's operation is compatible with an application running on a computing device, where the application is separate from an automated assistant accessible via the computing device. The method may further include causing a selectable graphical user interface (GUI) element to be rendered in a display interface of the computing device based on the assistant's operation being compatible with the application, where the selectable GUI element identifies the assistant's operation and is rendered in the foreground of the display interface of the computing device. The method may further include determining that a user has provided a verbal utterance directed to the automated assistant when the selectable GUI element is being rendered in the display interface of the computing device, where the verbal utterance specifies a particular value of a parameter of the assistant's operation without explicitly identifying the assistant's operation. The method may further include causing the automated assistant to control the application based on the assistant's operation and the particular value of the parameter in response to the verbal utterance from the user.

[0066] In some implementations, causing the selectable GUI elements to be rendered at the display interface of the computing device includes generating content to be rendered with the selectable GUI elements, the content including an icon selected based on an action of the assistant, the icon being selectable via touch input to the display interface of the computing device. In some implementations, causing the selectable GUI elements to be rendered at the display interface of the computing device includes generating content to be rendered with the selectable GUI elements, the content including natural language content that characterizes a partial command phrase that omits one or more parameter values ​​for an action of the assistant. In some implementations, determining that the action of the assistant is compatible with an application running at the computing device includes determining that additional selectable GUI elements rendered by the application control an application action that can be initialized by the automated assistant. In some implementations, causing the automated assistant to control the application based on the action of the assistant and a particular value of the parameter includes causing the application to render another application interface that is generated by the application based on a particular value of the parameter. In some implementations, causing the selectable GUI element to be rendered in a display interface of the computing device includes causing the selectable GUI element to be rendered simultaneously with the application rendering one or more application GUI elements of the application. [Explanation of symbols]

[0067] 100 Views 102 users 104 Computing Devices 106 In response to user input, initialize the home control application 108 108 Home Control Applications 110 Application Interface 112 Thermostat GUI 114 Identify one or more assistant-compatible actions 116 Hands 118 Display Interface 120 Views 122 Selectable GUI Elements 124 Oral Speech 126 Proposal Elements 140 Views 142 Thermostat GUI 144 Initialize Assistant adaptation operation 146 The updated thermostat GUI 142 is rendered as an update in the application interface 110 200 Views 202 users 204 Computing Devices 206 Initialized messaging application 208 in response to user input 208 Messaging Applications 210 Application Interface 212 Checkbox 214 causes one or more selectable GUI elements 222 and / or one or more selectable suggestions 224 to be rendered at the computing device 204. 218 Reply Icon 220 Views 222 Selectable GUI Elements 224 Selectable Proposals 226 Oral Speech 240 Views 242 Selectable GUI Elements 244 process 246 Oral Speech 248 Application Interface 250 selectable GUI elements 300 Systems 302 Computing Devices 304 Automated Assistants 306 Input Processing Engine 308 Voice Processing Engine 310 Data Parsing Engine 312 Parameter Engine 314 Output Generation Engine 316 Behavioral Detection Engine 318 GUI Element Content Engine 320 Assistant Interface 322 Assistant Call Engine 324 Action Execution Engine 326 GUI Element Duration Engine 330 Application Data 332 Device Data 334 Applications 336 Context Data 338 Assistant Data 400 ways 500 Block Diagram 510 Computer Systems 512 Bus Subsystem 514 processor 516 Network Interface Subsystem 520 User Interface Output Device 522 User Interface Input Devices 524 Storage Subsystem 525 Memory 526 File Storage Subsystem 530 Main Random Access Memory (RAM) 532 Read-Only Memory (ROM)

Claims

1. 1. A method implemented by one or more processors, comprising: Determining that the operation of the assistant is compatible with an application running on the computing device, the application is separate from an automated assistant accessible via the computing device; Steps and causing selectable graphical user interface (GUI) elements to be rendered in a display interface of the computing device based on the behavior of the assistant conforming to the application; the selectable GUI element identifies an action of the assistant and is rendered in the foreground of the display interface of the computing device; Steps and detecting, by the automated assistant, a selection by a user of the selectable GUI element via the display interface of the computing device; performing speech recognition on audio data capturing a verbal utterance provided by the user after the selection of the selectable GUI element and received at an audio interface of the computing device; the spoken utterance specifies a particular value for a parameter of the assistant's operation without explicitly identifying the assistant's operation; Steps and In response to the verbal utterance from the user, causing the automated assistant to control the application based on the operation of the assistant and the particular value of the parameter. A method comprising:

2. causing the selectable GUI elements to be rendered on the display interface of the computing device, generating content to be rendered using the selectable GUI elements, The content includes a text identifier or a graphical representation of an action of the assistant and a placeholder area indicating that a value of the parameter can be specified by the user; The method of claim 1 , comprising the steps of:

3. causing the selectable GUI elements to be rendered on the display interface of the computing device, causing the selectable GUI element to be rendered on an application interface of the application for a threshold duration; the threshold duration is based on an amount of interaction between the user and the application; 3. The method of claim 1 or claim 2, comprising the steps of:

4. 4. The method of claim 3, wherein the automated assistant does not respond to the spoken utterance when the spoken utterance is provided by the user after the threshold duration and after the selectable GUI element is no longer rendered in the display interface of the computing device.

5. determining that the operation of the assistant is compatible with the application running on the computing device, determining that additional selectable GUI elements rendered in an application interface of the application correspond to application actions that can be executed in response to initializing an operation of the assistant; 5. The method according to claim 1 , comprising:

6. determining that the operation of the assistant is compatible with the application running on the computing device, determining that the additional selectable GUI element includes a search icon or a search field and that the application action corresponds to a search action; 6. The method of claim 5, comprising:

7. causing the automated assistant to control the application based on the operation of the assistant and the particular value of the parameter; causing the automated assistant to cause the application to provide search results based on the particular values ​​of the parameters specified in the verbal utterances from the user to the automated assistant.

7. The method of claim 6, comprising:

8. 1. A method implemented by one or more processors, comprising: Determining that a user has provided a first spoken utterance to an automated assistant accessible via a computing device, comprising: the first spoken utterance includes a request to initialize an application that is separate from the automated assistant; Steps and in response to the first spoken utterance, initializing the application and causing the application to render an application interface in the foreground of a display interface of the computing device, the application interface includes content that identifies operations that can be controlled via the automated assistant; Steps and causing a selectable GUI element to be rendered on the application interface of the application based on the action being controllable via the automated assistant; the selectable GUI elements include textual identifiers or graphical representations of the actions that can be controlled by the automated assistant; Steps and Determining that the user has provided a second spoken utterance to the automated assistant, the second spoken utterance identifies parameters that can be utilized by the application in performing the action; and the second spoken utterance does not clearly identify the action; Steps and In response to the second spoken utterance, causing the automated assistant to initiate performance of the action via the application and using the parameters specified in the second spoken utterance. A method comprising:

9. Rendering the selectable GUI element on the application interface of the application, causing the text identifier to be rendered with a command phrase, the command phrase including a word identifying the action and a blank space indicating that a user-specifiable parameter has been omitted from the command phrase; 9. The method of claim 8, comprising:

10. causing initialization of an audio interface of the computing device to receive a particular verbal utterance from the user based on the operation being controllable via the automated assistant; When the audio interface is initialized, the user can provide the particular spoken utterance for controlling the automated assistant without explicitly identifying the automated assistant.

10. The method of claim 8 or claim 9, further comprising the step of:

11. Rendering the selectable GUI element on the application interface of the application, generating content to be rendered using the selectable GUI elements, the content includes the graphical representation of an action of the assistant that is selectable via touch input to the display interface of the computing device; 11. The method according to any one of claims 8 to 10, comprising the steps:

12. Rendering the selectable GUI element on an application interface of the application, causing the selectable GUI element to be rendered on the application interface of the application for a threshold duration; the threshold duration is based on an amount of interaction between the user and the automated assistant since the selectable GUI element is rendered on the application interface; 12. The method according to any one of claims 8 to 11, comprising the steps:

13. 13. The method of claim 12, wherein the automated assistant does not respond to an additional spoken utterance when the additional spoken utterance is provided by the user after the selectable GUI element is no longer rendered on the application interface.

14. Based on the action being controllable via the automated assistant, initializing an audio interface of the computing device to detect another spoken utterance that identifies one or more parameters for the action.

9. The method of claim 8, further comprising:

15. 1. A method implemented by one or more processors, comprising: Determining that the operation of the assistant is compatible with an application running on the computing device, the application is separate from an automated assistant accessible via the computing device; Steps and causing selectable graphical user interface (GUI) elements to be rendered in a display interface of the computing device based on the behavior of the assistant conforming to the application; the selectable GUI element identifies an action of the assistant and is rendered in the foreground of the display interface of the computing device; Steps and Determining that a user has provided a verbal utterance directed to the automated assistant when the selectable GUI element is being rendered on the display interface of the computing device, the spoken utterance specifies a particular value for a parameter of the assistant's operation without explicitly identifying the assistant's operation; Steps and In response to the verbal utterance from the user, causing the automated assistant to control the application based on the operation of the assistant and the particular value of the parameter. A method comprising:

16. causing the selectable GUI elements to be rendered on the display interface of the computing device, generating content to be rendered using the selectable GUI elements, The content includes icons selected based on an action of the assistant, the icons being selectable via touch input to the display interface of the computing device; The method of claim 15, comprising the steps of:

17. causing the selectable GUI elements to be rendered on the display interface of the computing device, generating content to be rendered using the selectable GUI elements, the content includes natural language content characterizing partial command phrases that omit one or more parameter values ​​for the operation of the assistant; The method of claim 15, comprising the steps of:

18. determining that the operation of the assistant is compatible with the application running on the computing device, determining that additional selectable GUI elements rendered by the application control application behaviors that can be initiated by the automated assistant; 18. The method of any one of claims 15 to 17, comprising:

19. causing the automated assistant to control the application based on the operation of the assistant and the particular value of the parameter; causing the application to render a different application interface that is generated by the application based on the particular value of the parameter.

19. The method of any one of claims 15 to 18, comprising:

20. causing the selectable GUI elements to be rendered on a display interface of the computing device, causing the selectable GUI element to be rendered simultaneously with the application rendering one or more application GUI elements of the application.

20. The method of any one of claims 15 to 19, comprising:

21. A computer program comprising instructions which, when executed by one or more processors of a computing system, cause the computing system to perform the method of any one of claims 1 to 20.

22. 21. A computing device configured to carry out the method of any one of claims 1 to 20.

Images