Fulfilling any actionable requirements before the user selects a particular autocomplete suggestion to complete the current user input

Abstract

To provide selectable autofill suggestions, which correspond to application actions that are at least partially fulfilled using server command data before a user selects a particular selectable autofill suggestion.SOLUTION: Proactively fulfilling command data in this way shortens the latency from user selection of a suggestion to fulfillment of a particular action. Initially, a partial input can be processed to generate autofill suggestions. The suggestions can be communicated to a server device for further processing. The autofill suggestions can also be rendered for selection at a touch display interface, thereby allowing a user to select one of the autofill suggestions. As the server provides command fulfillment data, the command fulfillment data can be available to a corresponding application so that any corresponding actions can be at least partially fulfilled before user selection.SELECTED DRAWING: Figure 1C

Description

[Technical Field]

[0001] Fulfilling possible requests before a user selects a particular autocomplete suggestion. [Background technology]

[0002] Humans may engage in human-computer interactions using interactive software applications referred to herein as "automated assistants" (also referred to as "digital agents," "chatbots," "interactive personal assistants," "intelligent personal assistants," "conversational agents," etc.). For example, a human (who may be referred to as a "user" when interacting with an automated assistant) may provide commands and / or requests using spoken natural language input (utterances), which may be converted to text and then processed and / or processed by providing textual (e.g., typed) natural language input.

[0003] In some cases, an automated assistant or other application may provide a feature that suggests to the user specific commands that the user can issue to the automated assistant or application. However, in many cases, the suggestions rendered to the user may not be easily interactive because there are several additional actions that need to be performed. For example, to respond to a suggested command selected by a user, the application may need to retrieve data from one or more server computing devices or applications and / or buffer certain data at the client computing device. This may increase the latency between the user selecting the suggestion and the application fulfilling the request. Such latency may delay certain resource-conserving actions, causing harm to other applications and / or computing devices. For example, a suggested command corresponding to powering down an appliance may not be executed quickly after the user selects the suggested command. As a result, the appliance may continue to consume energy and / or other resources until the respective application fulfills the suggested command. Furthermore, messages and / or other communications between users may be delayed as a result of such latency, which may result in unintended schedule changes and / or other consequences that may lead to loss of time and / or other resources. Summary of the Invention [Means for solving the problem]

[0004] Implementations described herein relate to the proactive fulfillment of executable requests when such requests correspond to autocomplete suggestions being presented to a user who has provided at least partial input to a user interface. Partial input may, for example, be a portion of input provided to a text field interface of a computing device (e.g., "on..."). Partial input may include one or more characters received at a computing device interface prior to an indication that the user has completed providing the corresponding input. The indication that the user has completed input may be, but is not limited to, selecting a GUI element, providing speech input, tactile input, interacting with a hardware button by providing a threshold amount of time elapsed since a particular input, and / or other interactions that may occur at the computing device. Partial input may be characterized by client data that is processed at a client computing device and also provided to a server computing device for processing. Each of the client and server computing devices may rely on a respective natural language understanding (NLU) process (e.g., a client NLU process and a server NLU process). In some implementations, the server NLU process may be more robust than the client NLU process.

[0005] The client computing device can use, for example, a matching process to generate one or more autocomplete suggestions and / or other instances of natural language content from the partial input. Each autocomplete suggestion, when attached as an addendum to the partial input, can create a respective input command that is specific as to the specific application action to be performed or the result to be achieved in response to the device and / or application receiving the respective input command. The client computing device can then render one or more selectable suggestions (e.g., selectable GUI elements and / or other selectable suggestion elements) in an interface of the client computing device or another interface of another computing device based on the autocomplete suggestions.

[0006] While the client computing device is rendering the selectable suggestions, the server computing device can process client data provided by the client computing device (e.g., partial interface input and / or one or more autocomplete suggestions). The client data can be processed by the server computing device using a server NLU process to facilitate generation of server suggestion data, which can include content corresponding to the autocomplete suggestions. The server suggestion data can then be communicated to the client computing device. The client computing device can process the server suggestion data to enhance the functionality of the one or more selectable suggestions rendered on the client computing device. For example, if the client data includes an autocomplete suggestion such as "Turn on the security lights," the server suggestion data can feature content indicating the status of the security lights. The content can include graphical data, such as a light bulb icon and / or a light switch icon, to visually indicate the current status of the security lights.

[0007] In some cases, the aforementioned process can occur while the user continues to provide the remaining partial input to the client computing device interface, or immediately after providing the partial input but before completing the input. During this time, the server computing device can identify and / or generate fulfillment data usable by the client computing device and / or one or more client applications to fulfill one or more executable requests corresponding to each respective selectable suggestion. For example, a particular selectable suggestion identified by the server computing device can be "Turn on the security lights." The fulfillment data corresponding to this particular selectable suggestion can characterize the current operational state of the "security lights" and / or the embedded link. When executed, a request to turn on the security lights is issued to an application or a controller controlling the security lights.

[0008] Once the server computing device generates the fulfillment data, it can communicate the fulfillment data to a client computing device for proactively fulfilling one or more selectable suggestions before the user selects one of the selectable suggestions. Once the client computing device receives the fulfillment data, each respective portion of the fulfillment data can be stored in association with one or more corresponding selectable suggestions. For example, a selectable suggestion labeled with natural language content, such as "Turn on the security lights," can be linked to a portion of the fulfillment data that characterizes an embedded link for turning on a particular security light to which the user may refer. Additionally or alternatively, another selectable suggestion labeled with other natural language content, such as "Turn on the air purifier," can be associated with another portion of the fulfillment data that characterizes command data for turning on a Wi-Fi-enabled air purifier. In some cases, the fulfillment data can be linked or associated with multiple different selectable suggestions while providing user input and / or before the user selects one or more of the selectable suggestions.

[0009] As a result, the latency between a user selecting a particular selectable suggestion and the corresponding action being initialized and / or completed can be reduced. For example, by actively searching for an embedded link for a particular selectable suggestion, a client computing device can bypass the operation of requesting an embedded link from a server computing device after a user selects a particular selectable suggestion. Bypassing this operation after receiving a selection of a selectable suggestion can also reduce the amount of network bandwidth and processing bandwidth consumed in response to receiving the selectable suggestion. Furthermore, bypassing this operation can also conserve computational resources for subsequent application processes that may be initiated when a user provides a series of requests to an application, such as an automated assistant.

[0010] The above description is provided as a summary of some implementations of the present disclosure. Further descriptions of these and other implementations are set forth in more detail below.

[0011] Other implementations may include a non-transitory computer-readable storage medium that stores instructions executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) to perform methods such as one or more of the methods described above and / or elsewhere herein. Still other implementations may include systems of one or more computers including one or more processors operable to execute the stored instructions to perform methods such as one or more of the methods described above and / or elsewhere herein.

[0012] It should be understood that all combinations of the foregoing concepts, and additional concepts described in more detail herein, are considered to be part of the subject matter disclosed herein, for example, all combinations of claimed subject matter appearing at the end of this disclosure are considered to be part of the subject matter disclosed herein. [Brief explanation of the drawings]

[0013] [Figure 1A] 1 shows a view of a user interacting with an interface that provides actively fulfilled, selectable autofill suggestions. [Figure 1B] 1 shows a view of a user interacting with an interface that provides actively fulfilled, selectable autofill suggestions. [Figure 1C] 1 shows a view of a user interacting with an interface that provides actively fulfilled, selectable autofill suggestions. [Figure 2]1 illustrates a system that provides selectable suggestions corresponding to automated assistant actions and / or application actions that are fulfilled at least in part using command data provided by a server device before a user selects a particular selectable suggestion. [Figure 3A] A method is shown for proactively initializing the performance of one or more actions suggested to a user in response to the user providing at least partial input to an interface of a computing device, prior to user selection. [Figure 3B] A method is shown for proactively initializing the performance of one or more actions suggested to a user in response to the user providing at least partial input to an interface of a computing device, prior to user selection. [Figure 4] FIG. 1 is a block diagram of an exemplary computer system. DETAILED DESCRIPTION OF THE INVENTION

[0014] 1A, 1B, and 1C illustrate views 100, 120, and 140 of a user 104 interacting with an interface 110 that provides actively fulfilled, selectable autofill suggestions. Each autofill suggestion may correspond to a respective application action that the user 104 can actively fulfill before selecting the respective autofill suggestion. One or more processes for actively initializing and / or fulfilling application actions may use a remote server device that processes the one or more autofill suggestions to generate command data that can be used to fulfill and / or initialize a particular action. For example, as provided in view 100 of FIG. 1A, the user 104 may provide a partial input 106 in a text field 108 rendered in an interface 110 of the computing device 102. The partial input 106 may be, for example, “Chan…,” which may include one or more characters that the user 104 intends as the first part of a natural language input (e.g., an auxiliary command such as “Chan[ge the temperature]”). The partial input 106 may be considered “partial” if the user 104 has not finished typing or providing input, additional input is provided over a threshold period selected to send the input to an application for processing, and / or the partial input 106 may otherwise indicate that the user 104 is incomplete with respect to subsequent input that the user 104 intends to provide (e.g., provided via speech and / or keyboard 116, which may receive touch input from the user's 104's hands 112).

[0015] FIG. 1B illustrates a view 120 in which a computing device 102 receives partial input 106 and communicates partial input data 132 from a user interface 122 to an application 124, such as an automated assistant. The computing device 102 and / or the application 124 can include a suggestion engine 126, which can execute a process for generating autofill suggestions 118, characterized by generating suggestion data 114 that can be rendered in the user interface 122. The process for generating autofill suggestions can include associating the partial input 106 with one or more previous inputs provided by the user 104 and / or one or more other users. Alternatively or additionally, the process for generating autofill suggestions can include processing the partial input data 132 using one or more trained machine learning models. Each autofill suggestion can then be ranked or otherwise prioritized based on other data accessible to the computing device 102. This other data can be, for example, contextual data and / or application data characterizing one or more interactions that have occurred or are occurring between the user 104 and the computing device 102, with prior permission from the user 104.

[0016] In some implementations, one or more autofill suggestions ranked or designated to be prioritized over one or more other autofill suggestions can be communicated to the server device 146 before the user 104 selects a particular autofill suggestion from the one or more autofill suggestions 130 rendered in the interface 110. For example, as shown in view 140 of FIG. 1C , the computing device 102 can communicate suggestion data 142 to the server device 146. The suggestion data 142 can characterize one or more autofill suggestions generated by the computing device 102. In some implementations, the suggestion data 142 can characterize a subset of the autofill suggestions generated by the suggestion engine 126 of the application 124. The subset of autofill suggestions can include, but is not limited to, one or more highest priority autofill suggestions from the autofill suggestions 130.

[0017] In response to receiving the suggestion data 142, the server device 146 can generate additional suggestion data 150 and / or command data 148, as shown in view 140 of FIG. 1C . The additional suggestion data 150 and / or command data 148 can be received by the computing device 102 after the autofill suggestions 130 are rendered in the interface 110, but before the user 104 selects one or more of the autofill suggestions. For example, the user 104 can continue to provide input using the keyboard 116 while the autofill suggestions 130 are being rendered in the interface 110, and even after the commands in the additional suggestion data 150 and 148 are received from the server device 146. Once the computing device 102 and / or the application 124 receive the command data 148, the application 124 and / or one or more other applications can access the command data 148 to facilitate the performance of one or more application actions associated with the autofill suggestions 130. For example, a smart thermostat application may access command data 148 in preparation for performing a temperature change action corresponding to an autofill suggestion of "change the temperature." Alternatively, or additionally, a smart home application may access command data 148 in preparation for receiving instructions from application 124 to change the brightness of lights in the home of user 104. These instructions may correspond to an autofill suggestion of "change the brightness in the living room."

[0018] In some implementations, the server device 146 can provide additional suggestion data 150, which can include content that can be rendered in the interface 110. The content can be, but is not limited to, graphical data, audio data, and / or other data that the user 104 can use to accomplish the completion of one or more application actions associated with the autofill suggestion 130. For example, the additional suggestion data 150 can be one or more interactive GUI elements 152, such as a slide bar and / or a rotatable dial. These GUI elements can be selected by the server device 146 using the suggestion data 142 and / or other data accessible to the server device. In some implementations, the server device 146 can access state data associated with particular devices and / or applications to generate GUI elements that characterize the state of the particular devices and / or applications. For example, in response to receiving the suggestion data 142, the server device 146 can access the state data or other content and generate the additional suggestion data 150 to characterize the state data or other content. In some cases, speakers accessible to computing device 102 may be capable of rendering audio at a volume level of "8," and therefore server device 146 may generate additional suggestion data 150 characterizing the volume level (e.g., an adjustable dial set to "8," as shown in FIG. 1C).

[0019] In another example, a thermostat accessible to the computing device 102 may be set to a temperature of 76 degrees, and thus the server device 146 may generate additional suggestion data 150 characterizing the temperature (e.g., an editable text field with “76°” populated in the editable text field). In some cases, content from the Internet may be accessed based on the suggestion data 142 to provide additional content that may characterize a particular autofill suggestion. For example, an image of an entree retrieved from the Internet may be stored in association with an autofill suggestion corresponding to a recipe (e.g., “Chana Dal Recipe”). In this way, the user 104 may more easily identify the purpose of a particular autofill suggestion, and portions of the Internet webpage (e.g., graphical content characterizing the entree) may already be buffered in the memory of the computing device 102 before the user 104 selects a particular autofill suggestion.

[0020] In some implementations, each autofill suggestion 130 and / or each interactive GUI element can be stored in association with command data 148, which can include embedded links, network data for establishing a connection to another device, a medium, and / or any other data accessible when the application is performing one or more actions. For example, if the autofill suggestion includes "change the brightness in the living room," the command data 148 can include network connection data accessible by the application 124 and / or any other application to establish a communication channel between the computing device 102 and one or more smart lights in the user's 104 home. Providing such command data 148 before the user 104 selects a particular autofill suggestion can conserve computational resources associated with performing one or more actions on the autofill suggestion. For example, the latency between the user's selection and the performance of the action can limit processing and network bandwidth for a certain period of time. Furthermore, such bandwidth limitations can consume electrical storage, which can be limited on certain portable electronic devices.

[0021] FIG. 2 illustrates a system 200 that provides selectable suggestions corresponding to automated assistant actions and / or application actions that are fulfilled at least in part using command data provided by a server device before a user selects a particular selectable suggestion. Automated assistant 204 can operate as part of an assistant application provided on one or more computing devices, such as computing device 202 and / or a server device. A user can interact with automated assistant 204 through assistant interface 220, which can be a microphone, a camera, a touchscreen display, a user interface, and / or any other device capable of providing an interface between a user and an application. For example, a user can initialize automated assistant 204 by providing verbal, textual, and / or graphical input to assistant interface 220 to cause automated assistant 204 to initiate one or more actions (e.g., providing data, controlling peripheral devices, accessing agents, generating input and / or output, etc.). Alternatively, automated assistant 204 can be initialized based on processing of context data 236 using one or more trained machine learning models. The context data 236 may characterize one or more characteristics of the environment to which the automated assistant 204 has access and / or one or more characteristics of a user who is predicted to intend to interact with the automated assistant 204.

[0022] The computing device 202 can include a display device, which may be a display panel including a touch interface for receiving touch input and / or gestures to enable a user to control applications 234 of the computing device 202 via the touch interface. In some implementations, the computing device 202 may lack a display device, thereby providing an audible user interface output without providing a graphical user interface output. Additionally, the computing device 202 can provide a user interface, such as a microphone, to receive spoken natural language input from a user. In some implementations, the computing device 202 can include a touch interface and lack a camera, but can optionally include one or more other sensors.

[0023] The computing device 202 and / or other third-party client devices can communicate with the server device over a network such as the Internet. Additionally, the computing device 202 and any other computing devices can communicate with each other over a local area network (LAN), such as a Wi-Fi network. The computing device 202 can offload computational tasks to the server device to conserve computational resources at the computing device 202. For example, the server device can host the automated assistant 204 and / or the computing device 202 can send inputs received at one or more assistant interfaces 220 to the server device. However, in some implementations, the automated assistant 204 can be hosted on the computing device 202, and various processes that can be associated with automated assistant operations can be executed on the computing device 202.

[0024] In various implementations, all or some aspects of the automated assistant 204 can be implemented on the computing device 202. In some of these implementations, aspects of the automated assistant 204 are implemented via the computing device 202 and can interface with a server device that can implement other aspects of the automated assistant 204. The server device can optionally serve multiple users and their associated assistant applications via multiple threads. In implementations in which all or some aspects of the automated assistant 204 are implemented via the computing device 202, the automated assistant 204 can be an application separate from the operating system of the computing device 202 (e.g., installed “on top of” the operating system)—or, alternatively, can be implemented directly by the operating system of the computing device 202 (e.g., considered an operating system application but integrated with the operating system).

[0025] In some implementations, the automated assistant 204 can include an input processing engine 206 that can use multiple different modules for processing input and / or output of the computing device 202 and / or the server device. For example, the input processing engine 206 can include a speech processing engine 208 that can process speech data received at the assistant interface 220 to identify text embodied in the speech data. The speech data can be transmitted from the computing device 202 to a server device, for example, to conserve computational resources at the computing device 202. Additionally or alternatively, the speech data can be processed exclusively at the computing device 202.

[0026] The process of converting the speech data to text can include speech recognition algorithms that can use neural networks and / or statistical models to identify groups of speech data that correspond to words or phrases. The text converted from the speech data is analyzed by the data analysis engine 210 and made available to the automated assistant 204 as text data that can be used to generate and / or identify command phrases, intents, actions, slot values, and / or any other content specified by the user. In some implementations, the output data provided by the data analysis engine 210 can be provided to the parameter engine 212 to determine whether the user has provided input (e.g., partial input or complete input), which corresponds to a particular intent, action, and / or routine that can be executed by the automated assistant 204 and / or an application or agent that can be accessed via the automated assistant 204. For example, the assistant data 238 can be stored on the server device and / or the computing device 202 and can include data defining one or more actions that can be executed by the automated assistant 204 and the parameters required to execute the action. The parameter engine 212 can generate one or more parameters for the intent, action, and / or slot value and provide the one or more parameters to the output generation engine 214. The output generation engine 214 can use the one or more parameters to communicate with the assistant interface 220 to provide output to the user and / or to communicate with one or more applications 234 to provide output to the one or more applications 234.

[0027] In some implementations, the automated assistant 204 can be an application that can be installed “on top of” the operating system of the computing device 202 and / or can itself form part of (or the entire) of the operating system of the computing device 202. The automated assistant application includes and / or has access to on-device speech recognition, on-device natural language understanding, and on-device fulfillment. For example, on-device speech recognition can be performed using an on-device speech recognition module that processes audio data (detected by a microphone) using an end-to-end speech recognition machine learning model stored locally on the computing device 202. The on-device speech recognition generates recognized text for utterances (if any) present in the audio data. Also, for example, on-device natural language understanding (NLU) can be performed using an on-device NLU module that processes recognized text generated using on-device speech recognition and, optionally, contextual data to generate NLU data.

[0028] The NLU data can include an intent corresponding to the utterance and, optionally, parameters of the intent (e.g., slot values). On-device fulfillment can be performed using an on-device fulfillment module that utilizes the NLU data (from the on-device NLU), and optionally other local data, to determine actions to perform to resolve the intent of the utterance (and optionally, parameters of the intent). This can include determining local and / or remote responses (e.g., answers) to the utterance, interacting with locally installed applications to perform based on the utterance, sending commands to Internet of Things (IoT) devices (directly or through corresponding remote systems) based on the utterance, and / or other resolution actions to perform based on the utterance. The on-device fulfillment can then initialize local and / or remote performance / execution of the determined actions to resolve the utterance.

[0029] In various implementations, remote speech processing, remote NLU, and / or remote fulfillment can be at least selectively utilized. For example, recognized text can at least selectively be sent to a remote automated assistant component for remote NLU and / or remote fulfillment. For example, recognized text can optionally be sent for remote performance in parallel with on-device performance or in response to failure of on-device NLU and / or on-device fulfillment. However, on-device speech processing, on-device NLU, on-device fulfillment, and / or on-device execution can be prioritized due to at least the reduced latency provided in resolving utterances (because resolving utterances does not require a client-server round trip). Furthermore, on-device functionality can be the only functionality available in situations where network connectivity is absent or limited.

[0030] In some implementations, the computing device 202 may include one or more applications 234 that may be provided by a third-party entity different from the entity that provided the computing device 202 and / or the automated assistant 204. The application state engine of the automated assistant 204 and / or the computing device 202 may access the application data 230 to determine one or more actions that may be performed by the one or more applications 234, as well as the state of each application of the one or more applications 234 and / or the state of each device associated with the computing device 202. The device state engine of the automated assistant 204 and / or the computing device 202 may access the device data 232 to determine one or more actions that may be performed by the computing device 202 and / or one or more devices associated with the computing device 202. Additionally, the application data 230 and / or any other data (e.g., device data 232) can be accessed by the automated assistant 204 to generate context data 236 that can characterize the context in which a particular application 234 and / or device is running and / or the context in which a particular user is accessing the computing device 202, accessing the application 234, and / or accessing any other device or module.

[0031] While one or more applications 234 are executing on the computing device 202, the device data 232 may characterize the current operational state of each application 234 executing on the computing device 202. Additionally, the application data 230 may characterize one or more features of the executing applications 234, such as the content of one or more interfaces rendered at the direction of the one or more applications 234. Alternatively or additionally, the application data 230 may characterize action schemas that may be updated by the respective applications and / or automated assistant 204 based on the current operational state of the respective applications and / or respective application actions. Alternatively or additionally, one or more action schemas of one or more applications 234 may remain static but may be accessed by the application state engine to determine appropriate actions to initiate via the automated assistant 204.

[0032] In some implementations, the computing device 202 may include an autofill suggestion engine 222 that can process various inputs received at the assistant interface 220 and / or any other computing device. For example, a user may begin providing typed input (e.g., one or more natural language characters) into a text field rendered in the interface of the computing device 202. In response to the user providing the partially typed input, the autofill suggestion engine 222 may identify one or more autofill suggestions that feature one or more character strings. When the one or more character strings are sequentially combined with the partially typed user input, a word or phrase is created that can be executed by the automated assistant 204 and / or another application 234. Each autofill suggestion identified by the autofill suggestion engine 222 may be rendered in the graphical user interface of the computing device 202, and each autofill suggestion may be selectable before the user finishes typing the typed input.

[0033] In some implementations, the computing device 202 and / or the automated assistant 204 can communicate each autofill suggestion to a server computing device before the user finishes typing the typed input. The server computing device can process each autofill suggestion to facilitate generating additional autofill suggestion content and / or command data associated with each respective autofill suggestion. The computing device 202 and / or the automated assistant 204 can include a command data engine 216 that can process command data received from the server computing device. The command data engine 216 can process the command data before the user selects a particular autofill suggestion. In some cases, processing the command data can include providing one or more respective applications 234 access to the command data. For example, the command data can include specific fulfillment data that a particular application 234 can use to fulfill an application action associated with a particular autofill suggestion before the user selects the particular autofill suggestion. Additionally or alternatively, the command data can include assistant command fulfillment data that the automated assistant 204 can use to fulfill an assistant action associated with a particular autofill suggestion before the user selects the particular autofill suggestion.

[0034] In some implementations, the first priority data can be generated by the suggestion ranking engine 218 using the application data 230, the device data 232, and / or the context data 236. For example, the current context of the user and / or the computing device 202 at the time the user provides the user input can be the basis for assigning a higher priority to the first autofill suggestion and a lower priority to the second autofill suggestion. Furthermore, the second priority data can be generated by the suggestion ranking engine of the server computing device using historical interaction data based on one or more previous interactions between one or more users and one or more applications and / or devices. For example, an autofill suggestion associated with an application that one or more users use frequently can be given a higher priority than an autofill suggestion associated with another application that one or more users use less frequently. The third priority data generated by the suggestion ranking engine 218 can then be based on the first priority data and the second priority data. For example, if a particular autofill suggestion is ranked highest by the first priority data and the second priority data, the third priority data can also rank that particular autofill suggestion highest. However, if a particular AutoFill suggestion is ranked highest in the first priority data but not in the second priority data, additional processing can be used to establish the final ranking of the particular AutoFill suggestion. For example, additional input provided by the user following the initial input that formed the basis of the set of AutoFill suggestions can be used as the basis for establishing the final ranking of the particular AutoFill suggestion.

[0035] In some cases, when partial input provided by a user is associated with rendering media data, the server computing device may rank a particular AutoFill suggestion higher based on information more readily available to the server computing device. For example, the server computing device may have access to information about newly released movies or music before the computing device 202 has access to such information. Thus, if the server ranking of an AutoFill suggestion is significantly higher than the client ranking of the AutoFill suggestion, this may be due to limited network access by the computing device 202 and / or more robust data available to the server computing device.

[0036] 3A and 3B illustrate methods 300 and 320 for proactively initializing the performance of one or more actions suggested to a user prior to user selection in response to the user providing at least partial input to an interface of a computing device. Methods 300 and 320 may be performed by one or more applications, devices, and / or any other apparatus for modules capable of responding to user input. Method 300 may include an operation 302 for determining whether a portion of user input has been received at the computing device. The user input may be text input including one or more characters received in an input field of an application interface. Thus, a portion of user input may be any portion of input that the user has not indicated, directly or indirectly, and / or explicitly or inherently, complete.

[0037] If a portion of the user input has been received, method 300 may proceed from operation 302 to operation 304. Otherwise, the computing device or other application may determine whether a portion of the user input has been received. Operation 304 may include causing one or more autofill suggestions to be rendered in an interface of the computing device. For example, the autofill suggestions may be used as addenda to the portion of the user input. The addenda may include natural language content that, when provided to the computing device along with the portion of the user input, causes the computing device to perform one or more actions. For example, the portion of the input may be "disable..." and the autofill suggestion may be "security system" or "dishwasher." In this manner, the combination of the partial input and the autofill suggestion, when provided to the computing device as spoken or typed input, may correspond to a complete command executable by the computing device.

[0038] Method 300 may proceed from operation 304 to operation 306, which may include providing autofill suggestion data to a server computing device. The autofill suggestion data may characterize one or more autofill suggestions generated at the computing device in response to user input. The computing device may provide the autofill suggestion data to the server computing device to benefit from additional computational resources and / or information available at the server computing device. For example, the server computing device may have access to a cloud storage service that may be used as a basis for generating additional autofill suggestion data and / or command data. The autofill suggestion data and / or command data may be provided to and / or processed by the computing device before a user selects a particular autofill suggestion rendered in the computing device's interface.

[0039] From operation 306, method 300 may proceed to operation 308, which may include determining whether an autofill suggestion has been received at the computing device. If an autofill suggestion has been received at the computing device, method 300 may proceed to operation 314. Operation 314 may include causing a particular application to perform one or more particular actions associated with the selected autofill suggestion. For example, a user may select an autofill suggestion such as "dishwasher," and in response, a smart device application (e.g., an IoT device application) may communicate a command to a smart dishwasher to disable the dishwasher.

[0040] However, if an autofill selection is not received at operation 308, method 300 may proceed to operation 310. Operation 310 may include determining whether command data has been received from a server computing device. The command data may be data that can be used by an application to at least partially fulfill an application action associated with a particular autofill suggestion before a user selects the particular autofill suggestion. The command data may include, but is not limited to, embedded links that are typically retrieved and / or executed by the application to modify the operation of the smart device. In some cases, the command data may include various fulfillment data for fulfilling one or more application actions corresponding to each of the one or more respective autofill suggestions.

[0041] If command data is received from the server computing device, method 300 may proceed to operation 312. Otherwise, method 300 may proceed from operation 310 via continuation element “A” to continuation element “A” provided in method 320 of FIG. 3B. Operation 312 may include enabling one or more applications to access the command data to perform one or more actions associated with the one or more autofill suggestions. If a particular application can access the command data, the particular application may operate to facilitate initialization of the performance of one or more actions associated with the autofill suggestion before a user selects the autofill suggestion. Method 300 may then proceed from operation 312 via continuation element “A” to optional operation 316 provided in method 320 of FIG. 3B. Operation 316 may include determining whether additional suggestion content has been received from the server computing device. The additional suggestion content may include data that can be rendered along with the one or more autofill suggestions to provide additional information to a user who may be viewing the one or more autofill suggestions. For example, the additional suggestion content may include graphical content that can characterize a state of a particular device associated with a particular autofill suggestion. For example, the additional suggestion content can be a graphical animation depicting a dishwasher actively washing dishes. Such information can therefore provide context that can further refine a user's decision to select a particular autofill suggestion from multiple different autofill suggestions. Alternatively or additionally, the additional suggestion content can include media data that can be rendered by a computing device to further characterize the various different autofill suggestions that are suggested to the user.

[0042] If the additional content is received from the server computing device, method 320 may optionally proceed from operation 316 to operation 318. Otherwise, the method of 320 may proceed from optional operation 316 to operation 308 via continuation element “B.” Operation 318 may include causing the computing device or another computing device to render the additional suggested content. By causing the additional suggested content to be rendered, each AutoFill suggestion can be enhanced with content that supplements information associated with the respective AutoFill suggestion. In some cases, this may reduce the latency between a user selecting an AutoFill suggestion and completing one or more actions in response to the AutoFill suggestion. For example, if the additional suggested content includes graphical content that features a preview of the suggested video, a certain amount of video may be buffered in the computing device's memory before the user selects the corresponding AutoFill suggestion. This reduced latency may be useful when a user is attempting to view security footage from a security camera and / or access other associated smart home devices.

[0043] From operation 318, the method may proceed via continuation element "B" to operation 308 provided in FIG. 3A . Method 300 may then continue from operation 308 accordingly. For example, if a user selects an autofill suggestion such as "disable the dishwasher...," method 300 may proceed to operation 314, causing a particular application to perform one or more actions related to disabling the "dishwasher." The particular application may be a smart home application for controlling various Wi-Fi-enabled devices in the user's home. The particular application may access particular command data prior to the user's suggestion selection, and the particular command data may characterize one or more actions executable by a module executing on the Wi-Fi-enabled dishwasher. For example, the command data may include an embedded link accessed by the particular application or another application (e.g., an automated assistant application) before the user selects the autofill suggestion.

[0044] 4 is a block diagram 400 of an exemplary computer system 410. The computer system 410 typically includes at least one processor 414 that communicates with a number of peripheral devices via a bus subsystem 412. These peripheral devices may include, for example, a storage subsystem 424 including memory 425 and a file storage subsystem 426, a user interface output device 420, a user interface input device 422, and a network interface subsystem 416. The input and output devices allow a user to interact with the computer system 410. The network interface subsystem 416 provides an interface to external networks and is coupled to corresponding interface devices in other computer systems.

[0045] The user interface input devices 422 may include pointing devices such as keyboards, mice, trackballs, touchpads, or other types of input devices, such as graphics tablets, scanners, touchscreens integrated into displays, voice recognition systems, microphones, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computer system 410 or communications network.

[0046] The user interface output devices 420 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computer system 410 to a user or to another machine or computer system.

[0047] Storage subsystem 424 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 424 may include logic to perform selected aspects of method 300, method 320, and / or implement one or more of system 200, computing device 102, server device 146, and / or other applications, devices, apparatuses, and / or modules described herein.

[0048] These software modules are typically executed by the processor 414 alone or in combination with other processors. The memory 425 used in the storage subsystem 424 may include several memories, including a main random access memory (RAM) 430 for storing instructions and data during program execution, and a read-only memory (ROM) 432 in which fixed instructions are stored. The file storage subsystem 426 may provide persistent storage for program and data files and may include a hard disk drive, a floppy disk drive, along with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of a particular implementation may be stored by the file storage subsystem 426 in the storage subsystem 424 or in another machine accessible by the processor 414.

[0049] Bus subsystem 412 provides a mechanism for enabling the various components and subsystems of computer system 410 to communicate with each other as intended. Although bus subsystem 412 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0050] Computer system 410 can be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computer system 410 shown in Figure 4 is intended only as a specific example to illustrate some implementations. Many other configurations of computer system 410 are possible, having more or fewer components than the computer system shown in Figure 4.

[0051] In situations where the systems described herein collect or may utilize personal information about users (or often referred to herein as “participants”), users may be provided with an opportunity to control whether a program or feature collects user information (e.g., information about the user's social networks, social actions or activities, occupation, user preferences, or the user's current geographic location) or whether and / or how to receive content from content servers that may be more relevant to the user. Also, certain data may be processed in one or more ways before being stored or used so that personally identifiable information is removed. For example, a user's ID may be processed so that the user's personally identifiable information cannot be identified, or the user's geographic location may be generalized by where the geographic location information is obtained (e.g., to the city, zip code, or state level) so that the user's specific geographic location cannot be identified. Thus, users have control over how information about them is collected and / or used.

[0052] While several implementations have been described and illustrated herein, various other means and / or structures for performing a function and / or obtaining a result and / or obtaining one or more of the advantages described herein may be utilized, and each such variation and / or modification is deemed to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are meant to be exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on the particular application or applications for which the teachings are used. Those skilled in the art will recognize or be able to ascertain using no more than routine experimentation many equivalents to the specific implementations described herein. Accordingly, it should be understood that the foregoing implementations are presented by way of example only, and that, within the scope of the appended claims and their equivalents, implementations may be practiced otherwise than as specifically described and claimed. Implementations of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. Furthermore, any combination of two or more such features, systems, articles, materials, kits, and / or methods is within the scope of the present disclosure, provided that such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent.

[0053] In some implementations, a method implemented by one or more processors is described as including an act such as receiving, at an interface of a client computing device, user input including one or more natural language characters. The method may further include an act of causing a selectable graphical user interface (GUI) element to be rendered at a display interface of the client computing device based at least in part on the user input, the selectable GUI element including natural language content characterizing autofill suggestions for the user input. The method may further include an act of providing the autofill suggestions from the client computing device to a server computing device. The method may further include an act of receiving, subsequent to causing the rendering of the selectable GUI element at the display interface but prior to receiving a user selection of the selectable GUI element, command data from the server computing device generated based on the autofill suggestions. The command data is accessible by one or more applications to facilitate performing one or more actions associated with the selectable GUI element. The method may further include an act of receiving, subsequent to receiving the command data, a user selection of the selectable GUI element. The method may further include an act of causing one or more applications to initiate performance of the one or more actions using the command data in response to receiving the user selection of the selectable GUI element.

[0054] In some implementations, the method may further include an act of receiving, from the server computing device, additional suggested content based on the autofill suggestions of the user input. The method may further include an act of causing the additional suggested content to be rendered on a display interface having the selectable GUI element before receiving a user selection of the selectable GUI element. In some implementations, the method may further include an act of causing the additional selectable GUI element to be rendered on a display interface of the client computing device based at least in part on the user input, the additional selectable GUI element including additional natural language content characterizing the additional autofill suggestions for the user input. In some implementations, the method may further include an act of providing the additional autofill suggestions from the client computing device to the server computing device, the command data received from the server computing device also being based on the additional autofill suggestions. In some implementations, the method may further include an act of determining, at the client computing device after causing the selectable GUI element and the additional selectable GUI element to be rendered on the display interface but before receiving a user selection of the selectable GUI element, a respective priority of each of the selectable GUI element and the additional selectable GUI element, the respective priority being based on historical interaction data characterizing one or more previous interactions between the user and one or more applications.

[0055] In some implementations, each respective priority is further based on other historical interaction data characterizing one or more other previous interactions between one or more other users and the one or more applications. In some implementations, the command data is received from the server computing device while the user is providing the additional portion of user input. In some implementations, the one or more applications include an automated assistant responsive to natural language input at one or more interfaces of the client computing device. In some implementations, the method may further include, after causing the selectable GUI element to be rendered at the display interface but before receiving a user selection of the selectable GUI element, causing the one or more applications to perform a function for buffering a portion of the command data to be rendered via the client computing device in response to the user selecting the selectable GUI element. In some implementations, providing the autofill suggestions to the server computing device includes causing the server computing device to generate the command data using the autofill suggestions and a server natural language understanding (NLU) process, the server NLU process being distinct from a client NLU process running on the client computing device.

[0056] In other implementations, a method implemented by one or more processors is described including an act such as receiving, by a server computing device and from a client computing device, client data characterizing one or more autofill suggestions for partial interface input received at the client computing device, wherein the client computing device renders the one or more autofill suggestions in response to the partial interface input at the client computing device. In some implementations, the method may further include an act of generating, based on the one or more autofill suggestions, command data characterizing one or more application actions that can be initiated at the client computing device by one or more respective applications. In some implementations, the method may further include an act of providing the command data to the client computing device before a user selects a particular autofill suggestion of the one or more autofill suggestions at the client computing device, wherein, in response to receiving the command data, the client computing device causes a particular application of the one or more respective applications to access the command data in preparation for performing the particular application action of the one or more application actions.

[0057] In some implementations, the method may further include an act of generating, at the server computing device, additional suggested content based on the one or more autofill suggestions. In some implementations, the method may further include an act of causing the user, at the client computing device, to render the additional suggested content through an interface of the client computing device before selecting a particular autofill suggestion of the one or more autofill suggestions. In some implementations, the command data is received by the client computing device while the user provides additional input to add partial interface input. In some implementations, the one or more respective applications include an automated assistant that responds to natural language input at one or more interfaces of the client computing device. In some implementations, causing a particular application of the one or more respective applications to access the command data in preparation for performing a particular application action of the one or more application actions includes causing the particular application to render a portion of the command data through one or more interfaces of the client computing device.

[0058] In yet another implementation, a method implemented by one or more processors is described as including an operation such as receiving, at an interface of a client computing device, user input characterizing a portion of an assistant command directed to an automated assistant. The automated assistant responds to the utterance received at the client computing device. The method may further include an operation of identifying autofill suggestions based on the portion of the assistant command in response to receiving the user input. The method may further include an operation of generating, based on the autofill suggestions, client command data characterizing each assistant action responsive to each autofill suggestion of the autofill suggestions. The method may further include an operation of communicating, by the client computing device, the autofill suggestions to a server computing device via a network connection, wherein, based on the server computing device receiving the autofill suggestions, the server computing device generates server command data characterizing each additional assistant action responsive to each autofill suggestion of the autofill suggestions and performs an operation including providing the server command data to the client computing device. The method may further include an operation of causing, by the client computing device, one or more selectable suggestion elements to be rendered at an interface of the client computing device or another interface, each selectable suggestion element of the one or more selectable suggestion elements including respective content based on the autofill suggestion. The method may further include an act of receiving, from the server computing device, assistant command fulfillment data after the client computing device renders one or more selectable suggestion elements but before the user selects a particular selectable suggestion element of the one or more selectable suggestion elements, wherein the assistant command fulfillment data is accessed by the automated assistant to facilitate fulfillment of the one or more assistant actions or the fulfillment of one or more additional assistant actions.

[0059] In some implementations, the assistant command fulfillment data is received from the server computing device while the user is providing the additional portion of the assistant command. In some implementations, the method can further include an operation of modifying each selectable suggestion of the one or more selectable suggestion elements based on the additional portion of the assistant command being provided by the user after the client computing device renders the one or more selectable suggestion elements but before the user selects a particular selectable suggestion element of the one or more selectable suggestion elements. In some implementations, the method can further include an operation of receiving additional suggestion content based on the autofill suggestion from the server computing device and causing the additional suggestion content to be rendered via an interface of the client computing device or another interface after the client computing device renders the one or more selectable suggestion elements but before the user selects a particular selectable suggestion element of the one or more selectable suggestion elements.

[0060] In some implementations, the additional suggestion content includes graphical content rendered on an interface of the client computing device or other interface, the graphical content characterizing a state of an operation being performed by a particular application and associated with a particular autofill suggestion of the autofill suggestion. In some implementations, the method can further include an operation in which, after the client computing device renders the one or more selectable suggestion elements but before the user selects a particular selectable suggestion element of the one or more selectable suggestion elements, the automated assistant uses the assistant command fulfillment data to complete a particular assistant action of the one or more additional assistant actions. In some implementations, communicating the autofill suggestions to a server computing device via a network connection includes having the server computing device generate the autofill suggestions and the assistant command fulfillment data using a server natural language understanding (NLU) process, the server NLU process being different from the client NLU process running on the client computing device.

[0061] In yet another implementation, a method implemented by one or more processors is described as including an operation such as receiving, at an interface of a client computing device, user input characterizing a portion of an assistant command directed to an automated assistant. The automated assistant responds to the utterance received at the client computing device. The method may further include an operation of identifying autofill suggestions based on the portion of the assistant command in response to receiving the user input. The method may further include an operation of generating client command data based on the autofill suggestions, the client command data characterizing each assistant action responsive to each autofill suggestion of the autofill suggestions. The method may further include an operation of selecting a first set of assistant actions from the client command data based on the client command data, the first set of assistant actions being a subset of the assistant actions identified in the client command data, the first set of assistant actions being selected further based on historical client data characterizing previous interactions between a user and the automated assistant at the client computing device. The method can further include an operation of communicating, by the client computing device, the autofill suggestions to a server computing device over a network connection, wherein, in response to the server computing device receiving a portion of the assistant commands, the server computing device performs an operation including generating server command data characterizing each additional assistant action responsive to each autofill suggestion and selecting a second set of assistant actions from the server command data. In some implementations, the second set of assistant actions is another subset of the assistant actions identified by the server command data, and the second set of assistant actions is identified based on server data characterizing previous interactions between one or more other users and the automated assistant.In some implementations, the method may further include an operation of initializing the performance of a particular assistant action identified in the first set of assistant actions and another particular assistant action identified in the second set of assistant actions. [Explanation of symbols]

[0062] 100 views 106 Partial Input 110 Interface 120 Views 140 Views 104 users 102 Computing Devices 122 User Interface 124 Applications 126 Suggestion Engine 128 Command Engine 116 keyboards 130 Autofill Suggestions 114 Proposal Data 118 Proposal generation 132 Partial Input Data 142 Proposal Data 146 Server Devices 148 Command Data 150 additional suggested data 152 Interactive GUI Elements 200 systems 202 Computing Devices 234 Applications 220 Assistant Interface 230 Application Data 232 Device Data 236 Context Data 238 Assistant Data 204 Automated Assistant 206 Input Processing Engine 208 Audio Processing Engine 210 Data Analysis Engine 212 Parameter Engine 214 Output Generation Engine 216 Command Data Engine 218 Suggestion Ranking Engine 222 Autofill Suggestion Engine 424 Memory Subsystem 425 Memory Subsystem 426 File Storage Subsystem 432 ROM 430 RAM 422 User Interface Input Devices 410 Computer Systems 412 Bus Subsystem 414 processor 416 Network Interface 420 User Interface Output Device

Claims

1. 1. A method implemented by one or more processors of a client device, comprising: receiving, via the client device, a first user input, the first user input being a partial user input including one or more natural language characters; in response to receiving the partial user input via the client device; determining one or more autofill suggestions based on the one or more natural language characters; displaying the one or more autofill suggestions in a user interface of the client device, the one or more autofill suggestions suggesting one or more actions that can be performed by one or more applications; communicating autofill suggestion data characterizing the one or more autofill suggestions to a server device; Steps and receiving from the server device additional suggested content generated by the server device based on the autofill suggestion data, the additional suggested content supplementing the one or more autofill suggestions; and displaying the additional suggested content along with the one or more autofill suggestions on the user interface of the client device after an initial display of the one or more autofill suggestions.

2. The method of claim 1 , wherein the additional suggested content includes graphical content that indicates a current state of a device controllable by a first of the one or more applications.

3. The method of claim 2 , wherein the graphical content includes interactive graphical user interface (GUI) elements that indicate the current state of the device, and the interactive GUI elements enable modifications to the current state of the device.

4. receiving a second user input selecting an autofill suggestion among the one or more autofill suggestions after the additional suggestion content is displayed in the user interface to supplement the one or more autofill suggestions; 2. The method of claim 1, further comprising: in response to receiving the second user input, causing an application action suggested by the selected autofill suggestion to be performed.

5. The method of claim 1 , wherein the one or more autofill suggestions include a particular autofill suggestion suggesting particular media content, and the additional suggested content includes a preview of the media content.

6. 6. The method of claim 5, wherein the additional suggested content further includes a portion of the media content, the portion of the media content being buffered in memory of the client device in response to receiving the additional suggested content.

7. 2. The method of claim 1, further comprising receiving from the server device command data generated by the server device based on the autofill suggestion data, the command data being received along with the additional suggested content, and the command data enabling the one or more applications to perform the one or more actions.

8. 8. The method of claim 7, wherein the command data includes an embedded link that, when executed, causes a first of the one or more applications to perform a corresponding action.

9. The method of claim 7 , wherein the command data includes network data for establishing a connection between the client device and one or more devices controllable by the one or more applications.

10. 1. A method implemented by one or more processors of a client device, comprising: receiving, via the client device, a partial user input comprising one or more natural language characters; in response to receiving the partial user input via the client device; determining one or more autofill suggestions based on the one or more natural language characters; displaying the one or more autofill suggestions in a user interface of the client device, the one or more autofill suggestions suggesting one or more actions that can be performed by one or more applications; communicating autofill suggestion data characterizing the one or more autofill suggestions to a server device; Steps and receiving from the server device command data and additional suggested content generated by the server device based on the AutoFill suggestion data; the command data enabling the one or more applications to perform the one or more actions; the additional suggested content supplements the one or more autofill suggestions; and and displaying the additional suggested content along with the one or more autofill suggestions on the user interface of the client device after an initial display of the one or more autofill suggestions.

11. receiving, after receiving the command data from the server device, further user input via the client device selecting an autofill suggestion among the one or more autofill suggestions, the selected autofill suggestion suggesting a first action executable by a first application of the one or more applications; 11. The method of claim 10, further comprising: in response to receiving the further user input, causing the first application to perform the first action using the command data.

12. The method of claim 11 , wherein the further user input is received after the additional suggested content is displayed on the user interface of the client device.

13. The method of claim 11 , wherein the additional suggested content includes graphical content that indicates a current state of a device controllable by a first of the one or more applications.

14. The method of claim 13 , wherein the graphical content includes interactive graphical user interface (GUI) elements that indicate the current state of the device, and the interactive GUI elements enable modifications to the current state of the device.

15. The method of claim 10 , wherein the one or more autofill suggestions include a particular autofill suggestion suggesting particular media content, and the additional suggested content includes a preview of the media content.

16. 16. The method of claim 15, wherein the additional suggested content further includes a portion of the media content, the portion of the media content being buffered in memory of the client device in response to receiving the additional suggested content.

17. 1. A method implemented by one or more processors of a client device, comprising: receiving partial input via an input interface of the client device, the partial input including one or more natural language characters; in response to receiving partial user input via the input interface of the client device; determining one or more autofill suggestions based on the one or more natural language characters; displaying the one or more determined autofill suggestions in a user interface of the client device, the one or more autofill suggestions suggesting one or more actions that can be performed by one or more applications; communicating autofill suggestion data characterizing the one or more autofill suggestions to a server device; Steps and receiving from the server device command data generated by the server device based on the AutoFill suggestion data, the command data enabling the one or more applications to perform the one or more actions; receiving, after receiving the command data from the server device, further user input via the client device selecting an autofill suggestion among the one or more autofill suggestions, the selected autofill suggestion suggesting a first action executable by a first application of the one or more applications; and in response to receiving the further user input, causing the first application to perform the first action using the command data.

18. 20. The method of claim 17, wherein the command data includes an embedded link that, when executed, causes the first application to perform the first action.

19. 20. The method of claim 17, wherein the command data includes network data for establishing a connection between the client device and a device controllable by the first application.

20. 18. The method of claim 17, wherein the first action is to modify a current state of a device controlled by the first application and separate from the client device and the server device, and the command data indicates the current state of the device.

21. 1. A method implemented by one or more processors of a client device, comprising: receiving, via the client device, a first user input, the first user input being a partial user input including one or more natural language characters; in response to receiving the partial user input via the client device; determining one or more autofill suggestions based on the one or more natural language characters; displaying the one or more autofill suggestions on a user interface of the client device, the one or more autofill suggestions suggesting one or more application actions that can be performed by one or more applications; communicating autofill suggestion data to a server device characterizing the subset of autofill suggestions, the subset of autofill suggestions selected from the one or more autofill suggestions and suggesting a subset of application actions from the one or more application actions; Steps and receiving from the server device additional suggestion content generated by the server device based on the autofill suggestion data characterizing the subset of the autofill suggestions, the additional suggestion content supplementing the subset of the autofill suggestions; and causing the additional suggestion content to be displayed in the user interface of the client device for the subset of autofill suggestions to supplement the subset of autofill suggestions after the initial display of the one or more autofill suggestions and before receiving a second user input selecting any of the one or more autofill suggestions.

22. ranking the one or more autofill suggestions; and selecting one or more top-ranked autofill suggestions from the ranked one or more autofill suggestions as the subset of the autofill suggestions.

23. 23. The method of claim 22, wherein the step of ranking the one or more autofill suggestions is based on data accessible to the client device, the data accessible to the client device including contextual or application data characterizing one or more interactions that have occurred or are occurring between a user of the client device and the client device.

24. 23. The method of claim 22, wherein the one or more autofill suggestions include a first autofill suggestion and a second autofill suggestion, and wherein the first autofill suggestion associated with an application that is used more frequently than an application associated with the second autofill suggestion is ranked higher than the second autofill suggestion.

25. 22. The method of claim 21, wherein the additional suggestion content includes graphical content that indicates a current state of a device controllable by a first application of the one or more applications that corresponds to a first autofill suggestion from the subset of autofill suggestions.

26. 26. The method of claim 25, wherein the graphical content includes interactive graphical user interface (GUI) elements that indicate the current state of the device, and the interactive GUI elements enable modifications to the current state of the device.

27. receiving a second user input selecting an autofill suggestion from the subset of autofill suggestions after the additional suggestion content is displayed in the user interface to supplement the subset of autofill suggestions; 22. The method of claim 21, further comprising: in response to receiving the second user input, causing an application action suggested by the selected autofill suggestion to be performed.

28. 22. The method of claim 21, wherein the subset of autofill suggestions includes a specific autofill suggestion that suggests specific media content, and the additional suggested content includes a preview of the specific media content that complements the specific autofill suggestion.

29. 30. The method of claim 28, wherein the additional suggested content further includes a portion of the particular media content, and the portion of the particular media content is buffered in memory of the client device in response to receiving the additional suggested content.

30. 22. The method of claim 21, further comprising receiving from the server device command data generated by the server device based on the autofill suggestion data, the command data being received along with the additional suggested content, and the command data facilitating execution of the subset of application actions.

31. 31. The method of claim 30, wherein the command data includes an embedded link that, when executed, causes a corresponding application from the one or more applications to perform a corresponding application action from the subset of application actions.

32. 21. The method of claim 20, wherein the command data includes network data for establishing a connection between the client device and one or more devices controllable by the subset of applications that correspond to the subset of autofill suggestions.

33. 1. A method implemented by one or more processors of a client device, comprising: receiving, via the client device, a first user input, the first user input being a partial user input including one or more natural language characters; in response to receiving the partial user input via the client device; determining one or more autofill suggestions based on the one or more natural language characters; displaying the one or more autofill suggestions on a user interface of the client device, the one or more autofill suggestions suggesting one or more application actions that can be performed by one or more applications; communicating autofill suggestion data to a server device characterizing the subset of autofill suggestions, the subset of autofill suggestions selected from the one or more autofill suggestions, and suggesting a subset of application actions from the one or more application actions; receiving from the server device command data generated by the server device based on the AutoFill suggestion data; the command data is accessible by a subset of applications from the one or more applications to invoke the subset of application actions suggested by the subset of the autofill suggestions; and causing at least one application of the subset of applications to access the command data in preparation for performing at least one application action of the subset of application actions before receiving a second user input selecting one of the one or more autofill suggestions.

34. ranking the one or more autofill suggestions; and selecting one or more top-ranked autofill suggestions from the ranked one or more autofill suggestions as the subset of the autofill suggestions.

35. 34. The method of claim 33, further comprising receiving from the server device additional suggested content generated by the server device based on the AutoFill suggestion data, the additional suggested content supplementing the subset of AutoFill suggestions.

36. 36. The method of claim 35, wherein the additional suggestion content includes graphical content that indicates a current state of a device controllable by a first application of the one or more applications that corresponds to a first autofill suggestion from the subset of autofill suggestions.

37. 37. The method of claim 36, wherein the graphical content includes interactive graphical user interface (GUI) elements that indicate the current state of the device, and the interactive GUI elements enable modifications to the current state of the device.

38. 1. A method implemented by one or more processors of a client device, comprising: receiving, via the client device, user input characterizing a portion of an assistant command directed to an automated assistant; in response to receiving the user input via the client device characterizing the portion of the assistant command; determining autofill suggestions based on the portion of the assistant command, the autofill suggestions suggesting assistant actions that can be performed by the automated assistant; and generating, based on the autofill suggestions, client command data characterizing each assistant action among the assistant actions suggested by the autofill suggestions; selecting a first set of assistant actions from the client command data based on the client command data; communicating autofill suggestion data characterizing the autofill suggestions to a server device; receiving data from the server device indicating a selection of a second set of assistant actions from server command data, the server command data being generated by the server device in response to receiving the autofill suggestion data characterizing the autofill suggestions, the server command data characterizing each assistant action of the assistant actions suggested by the autofill suggestions; and initiating execution of a particular assistant action within the first set of assistant actions and execution of another particular assistant action within the second set of assistant actions.

39. 39. The method of claim 38, wherein the first set of assistant actions is selected further based on historical client data characterizing previous interactions between a user of the client device and the automated assistant installed on the client device.

40. 39. The method of claim 38, wherein the second set of assistant actions is identified based on previous interactions between one or more users and the automated assistant.

41. 41. A client device comprising: a memory for storing instructions; and one or more processors operable to execute said instructions to perform the method of any one of claims 1 to 40.

Images