Supplement of voice input to automated assistant by selected suggestion

Abstract

To provide a suggestion for completing spoken utterance directed to an automated assistant through a display modality and in some cases, to continuously provide an updated suggestion based on various factors while spoken utterance is in progress.SOLUTION: A method compiles a user request from a content of ongoing spoken utterance and a content of an arbitrary and selected suggestion element, performs arbitrary action corresponding to a currently compiled portion of the user request through an automatic assistant when the currently compiled portion of the user request can be performed through the automatic assistant and uses the arbitrary content resulting from performance of the actions to provide further suggestions with an arbitrary recognizable context.SELECTED DRAWING: Figure 2B

Description

[Background technology]

[0001] Humans may engage in human-computer dialogues with interactive software applications, referred to herein as "automated assistants" (also referred to as "digital agents," "chatbots," "conversational personal assistants," "intelligent personal assistants," "assistant applications," "conversational agents," etc.). For example, a human (who may be referred to as a "user" when interacting with an automated assistant) may provide commands and / or requests to the automated assistant using verbal natural language input (i.e., speech), which may in some cases be converted to text and then processed, and / or by providing textual (e.g., typed) natural language input. The automated assistant responds to the request by providing a responsive user interface output, which may include audible and / or visual user interface output. Thus, the automated assistant may provide a voice-based user interface.

[0002] In some instances, a user may provide a spoken utterance that is intended by the user to cause the performance of an automated assistant action, but does not result in the performance of the intended automated assistant action. For example, the spoken utterance may be provided in a syntax that is not understandable by the automated assistant and / or may lack required parameters for the automated assistant action. As a result, the automated assistant may be unable to fully process the spoken utterance and determine that it is a request for an automated assistant action. This leads to the automated assistant not providing a response to the spoken utterance or providing an error in the response, such as "Sorry, I can't help with that," and / or an error sound. Despite the automated assistant failing to perform the intended action of the spoken utterance, various computer and / or network resources are nevertheless consumed in processing the spoken utterance and attempting to resolve an appropriate action. For example, audio data corresponding to the spoken utterance may be transmitted, subjected to speech-to-text processing, and / or subjected to natural language processing. Such consumption of resources is wasteful because the intended automated assistant action is not performed, and the user is likely to attempt to provide a reconstructed spoken utterance when later seeking to perform the intended automated assistant action. Moreover, such reconstructed spoken utterance would also have to be processed. Moreover, this would cause latency in the performance of the automated assistant action compared to if the user had instead provided a preferred spoken utterance to the automated assistant initially. Summary of the Invention [Means for solving the problem]

[0003] Implementations described herein relate to providing suggestions for completing a verbal utterance directed to an automated assistant via a display modality, and possibly continually providing updated suggestions based on various factors while the verbal utterance is ongoing. For example, suggestions may be provided to encourage the automated assistant to develop actionable dialogue phrases that will collectively cause the automated assistant to perform the automated assistant action originally intended by the user, while also streamlining the interaction between the user and the automated assistant. Additionally or alternatively, suggestions may be provided to encourage the user to employ the automated assistant to perform an action that may reduce the frequency and / or length of time that the user will participate in a current and / or subsequent dialogue session with the automated assistant.

[0004] In some implementations, during a dialog session between a user and an automated assistant, the user can provide verbal utterances to facilitate triggering the automated assistant to perform one or more actions. The verbal utterances can be captured by audio data and speech-to-text processing performed on the audio data to generate one or more text segments, each of which is a corresponding interpretation of the verbal utterance. The one or more text segments can be processed to determine whether the verbal utterance corresponds to a complete request and / or an actionable request. As used herein, a "complete request" or an "actionable request" is one that, when fully processed by the automated assistant, triggers the performance of one or more corresponding automated assistant actions, where the performed automated assistant action is not a default error type action (e.g., is not an audible response of "Sorry, I can't help with that"). For example, an actionable request can cause one or more smart devices to be controlled, can cause certain audible and / or graphical content to be rendered, etc. When it is determined that the verbal utterance corresponds to an incomplete request, the one or more text segments can be further processed to determine and provide one or more suggestions for completing the request. Each suggestion of the one or more suggestions may include additional text that, when combined with a corresponding one of the one or more text segments, provides a complete request. In other words, if the user continues his / her dialog session by repeating the text of a particular suggestion (or an alternative text that generally matches the text based on the particular suggestion), the automated assistant will perform the corresponding automated assistant action. Thus, the generation of suggestions assists the user in performing the task. While the user is entering the request, the user is provided with information that guides the user in interacting with the assistant.The information presented to the user is based on the user's existing input and therefore uses objective analysis to provide information to the user, so the user is provided with objectively relevant information to assist the user in performing a task.

[0005] In some implementations, the computing device can generate different interpretations of the spoken utterance, and variations in those interpretations can result in corresponding variations to the suggestions provided in response to the spoken utterance. For example, a spoken utterance such as "Assistant, can you change..." can correspond to multiple interpretations by the computing device. The above spoken utterance can be interpreted as a request to change an IoT device setting via an automated assistant or as a request to change an application setting via an automated assistant. Each interpretation can be processed to generate a corresponding suggestion that can be presented to a user via a display panel in communication with the computing device. For example, the computing device can cause the display panel to show the initial text of the spoken utterance (e.g., "Assistant, can you change...") along with a list of suggestions based on both interpretations of the spoken utterance. The suggestions can include at least one suggestion based on one interpretation and at least one other suggestion based on another interpretation.

[0006] For example, the at least one suggestion based on the “change IoT device settings” interpretation may be “set thermostat to 72,” which, when spoken by a user, may be text that supplements the original verbal utterance. As an example, the display panel may present the original verbal utterance and the at least one suggestion together in response to the user speaking the text of the at least one suggestion. In other words, the display panel may present the following text: “Assistant, can you change the thermostat to 72?” The user may view the presented text and identify that the portion (i.e., “set thermostat to 72”) was suggested and was not included within his or her original verbal utterance (i.e., “Assistant, can you change…”). Upon identifying the suggested portion, the user may repeat the text of the suggested portion and trigger the automated assistant to act to facilitate completing an action, such as changing the setting of the user’s thermostat to 72. In some implementations, the display panel can present as a suggestion text that includes only the suggested, unspoken portion (e.g., "Turn the thermostat to 72" without presenting "Assistant, can you?"). This can allow the suggestion to be easily glanced at and enable a user to quickly see the suggestion and confirm further verbal input that the user can provide to construct an actionable request, such as "Turn the thermostat to 72" or further verbal input of "Turn the thermostat to 70" (different from the suggestion but confirmable based on the suggestion).

[0007] In various implementations, the text portion of the suggestion may be presented, possibly in combination with an icon or other graphical element that indicates an action that will be performed as a result of providing further verbal input that matches the text portion of the suggestion. As one example, a suggestion may include a text portion that reads "Change the thermostat to 72," along with a graphical indication of a thermostat and up and down arrows to indicate that further verbal input that matches the suggestion will result in a change in the setpoint of the thermostat. As another example, a suggestion may include a text portion that reads "Play a TV station on the living room TV," along with the outline of a television with a play icon inside it to indicate that further verbal input that matches the suggestion will result in the playing of streaming content on the television. As yet another example, a suggestion may include "Turn off the living room lights," along with a light bulb icon with an X superimposed on it to indicate that further verbal input that matches the suggestion will result in the lights being turned off. Such icons may also allow suggestions to be glanced at more easily, allowing a user to quickly see the icon first and, if the icon matches the user's desired action, to see the text portion of the suggestion. This may allow a user to quickly glance at the graphical elements of multiple simultaneously presented suggestions (without first reading the corresponding text portions) to see which ones match the user's intent, and then read only that corresponding text portion. This may reduce the latency in providing further verbal input when a user is presented with multiple suggestions, and consequently reduce the latency in performing the resulting action.

[0008] In some implementations, a spoken utterance may be interpreted differently as a result of how a user articulates and / or pronounces a particular word. For example, a computing device receiving the above spoken utterance "Assistant, can you change..." may determine with X% certainty that the spoken utterance includes the word "change" and with Y% certainty (where Y is less than X) that the spoken utterance includes the word "tidy up" and thus refers to a request to perform an "tidy up" function. For example, speech-to-text processing of the spoken utterance may yield two separate interpretations ("change" and "tidy up") with different certainties. As a result, an automated assistant tasked with responding to the spoken utterance may present suggestions based on each interpretation ("change" and "tidy up"). For example, the automated assistant may graphically present a first suggestion, such as "...Thermostat to 72," corresponding to the "change" interpretation, and a second suggestion, such as "...My Desktop File Folder," corresponding to the "organize" interpretation. Furthermore, if the user later repeats the content of the first suggestion by repeating "Thermostat to 72" (or similar content, such as "Thermostat to 80"), the user may cause the "change" function to be performed via the automated assistant (and based on the user's further repetition). However, if the user later repeats the content of the second suggestion by repeating "My Desktop File Folder" (or similar content, such as "My Documents Folder"), the user may cause the "organize" function to be performed via the automated assistant (and based on the user's further repetition). In other words, the automated assistant will utilize the interpretation ("change" or "organize") that matches the repeated suggestion.For example, even if "organize" is predicted with only 20% probability and "change" is predicted with 80% probability, "organize" may nevertheless be utilized if the user further speaks "My Desktop file folder." As another example, and as a variation of the "change" and "organize" examples, a first suggestion "change thermostat to 75" and a second suggestion "organize My Desktop file folder" may be presented. If the user selects "organize My Desktop file folder" through touch input or provides further verbal input, such as "organize My Desktop file folder" or "second," the "organize" interpretation will be utilized instead of the "change" interpretation. In other words, even though "change" is initially selected as the correct interpretation based on a higher probability, "organize" may nevertheless supersede that selection based on a user selection or further verbal input directed to the "organize" suggestion. In these and other ways, speech-to-text accuracy can be improved and less certain interpretations can be utilized in some situations. This can preserve computational resources that might otherwise be expended if an inaccurate interpretation of the initial verbal utterance were selected and the user had to repeat his or her initial verbal utterance in an attempt to have the correct interpretation selected. Additionally, this can reduce the amount of time a user spends engaging with an automated assistant, given that the user is less likely to repeat themselves and wait for the automated assistant to render an audible response.

[0009] In some implementations, the provided suggestion element may be used as a basis for biasing speech-to-text processing of further verbal utterances received in response to providing the suggestion. For example, the speech-to-text processing may be biased toward the terms of the provided suggestion element and / or toward terms that fit the expected category or other type of content corresponding to the suggestion. For example, for a provided suggestion of "[musician's name]", the speech-to-text processing may be biased toward the musician's name (generally or to musicians in the user's library); for a provided suggestion of "at 2:00", the speech-to-text processing may be biased toward "2:00" or toward the time more generally; for a provided suggestion of "[smart device]", the speech-to-text processing may be biased toward the name of the user's smart device (e.g., as ascertained from the user's stored device topology). Speech-to-text processing can include, for example, selecting a particular speech-to-text model (from multiple candidate speech-to-text models) for use in performing speech-to-text processing, boosting a candidate transformation (generated during speech-to-text processing) based on a candidate transformation corresponding to a term, biasing toward a term by boosting a path of a state decoding graph (used in speech-to-text processing) corresponding to a term, and / or additional or alternative speech-to-text biasing techniques. As an example, a user can provide an initial verbal utterance such as "Assistant, set a calendar event for..." and in response to receiving the initial verbal utterance, the automated assistant can cause multiple suggestion elements to appear on a display panel of the computing device. The suggestion elements can be, for example, "August...September...[current month]..." and the user can select a particular suggestion element by providing a subsequent verbal utterance such as "August."In response to determining that the suggestion element includes January-December, the automated assistant can bias speech-to-text processing of a subsequent verbal utterance toward January-December. As another example, in response to providing a suggestion element that includes (or indicates) a numerical input, the speech-to-text processing can be biased toward the numerical value. In these and other ways, speech-to-text processing of a subsequent utterance can be biased in favor of the provided suggestion element, thereby increasing the likelihood that the predicted text from the speech-to-text processing is accurate. This can mitigate the occurrence of further dialogue opportunities (e.g., based on further input from the user to correct the inaccurate interpretation) that would otherwise be necessary if the predicted text was inaccurate.

[0010] In some implementations, the timing for providing the automated assistant with a suggestion element for completing a particular request may be based on the context in which the user initiated the particular request. For example, when a user provides at least a portion of a request via verbal utterance while driving his / her vehicle, the automated assistant may determine that the user is driving the vehicle and delay displaying suggestions for completing the request. In this manner, the user is less frequently distracted by graphics being presented on a display panel within the vehicle. Additionally or alternatively, the content of the suggestion element (e.g., candidate text segments) may also be based on the context in which the user initiated the verbal utterance. For example, the automated assistant may determine that the user provided a verbal utterance such as "Assistant, turn on..." from his / her living room with prior permission from the user. In response, the automated assistant may cause the computing device to generate a suggestion element that identifies a particular device in the user's living room. The suggestion element may include natural language content such as "TV in the living room...lights in the living room...stereo in the living room." The user can select one of the suggestion elements by providing a subsequent verbal utterance such as "living room lights," in response to which the automated assistant can turn on the living room lights. In some implementations, the devices that the suggestion elements identify can be based on device topology data that characterizes relationships between various devices associated with the user and / or the placement of computing devices within the user's location, such as a home or office. The device topology data can include identifiers for various areas within the location, as well as device identifiers that characterize whether devices are within each area.For example, the device topology data may identify an area such as the "Living Room" and also identify devices within the living room, such as the "TV," "lights," and "stereo." Thus, in response to a user providing the above-mentioned utterance, "Assistant, turn it on," the device topology may be accessed and compared to the area in which the user is located (e.g., identifying the "Living Room" based on the utterance being provided through a device defined by the device topology data as being within the "Living Room") in order to provide suggestions that correspond to specific devices within that area.

[0011] In some implementations, an action corresponding to at least a portion of the received request may be performed while a suggestion element is presented to the user to complete the request. For example, the user may provide a verbal utterance such as "Assistant, create a calendar event for...". In response, the automated assistant may cause a default calendar to be created with default content and may also cause a suggestion element to be created to complete the request to create a calendar event. For example, the suggestion element may include "Tonight...Tomorrow...Saturday...", and the suggestion element may be presented before, during, or after the default calendar event is created. When the user selects one of the suggestion elements, the previously created default calendar event may be modified according to the suggestion element selected by the user.

[0012] In some implementations, the automated assistant can provide suggestions to reduce the amount of voice processing and / or assistant interaction time typically associated with some input from a user. For example, when a user completes a dialog session with an automated assistant to create a calendar event, the automated assistant can nonetheless provide further suggestions. For example, assume that the user provides a verbal utterance such as, "Assistant, create a calendar event for Matthew's birthday next Monday." In response, the automated assistant can cause a calendar event to be generated and also cause a suggestion element such as "and repeat every year" to be presented to invite the user to select the suggestion element, thereby causing the calendar event to repeat every year. This can reduce the expected number of interactions between the user and the automated assistant since the user does not need to create the calendar event every year. This can also teach the user that "and repeat every year" can be provided via voice in the future, thereby enabling the user to provide future verbal input including "repeat every year," thereby making such future verbal planning more efficient. Furthermore, in some implementations, the suggestion elements may be suggested as abridged versions of previous spoken utterances provided by the user. In this way, computational resources such as power and network bandwidth may be conserved by reducing the amount of time that a user interacts with their automated assistant.

[0013] In some implementations, the suggestion elements presented to the user may be provided to avoid some verbal utterances with which the user is already familiar. For example, when a user has a history of providing verbal utterances such as "Assistant, dim my lights," the automated assistant may not provide the suggestion "my lights" in response to a user who later says "Assistant, dim...". Rather, the user may be presented with other suggestions with which the user may not be familiar, such as "My monitor...my tablet screen...the amount of [color] in my lights." Alternatively or additionally, the suggestions described above may be presented to the user in situations when the user provides verbal utterances that are incomplete (e.g., "Assistant, play...") and in other situations when the user provides verbal utterances that are complete (e.g., "Assistant, play some music").

[0014] In various implementations, the user will not explicitly indicate when the spoken utterance is considered complete by the user. For example, the user will not press the "Submit" button or speak "Finished", "Done", "Done", or other closing phrases when the spoken utterance is considered complete by the user. Thus, in these various implementations, graphical suggestions may be presented in response to an already made spoken utterance, and the automated assistant will need to decide whether to provide the user with more time to provide a further spoken utterance that matches one of the suggestions and is a continuation of the already provided spoken utterance (or to select one of the suggestions through touch input), or instead, to act only on the already provided spoken utterance.

[0015] In some of those implementations, the automated assistant will act only on the already provided verbal utterance in response to detecting a duration of lack of verbal input from the user. The duration may be dynamically determined in various implementations. For example, the duration may depend on whether the already provided verbal utterance is an "incomplete" request or a "complete request" and / or may depend on the characteristics of the "complete request" of the already provided verbal utterance. As described above, a "complete request" is one that, when fully processed by the automated assistant, causes the implementation of one or more corresponding automated assistant actions, where the implemented automated assistant action is not a default error type action (e.g., not an audible response of "Sorry, I can't help you with that"). An "incomplete request" is one that, when fully processed by the automated assistant, causes the implementation of a default error type action (e.g., an error sound and / or a default verbal response such as "Sorry, I can't help you with that").

[0016] In various implementations, the duration may be greater when the verbal utterance already made is an incomplete request compared to when it is a complete request. For example, the duration may be 5 seconds when the verbal utterance already made is an incomplete request, but may be less when the verbal utterance already made is a complete request. For example, the suggestions "Calendar entry for X:00 on [date]" and "Reminder to do X at [time] or [location]" may be presented in response to the verbal utterance "Create a". Because the verbal utterance "Create a" is incomplete, the automated assistant may wait 5 seconds for further verbal utterances, and only when 5 seconds have elapsed without further verbal input will the automated assistant fully process the incomplete verbal utterance and perform the default error type action. Also, for example, the suggestion "for X:00 on [date]" may be presented in response to the verbal utterance "Create a calendar entry for call mom". Because the spoken utterance "Create a calendar entry that says Call Mom" ​​is complete (i.e., will lead to the creation of a calendar entry with a title, and possibly further prompts for date and time details), the automated assistant may wait for three seconds, or other shortened duration, for further spoken utterance, and only when three seconds have elapsed without further verbal input will the automated assistant fully process the spoken utterance already made, create the calendar entry, and possibly further prompt the user to provide a date and / or time for the calendar entry.

[0017] As another example, the duration may be shorter for a complete request that includes all required parameters compared to the duration for a complete request that does not include all required parameters (and would result in further prompting for required parameters). For example, a complete request to create a calendar entry may include the required parameters "title", "date", and "time". A spoken utterance of "Create a calendar entry that says call mom at 2 PM today" includes all required parameters, but may nevertheless provide suggestions for optional parameters (e.g., a suggestion that says "and repeat every day"). Because all required parameters are included in the spoken utterance, the duration may be shorter than if the spoken utterance had been, for example, "Create a calendar entry that says call mom" that did not include all required parameters (and would result in a suggestion such as "@X:00 on [date]" being presented). As yet another example, the duration may be shorter for a complete request that results in a specific automated assistant agent being invoked rather than a general search agent that issues a general search based on the request and displays the search results. For example, the spoken utterance "What is time" is a complete request that results in a general search and the provision of search results that provide information related to the concept of "time." The suggestion "In [city]" may be displayed in response to the spoken utterance, and may be displayed for two seconds or some other duration. On the other hand, the spoken utterance "Turn on the kitchen light" is a complete request that invokes a specific automated assistant agent that causes the "kitchen light" to be turned on. The suggestion "To X% brightness" may be displayed in response to the spoken utterance, but may be displayed for only one second (or other shortened duration), because the spoken utterance already provided is complete and invokes a specific automated assistant agent (rather than causing a general search to be performed).

[0018] The above description is provided as an overview of some implementations of the present disclosure. Further description of those implementations, as well as other implementations, are described in more detail below.

[0019] Other implementations may include a non-transitory computer-readable storage medium storing instructions executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) to perform a method such as one or more of the methods described above and / or elsewhere herein. Still other implementations may include one or more computers, including one or more processors, and / or one or more robotic systems operable to execute the stored instructions to perform a method such as one or more of the methods described above and / or elsewhere herein.

[0020] It should be appreciated that all combinations of the above concepts and additional concepts described in more detail herein are contemplated as being part of the subject matter disclosed herein, for example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein. [Brief description of the drawings]

[0021] [Figure 1A] FIG. 1 illustrates a diagram in which a user provides an oral utterance and an automated assistant provides suggestions for completing the oral utterance. [Figure 1B] FIG. 1 illustrates a diagram in which a user provides an oral utterance and an automated assistant provides suggestions for completing the oral utterance. [Figure 2A] FIG. 1 illustrates a user receiving category suggestions in response to a verbal utterance and / or based on one or more actions of an automated assistant. [Figure 2B]FIG. 1 illustrates a user receiving category suggestions in response to a verbal utterance and / or based on one or more actions of an automated assistant. [Diagram 3] FIG. 13 illustrates a user receiving suggestions for completing a verbal utterance instead of an audible prompt being rendered by the automated assistant. [Figure 4] FIG. 1 is a diagram of a system for providing suggestions via a display modality and completing an oral utterance via an automated assistant. [Diagram 5] FIG. 1 illustrates a method for providing one or more suggestion elements to complete and / or supplement a verbal utterance provided by a user to an automated assistant, a device, an application, and / or any other apparatus or module. [Figure 6] FIG. 1 is a block diagram of an exemplary computer system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] 1A and 1B show diagrams 100 and 120 in which a user provides an oral utterance and an automated assistant provides suggestions for completing the oral utterance. A user 112 can provide an oral utterance 116 to an automated assistant interface of a computing device 114, but the oral utterance 116 may be incomplete, at least to the automated assistant. For example, a user can provide an oral utterance 116 such as "Assistant, can you put it down...", which can be processed at the computing device 114 and / or a remote computing device, such as a server device, and rendered graphically at the display panel 110 of the computing device 114. The graphical rendering of the oral utterance 116 can be presented as a graphical element 106 at the assistant interaction module 104 of the computing device 114. In response to receiving the oral utterance 116, the computing device can determine that the oral utterance 116 is an incomplete oral utterance. An incomplete spoken utterance may be an utterance in which at least one action does not result in being performed via the automated assistant and / or any other application or assistant agent. Alternatively or additionally, an incomplete spoken utterance may be an utterance in which one or more parameter values ​​necessary for a function to be performed or otherwise controlled are missing. For example, an incomplete spoken utterance may be an utterance in which one or more parameter values ​​necessary for an assistant function to be performed via an automated assistant are missing, and therefore require further or supplemental input from a user or other source.

[0023] In response to determining that the spoken utterance 116 is an incomplete spoken utterance, the automated assistant can cause one or more suggestion elements 108 to be presented in the first user interface 102. However, in some implementations, the one or more suggestion elements 108 and / or their content can be rendered audibly to a user who may be visually impaired or otherwise unable to easily see the first user interface 102. The suggestion elements 108 can include natural language content that characterizes a candidate text segment from which the user 112 can select by tapping on the first user interface 102, verbally repeating the text of the candidate text segment, and / or describing at least one suggestion element 108 as it is presented in the first user interface 102. For example, rather than providing an additional spoken utterance such as "sound," the user 112 can provide a different additional spoken utterance, such as "the first one." Alternatively, if the user wishes to select a different suggestion element 108, the user 112 can provide a touch input to the suggestion element 108 that reads "lights," tap the first user interface 102 at a location corresponding to the desired suggestion element 108, and / or provide a different additional verbal utterance, such as "second one." In various implementations, one or more of the suggestion elements 108 may be presented with icons and / or other graphical elements to indicate actions that will be performed in response to their selection. For example, "sound" may be presented with a speaker icon, "lights" may be presented with a light icon, and / or "garage door" may be presented with a garage door icon.

[0024] The natural language content of the text segment provided with each suggestion element 108 may be generated based on whether the combination of the incomplete verbal utterance 116 and the candidate text segment would cause the automated assistant to perform an action. For example, each candidate text segment may be generated based on whether the combination of the incomplete verbal utterance 116 and the candidate text segment would cause an action to be performed via the automated assistant when spoken together by the user 112. In some implementations, when the user 112 provides a complete verbal utterance, the automated assistant, the computing device 114, and / or the associated server device may bypass generating the suggestion element 108. Alternatively, when the user 112 provides a complete verbal utterance, the automated assistant, the computing device 114, and / or the associated server device may generate a suggestion element for provisioning in the first user interface 102.

[0025] In some implementations, device topology data may be used as a basis for generating candidate text segments of the suggestion element 108. For example, the computing device 114 may have access to device topology data that identifies one or more devices associated with the user 112. Furthermore, the device topology data may indicate the location of each device of the one or more devices. The location of the user 112 may be used to determine the context of the user 112, together with prior permission from the user 112, at least because the context characterizes the location of the user 112. The device topology data may be compared to the location data to determine devices that may be near the user 112. Furthermore, once nearby devices are identified, the status of each nearby device may be determined to compare the natural language content of the incomplete spoken utterance 116 with each status of each nearby device. For example, as shown in FIG. 1A, the user 112 provided a request for the assistant to “lower” something, so that devices having a status that may be modified via the action of “lowering” may be used as a basis for generating the suggestion element 108. Thus, if a speaker is playing music nearby, a suggestion of "sound" may be provided. Additionally, if a light is on near the user, a suggestion of "light" may be provided. Additionally or alternatively, device topology data may characterize devices that may not be near the user 112, but may nevertheless be associated with the incomplete spoken utterance 116. For example, the user 112 may be in his or her bedroom, but the incomplete spoken utterance 116 may be associated with a garage door in the garage of the user's 112 home. Thus, the automated assistant may also provide a suggestion element 108 that enables the user to "lower the garage door" even though the user is not in the garage.

[0026] FIG. 1B illustrates a diagram 120 in which a user 112 provides an additional spoken utterance 128 to facilitate the selection of a suggestion element 108 provided in the first user interface 102 illustrated in FIG. 1A. The user 112 can select one of the previously displayed suggestion elements 108 by providing the additional spoken utterance 128, such as “garage door.” In response to receiving the spoken utterance in the automated assistant interface of the computing device 114, candidate text corresponding to the selected suggestion element 108 can be presented adjacent to the natural language content of the incomplete spoken utterance 116. For example, as illustrated in FIG. 1B, a completed candidate text 122 can be presented in the second user interface 126 as a combination of the natural language content from the incomplete spoken utterance 116 and the selected suggestion element 108 (e.g., the natural language content of the additional spoken utterance 128).

[0027] The computing device 114, the automated assistant, and / or the server device can determine that the additional spoken utterance 128 combined with the initial spoken utterance 116 resulted in a complete spoken utterance. In response, one or more actions can be implemented via the automated assistant. For example, when the completed spoken utterance corresponds to the completed candidate text 122, the automated assistant can cause a garage door at the home of the user 112 to electromechanically lower to a closed position. In some implementations, when a complete spoken utterance is formed and / or when an incomplete spoken utterance is formed, an additional suggestion element 124 can be provided in the second user interface 126. For example, even though the completed candidate text 122 is actionable by the automated assistant, the computing device 114, the automated assistant, and / or the server device can cause the additional suggestion element 124 to be presented in the second user interface 126. As described herein, in various implementations, the automated assistant can perform an action (i.e., cause the garage door to be lowered) based on the completed candidate text 122 without any further verbal input or selection of any one of the additional suggestion elements 124 if the duration of time has elapsed without any further verbal input or selection of any one of the additional suggestion elements 124. As also described herein, the duration of time can be shorter than the duration of time during which the automated assistant would wait in FIG. 1A. This shorter duration of time can be based, for example, on the completed candidate text 122 being a "complete request" whereas the candidate text 106 (FIG. 1A) being an "incomplete request."

[0028] In some implementations, a determination may be made as to whether there are additional functions or actions that may be performed and associated with the completed candidate text 122. Alternatively or additionally, the determination may be based on whether additional functions or actions may reduce such interactions in order to limit the amount of computational and / or network resources that may be expended in any subsequent amount of interaction between the user 112 and the automated assistant. For example, in some implementations, the content of the completed candidate text 122 may be compared to the content of a previous interaction between the user 112 and the automated assistant, with prior permission from the user 112 and one or more other users, and / or with one or more other users and the automated assistant. Furthermore, the interaction time between the user and the user's respective automated assistant may be compared when participating in an interaction related to the candidate text 122. For example, another user may only interact with another user's respective automated assistant by requesting that a periodic action be performed, such as requesting that an action be performed in response to a daily activity. Thus, the amount of time that the other user spends interacting with the respective automated assistant of the other user may be relatively less than that of the user 112 who does not request that the periodic and / or conditional action be performed, at least because the interaction is related to the subject matter of the candidate text 122. Thus, the second user interface 126 may be generated with the additional suggestion elements 124 so that the user may select one or more of the additional suggestion elements 124, thereby resulting in reduced consumption of computational and network resources. For example, if the user 112 selects the first additional suggestion element 124 by speaking the phrase 130 "every morning after I leave for work", it does not need to be displayed to the user 112. Furthermore, the spoken utterance 128 no longer needs to be processed and / or the completed candidate text 122 no longer needs to be generated for display in the computing device 114.Moreover, the user 112 may know additional conditional statements, such as "every night at 9 PM," and "every night, when my spouse and I are at home," that may be used to alleviate the drain on computational resources associated with other commands that the user frequently requests. For example, by seeing the additional suggestion elements 124, the user 112 may remember them and use them later when providing a verbal utterance, such as "Assistant, can you turn on the home security system?"

[0029] 2A and 2B provide diagrams 200 and 220 illustrating a user 214 receiving a category suggestion in response to a verbal utterance 218 and / or based on one or more actions of an automated assistant. As provided in FIG. 2A, the user 214 can provide the verbal utterance 218 to an automated assistant interface of a computing device 216. The verbal utterance 218 can be, for example, "Assistant, can you turn it on?" In response to receiving the verbal utterance 218, the computing device 216, a remote computing device in communication with the computing device 216, and / or the automated assistant can cause a first user interface 204 to be presented on a display panel 212 of the computing device 216. The first user interface 204 can include an assistant interactivity module 206 that can present various graphical elements according to one or more actions of the automated assistant. For example, an interpretation of the verbal utterance 218 can be presented as an incomplete text segment 208 in the first user interface 204.

[0030] The first user interface 204 may also include one or more suggestion elements 210 that may include content that characterizes candidate categories for additional input to complete the incomplete text segment 208. The candidate categories characterized by the content of the suggestion elements 210 may be based on data from one or more different sources. For example, the content of the suggestion elements 210 may be based on a context in which a user provided an oral utterance 218. The context may be characterized by contextual data available at the computing device 216, a remote device (such as a server device) in communication with the computing device 216, an automated assistant, and / or any other application or device associated with the user 214. For example, device topology data may indicate that the computing device 216 is paired with or otherwise in communication with one or more other devices. Based on this device topology data, one or more of the suggestion elements 210 may provide a candidate category, such as “[device name].” Alternatively or additionally, the data from one or more sources may include application data indicating media recently accessed by the user 214. When the recent media includes movies, the suggestion element 210 may be generated to include content such as "[name of movie]". Alternatively or additionally, when the recent media includes music, the suggestion element 210 may be generated to include content such as "[name of song and artist]".

[0031] 2B illustrates a diagram 220 in which a user 214 provides an additional verbal utterance 228 for selecting a candidate category from the suggestion element 210 provided in FIG. 2A. As shown, the user 214 can provide the additional verbal utterance 228, “lights in the kitchen.” In response to receiving the additional verbal utterance 228, an additional suggestion element 226 can be presented to the user 214 via the display panel 212. The additional suggestion element 226 can be presented to the user 214 in response to the user completing the initial verbal utterance 218 by providing the additional verbal utterance 228, or when the additional verbal utterance 228 does not complete the initial verbal utterance 218. In some implementations, the additional suggestion element 226 can include content, such as a candidate category, that can be of a similar format as the suggestion element 210. Additionally or alternatively, the additional suggestion element 226 may include content such that, as the user speaks, the content of the speech may be used to facilitate completing, modifying, and / or initializing one or more actions via the automated assistant. Thus, the assistant interaction module 222 may provide various formats of candidate text segments, which may include candidate categories (e.g., those provided in the suggestion element 210), candidate requests (e.g., those provided in the additional suggestion element 226), and / or any other content that may be used for inclusion in commands provided to the automated assistant and / or other applications or devices.

[0032] In some implementations, the provisioning of one or more suggestion elements and / or the order of presentation of one or more suggestion elements may be based on priorities assigned to certain content, actions, verbal utterances, candidate text segments, and / or any other data from which suggestions may be based. For example, in some implementations, the provisioning of certain content in the suggestion element 226 may be based on historical data characterizing previous interactions between the user 214 and an automated assistant, between the user 214 and the computing device 216, between the user 214 and one or more other devices, between the user 214 and one or more other applications, and / or between one or more other users and one or more devices and / or applications. For example, in response to receiving a selection of the suggestion element 210 with content “[device name]”, the computing device 216 and / or the server device may access historical data characterizing previous interactions between the user 214 and one or more devices associated with the user 214. The historical data may be used as the basis for generating content for the suggestion element 226. For example, even though a user selects "kitchen lights," the historical data may be used to generate suggestions for other devices that the user may be interested in controlling and / or other settings (other than "on") that the user may be interested in modifying for devices that the user did not explicitly select.

[0033] As an example, and with prior permission from the user 214, the historical data can characterize one or more previous interactions in which the user 214 turned off her kitchen lights a few minutes before going to bed. Based on these interactions characterized by the historical data, the automated assistant can assign the highest priority to such interactions as they relate to a completed request 224, such as, "Assistant, can you turn on the kitchen lights?" Additionally, the automated assistant can generate content for a suggestion element 226, such as, "Until I go to bed." Based on the content of the historical data characterizing the highest priority interactions, at least as its content relates to the completed request 224, the automated assistant can provide the suggestion element 226 "Until I go to bed" as the highest priority or highest positioned suggestion element 226 in the second user interface 232.

[0034] Alternatively or additionally, when the historical data characterizes one or more prior interactions in which the user 214 requested that the automated assistant turn the kitchen lights to a half brightness setting or a red setting, but those requests occurred less frequently than the above-mentioned request to turn the lights off before going to bed, the content of those requests may be assigned a priority that is less than the prior priority assigned to the content of the above-mentioned request. In other words, because the user 214 requested that the kitchen lights be turned off when going to bed more frequently, the suggestion element "until I go to bed" will be prioritized and / or otherwise have a higher position than other suggestion elements such as "and at half brightness setting" and / or "and at a red setting." In some implementations, the content of the suggestion element may be based on features of the device or application not previously used by the user 214 and / or one or more updates pushed to the application or device by a third-party entity. The term third-party entity may refer to an entity different from the manufacturer of the computing device 216, the automated assistant, and / or the server device in communication with the computing device 216. Thus, when the manufacturer of the kitchen light (i.e., the third-party entity) pushes an update that enables the kitchen light to operate according to a red setting, the automated assistant can access data characterizing this update and generate the suggestion element 226, "and on the red setting," based on the update provided by the third-party manufacturer (i.e., the third-party entity).

[0035] Alternatively or additionally, the content of the suggestion element 226 based on the update may be assigned a priority that is less than another priority assigned to content corresponding to another suggestion element 226 based on one or more actions that the user 214 frequently requests to be performed. However, the content based on the update may be assigned a higher priority than content that the user 214 has previously presented via the display panel 212 and / or via the automated assistant but has elected not to select the previously presented content. In this manner, the suggestion elements provided to the user may be cycled according to the assigned priority to adapt the suggestions to the user. Furthermore, when those suggestions are relevant to reducing interaction X between the user 214 and the automated assistant, computational and network resources may be conserved by providing more suggestions that reduce such interaction while limiting suggestions that do not reduce such interaction or that are otherwise not relevant to the user.

[0036] 3 shows a diagram 300 in which a user 312 receives a suggestion for completing a spoken utterance 316 instead of an audible prompt being rendered by the automated assistant. The user 312 can provide the spoken utterance 316, such as, "Assistant, can you create a calendar entry for me?" In response, an assistant interactive module 304, which may be controlled via the automated assistant, can provide a graphical representation 306 of the spoken utterance 316, which can be presented on a display panel 320 of the computing device 314.

[0037] The spoken utterance 316 may correspond to a request for a function to be performed, which may have one or more required parameters that must be specified by the user 312 or otherwise satisfied by one or more available values. To prompt the user to provide one or more values ​​to satisfy the one or more required parameters, the automated assistant may provide a graphical prompt 308 that, in some cases, requests a value for a particular parameter. For example, the prompt 308 may include content such as "What will be the title?", thereby indicating that the user 312 must provide a title for the calendar entry. However, to conserve the power and computational resources of the computing device 314, the automated assistant may cause the computing device 314 to bypass rendering an audible prompt for the user 312. Furthermore, this may eliminate delay time because the user 312 does not have to wait until the audible prompt is fully rendered.

[0038] Instead of the automated assistant bypassing providing an audible prompt, the automated assistant can generate one or more suggestion elements 310 that may be based on one or more required parameters for the function requested by the user 312. For example, if the function corresponds to generating a calendar entry and the parameter to be filled is the "title" of the calendar entry, the suggestion element 310 may include a title suggestion for the calendar entry. For example, the content of the suggestion may include "for my mom's birthday," "to take out the trash," and "to pick up the dry cleaning tomorrow."

[0039] To select one of the suggestion elements 310, the user 312 can provide an additional spoken utterance 318 that identifies the content of at least one of the suggestion elements 310. For example, the user 312 can provide the additional spoken utterance 318 of "To pick up the dry cleaning tomorrow." As a result, and from the perspective of the computing device 314 and / or the automated assistant, only the dialogue rendered during the interaction between the user 312 and the automated assistant will have been rendered by the user 312. Thus, rendering of audio will not be necessary or otherwise employed by the automated assistant. Furthermore, in response to the provision of the additional spoken utterance 318 by the user, a different prompt may be provided on the display panel 320 to request that the user 312 provide values ​​for another parameter, if necessary for the requested function. Furthermore, when another parameter is required for the function, a different suggestion element may be presented on the display panel 320 with different content. Moreover, subsequent prompts and subsequent suggestion elements may be presented without any audio being rendered at the computing device 314 or otherwise via an automated assistant. Thus, the user 312 is ostensibly providing ongoing verbal utterances while simultaneously viewing the display panel 320 to identify content for then further completing the original function requested by the user.

[0040] 4 illustrates a system 400 that provides suggestions via a display modality for completing an oral utterance via an automated assistant 408. In some implementations, the suggestions may be provided to a user to complete the oral utterance or at least supplement the user's previous oral utterance. In some implementations, the suggestions may be provided to reduce the frequency and / or length of time that the user will engage in a subsequent dialog session with the automated assistant.

[0041] The automated assistant 408 can operate as part of an automated assistant application provided in one or more computing devices, such as the client device 434 and / or the server device 402. A user can interact with the automated assistant 408 via one or more assistant interfaces 436, which may include one or more of a microphone, a camera, a touchscreen display, a user interface, and / or any other device capable of providing an interface between a user and an application. For example, a user can initialize the automated assistant 408 by providing verbal, textual, and / or graphical input to the assistant interface, causing the automated assistant 408 to perform a function (e.g., provide data, control a device (e.g., control an IoT device 442), access an agent, modify settings, control an application, etc.). The client device 434 and / or the IoT device 442 may include a display device, which may be a display panel including a touch interface for receiving touch input and / or gestures to allow a user to control an application of the client device 434 and / or the server device 402 via a touch interface. Touch input and / or other gestures (e.g., verbal speech) may also enable a user to interact with the automated assistant 408, the automated assistant 438, and / or the IoT device 442 via the client device 434.

[0042] In some implementations, the client device 434 and / or the IoT device 442 may lack a display device but may include an audio interface (e.g., a speaker and / or a microphone) to provide audible user interface output without providing graphical user interface output, as well as to provide a user interface, such as a microphone, for receiving verbal natural language input from a user. For example, in some implementations, the IoT device 442 may include one or more tactile input interfaces, such as one or more buttons, and omit a display panel over which graphical data from a graphics processing unit (GPU) is provided. In this manner, energy and processing resources may be conserved compared to a computing device that includes a display panel and a GPU.

[0043] The client device 434 may be in communication with the server device 402 over a network 446, such as the Internet. The client device 434 may offload computational tasks to the server device 402 to conserve computational resources at the client device 434 and / or the IoT device 442. For example, the server device 402 may host the automated assistant 408, and the client device 434 may send one or more assistant interfaces and / or inputs received at the user interface 444 of the IoT device 442 to the server device 402. However, in some implementations, the automated assistant 408 may be hosted at the client device 434. In various implementations, all or less than all aspects of the automated assistant 408 may be implemented on the server device 402 and / or the client device 434. In some of those implementations, aspects of the automated assistant 408 are implemented via a local automated assistant 438 of the client device 434, interfacing with the server device 402, which may implement other aspects of the automated assistant 408. The server device 402 can service multiple users and their associated assistant applications, possibly via multiple threads. In implementations in which all or less than all aspects of the automated assistant 408 are implemented via a local automated assistant 438 on the client device 434, the local automated assistant 438 can be an application that is separate from the operating system of the client device 434 (e.g., installed "on top of" the operating system)--or alternatively, can be implemented directly by the operating system of the client device 434 (e.g., an operating system application, but considered integral to the operating system).

[0044] In some implementations, the automated assistant 408 and / or the automated assistant 438 may include an input processing engine 412 that can employ multiple different engines to process input and / or output for the client device 434, the server device 402, and / or one or more IoT devices 442. For example, the input processing engine 412 may include a speech processing engine 414 that can process audio data received at the assistant interface 436 to identify text embedded in the audio data. The audio data may be transmitted from the client device 434 to the server device 402, for example, to conserve computational resources at the client device 434.

[0045] The process for converting the audio data to text may include a speech recognition algorithm, which may employ neural networks, word2vec algorithms, and / or statistical models to identify groups of audio data that correspond to words or phrases. The text converted from the audio data may be parsed by the data parsing engine 416 and made available to the automated assistant 408 as text data that may be used to generate and / or identify command phrases from the user. In some implementations, the output data provided by the data parsing engine 416 may be provided to the action engine 418 to determine whether the user has provided input corresponding to a particular action and / or routine that may be performed by the automated assistant 408 and / or an application, agent, and / or device that may be accessed by the automated assistant 408. For example, the assistant data 422 may be stored as client data 440 at the server device 402 and / or the client device 434 and may include data defining one or more actions that may be performed by the automated assistant 408, as well as parameters involved in performing those actions.

[0046] When the input processing engine 412 determines that a user has requested that a particular action or routine be performed, the action engine 418 can determine one or more parameters for the particular action or routine, and the output generation engine 420 can then provide an output to the user based on the particular action, routine, and / or the one or more parameters. For example, in some implementations, in response to a user input, such as a gesture directed at the assistant interface 436, the automated assistant 438 can cause data characterizing the gesture to be transmitted to the automated assistant 408 and / or the server device 402 to determine the action that the user intends the automated assistant 438 to perform.

[0047] The client device 434 and / or the IoT device 442 may each include one or more sensors capable of providing an output in response to a user input from a user. For example, the one or more sensors may include an audio sensor (i.e., an audio response device) that provides an output signal in response to an audio input from a user. The output signal may be transmitted to the client device 434, the server device 402, the IoT device 442 via a communication protocol, such as Bluetooth, LTE, Wi-Fi, and / or any other communication protocol. In some implementations, the output signal from the client device 434 may be converted into user input data by one or more processors of the client device 434. The user input data may be transmitted to the IoT device 442 (e.g., a Wi-Fi enabled light bulb) via the communication protocol and either processed at the client device 434 and / or transmitted to the server device 402 via the network 446. For example, the client device 434 and / or the server device 402 may determine, based on the user input data, a function of the IoT device 442 to be controlled via the user input data.

[0048] A user may provide input, such as a verbal utterance provided by the user, that is received at the assistant interface 436 and that facilitates controlling the IoT device 442, the client device 434, the server device 402, an automated assistant, an agent assistant, and / or any other device or application. In some implementations, the system 400 may provide suggestions to the user on how to complete the verbal utterance, supplement the verbal utterance, and / or otherwise assist the user. For example, a user may provide an incomplete verbal utterance along with an intent to cause the IoT device 442 to perform a particular function. The incomplete verbal utterance may be received at the assistant interface 436 and converted to audio data, which may be transmitted to the server device 402 for processing. The server device 402 may include a suggestion engine 424 that may receive the audio data or other data based on the incomplete verbal utterance and determine one or more suggestions for completing or otherwise supplementing the incomplete verbal utterance.

[0049] In some implementations, the timing for providing the one or more suggestion elements and the display panel of the client device 434 and / or IoT device 442 can be determined by the suggestion engine 424. The suggestion engine 424 can include a timing engine 428, which can determine when the user should be presented with one or more suggestions for additional verbal utterances. In some implementations, one or more thresholds can be characterized by the action data 444 available to the suggestion engine 424. For example, the action data 444 can characterize a threshold corresponding to the time a user may delay after providing a portion of a verbal utterance. If the user delays after providing an incomplete verbal utterance, the amount of delay time can be compared to the threshold value characterized by the action data of 444. When the delay meets the threshold, the timing engine 428 can cause one or more suggestion elements to be presented to the user to complete the incomplete verbal utterance.

[0050] In some implementations, the timing engine 428 can adapt a value for the delay time between the user providing an incomplete verbal utterance and receiving a suggestion to complete the verbal utterance. For example, the suggestion engine 424 can determine a context of the user based on contextual data incorporated in the assistant data 422, the action data 444, and / or the client data 440, and use the context as a basis for determining a value for the delay time. For example, when the user is in a first context characterized by the user being at home and watching TV, the timing engine 428 can set the delay time to a first value. However, when the user is in a second context characterized by the user driving and interacting with a vehicle computing device, the timing engine 428 can set the delay time to a second value greater than the first value. In this manner, the suggestion element for completing the verbal utterance can be relatively delayed to limit the user's distraction while the user is driving.

[0051] In some implementations, the value for the delay time may be proportional to or adapted according to how often a user provides incomplete verbal utterances over time. For example, initially, when a user begins interacting with an automated assistant, the user may provide X number of incomplete verbal utterances per period Y. Sometime later, the user may have a history of providing Z number of incomplete utterances Y. If Z is less than X, the timing engine 428 may increase the value for the delay time threshold over time. Thus, as the delay time threshold increases, computational resources at the server device 402 and / or the client device 434 may be conserved because there are fewer changes to what is presented on the display panel over time, thereby reducing the load on the GPU responsible for processing the graphical data for the display panel.

[0052] In some implementations, the suggestion engine 424 may include a speech-text model engine 430, which may select a speech-text model from a plurality of different speech-text models according to an expected verbal utterance from the user. Additionally or alternatively, the speech-text model engine 430 may select a speech-text model from a plurality of different speech-text models according to an expected type of verbal utterance from the user. Alternatively or additionally, the speech-text model engine 430 may bias the selected speech-text model based on the expected verbal utterance from the user. In some implementations, when a suggestion element provided to the user includes content having at least some numerical text, the speech-text model engine 430 may select a speech-text model adapted for processing verbal utterances that feature numerical information based on the content. Alternatively or additionally, when the suggestion elements provided to the user include content that includes at least a proper name, such as a city or person's name, the speech-to-text model engine 430 can bias the selected speech-to-text model to more easily interpret proper names. For example, the selected speech-to-text model can be biased to reduce error rates that would otherwise result from processing proper names. This can eliminate latency that might otherwise result and / or reduce the frequency of irrelevant suggestions from the suggestion engine 424.

[0053] In some implementations, the content of each of the suggestion elements of one or more suggestions provided to the user in response to an incomplete or complete verbal utterance may be based on the user's context. For example, the user's context may be characterized by data generated by the device topology engine 426. The device topology engine 426 may identify a device that has received an incomplete or complete verbal utterance from the user and determine information associated with the device, such as other devices connected to the receiving device, the location of the receiving device relative to the other devices, and an identifier for the receiving device, one or more different users associated with the receiving device, the capabilities of the receiving device, the capabilities of one or more other devices paired with or otherwise in communication with the receiving device, and / or any other information that may be associated with the device that received the verbal utterance. For example, when the receiving device is a standalone speaker device located in the living room of the user's home, the device topology engine 426 may determine that the standalone speaker device is in the living room and may also determine other devices located in the living room.

[0054] The suggestion engine 424 can use information such as descriptions of other devices in the living room to generate a suggestion element to be presented to the user. For example, in response to receiving an incomplete spoken utterance from a user by the stand-alone speaker device, the device topology engine 426 can determine that a television is in the living room with the stand-alone speaker device. The suggestion engine 424 can generate content for a suggestion element to complete the incomplete spoken utterance based on the determination that the television is in the same room. For example, when the incomplete spoken utterance is "Assistant, change," the content of the suggestion element can include "TV channel." Alternatively or additionally, the device topology engine 426 can determine that a television is in the same room as the stand-alone speaker device and also determine that the television is capable of graphically representing the suggestion element, while the stand-alone speaker device is not capable of graphically representing the suggestion element. Thus, in response to an incomplete verbal utterance, the suggestion engine 424 can cause the television display panel to present one or more suggestion elements, including content such as "change my TV channel." When the user repeats the content of the suggestion element "change my TV channel," the automated assistant can cause the television channel to be changed while also generating additional suggestion elements that can be presented simultaneously as the new television channel is rendered on the television.

[0055] In some implementations, data such as assistant data 422, action data 444, and / or client data 440 can characterize past interactions between the user and the automated assistant 408 and / or the automated assistant 438. The suggestion engine 424 can use the above-mentioned data to generate content for suggestion elements that are presented to the user in response to providing an incomplete or complete spoken utterance to the assistant interface 436. In this manner, if the user temporarily forgets some preferred command, natural language text can be provided via the suggestion element to remind the user of the user's particular preferred command. Such preferred commands can be identified by comparing the content to previously received incomplete or complete spoken utterances and comparing the content to previous natural language inputs received from the user. The most frequently occurring natural language inputs can be considered as corresponding to the preferred command. Thus, when the content of a complete or incomplete spoken utterance corresponds to such natural language input, the suggestion element can be rendered and / or generated based on the content of those natural language inputs. Furthermore, by identifying the most frequently processed natural language inputs and / or identifying preferred commands for a particular user, the suggestion engine 424 can provide the user, via the automated assistant, with suggestions based on the most frequently and most successfully performed commands. Thus, providing such suggestions can result in reduced processing of the least successful commands, thereby conserving computing resources.

[0056] FIG. 5 illustrates a method 500 for providing one or more suggestion elements to complete and / or supplement a verbal utterance provided by a user to an automated assistant, a device, an application, and / or any other device or module. The method 500 may be implemented by one or more computing devices, applications, and / or any other device or module that may be associated with an automated assistant. The method 500 may include an operation 502 of determining that a user has provided at least a portion of a verbal utterance. The verbal utterance may be a verbal natural language input audibly provided by a user to one or more automated assistant interfaces of one or more computing devices. The verbal utterance may comprise one or more words, one or more phrases, or any other construct of verbal natural language. For example, the automated assistant interface may include a microphone, and the computing device may be a smart device including a touch display panel that may also operate as another automated assistant interface. In some implementations, the computing device, or a server device in communication with the computing device, may determine that a user has provided a verbal utterance.

[0057] The method 500 may further include an operation 504 of determining whether the spoken utterance is complete. Determining whether the spoken utterance is complete may include determining whether each parameter of one or more parameters of the function has an assigned value. For example, when the spoken utterance is "Assistant, turn it down," the user may be requesting that an action be performed to turn down the output modality of the device. The action may correspond to a function such as a "turn down volume" function or a "turn down brightness" function, which may require the user to specify a device name. Alternatively or additionally, determining whether the spoken utterance is complete may include determining whether the automated assistant can perform the action in response to receiving the spoken utterance.

[0058] In some cases, operation 504 may include determining whether some categories of actions can be performed in response to the spoken utterance to determine whether the spoken utterance is complete. For example, an action such as an automated assistant asking the user to repeat what the user said for clarity may be considered not to indicate that the user provided a complete spoken utterance, even though the automated assistant responds to the spoken utterance. However, an action such as performing a web query, controlling another device, controlling an application, and / or possibly requesting the automated assistant to perform an intended action may be considered to be a complete spoken utterance. For example, in some implementations, the spoken utterance may be considered complete when at least one request is identified from the spoken utterance and the request can be performed via the automated assistant.

[0059] When the spoken utterance is determined to be complete (e.g., "Assistant, turn down the TV"), the method 500 may proceed to an optional operation 512 of rendering and / or generating a suggestion for the user based on the action to be performed, and / or an operation 514 of performing an action corresponding to the spoken utterance. For example, a computing device that receives the spoken utterance via an automated assistant interface may initiate the performance of one or more actions specified via the spoken utterance. However, when the spoken utterance is determined to be incomplete, the method 500 may proceed to operation 506.

[0060] Operation 506 may be an optional operation that includes determining whether a threshold duration of delay occurred after the incomplete oral utterance. For example, whether the duration of spoken input silence meets the threshold may be determined further based on the user providing the oral utterance, the user's context, and / or any other information associated with the incomplete oral utterance, along with permission from the user. When the amount of spoken input silence meets the threshold duration of delay, method 500 may proceed to operation 508. However, if the amount of spoken input silence does not meet the threshold duration of delay, method 500 may proceed to operation 510.

[0061] Method 500 may optionally proceed from operation 504 to operation 508, or may optionally proceed from operation 504 to operation 506. When it is determined at operation 506 that the threshold duration of the delay is met, method 500 may proceed to operation 508, which renders one or more suggestions for completing the verbal utterance. The content of the one or more suggestions for completing the verbal utterance may be based on the user who provided the incomplete verbal utterance, the content of the incomplete verbal utterance, the context in which the verbal utterance was provided, data representing a device topology associated with the device that received the verbal utterance, interaction history data characterizing a previous interaction between the user and the automated assistant (or one or more users and the automated assistant), and / or any other information from which the suggestions may emerge. For example, when the incomplete spoken utterance is "Assistant, lower it," the content of the suggestion provided to the user may include "TV," which may be based on device topology data indicating that the TV and / or the user are located in the same room as the computing device that received the incomplete spoken utterance. Alternatively, when the incomplete spoken utterance is "Assistant, send a message to," the content of the suggestion provided to the user may be based on interaction history data characterizing a previous dialog session in which the user requested the automated assistant to send a message to the user's spouse and sibling. Thus, the content of the suggestion may include "my brother" and "my wife." In some implementations, the content of one or more suggestion elements may be determined and / or generated after operation 502, but before optional operation 506 and before operation 508.

[0062] From operation 506 or operation 508, method 500 may proceed to operation 510, which determines whether another spoken utterance has been provided by the user. For example, the user may provide an additional spoken utterance after providing an incomplete spoken utterance. In such a case, method 500 may proceed from operation 510 back to operation 504. For example, the user may provide an additional spoken utterance that corresponds to the content of the suggestion element, thereby indicating to the automated assistant that the user selected the suggestion element. This may result in a compilation of spoken utterances, which may consist of the initial incomplete spoken utterance and the additional spoken utterance that corresponds to the content of the suggestion element selected by the user. This compilation of spoken utterances may be subjected to operation 504, which determines whether the compilation of spoken utterances corresponds to a complete spoken utterance. In implementations, at operation 510, the system may wait for another verbal utterance for the duration, and may proceed to operation 514 (or 512) if no other verbal utterance is received in the duration. If another verbal utterance is received in the duration, the system may proceed back to operation 504. In some of those implementations, the duration may be dynamically determined. For example, the duration may depend on whether the verbal utterance already provided is an "incomplete" request or a "complete request" and / or may depend on the "complete request" characteristics of the verbal utterance already provided.

[0063] When the compilation of spoken utterances corresponds to a complete spoken utterance, the method 500 may proceed to optional operation 512 and / or operation 514. However, if the compilation of spoken utterances is not determined to be a complete spoken utterance, the method 500 may possibly proceed to operation 506 and / or possibly proceed to operation 508. For example, when the method 500 again proceeds to operation 508, the suggestions for completing the spoken utterance may be based at least on the content of the compilation of spoken utterances. For example, if the user selects the suggestion "my brother", the combination of the incomplete spoken utterance "Assistant, send a message to" and "my brother" may be used as a basis for further suggestions to be provided. In this manner, a cycling of suggestions may be provided as the user continues to select suggestions for completing each of the user's spoken utterances. For example, a subsequent suggestion may include "I'm on my way" based on interaction history data that indicates the user previously sent the same message to the user through an application that is separate from the automated assistant.

[0064] In some implementations, method 500 may proceed from operation 510 to an optional operation 512 that renders suggestions for the user based on the action to be performed. For example, when an automated assistant is requested to perform an action, such as turning on an alarm system for a user, the action and / or any information associated with the action may be used as a basis for providing additional suggestion elements for the user to select from. For example, if the complete verbal utterance to be performed via the automated assistant is "Assistant, turn on my alarm system," one or more other suggestion elements may be rendered in operation 512 and may include content such as "and set my alarm for," "turn off all my lights," "and turn on my audiobook," and / or any other suitable suggestion content for the user. In implementations, at operation 512, the system proceeds to operation 514 in response to receiving further verbal input or in response to non-reception of other verbal utterances for a duration. In some of these implementations, the duration may be dynamically determined. For example, the duration may depend on whether the verbal utterances already provided constitute an "incomplete" request or a "complete request" and / or may depend on the "complete request" characteristics of the verbal utterances already provided.

[0065] In some implementations, suggestions may be rendered and / or generated to reduce the amount of interaction between the user and the automated assistant, thereby conserving power, computational resources, and / or network resources. For example, the content of the supplemental suggestion element rendered in operation 512 may include "every night, after 10 PM." In this manner, if it is determined that the user has selected the supplemental suggestion element (e.g., via another verbal utterance), as determined in operation 510, the method 500 may proceed to operation 504 and thereafter to operation 514, causing the automated assistant to generate a setting in which the alarm system is turned on every night, after 10 PM. Thus, during subsequent nights, the user need not provide the same initial verbal utterance of "Assistant, turn on...", but rather may simply rely on the setting adopted via the other selected suggestion elements.

[0066] When the user does not provide additional verbal utterances after the first complete verbal utterance or otherwise indicates a preference that one or more commands corresponding to a compilation of verbal utterances be executed, method 500 can proceed from operation 510 to optional operation 512 and / or operation 514. For example, when a user provides a verbal utterance such as "Assistant, change" and later selects a suggestion element including content such as "TV channel," the user may elect to have the request executed even though the user does not specify a particular channel to change the television. As a result and operation 514, the television channel may be changed even though the user does not select any supplemental suggestion elements to further narrow the resulting instructions in the command (e.g., the channel change may be based on the user's learned preferences derived from the interaction history data).

[0067] 6 is a block diagram of an exemplary computer system 610. Computer system 610 typically includes at least one processor 614 that communicates with several peripheral devices via a bus subsystem 612. These peripheral devices may include, for example, a storage subsystem 624 including memory 625 and a file storage subsystem 626, user interface output devices 620, user interface input devices 622, and a network interface subsystem 616. The input and output devices enable user interaction with computer system 610. Network interface subsystem 616 provides an interface to external networks and is coupled to corresponding interface devices in other computer systems.

[0068] The user interface input devices 622 may include keyboards and pointing devices such as a mouse, trackball, touchpad, or graphics tablet, scanners, touch screens integrated into displays, and audio input devices such as voice recognition systems, microphones, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computer system 610 or onto a communications network.

[0069] The user interface output devices 620 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computer system 610 to a user or to another machine or computer system.

[0070] The storage subsystem 624 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 624 may include logic for implementing selected aspects of the method 500 and / or the computing device 114, the server device, the computing device 216, the assistant interaction module, the client device 434, the computing device 314, the server device 402, the IoT device 442, the automated assistant 408, the automated assistant 438, the automated assistant, and / or any other device, apparatus, application, and / or module described herein.

[0071] These software modules are generally executed by the processor 614 alone or in combination with other processors. The memory 625 used within the storage subsystem 624 may include several memories, including a main random access memory (RAM) 630 for storing instructions and data during program execution, and a read-only memory (ROM) 632 in which fixed instructions are stored. The file storage subsystem 626 may provide persistent storage for program and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules implementing the functionality of some implementations may be stored by the file storage subsystem 626 within the storage subsystem 624 or within other machines accessible by the processor 614.

[0072] Bus subsystem 612 provides a mechanism for allowing the various components and subsystems of computer system 610 to communicate with each other as intended. Although bus subsystem 612 is shown diagrammatically as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0073] The computer system 610 can be of different types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of the computer system 610 shown in Figure 6 is only a specific example to illustrate some implementations. Many other configurations of the computer system 610 are possible, having more or fewer components than the computer system shown in Figure 6.

[0074] In situations where the systems described herein may collect or utilize personal information about users (or as they are often referred to herein as “participants”), the user may be provided with an opportunity to control whether a program or feature collects user information (e.g., information about the user's social network, social actions or activities, occupation, user preferences, or the user's current geographic location) or to control whether and / or how to receive content from a content server that may be more relevant to the user. Also, certain data may be treated in one or more ways before being stored or used such that personally identifiable information is removed. For example, the user's identity may be treated such that personally identifiable information cannot be determined about the user, or the user's geographic location may be generalized, in which case geographic location information is obtained (such as to the city, zip code, or state level) such that the user's specific geographic location cannot be determined. Thus, the user may have control over how information is collected and / or used about the user.

[0075] In some implementations, a method implemented by one or more processors is provided, comprising performing speech-to-text processing on data characterizing an oral utterance provided by a user. The oral utterance includes natural language content and is received via an automated assistant interface of a computing device connected to a display panel. The method further comprises determining whether the oral utterance is complete based on performing speech-to-text processing on the data characterizing the oral utterance, including at least determining whether the automated assistant can cause one or more actions to be accomplished based on the natural language content. When the oral utterance is determined to be incomplete, the method further comprises causing the display panel of the computing device to provide one or more suggestion elements in response to determining that the oral utterance is incomplete. The one or more suggestion elements include a specific suggestion element that, when spoken by a user to the automated assistant interface, provides other natural language content via the display panel that causes the automated assistant to operate to facilitate completing the action. The method further includes, after causing the display panel of the computing device to provide one or more suggestion elements, determining that the user has provided another verbal utterance associated with other natural language content of the particular suggestion element. The method further includes determining whether a combination of the verbal utterance and the another verbal utterance is complete in response to determining that the user has provided the other verbal utterance. The method further includes causing one or more actions to be performed via the automated assistant based on the natural language content and the other verbal utterance when the combination of the verbal utterance and the other verbal utterance is determined to be complete.

[0076] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0077] In some implementations, performing speech-to-text processing on data characterizing the oral utterance provided by the user includes generating one or more candidate text segments from the data, and the method further includes causing a display panel of the computing device to provide a graphical representation of at least one of the one or more candidate text segments when the oral utterance is determined to be incomplete. In some versions of those implementations, the method further includes causing a display panel of the computing device to bypass providing the graphical representation of the at least one of the one or more candidate text segments and to bypass providing the one or more suggestion elements when the oral utterance is determined to be complete and includes all required parameters for the one or more actions. In some other versions of those implementations, the method further includes: identifying a natural language command that, when spoken by one or more users to the automated assistant interface, results in a reduced amount of speech-to-text processing compared to an amount of speech-to-text processing associated with the oral utterance from the user when the oral utterance is determined to be complete; generating one or more other suggestion elements that characterize the natural language command based on identifying the natural language command; and causing a display panel of the computing device to provide the one or more other suggestion elements at least in response to determining that the oral utterance is complete.

[0078] In some implementations, the method further includes selecting a speech-to-text processing model from a plurality of different speech-to-text processing models based on a type of oral utterance expected from the user when the oral utterance is determined to be incomplete, and when at least one suggestion element is provided via the display panel.

[0079] In some implementations, the method further includes biasing the speech-to-text processing toward one or more terms of the one or more suggestion elements and / or toward one or more expected types of content corresponding to the one or more suggestion elements when the verbal utterance is determined to be incomplete.

[0080] In some implementations, the method further includes determining whether a threshold duration of verbal input silence followed the verbal utterance. In some of those implementations, causing a display panel of the computing device to provide one or more suggestion elements is further responsive to determining that at least a threshold duration of verbal input silence followed the verbal utterance.

[0081] In some implementations, performing speech-to-text processing includes determining a first candidate text segment and a second candidate text segment, where the first candidate text segment and the second candidate text segment correspond to different interpretations of the oral utterance. In some of those implementations, a particular suggestion element is determined based on the first candidate text segment and at least one other suggestion element is determined based on the second candidate text segment. In some versions of those implementations, causing a display panel connected to the computing device to provide one or more suggestion elements includes causing a display panel connected to the computing device to graphically represent the first candidate text segment adjacent to the particular suggestion element and graphically represent the second candidate text segment adjacent to the at least one other suggestion element. In some of those versions, determining that the user has provided another verbal utterance associated with other natural language content of the particular suggestion element includes determining whether the user has identified a first candidate text segment or a second candidate text segment based on the another verbal utterance.

[0082] In some implementations, when the oral utterance is determined to be incomplete, the method further includes, in response to determining that the oral utterance is incomplete, generating other natural language content based on historical data characterizing one or more prior interactions in which a prior action was accomplished via the automated assistant and in which the user identified at least a portion of the other natural language content during the one or more prior interactions between the user and the automated assistant.

[0083] In some implementations, the method further includes, when the verbal utterance is determined to be incomplete, generating other natural language content based on device topology data characterizing relationships between various devices associated with the user in response to determining that the verbal utterance is incomplete, where the particular suggestion element is based on the device topology data and identifies one or more devices among the various devices associated with the user.

[0084] In some implementations, causing one or more actions to be performed via the automated assistant based on the natural language content and another oral utterance includes controlling a device.

[0085] In some implementations, a particular suggestion element further includes a graphical element that indicates the action.

[0086] In some implementations, the method further includes, when the verbal utterance is determined to be incomplete, determining a specific duration for waiting for another verbal utterance after provisioning of one or more suggestion elements, where the specific duration is determined based on the verbal utterance being determined to be incomplete. In some versions of those implementations, the method further includes, when the verbal utterance is determined to be complete, generating one or more other suggestion elements based on the complete verbal utterance, causing a display panel of the computing device to provide the one or more other suggestion elements, and determining an alternative specific duration for waiting for further verbal utterance after provisioning of the one or more other suggestion elements. The alternative specific duration is shorter than the specific duration, and the alternative specific duration is determined based on the verbal utterance being determined to be complete.

[0087] In some implementations, a method implemented by one or more processors is provided, comprising performing speech-to-text processing on data characterizing a verbal utterance provided by a user to facilitate triggering an automated assistant to perform an action. The verbal utterance includes natural language content and is received via an automated assistant interface of a computing device connected to a display panel. The method further comprises determining whether the verbal utterance is complete based on performing speech-to-text processing on the data characterizing the verbal utterance, including at least determining whether the natural language content is missing one or more parameter values ​​for controlling a function associated with the action. When the verbal utterance is determined to be incomplete, the method further comprises causing a display panel connected to the computing device to provide one or more suggestion elements in response to determining that the verbal utterance is incomplete. The one or more suggestion elements include a specific suggestion element that, when spoken by a user to the automated assistant interface, provides other natural language content via the display panel that causes the automated assistant to operate to facilitate completing the action. The method further includes determining that the user has selected a particular suggestion element of the one or more suggestion elements through another spoken utterance received at the automated assistant interface. The method further includes, in response to determining that the user has selected the particular suggestion element, causing an action to be performed based on other natural language content of the particular suggestion element and the natural language content of the spoken utterance. When the spoken utterance is determined to be complete, the method further includes, in response to determining that the spoken utterance is complete, causing an action to be performed based on the natural language content of the spoken utterance.

[0088] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0089] In some implementations, the method further includes, in response to determining that the user selected the particular suggestion element when the oral utterance is determined to be incomplete, causing a priority associated with the other natural language content to be modified from a previous priority associated with the other natural language content before the oral utterance was determined to be incomplete. In some of those implementations, an order of presentation of the particular suggestion element relative to at least one other suggestion element on the display panel is based at least in part on the assigned priority to the other natural language content.

[0090] In some implementations, a method implemented by one or more processors is provided, comprising performing speech-to-text processing on data characterizing a spoken utterance provided by a user to facilitate triggering an automated assistant to perform an action. The spoken utterance includes natural language content and is received through an automated assistant interface of a computing device connected to a display panel. The method further comprises determining whether the spoken utterance is complete based on performing speech-to-text processing on the data characterizing the spoken utterance, including at least determining whether the natural language content is missing one or more parameter values ​​for controlling a function associated with the action. The method further includes, when the verbal utterance is determined to be incomplete, determining context data, based on context data accessible via the computing device, characterizing a context in which the user provided the verbal utterance to the automated assistant interface, determining a time for presenting one or more suggestions via a display panel based on the context data, where the context data indicates that the user previously provided a separate verbal utterance to the automated assistant in the context, and causing the display panel of the computing device to provide one or more suggestion elements based on determining the time for presenting the one or more suggestions via the display panel. The one or more suggestion elements include a specific suggestion element that, when spoken by the user to the automated assistant interface, provides other natural language content via the display panel that causes the automated assistant to operate to facilitate completing the action.

[0091] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0092] In some implementations, determining the time for presenting the one or more suggestions includes comparing the context data to other context data characterizing prior instances in which the user, or one or more other users, provided one or more other verbal utterances while in the context.

[0093] In some implementations, the method further includes, when the spoken utterance is determined to be incomplete, identifying a natural language command that, when spoken by one or more users to the automated assistant interface, causes a reduced amount of speech-to-text processing compared to the amount of speech-to-text processing associated with the spoken utterance from the user and causes the automated assistant to operate to facilitate completing the action. The natural language command is incorporated by some content of another of the one or more suggestion elements.

[0094] In some implementations, a method implemented by one or more processors is provided, comprising performing speech-to-text processing on audio data characterizing an oral utterance provided by a user. The oral utterance includes natural language content and is received via an automated assistant interface of a computing device connected to a display panel. The method further includes causing a first action to be performed based on the natural language content based on performing speech-to-text processing on the audio data characterizing the oral utterance. The method further includes causing a display panel of the computing device to provide one or more suggestion elements in response to determining that the user has provided an oral utterance including natural language content. The one or more suggestion elements include a specific suggestion element that, when spoken by the user to the automated assistant interface, provides other natural language content via the display panel that causes the automated assistant to operate to facilitate completing the action. The method further includes determining that the user has selected a particular suggestion element of the one or more suggestion elements via a subsequent verbal utterance received at the automated assistant interface, and, in response to determining that the user has selected the particular suggestion element, causing a second action to be performed based on other natural language content identified by the particular suggestion element.

[0095] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0096] In some implementations, performance of the second action results in action data that is a modification of and / or supplements the other action data resulting from performance of the first action. In some of those implementations, the method further includes generating other natural language content in response to determining that the user has provided an oral utterance, where the other natural language content identifies at least one suggested value for a parameter used during performance of the second action.

[0097] In some implementations, causing the first action to be performed includes causing data based on predetermined default content associated with the user to be provided at the computing device. In some of those implementations, the method further includes modifying the default content to facilitate the modified default content being available when a subsequent incomplete request from the user is received based on the user selecting the particular suggestion element.

[0098] In some implementations, a method implemented by one or more processors is provided, which includes generating text based on performing speech-to-text processing on audio data capturing an oral utterance provided by a user. The oral utterance is received through an automated assistant interface of a computing device connected to a display panel. The method further includes determining whether the oral utterance is complete or incomplete based on the text. The method further includes determining a specific duration based on whether the oral utterance is determined to be complete or incomplete. The specific duration is shorter when the oral utterance is determined to be complete than when the oral utterance is determined to be incomplete. The method further includes generating one or more suggestion elements based on the text, where the one or more suggestion elements each indicate corresponding additional text that, when combined with the text, causes the automated assistant to operate to promote a corresponding action. The method further includes causing a display panel of the computing device to provide the one or more suggestion elements, and monitoring for further oral input for the determined duration after provisioning of the one or more suggestion elements. The method further includes, when the further verbal input is received within the duration, generating an automated assistant command based on the candidate text and additional text generated from performing speech-to-text processing on the further audio data incorporating the further verbal input, and causing the automated assistant to execute the automated assistant command. The method further includes, when the further verbal input is not received within the duration, causing the automated assistant to execute an alternative command based only on the candidate text.

[0099] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0100] In some implementations, determining that the verbal utterance is complete and determining the specific duration based on whether the verbal utterance is determined to be complete or incomplete further includes determining that the specific duration is a first specific duration when the verbal utterance is determined to be complete and directed to a specific automated assistant agent that is not a general search agent, and determining that the specific duration is a second specific duration when the verbal utterance is determined to be complete and directed to a general search agent, the second specific duration being longer than the first specific duration.

[0101] In some implementations, the verbal utterance is determined to be complete, and determining the particular duration based on whether the verbal utterance is determined to be complete or incomplete further includes determining the particular duration to be a first particular duration when the verbal utterance is determined to be complete and includes all required parameters, and determining the particular duration to be a second particular duration when the verbal utterance is determined to be complete but lacks all required parameters, the second particular duration being longer than the first particular duration.

[0102] In each implementation, the verbal utterance may be determined to be incomplete and associated processing may be performed, while in other implementations, the verbal utterance may be determined to be complete and associated processing may be performed. [Explanation of symbols]

[0103] 102, 204 First user interface 104, 206, 222, 304 Assistant interactive module 106 Graphical Elements, Suggested Text 108, 210, 310 Suggestion elements 110, 212, 320 Display Panel 112, 214, 312 users 114, 216, 314 Computing devices 116 Oral utterances, incomplete oral utterances, initial oral utterances 122 Completed candidate text, candidate text 124 Additional Suggestion Elements, 1st Additional Suggestion Element 126, 232 Secondary user interface 128 Additional verbal utterances, verbal utterances 130 phrases 208 Incomplete text segment 218 Oral utterance, first oral utterance 224 Completed demand, completed demand 226 Additional Suggestion Elements, Suggestion Elements 228, 318 additional oral utterances 230 "Until I go to bed." 306 Graphical Representation 308 Graphical prompt, prompt 316 Oral Utterances 400 Systems 402 Server Device 408 Automated Assistant 412 Input Processing Engine 414 Voice Processing Engine 416 Data Parsing Engine 418 Action Engine 420 Output Generation Engine 422 Assistant Data 424 Suggestion Engine 426 Device Topology Engine 428 Timing Engine 430 Speech-to-Text Model Engine 432 Action Engine 434 client device 436 Assistant Interface 438 Automated Assistants, Local Automated Assistants 440 Client Data 442 IoT devices 444 User Interface, Action Data 446 Network 610 Computer Systems 612 Bus Subsystem 614 processor 616 Network Interface Subsystem 620 User Interface Output Device 622 User Interface Input Devices 624 Memory Subsystem 625 Memory 626 File Storage Subsystem 630 Main Random Access Memory (RAM) 632 Read-Only Memory (ROM)

Claims

1. 1. A method implemented by one or more processors, comprising: performing speech-to-text processing on data characterizing the oral utterance provided by the user, the spoken utterance includes natural language content and is received via an automated assistant interface of a computing device connected to a display panel; determining whether the oral utterance is complete based on performing the speech-to-text processing on the data characterizing the oral utterance, determining whether an automated assistant can cause one or more actions to be accomplished based on the natural language content; when the oral utterance is determined to be incomplete, in response to determining that the oral utterance is incomplete, causing the display panel of the computing device to provide one or more suggestion elements; The one or more suggestion elements include a specific suggestion element that, when spoken by the user to the automated assistant interface, provides other natural language content via the display panel that causes the automated assistant to operate to facilitate completing an action; determining, after causing the display panel of the computing device to provide the one or more suggestion elements, that the user has provided another verbal utterance that is associated with the other natural language content of the particular suggestion element; determining whether a combination of the verbal utterance and the further verbal utterance is complete in response to determining that the user has provided the further verbal utterance; when the combination of the verbal utterance and the further verbal utterance is determined to be complete; causing the one or more actions to be performed via the automated assistant based on the natural language content and the other oral utterance; The method includes:

2. and performing speech-to-text processing on the data characterizing the oral utterance provided by the user, the method comprising: generating one or more candidate text segments from the data, the method comprising: when the oral utterance is determined to be incomplete, causing the display panel of the computing device to provide a graphical representation of at least one candidate text segment of the one or more candidate text segments. The method of claim 1, further comprising:

3. when the verbal utterance is determined to be complete and includes all required parameters for the one or more actions; causing the display panel of the computing device to bypass providing the graphical representation of the at least one candidate text segment of the one or more candidate text segments and to bypass providing the one or more suggestion elements.

3. The method of claim 2, further comprising:

4. When the spoken utterance is determined to be complete, Identifying natural language commands that, when spoken by one or more users to the automated assistant interface, result in a reduced amount of speech-to-text processing compared to an amount of speech-to-text processing associated with the verbal utterances from the users, and cause the automated assistant to operate to facilitate completing the action; generating one or more other suggestion elements characterizing the natural language command based on the step of identifying the natural language command; in response to at least determining that the spoken utterance is complete, causing the display panel of the computing device to provide the one or more other suggestion elements.

4. The method of claim 2 or 3, further comprising:

5. when the oral utterance is determined to be incomplete, selecting a speech-to-text processing model from a plurality of different speech-to-text processing models based on a type of oral utterance expected from the user when at least one suggestion element is provided via the display panel.

5. The method of claim 1, further comprising:

6. when the oral utterance is determined to be incomplete, Biasing the speech-to-text processing towards one or more terms of the one or more suggestion elements and / or towards one or more expected types of content corresponding to the one or more suggestion elements.

6. The method of claim 1, further comprising:

7. determining whether a threshold duration of oral input silence has followed said oral utterance; causing the display panel of the computing device to provide the one or more suggestion elements is further responsive to determining that at least a threshold duration of verbal input silence has followed the verbal utterance.

7. The method of claim 1, further comprising:

8. the step of performing speech-to-text processing includes determining a first candidate text segment and a second candidate text segment, the first candidate text segment and the second candidate text segment corresponding to different interpretations of the spoken utterance; 8. The method of claim 1, wherein the particular suggestion element is determined based on the first candidate text segment and at least one other suggestion element is determined based on the second candidate text segment.

9. causing the display panel coupled to the computing device to provide the one or more suggestion elements, 9. The method of claim 8, further comprising: causing the display panel coupled to the computing device to graphically represent the first candidate text segment adjacent to the particular suggestion element and to graphically represent the second candidate text segment adjacent to the at least one other suggestion element.

10. determining that the user has provided another spoken utterance associated with the other natural language content of the particular suggestion element, 10. The method of claim 9, further comprising determining whether the user has identified the first candidate text segment or the second candidate text segment based on the other spoken utterance.

11. when the oral utterance is determined to be incomplete, In response to determining that the oral utterance is incomplete, generating the other natural language content based on historical data characterizing one or more prior interactions in which a previous action was accomplished via the automated assistant and in which the user identified at least a portion of the other natural language content during one or more prior interactions between the user and the automated assistant.

11. The method of any one of claims 1 to 10, further comprising:

12. when the oral utterance is determined to be incomplete, in response to determining that the spoken utterance is incomplete, generating the other natural language content based on device topology data characterizing relationships between various devices associated with the user, the particular suggestion element is based on the device topology data and identifies one or more devices of the various devices associated with the user.

11. The method of any one of claims 1 to 10, further comprising:

13. 13. The method of claim 1, wherein causing the one or more actions to be performed via the automated assistant based on the natural language content and the other oral utterance comprises controlling a device.

14. The method of claim 1 , wherein the particular suggestion element further comprises a graphical element indicating the action.

15. when the oral utterance is determined to be incomplete, determining a specific duration for waiting for another verbal utterance after provisioning of the one or more suggestion elements, the specific duration being determined based on determining that the verbal utterance is incomplete.

14. The method of any one of claims 1 to 13, further comprising:

16. When the spoken utterance is determined to be complete, generating one or more other suggestion elements based on the complete spoken utterance; and causing the display panel of the computing device to provide the one or more other suggestion elements; determining an alternative specific duration for waiting for further verbal utterance after provisioning of the one or more other suggestion elements, the alternative specific duration being shorter than the specific duration, and the alternative specific duration being determined based on determining that the verbal utterance is complete; 16. The method of claim 15, further comprising:

17. 1. A method implemented by one or more processors, comprising: performing speech-to-text processing on data characterizing a verbal utterance provided by a user to facilitate causing an automated assistant to perform an action; the spoken utterance includes natural language content and is received via an automated assistant interface of a computing device connected to a display panel; determining whether the oral utterance is complete based on performing the speech-to-text processing on the data characterizing the oral utterance, determining whether the natural language content is absent one or more parameter values ​​for controlling a function associated with the action; when the oral utterance is determined to be incomplete, in response to determining that the oral utterance is incomplete, causing the display panel coupled to the computing device to provide one or more suggestion elements; The one or more suggestion elements include a specific suggestion element that, when spoken by the user to the automated assistant interface, provides other natural language content via the display panel that causes the automated assistant to operate to facilitate completing the action; Determining that the user selected the particular suggestion element of the one or more suggestion elements via another verbal utterance received at the automated assistant interface; in response to determining that the user has selected the particular suggestion element, causing the action to be performed based on the other natural language content of the particular suggestion element and the natural language content of the oral utterance; When the spoken utterance is determined to be complete, in response to determining that the verbal utterance is complete, causing the action to be performed based on the natural language content of the verbal utterance. The method includes:

18. when the oral utterance is determined to be incomplete, in response to determining that the user has selected the particular suggestion element, causing a priority associated with the other natural language content to be modified from a previous priority associated with the other natural language content before the oral utterance was determined to be incomplete.

20. The method of claim 17, further comprising:

19. 20. The method of claim 18, wherein an order of presentation of the particular suggestion element relative to at least one other suggestion element on the display panel is based at least in part on an assigned priority to the other natural language content.

20. determining whether a threshold duration of oral input silence has followed said oral utterance; causing the display panel of the computing device to provide the one or more suggestion elements is further responsive to determining that at least the threshold duration of verbal input silence has followed the verbal utterance.

20. The method of any one of claims 17 to 19, further comprising:

21. when the oral utterance is determined to be incomplete, In response to determining that the oral utterance is incomplete, generating the other natural language content based on historical data characterizing one or more prior interactions in which a previous action was accomplished via the automated assistant and in which the user identified at least a portion of the other natural language content during one or more prior interactions between the user and the automated assistant.

21. The method of any one of claims 17 to 20, further comprising:

22. when the oral utterance is determined to be incomplete, in response to determining that the spoken utterance was incomplete, generating the other natural language content based on device topology data characterizing relationships between various devices associated with the user, the particular suggestion element is based on the device topology data and identifies one or more devices of the various devices associated with the user.

21. The method of any one of claims 17 to 20, further comprising:

23. the step of performing speech-to-text processing includes determining a first candidate text segment and a second candidate text segment, the first candidate text segment and the second candidate text segment corresponding to different interpretations of the spoken utterance; 23. The method of claim 17, wherein at least one suggestion element of the one or more suggestion elements is determined based on the first candidate text segment and at least one other suggestion element is determined based on the second candidate text segment.

24. causing the display panel coupled to the computing device to provide one or more suggestion elements, 24. The method of claim 23, comprising causing the display panel coupled to the computing device to graphically represent the first candidate text segment adjacent to the at least one suggestion element and to graphically represent the second candidate text segment adjacent to the at least one other suggestion element.

25. 1. A method implemented by one or more processors, comprising: performing speech-to-text processing on data characterizing a verbal utterance provided by a user to facilitate causing an automated assistant to perform an action; the spoken utterance includes natural language content and is received via an automated assistant interface of a computing device connected to a display panel; determining whether the oral utterance is complete based on performing the speech-to-text processing on the data characterizing the oral utterance, determining whether the natural language content is absent one or more parameter values ​​for controlling a function associated with the action; when the oral utterance is determined to be incomplete, Determining context data characterizing a context in which the user provided the verbal utterance to the automated assistant interface based on context data accessible via the computing device; determining a time for presenting one or more suggestions via the display panel based on the context data; The context data indicates that the user has previously provided a distinct verbal utterance to the automated assistant in the context; causing the display panel of the computing device to provide one or more suggestion elements based on the determining the time for presenting the one or more suggestions via the display panel; the one or more suggestion elements include a particular suggestion element that, when spoken by the user to the automated assistant interface, provides other natural language content via the display panel that causes the automated assistant to operate to facilitate completing the action; The method includes:

26. determining the time for presenting the one or more suggestions comprises:

26. The method of claim 25, comprising comparing the contextual data to other contextual data characterizing prior instances in which the user, or one or more other users, provided one or more other verbal utterances while in the context.

27. when the oral utterance is determined to be incomplete, Identifying natural language commands that, when spoken by one or more users to the automated assistant interface, result in a reduced amount of speech-to-text processing as compared to an amount of speech-to-text processing associated with the verbal utterances from the users and cause the automated assistant to operate to facilitate completing the action; the natural language command is incorporated by some content of another of the one or more suggestion elements.

27. The method of claim 25 or 26, further comprising:

28. The other natural language content is based on a previous interaction between one or more users and the automated assistant that occurred before at least the action or at least another action was performed via the automated assistant; 28. A method according to any one of claims 25 to 27, wherein the action or the further action corresponds to execution of the function or a further function, respectively, on the computing device or on a separate computing device.

29. when the oral utterance is determined to be incomplete, selecting a speech-to-text processing model from a plurality of different speech-to-text processing models based on at least one suggestion element of the one or more suggestion elements provided via the display panel based on a type of oral utterance expected from the user when the at least one suggestion element is provided via the display panel.

29. The method of any one of claims 25 to 28, further comprising:

30. receiving a user input completing the oral utterance based on the one or more suggestions after causing the display panel of the computing device to provide the one or more suggestion elements; causing the automated assistant to operate to facilitate completing the action; 30. The method of any one of claims 25 to 29, further comprising:

31. 1. A method implemented by one or more processors, comprising: performing speech-to-text processing on audio data characterizing an oral utterance provided by a user, the spoken utterance includes natural language content and is received via an automated assistant interface of a computing device connected to a display panel; causing a first action to be performed based on the natural language content based on the performing of the speech-to-text processing on the audio data characterizing the oral speech; in response to determining that the user has provided the oral utterance that includes the natural language content, causing the display panel of the computing device to provide one or more suggestion elements; The one or more suggestion elements include a specific suggestion element that, when spoken by the user to the automated assistant interface, provides other natural language content via the display panel that causes the automated assistant to operate to facilitate completing an action; Determining that the user has selected the particular suggestion element of the one or more suggestion elements via a subsequent verbal utterance received at the automated assistant interface; in response to determining that the user selected the particular suggestion element, causing a second action to be performed based on the other natural language content identified by the particular suggestion element; The method includes:

32. 32. The method of claim 31 , wherein performance of the second action results in action data that is a modification of other action data resulting from the performance of the first action and / or that supplements the other action data resulting from the performance of the first action.

33. generating the other natural language content in response to determining that the user has provided the oral utterance, the other natural language content identifying at least one suggested value for a parameter used during performance of the second action.

33. The method of claim 32, further comprising:

34. modifying a priority associated with each suggestion element of the one or more suggestion elements in response to determining that the user has selected the particular suggestion element of the one or more suggestion elements; each priority indicating whether each corresponding suggestion element will be presented on the display panel in response to a subsequent request by the automated assistant to perform the action; 34. The method of any one of claims 31 to 33, further comprising:

35. 35. The method of claim 31, wherein causing the first action to be performed comprises causing data based on predetermined default content associated with the user to be provided at the computing device.

36. modifying the default content to facilitate the availability of the modified default content when a subsequent incomplete request from the user is received based on the user's selection of the particular suggestion element.

36. The method of claim 35, further comprising:

37. 37. The method of any one of claims 31 to 36, wherein the other natural language content is based on a previous interaction between the user and the automated assistant that occurred before at least the first action was performed via the automated assistant.

38. 1. A method implemented by one or more processors, comprising: generating text based on performing speech-to-text processing on audio data capturing oral speech provided by a user, The spoken utterance is received via an automated assistant interface of a computing device connected to a display panel; determining whether the spoken utterance is complete or incomplete based on the text; determining a particular duration based on whether the oral utterance is determined to be complete or incomplete, the particular duration being shorter when the verbal utterance is determined to be complete than when the verbal utterance is determined to be incomplete; generating one or more suggestion elements based on the text, The one or more suggestion elements each indicate corresponding additional text that, when combined with the text, causes an automated assistant to operate to facilitate a corresponding action; and causing the display panel of the computing device to provide the one or more suggestion elements; monitoring for further verbal input for the determined duration after provisioning of the one or more suggestion elements; when the further verbal input is received within the duration, generating an automated assistant command based on the candidate text and additional text generated from performing speech-to-text processing on further audio data incorporating the further verbal input; causing the automated assistant to execute the automated assistant command; when said further verbal input is not received within said duration; causing the automated assistant to execute an alternative command based solely on the candidate text; The method includes:

39. determining that the oral utterance is complete and determining the particular duration based on whether the oral utterance is determined to be complete or incomplete; determining that the particular duration is a first particular duration when the spoken utterance is determined to be complete and directed to a particular automated assistant agent that is not a general search agent; determining, when the spoken utterance is determined to be complete and directed to the general search agent, that the particular duration is a second particular duration; the second particular duration being greater than the first particular duration; 39. The method of claim 38, further comprising:

40. determining that the oral utterance is complete and determining the particular duration based on whether the oral utterance is determined to be complete or incomplete; determining that the particular duration is a first particular duration when the spoken utterance is determined to be complete and includes all required parameters; determining, when the oral utterance is determined to be complete but lacking all of the required parameters, that the particular duration is a second particular duration, the second particular duration being greater than the first particular duration; 39. The method of claim 38, further comprising:

41. 41. A computer program comprising instructions which, when executed by one or more processors, cause the one or more processors to perform a method according to any one of claims 1 to 40.

42. 41. A computer-readable storage medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 40.

43. 41. A system comprising one or more processors for performing the method of any one of claims 1 to 40.

Images