Dynamic adaptation of graphical user interface elements by automated assistant as user iteratively provides spoken utterance, or sequence of spoken utterances

Abstract

To provide dynamic adaptation of graphical user interface elements by an automated assistant when iteratively providing a spoken utterance or a sequence of spoken utterances.SOLUTION: Implementations described herein relate to an automated assistant that iteratively renders various GUI elements as a user iteratively provides a spoken utterance, or a sequence of spoken utterances. In some implementations, a generic container graphical element associated with candidate intents can be initially rendered at a display interface and dynamically adapted with tailored container graphical elements as a particular intent is determined while the user iteratively provides the spoken utterance. In additional or alternative implementations, the tailored container graphical elements can include a current status of one or more settings associated with a computing device such that the user can view the current status while completing the spoken utterance.SELECTED DRAWING: Figure 2B

Description

[Background technology]

[0001] A human may engage in human-computer interactions with an interactive software application, referred to herein as an "automated assistant" (also referred to as a "digital agent," "chatbot," "interactive personal assistant," "intelligent personal assistant," "assistant application," "conversational agent," etc.). For example, a human (sometimes referred to as a "user" when interacting with an automated assistant) may provide commands and / or requests to the automated assistant using voice natural language input (i.e., speech), which in some cases may be converted to text and then processed, and / or by providing textual (e.g., typed) natural language input.

[0002] In many cases, an automated assistant may be invoked by a user who may not have a complete command phrase in mind. For example, assume that a user gives a voice utterance of "Assistant, set ..." that includes a portion of a request that the automated assistant responds to. In this example, the portion of the request may indicate that the user intends to set the volume for a smart speaker, set the temperature for a smart thermostat, set the brightness level for a smart light bulb, etc. However, in many of these examples, if the user does not utter the complete command phrase within a certain time window, the automated assistant may not respond based on the request because it is too vague, or the automated assistant may respond based on the request and perform some action that the user did not intend. Thus, the user may need to re-invoke the automated assistant and give the voice utterance again with the complete command phrase, thereby extending one or more interaction sessions between the user and the automated assistant and increasing the amount of user input received at the computing device.

[0003] In some cases, an automated assistant may be invoked by a user who may have a complete command phrase in mind, but not a particular slot value associated with the command phrase. For example, assume a user gives a voice utterance of "Assistant, set the volume to ..." that includes a portion of a request that the automated assistant responds to. In this example, the portion of the request may indicate that the user intends to set the volume for the speaker as a particular slot value associated with a command to set the volume for the speaker. However, in many of these examples, the user may not know the current state of the speaker volume and, as a result, may not know how to modify the speaker volume for the current status. Thus, after providing the first portion of a voice utterance (e.g., "Assistant, set the volume to..."), the user may pause to think about how to modify the speaker volume. Similar to the above example, if the user does not pronounce a particular slot value within a certain time window, the automated assistant may not act on the request because it is too vague, or the automated assistant may act on the request and perform some action that the user did not intend. Again, the user may need to re-invoke the automated assistant and again provide the voice utterance with the complete command phrase and the particular slot value, thereby extending one or more interaction sessions between the user and the automated assistant and increasing the amount of user input received at the computing device. As a result, in such an example, processing the incomplete voice utterance may waste computational resources and require the user to repeat the interaction session with the automated assistant. Summary of the Invention [Means for solving the problem]

[0004] Implementations described herein relate to an automated assistant that may dynamically adapt graphical user interface (GUI) elements based on a user repeatedly providing a voice utterance, or a sequence of voice utterances, that includes a request directed to an automated assistant that at least partially executes on the user's computing device. The GUI elements may characterize portions of an incoming request from the user and / or provide suggestions that may help the user more efficiently and accurately describe the request to the automated assistant. In some implementations, based on processing portions of the request, candidate intents may be determined, and generic container graphical elements associated with the candidate intents may be rendered in a display interface of the computing device before the user completes the request. Furthermore, based on processing additional portions of the request, specific intents may be determined from the candidate intents, and the generic container graphical elements may be dynamically adapted to specific tailored container graphical elements associated with the specific intent without rendering a different display interface on the computing device. In additional or alternative implementations, specific words or phrases included in portions of the request may be directly mapped to generic container graphical elements without the need to determine candidate intents. In additional or alternative implementations, a particular adjusted container graphical element may include a current state of one or more settings of the computing device and / or additional computing devices in communication with the computing device (e.g., slot values ​​associated with the current state of the one or more settings) in response to a determination that a portion of the request relates to modifying a current state of one or more settings of the computing device and / or additional computing devices.

[0005] For example, assume that a user begins providing a voice utterance, "Assistant, set ...," which includes a portion of a request that the automated assistant adjust a state of a device, such as the volume for a smart speaker, the temperature for a smart thermostat, the brightness level for a smart light bulb, etc. As the user provides the portion of the request, the automated assistant may use a streaming automatic speech recognition (ASR) model to process a stream of audio data capturing the portion of the request and generate an ASR output. Additionally, the automated assistant may use a natural language understanding (NLU) model to process the ASR output and generate an NLU output. The generic container graphical element may be rendered in a display interface of the computing device based on the ASR output (e.g., indicating that the portion of the request includes "set" or another specific word or phrase) and / or the NLU output (e.g., indicating that the portion of the request includes a candidate intent associated with the generic container graphical element).

[0006] Further assume that the user continues to provide the voice utterance "... the volume for the speakers ..." (or as additional voice utterances following the voice utterance) that includes an additional portion of the request for the automated assistant to adjust the state of the device. Similarly, as the user provides the additional portion of the request, the automated assistant may use the streaming ASR model to process a stream of audio data that also captures the additional portion of the request to generate an additional ASR output. Furthermore, the automated assistant may use the NLU model to process the additional ASR output to generate an additional NLU output. Based on the additional ASR output and / or the additional NLU output, the automated assistant may determine that the user wants to set the volume for the smart speaker. Thus, the generic container graphical element may be dynamically adapted to an adjusted container graphical element specific to setting the volume for the smart speaker. For example, a tailored container graphical element specific to setting the volume for a smart speaker may include a current state of the volume for the smart speaker, a volume control graphical element that allows a user to set the volume using touch input, media content indicating that the volume is being set for the smart speaker, a device identifier associated with the smart speaker, and / or any other content associated with the smart speaker.

[0007] On the other hand, if the user indicates "... the temperature ..." while continuing to provide the voice utterance, the generic container graphical element may be dynamically adapted to a tailored container graphical element specific to setting a temperature for a smart thermostat that is separate from the tailored container graphical element specific to setting a volume for a smart speaker. For example, the tailored container graphical element specific to setting a temperature for a smart thermostat may include the current state of the temperature, media content indicating that the temperature is being set for the smart thermostat, a temperature control graphical element that allows the user to set the temperature using touch input, a device identifier associated with the smart thermostat, and / or any other content associated with the smart thermostat. Yet in either of these examples, the same generic container graphical element may be dynamically adapted to these various tailored container graphical elements without rendering any additional user interface.

[0008] The generic container graphical element may act as a placeholder for any one of a plurality of disparate tailored container graphical elements, each associated with a corresponding one of a plurality of disparate intents, or directly mapped to a particular word or phrase. Thus, as the user continues to provide requests to the automated assistant, the generic container graphical element may dynamically and seamlessly adapt to a particular tailored graphical element associated with a particular intent determined based on processing additional portions of the request. For example, in the above example, in response to the portion of the request "assistant, set ...", the generic container graphical element may initially be rendered in the display interface. The generic container graphical element may include, for example, an array of graphical elements (e.g., an array of dot shapes) to indicate a range of values. Thereafter, when the user provides additional portions of the request (e.g., "...speaker volume...", "...brightness...", "...temperature...", etc.), the automated assistant may adapt the array of graphical elements based on the additional portions of the request. For example, based on the user providing an additional portion of the request of "...speaker volume...", the array of graphical elements may be adapted to reflect a range of values ​​associated with the volume of the smart speaker and to include a current state of the volume of the smart speaker to assist the user in determining how to modify the volume. Further, for example, based on the user providing an additional portion of the request of "...brightness...", the array of graphical elements may be adapted to reflect a range of values ​​associated with the brightness of the smart light bulb and to include a current state of the brightness of the smart light bulb to assist the user in determining how to modify the brightness.

[0009] In some implementations, the automated assistant may process the candidate intent to identify a particular device and / or application that the user may be attempting to control in providing the request. When the automated assistant identifies a particular device and / or application, the automated assistant may cause an arrangement of graphical elements within the generic container graphical element to represent the current state of the particular application and / or device, resulting in an adjusted container graphical element. For example, the arrangement of graphical elements may include seven filled circles followed by three empty circles, thereby indicating that a smart light bulb associated with the particular device and / or application is currently at 70% of the maximum brightness level as the current state of the smart light bulb's brightness setting. Alternatively or additionally, the automated assistant may identify an icon representing the particular device and / or application that the user is expected to be referring to (e.g., an icon representing a kitchen light). The automated assistant may include an icon for the adjusted container graphical element to identify the particular device and / or application that the automated assistant selected to control in response to the request from the user. In this way, the user may avoid providing another portion of the request via voice utterance to specify a particular application and / or device (e.g., to change brightness from 70% to 50%) and instead choose to utilize touch input, thereby preserving computational resources that would otherwise be consumed when processing the voice utterance or additional voice utterances.

[0010] In some implementations, the user may have the automated assistant control a particular device and / or application based on witnessing the current state indicated by the array of graphical elements contained within the adjusted container graphical element by completing the request via one or more additional voice utterances. For example, by witnessing the array of graphical elements contained within the adjusted container graphical element, the user may consider the final portion of the user's request. The user may give a final voice utterance such as "... to 30%", thereby instructing the automated assistant to control the particular device and / or application to adjust the brightness level from 70% to 30%. Alternatively or additionally, the user may tap the portion of the array of graphical elements corresponding to the "30% dot" to similarly cause the automated assistant to adjust the brightness level from 70% to 30%.

[0011] In some implementations, in response to an initial portion of a request being given by a user, the automated assistant may cause multiple adjusted container elements to be rendered. For example, when a user gives an initial portion of a request via a voice utterance such as "Assistant, play [song title 1] by ...", the automated assistant may cause multiple different adjusted container elements to be rendered in a display interface of a computing device. Each of the adjusted container graphical elements may correspond to a different action and / or interpretation that may be associated with the request. For example, a first adjusted container graphical element may correspond to an action to play "[song title 1]" by "[artist 1]" on the computing device or the additional computing device, and a second adjusted container graphical element may correspond to another action to play "[song title 1]" by "[artist 2]" on the computing device or the additional computing device. In some implementations, additionally or alternatively, each of the adjusted container graphical elements includes a current state of the computing device or the additional computing device (e.g., what is currently playing on the first device and / or the second device). The user can complete the request (e.g., via voice utterances and / or touch input), and the automated assistant can have the request fulfilled accordingly.

[0012] In some implementations, the ASR output generated using the streaming ASR model may include, for example, predicted speech hypotheses predicted to correspond to various parts of the request, predicted phonemes predicted to correspond to various parts of the request, predicted ASR measures indicating how likely the predicted speech hypotheses and / or predicted phonemes correspond to various parts of the request, and / or other ASR outputs. Furthermore, the NLU output generated using the NLU model may include, for example, candidate intents predicted to correspond to the user's actual intent in providing the various parts of the request, one or more slot values ​​for corresponding parameters associated with the candidate intents, and / or other NLU outputs. Furthermore, one or more structured requests may be generated based on the NLU output and processed by various devices and / or applications to generate fulfillment data for the request. The fulfillment data, when implemented, may cause the automated assistant to fulfill the request provided by the user.

[0013] In some implementations, the generic container graphical element and / or the adjusted container graphical element described herein may only be rendered in the display interface of the computing device in response to a determination that the user has paused from providing the request. The automated assistant may determine that the user has paused from providing the request, for example, based on NLU data and / or audio-based characteristics associated with the portion of the request received at the computing device. The audio-based characteristics associated with the portion of the request may include one or more of intonation, tone, stress, rhythm, tempo, pitch, and elongated syllables. For example, assume that the user provides a request "Assistant, set the volume to..." contained within a voice utterance directed to the automated assistant. In this example, the automated assistant may determine that the user has paused, for example, based on a threshold duration that has elapsed since the aforementioned "to" and an NLU output indicating that the user has not provided a slot value for a volume parameter associated with an intent to change the predicted smart speaker volume. In response, the automated assistant may cause a volume container graphical element for the smart speaker to be rendered in the display interface of the computing device. A volume container graphical element for the smart speaker rendered in the display interface of the computing device may include the current volume of the smart speaker to assist the user in determining how to modify the volume relative to the current volume. Alternatively or additionally, assume further that the user includes a drawn-out syllable in providing the "to" (e.g., "Assistant, set the volume toooooo ..."). In this example, the automated assistant may determine that the user has paused based on audio-based characteristics reflecting uncertainty about how to modify the volume relative to the current volume, for example, based at least on the drawn-out syllable in providing the request.Thus, the volume container graphical element may assist the user in determining how to modify the volume relative to the current volume.

[0014] By using the techniques described herein, various technical advantages may be achieved. As a non-limiting example, the techniques described herein may enable an automated assistant to dynamically adapt various GUI elements from a generic GUI element to an adjusted GUI element while a user provides a voice utterance or a sequence of voice utterances. For example, a user may provide a voice utterance including a portion of a request, and the automated assistant may render a generic GUI element, which is then adapted to an adjusted GUI element based on processing an additional portion of the voice utterance or an additional voice utterance. Such an adjusted GUI element may assist the user in completing the request, thereby allowing an interaction session between the user and the automated assistant to be completed more quickly and efficiently, and reducing the amount of user input received at the computing device. Furthermore, cases in which an automated assistant fails because a user does not complete a request within a certain time window may be mitigated. As a result, computational resources at the computing device may be preserved, and latency in fulfilling requests may be reduced.

[0015] The above description is provided as a summary of some implementations of the present disclosure. More detailed descriptions of those implementations, as well as other implementations, are described in more detail below. [Brief description of the drawings]

[0016] [Figure 1A] 1A-1C are diagrams illustrating a user iteratively providing example requests to an automated assistant, with the automated assistant iteratively rendering graphical elements that correspond to portions of the example requests, according to various implementations. [Figure 1B] 1A-1C are diagrams illustrating a user iteratively providing example requests to an automated assistant, with the automated assistant iteratively rendering graphical elements that correspond to portions of the example requests, according to various implementations. [Figure 1C] 1A-1C are diagrams illustrating a user iteratively providing example requests to an automated assistant, with the automated assistant iteratively rendering graphical elements that correspond to portions of the example requests, according to various implementations. [Figure 2A] 1A-1C are diagrams illustrating a user iteratively providing additional example requests to an automated assistant, and the automated assistant iteratively rendering graphical elements that correspond to portions of the additional example requests, according to various implementations. [Figure 2B] 1A-1C are diagrams illustrating a user iteratively providing additional example requests to an automated assistant, and the automated assistant iteratively rendering graphical elements that correspond to portions of the additional example requests, according to various implementations. [Figure 2C] 1A-1C are diagrams illustrating a user iteratively providing additional example requests to an automated assistant, and the automated assistant iteratively rendering graphical elements that correspond to portions of the additional example requests, according to various implementations. [Diagram 3] 1 illustrates a system that provides an automated assistant that iteratively builds graphical elements as a user provides requests directed to the automated assistant, according to various implementations. [Figure 4A] 1A-1C illustrate methods for controlling an automated assistant to repeatedly present graphical elements in a display interface when a user provides a request directed to the automated assistant, according to various implementations. [Figure 4B] 1A-1C illustrate methods for controlling an automated assistant to repeatedly present graphical elements in a display interface when a user provides a request directed to the automated assistant, according to various implementations. [Diagram 5] FIG. 1 is a block diagram of an exemplary computer system according to various implementations. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] 1A, 1B, and 1C respectively show views 100, 120, and 140 in which a user 102 repeatedly provides an example request to an automated assistant, and the automated assistant repeatedly renders graphical elements corresponding to portions of the example request. The request may be included in a voice utterance directed to the automated assistant, or in a sequence of voice utterances directed to the automated assistant. For example, in the example of FIG. 1A, assume that the user 102 provides a voice utterance "assistant, set ..." that includes a first portion of a request 108 directed to the automated assistant that is accessible via a computing device 104 in the user's home. The user 102 may provide a voice utterance to prompt the automated assistant to perform a particular assistant action and / or execute a particular intent associated with the first portion of the request 108. The automated assistant may use a streaming automatic speech recognition (ASR) model to process a stream of audio data capturing the first portion of the request 108 to generate an ASR output. Additionally, the automated assistant may process the ASR output using a natural language understanding (NLU) model to generate an NLU output. In particular, such operations may be performed while the user 102 is providing a voice utterance that includes the first portion of the request 108.

[0018] In some implementations, the automated assistant may determine a candidate intent associated with the first portion of the request 108 based on the NLU output based on processing the first portion of the request 108. Furthermore, the automated assistant may determine a generic container graphical element 110 to be rendered in the display interface 106 of the computing device 104 based on the candidate intent. For example, the generic container graphical element 110 may be selected to represent a graphical user interface (GUI) element that may be dynamically adapted to a particular tailored container graphical element associated with a particular intent selected from among the candidate intents based on processing additional portions of the request (e.g., as shown in FIG. 1A and FIG. 1B). In additional or alternative implementations, the automated assistant may determine based on the ASR output (and without considering the NLU output) that the first portion of the request indicates that it includes a particular word or phrase (e.g., "set it"). For example, when a particular word or phrase is detected in the ASR output, the particular word or phrase may be mapped to a generic container graphical element 110 in an on-device memory of the computing device 104 such that the generic container graphical element 110 may be rendered on the display interface 106 of the computing device 104. The generic container graphical element 110 may likewise be dynamically adapted to a particular tailored container graphical element associated with a particular intent based on processing additional portions of the request (e.g., as shown in Figures 1B and 1C).In other words, because the first portion of the request 108 is predicted to correspond to a request to control an application and / or device involving a "setting" (e.g., based on the first portion of the request 108 including the words "set," "change," and / or other specific words or phrases), which may be represented by an array of graphical elements (e.g., an array of empty or filled circles as shown in FIG. 1A), the generic container graphical element 110 may be rendered in the display interface 106 of the computing device 104.

[0019] In some implementations, in response to determining that the user 102 has paused from providing a speech utterance including the first portion of the request 108, the generic container graphical element 110 may be rendered in the display interface 106 of the computing device 104. The automated assistant may determine that the user 102 has paused from providing a speech utterance including the first portion of the request 108, for example, based on NLU output generated in processing the speech utterance, audio-based characteristics determined based on processing the speech utterance, and / or a threshold duration that has elapsed since the user 102 provided the first portion of the request 108. For example, the automated assistant may determine that the user 102 has paused from providing a speech utterance (e.g., the user 102 said "set it" but failed to provide any indication as to what to "set") based on a threshold duration that has elapsed since the user 102 provided the first portion of the request 108 and based on an NLU output indicating that a slot value for a corresponding parameter associated with a candidate intent is unknown. In some versions of such implementations, the automated assistant may also consider one or more terms or phrases surrounding the predicted pause (e.g., whether the pause occurred after a preposition, or a disfluent speech (e.g., uhmmm, uhhh, etc.)). Alternatively or additionally, the audio-based characteristics may indicate that the manner in which the user 102 provided the first portion of the request 108 indicates that the user 102 paused to consider how to phrase the natural language provided to the automated assistant to complete the request.

[0020] To facilitate providing a complete request to the automated assistant, assume further that the user 102 continues to provide the voice utterance or provides an additional voice utterance including the second part of the request 122 by providing "...lights..." as shown in overview 120 of FIG. 1B. In this example, the stream of audio data capturing the second part of the request 122 may be processed to generate an additional ASR output and an additional NLU output. Based on the additional NLU output, the automated assistant may select a specific intent from among the candidate intents described with respect to FIG. 1A that indicates to the automated assistant that the user 102 wants to modify the current state of the smart light bulb in the user's 102 home. Thus, in response to receiving the second part of the request 122, the automated assistant may dynamically adapt the generic container graphical element 110 of FIG. 1A to the content associated with the specific intent determined based on processing the second part of the request 122, resulting in a specific tailored graphical container element 112 in a seamless manner.

[0021] The particular adjusted graphical container element 112 may be one of a plurality of disparate adjusted graphical container elements to which the generic graphical container element 110 may be dynamically adapted, and may be specific to a particular intent selected based on processing the second portion of the request 122. In other words, the particular adjusted graphical container element 112 shown in FIG. 1B may be specific to a particular intent associated with modifying a current state of the smart light bulb, and other intents may be associated with adjusted graphical container elements (e.g., modifying a temperature intent, modifying a speaker volume intent, etc.) that are different from the particular adjusted graphical container element 112 shown in FIG. 1B. The particular tailored graphical container element 112 may include, for example, media content 124 indicating a device and / or application that the automated assistant may associate with the second portion of the request 122 (e.g., the light bulb icon shown in FIG. 1B), a current state 120 indicating a current state of a device and / or application that the automated assistant may associate with the second portion of the request 122 and that is dynamically adapted from the arrangement of graphical elements shown in association with the generic container graphical element 110 of FIG. 1A (e.g., the seven filled circles shown in FIG. 1B to indicate that the smart light bulb is at 70% brightness), one or more control elements 144 that, when selected via touch input of the user 102, cause the current state 120 of the device and / or application to be controlled (e.g., as described in association with FIG. 1C), device identifiers associated with the various smart light bulbs to be controlled, and / or other content. In particular, when the user 102 provides a request, the generic container graphical element 110 is dynamically adapted to the specific tailored container graphical element 112, so that the specific tailored container graphical element 112 may be rendered on the display interface 106 of the computing device 104 before the user 102 completes the request and / or before the automated assistant completes fulfillment of the request.

[0022] In contrast to the example shown in FIG. 1B, assume that the user 102 continues to provide the voice utterance or provides an additional voice utterance including the second part of the request 122, such as "...speaker volume...". In this example, the second part of the request 122 may be processed in the same or similar manner as described above. However, the resulting particular adjusted container graphical element will be different from the particular adjusted container graphical element 112 shown in FIG. 1B. For example, the particular adjusted graphical container element in this contrasting example may include, for example, media content (e.g., a speaker icon) indicative of a device and / or application that the automated assistant may associate with the second part of the request 122, a current state (e.g., a filled circle may correspond to a speaker volume level) indicative of a current state of a device and / or application that the automated assistant may associate with the second part of the request 122 and dynamically adapted from the arrangement of graphical elements shown in association with the generic container graphical element 110 of FIG. 1A, one or more control elements that, when selected via touch input of the user 102, cause the current state of the device and / or application to be controlled, and / or other content. Thus, the same generic container graphical element 110 shown in FIG. 1B may be adapted differently based on the second portion of the request 122 .

[0023] Assume further that the user 102 completes the speech utterance, provides the additional speech utterance, or provides a complete request to the automated assistant by providing another additional speech utterance including a third portion of the request 142 by providing "... to 30 percent" as shown in overview 140 of FIG. 1C. In this example, a stream of audio data capturing the third portion of the request 142 may be processed to generate another additional ASR output and another additional NLU output. Based on the another additional NLU output, the automated assistant may generate a structured request to be utilized in fulfilling the speech utterance of the user 102. For example, in response to receiving the third portion of the request 142 from the user 102, the automated assistant may send one or more structured requests to the smart light bulb and / or an application associated with the smart light bulb accessible at the computing device 104 to modify the current state 122 of the smart light bulb from 70% brightness to 30% brightness. Further, in response to receiving the third part of the request 142, the automated assistant may reflect the change in brightness of the smart bulb by adapting the current state 122 to reflect that the smart bulb has changed from 70% brightness to 30% brightness (e.g., the seven filled circles shown in FIG. 1B to indicate that the smart bulb is at 70% brightness changes to three filled circles in FIG. 1C to indicate that the smart bulb is now at 30%).

[0024] 1C, the user 102 may direct a touch input to a given one of the one or more control graphical elements 144 (e.g., the third circle). Similarly, in response to the user 102 directing a touch input to a given one of the one or more control graphical elements, the automated assistant may generate a structured request to be utilized in fulfilling the user's 102 voice utterance to modify the current state 122 of the smart light bulb from 70% brightness to 30% brightness (assuming the user 102 directs the touch input to the third circle) and reflect the change in brightness of the smart light bulb by adapting the current state 122 to reflect the change in brightness of the smart light bulb from 70% brightness to 30% brightness as described above.

[0025] It should be understood that while the examples of FIG. 1A, FIG. 1B, and FIG. 1C are described in connection with a request to change the brightness of the smart light bulb in various portions (e.g., the first portion 108, the second portion 122, and the third portion 142), this is for illustrative purposes and is not meant to be limiting. For example, even if the user 102 does not pause while providing the request, the automated assistant may dynamically adapt GUI elements as described above to indicate to the user 102 that the automated assistant is listening to the user 102 and fulfilling the request. As a result, instances in which the user 102 repeats one or more portions of the request may be mitigated because the user 102 knows that the automated assistant is fulfilling the request based on the dynamic GUI elements described herein. However, if the user 102 pauses in providing the request, the dynamic GUI elements described herein may assist the user 102 in completing the request by at least providing the current state 122 of the smart light bulb to present to the user 102 when the request is provided. As a result, an interaction session between the user 102 and the automated assistant may be completed more quickly and efficiently. Additionally, while the examples of Figures 1A, 1B, and 1C are described in connection with a request to change the brightness of a smart light bulb, it should be understood that this is for purposes of illustration and not limitation. Rather, it should be understood that the techniques described herein may be utilized to provide dynamic GUI elements for any request directed to an automated assistant.

[0026] 2A, 2B, and 2C respectively show overviews 200, 220, and 240 in which a user 202 repeatedly provides an additional exemplary request to an automated assistant, and the automated assistant repeatedly renders graphical elements corresponding to portions of the additional exemplary request. The request may be included in a voice utterance directed to the automated assistant, or in a sequence of voice utterances directed to the automated assistant. For example, in the example of FIG. 2A, assume that the user 202 provides a voice utterance "assistant, set ..." that includes a first portion of a request 208 directed to an automated assistant that is accessible via a computing device 204 in the user's home. The user 202 may provide a voice utterance to prompt the automated assistant to perform a specific assistant action and / or execute a specific intent associated with the first portion of the request 208. 1A, the automated assistant may use a streaming automatic speech recognition (ASR) model to process the stream of audio data capturing the first portion of the request 208 to generate an ASR output, and may use a natural language understanding (NLU) model to process the ASR output to generate an NLU output. In response to receiving the first portion of the request 208, the automated assistant may cause the display interface 206 of the computing device 204 to render a generic container graphical element 210 in the same or similar manner as described above in connection with FIG. 1A.

[0027] Again, to facilitate providing a complete request to the automated assistant, assume further that the user 102 continues to provide the voice utterance or provides an additional voice utterance including the second part of the request 226 by providing "...lights..." as shown in overview 220 of FIG. 2B. Similarly, the stream of audio data capturing the second part of the request 226 may be processed to generate additional ASR output and additional NLU output. Based on the additional NLU output, the automated assistant may select a specific intent that indicates to the automated assistant that the user 202 wants to modify the current state of the smart light bulbs in the user's 202 home. Thus, in response to receiving the second part of the request 226. However, in contrast to the example of FIG. 1A, assume that the user 202 has multiple groups of smart light bulbs grouped together throughout the user's 202 home (e.g., a "kitchen" group of smart light bulbs, a "basement" group of smart light bulbs, and a "hallway" group of smart light bulbs). Thus, in the example of FIG. 2A, the automated assistant may dynamically adapt multiple instances of generic container graphical element 210 of FIG. 2A to content associated with a specific intent determined based on processing the second part of request 226 and based on multiple groups of smart light bulbs, resulting in multiple specific tailored graphical container elements 222A, 222B, and 222C.

[0028] For example, a first coordinated graphical container element 222A may be associated with a “Kitchen” group of smart light bulbs and may include a device identifier 224 of “Kitchen” and other content (e.g., a current state of the “Kitchen” lights at 50% brightness, indicated by five filled circles, control graphical elements associated with the “Kitchen” group of smart light bulbs, media content, and / or other content), and a second coordinated graphical container element 222B may be associated with a “Basement” group of smart light bulbs and may include a device identifier 228 of “Basement” and other content (e.g., nine filled circles, a current state of the "Basement" lights at 90% brightness, indicated by the 7 filled circles, control graphical elements, media content, and / or other content associated with the "Basement" group of smart light bulbs, and a third adjusted graphical container element 222C associated with the "Hallway" group of smart light bulbs and may include a device identifier 230 of "Hallway" and other content (e.g., a current state of the "Hallway" lights at 70% brightness, indicated by the 7 filled circles, control graphical elements, media content, and / or other content associated with the "Hallway" group of smart light bulbs). In some implementations, the automated assistant may retrieve the current state of each of the group of smart light bulbs from an application associated with the smart light bulbs, while in other implementations, the automated assistant may retrieve the current state of each of the group of smart light bulbs directly from the smart light bulbs. Thus, in the example time that user 202 provides the second portion of request 226, the automated assistant may dynamically adapt multiple instances of generic container graphical element 210, resulting in multiple distinct, tailored container graphical elements to dynamically provide instructions to user 202 on how to control the smart light bulbs based on the current status of each of the group of smart light bulbs.

[0029] In some implementations, when the user 202 provides a request, natural language content 232 corresponding to the request from the user 202 may be rendered in the display interface 206. For example, the natural language content 232 may include a streaming transcription (e.g., “Assistant, set the lights ...” as shown in the display interface 206 of the computing device 204 in FIG. 2B ) determined based on an ASR output generated using a streaming ASR model in processing a stream of audio data capturing the first portion of the request 208 and the second portion of the request 226. In particular, the natural language content may be rendered simultaneously with the multiple coordinated container graphical elements 222A, 222B, and 222C being rendered in the display interface 206 of the computing device 204. In these and other ways, the automated assistant may invoke various graphical elements (e.g., coordinated container graphical elements 222A, 222B, and 222C, natural language content 232, and / or other graphical elements) in an attempt to assist the user 202 in completing a request quickly and efficiently, thereby reducing the duration of an interaction session between the user 202 and the automated assistant.

[0030] To facilitate providing a complete request to the automated assistant, further assume that the user 202 completes the request by providing a third portion of the request 242, "... in the basement to 30 percent," as shown in overview 240 of FIG. 2C. In this example, a stream of audio data capturing the third portion of the request 242 may be processed to generate an ASR output and an NLU output. Based on the NLU output, the automated assistant may generate a structured request to be utilized in fulfilling the voice utterance of the user 202. For example, in response to receiving the third portion of the request 242 from the user 202, the automated assistant may send one or more structured requests to the smart light bulbs in the "basement" group and / or applications associated with the smart light bulbs accessible at the computing device 204 to modify the current state of the smart light bulbs in the "basement" group from 90% brightness to 30% brightness. Further, in response to receiving the third part of request 242, the automated assistant may reflect the change in brightness of the smart light bulbs by adapting the current state to reflect that the smart light bulbs have changed from 90% brightness to 30% brightness (e.g., the nine filled circles shown in FIG. 2B to indicate that the smart light bulbs in the “Basement” group are at 90% brightness change to the three filled circles in FIG. 2C to indicate that the smart light bulbs in the “Basement” group are now at 30%).

[0031] In some implementations, as shown in FIG. 2C, when the ASR output indicates that the user selected a particular group of smart light bulbs (e.g., the “basement” group in the example of FIG. 2C), the automated assistant may remove the other adjusted container graphical elements 222A and 222C from the display interface 206 of the computing device 204 such that the adjusted container graphical element 222B is the only remaining adjusted container graphical element associated with the particular intent of the user 202. In some implementations, the ASR and / or NLU process performed via the automated assistant may be biased according to the content of the container graphical element. For example, in response to receiving the third part of the request 242, the automated assistant may bias the ASR processing and / or ASR output toward “kitchen”, “basement”, and “hallway”.

[0032] FIG. 3 illustrates a system 300 that provides an automated assistant 304 that iteratively builds graphical elements as a user provides a request directed to the automated assistant 304. The automated assistant 304 may operate as part of an assistant application provided on one or more computing devices, such as the computing device 302 (e.g., an example of the computing device 104, an example of the computing device 204, etc.) and / or a server device. A user may interact with the automated assistant 304 through an assistant interface 320, which may be a microphone, a camera, a touch screen display, a user interface, and / or any other device capable of providing an interface between a user and an application. For example, a user may initialize the automated assistant 304 by providing verbal, textual, and / or graphical input to the assistant interface 320 to cause the automated assistant 304 to initialize one or more actions (e.g., provide data, control a peripheral device, access an agent, generate input and / or output, etc.). Alternatively, the automated assistant 304 may be initialized based on processing of the context data 336 using one or more trained machine learning models. The context data 336 may characterize one or more features of an environment in which the automated assistant 304 is accessible and / or one or more features of a user predicted to intend to interact with the automated assistant 304. The computing device 302 may include a display device, which may be a display panel including a touch interface (e.g., display interface 106 of computing device 104, display interface 206 of computing device 204) for receiving touch inputs and / or gestures to enable a user to control the applications 334 of the computing device 302 via a touch interface.In some implementations, the computing device 302 may lack a display device, thereby providing an audible user interface output without providing a graphical user interface output. Additionally, the computing device 302 may provide a user interface, such as a microphone, for receiving speech natural language input from a user. In some implementations, the computing device 302 may include a touch interface and may lack a camera, but may optionally include one or more other sensors.

[0033] The computing device 302 and / or other third party client devices may be in communication with a server device over a network such as the Internet. Additionally, the computing device 302 and any other computing devices may be in communication with each other over a local area network (LAN) such as a Wi-Fi network. The computing device 302 may offload computational tasks to the server device to preserve computational resources at the computing device 302. For example, the server device may host the automated assistant 304 and / or the computing device 302 may send inputs received at one or more assistant interfaces 320 to the server device. However, in some implementations, the automated assistant 304 may be hosted locally at the computing device 302 and various processes related to automated assistant operation may be performed exclusively at the computing device 302.

[0034] In various implementations, all or less than all aspects of the automated assistant 304 may be implemented on the computing device 302. In some such implementations, aspects of the automated assistant 304 are implemented via the computing device 302 and may interface with a server device, which may implement other aspects of the automated assistant 304. Optionally, the server device may service multiple users and their associated assistant applications via multiple threads. In implementations in which all or less than all aspects of the automated assistant 304 are implemented via the computing device 302, the automated assistant 304 may be an application separate from (e.g., installed "on") the operating system of the computing device 302, or alternatively may be implemented directly by (e.g., considered an application of the operating system but integrated with the operating system) the operating system of the computing device 302.

[0035] In some implementations, the automated assistant 304 may include an input processing engine 306, which may utilize multiple different modules for processing input and / or output to the computing device 302 and / or server device. For example, the input processing engine 306 may include a speech processing engine 308 that utilizes a streaming ASR model that may process a stream of audio data received at the assistant interface 320 to generate an ASR output, such as text embodied in the stream of audio data. Additionally, for example, the input processing engine 306 may use an audio-based machine learning model and / or a heuristic-based approach to determine audio-based characteristics associated with any voice utterances / requests captured in the stream of audio data. In some implementations, the stream of audio data may be transmitted, for example, from the computing device 302 to a server device to conserve computational resources at the computing device 302. Additionally or alternatively, the stream of audio data may be processed exclusively at the computing device 302.

[0036] The process for converting audio data to text may include speech recognition algorithms that may utilize neural networks and / or statistical models (e.g., streaming ASR models described herein) for identifying groups of audio data that correspond to words or phrases. The text converted from the audio data may be parsed by a data parsing engine 310 utilizing an NLU model and made available to the automated assistant 304 as text data that may be used to generate NLU output, such as command phrases, intents, actions, slot values, and / or any other content specified by the user. In some implementations, the output data provided by the data parsing engine 310 may be provided to a parameter engine 312 to determine whether the user has provided input that corresponds to a particular intent, action, and / or routine that may be performed by the automated assistant 304 and / or an application or agent that may be accessed via the automated assistant 304. For example, the assistant data 338 may be stored on the server device and / or the computing device 302 and may include data defining one or more actions that may be performed by the automated assistant 304, as well as parameters necessary to perform the action. The parameter engine 312 may generate one or more parameters for the intent, action, and / or slot value and provide the one or more parameters to the output generation engine 314. The output generation engine 314 may use the one or more parameters to communicate with the assistant interface 320 to provide output (e.g., visual output and / or audible output) to the user and / or to communicate with one or more applications 334 to provide output to the one or more applications 334.

[0037] In some implementations, as mentioned above, the automated assistant 304 may be an application that may be installed "on" the operating system of the computing device 302 and / or may itself form part (or the entirety) of the operating system of the computing device 302. The automated assistant application may include and / or have access to on-device ASR, on-device NLU, and on-device fulfillment. For example, the on-device ASR may be implemented using a streaming ASR model that processes a stream of audio data (detected by a microphone) using an end-to-end streaming ASR model stored locally on the computing device 302. The on-device speech recognition generates an ASR output, such as recognized text, for a voice utterance present (if present) in the stream of audio data. Additionally, the on-device NLU may be implemented to generate an NLU output, for example, using an on-device NLU model that processes the ASR output generated using the streaming ASR model and, optionally, contextual data. The NLU output may include a candidate intent corresponding to the voice utterance and, optionally, slot values ​​for corresponding parameters associated with the candidate intent.

[0038] Using an on-device fulfillment model or fulfillment rules utilizing the NLU output and optionally other local data, on-device fulfillment may be performed to determine a structured request for determining actions to take to resolve the intent of the voice utterance (and optionally slot values ​​for corresponding parameters associated with the candidate intent). This may include determining local and / or remote responses (e.g., answers) to the voice utterance and / or request, interactions with locally installed applications to perform based on the voice utterance and / or request, commands to send to an Internet of Things (IoT) device (directly or via a corresponding remote system) based on the voice utterance and / or request, and / or other resolution actions to perform based on the voice utterance and / or request. The on-device fulfillment then initiates local and / or remote implementation / execution of the determined actions to resolve the voice utterance and / or request.

[0039] In various implementations, remote speech processing, remote NLU, and / or remote fulfillment may be utilized at least selectively to conserve computational resources at the computing device 302. For example, recognized text may be at least selectively sent to a remote automated assistant component remote NLU and / or remote fulfillment. For example, optionally, recognized text may be sent for remote fulfillment in parallel with on-device fulfillment or in response to failure of on-device NLU and / or on-device fulfillment. However, on-device speech processing, on-device NLU, on-device fulfillment, and / or on-device execution may be prioritized at least for the reduced latency realized when resolving a voice utterance (because no client-server round trips are required to resolve the voice utterance). Furthermore, on-device functionality may be the only functionality available in situations without network connectivity or with limited network connectivity.

[0040] In some implementations, the computing device 302 may have access to one or more applications 334, which may be provided by a third party entity different from the entity that provided the computing device 302 and / or the automated assistant 304, or a first party entity that is the same entity that provided the computing device 302 and / or the automated assistant 304. The automated assistant 304 and / or the computing device 302 may access application data 330 to determine one or more actions that may be performed by the one or more applications 334, as well as the state of each application of the one or more applications 334 and / or the state of each device associated with the computing device 302. Additionally, the automated assistant 304 and / or the computing device 302 may access device data 332 to determine one or more actions that may be performed by the computing device 302 and / or one or more devices associated with the computing device 302. Additionally, the application data 330 and / or any other data (e.g., device data 332) may be accessed by the automated assistant 304 to generate context data 336 that may characterize the context in which a particular application 334 and / or device is running and / or the context in which a particular user is accessing the computing device 302, the application 334, and / or any other device or module.

[0041] While one or more applications 334 are running on the computing device 302, the device data 332 may characterize a current operational state of each application 334 running on the computing device 302. Additionally, the application data 330 may characterize one or more features of the running applications 334, such as the content of one or more graphical user interfaces being rendered at the direction of the one or more applications 334. Alternatively or additionally, the application data 330 may characterize action schemas, which may be updated by the respective applications and / or the automated assistant 304 based on the respective applications' current operational status. Alternatively or additionally, the one or more action schemas for the one or more applications 334 may remain static but may be accessed by the application state engine to determine appropriate actions to initialize via the automated assistant 304.

[0042] The computing device 302 may further include an assistant invocation engine 322 that may use one or more trained machine learning models to process the application data 330, the device data 332, the context data 336, and / or any other data accessible to the computing device 302. The assistant invocation engine 322 may process this data to determine whether to wait for the user to explicitly speak an invocation phrase to invoke the automated assistant 304, or to consider the data to indicate an intent by the user to invoke the automated assistant instead of requiring the user to explicitly speak an invocation phrase. For example, the one or more trained machine learning models may be trained using example training data based on a scenario in which the user is in an environment in which multiple devices and / or applications exhibit various operating states.

[0043] Examples of training data may be generated to capture training data that characterize contexts in which a user invokes an automated assistant and other contexts in which a user does not invoke an automated assistant. When one or more trained machine learning models are trained according to such examples of training data, the assistant invocation engine 322 may cause the automated assistant 304 to detect or limit detection of a voice invocation phrase from a user based on the context and / or environmental features. Additionally or alternatively, the assistant invocation engine 322 may cause the automated assistant 304 to detect or limit detection of one or more assistant commands from a user based on the context and / or environmental features. In some implementations, the assistant invocation engine 322 may be disabled or limited based on the computing device 302 detecting an assistant suppression output from another computing device. In this manner, when the computing device 302 detects an assistant suppression output, the automated assistant 304 will not be invoked based on the context data 336, and otherwise the automated assistant 304 would be invoked if the assistant suppression output was not detected.

[0044] In some implementations, the system 300 may include a candidate intent engine 316 for determining one or more candidate intents that may be associated with one or more portions of a request provided by a user to the automated assistant 304. For example, when a user provides a voice utterance such as "Assistant, play ...", the candidate intent engine 316 may identify one or more intents that may be associated with the voice utterance based on the aforementioned NLU output. Alternatively or additionally, the candidate intent engine 316 may eliminate some candidate intents that may not be associated with the voice utterance provided by the user.

[0045] In some implementations, the system 300 may include a generic container engine 318 that may retrieve and / or generate one or more generic container graphical elements based on one or more candidate intents identified by the candidate intent engine 316. The generic container graphical elements may be retrieved or generated (e.g., from an on-device memory of the computing device 302) in response to receiving a voice utterance that includes a request for the automated assistant 304 to fulfill. The generic container graphical elements may be assigned other elements and / or features associated with one or more requests that the user is predicted to be giving via the voice utterance. For example, an initial voice utterance such as "Assistant, play..." may not identify a particular slot value (e.g., a song, artist, movie, streaming service, etc.), but the generic container engine 318 may identify a type of slot value predicted to be associated with the request. Based on this type of slot value, the generic container engine 318 may retrieve and / or generate a generic container graphical element that may be dynamically adapted based on the portion of the request yet to be received at the computing device 302. For example, an initial voice utterance that includes the term "play" may be associated with a type of slot value for controlling media playback. Thus, a generic container graphical element associated with a "media playback" feature may be selected by the generic container engine 318 and then dynamically adapted to a feature associated with media playback. Alternatively, an initial voice utterance including the term "turn" may be associated with a type of slot value for controlling an application and / or device output level. Thus, a generic container graphical element associated with adjusting an application and / or device setting may be selected by the generic container engine 318 and then dynamically adapted to a feature associated with an application and / or device output level control setting.

[0046] In some implementations, the system 300 may include an adjusted container engine 326 that may obtain and / or generate control elements for assigning dynamically adapting generic container graphical elements, resulting in an adjusted container graphical element. For example, as the user provides additional portions of a request to the automated assistant 304, the adjusted container engine 326 may iteratively assign and / or remove control elements in the generic container graphical element. Each control element may be associated with a slot value and / or a type of slot value that is determined to correspond to the request that the user is predicted to be providing to the automated assistant 304. For example, when the automated assistant 304 predicts that the user is requesting to modify a device setting that may have a range of numerical values, the adjusted container engine 326 may select to assign a “sliding” GUI element (or any other element suitable for controlling a device setting) to the container graphical element. Alternatively, when a user is predicted to have provided a request to the automated assistant 304 to control playback of media content (e.g., audio and / or video), the tuned container engine 326 may select to assign one or more media playback control elements (e.g., a pause button, a play button, a skip button, etc.) to the container graphical element.

[0047] In some implementations, when the user provides the automated assistant 304 with additional portions of the request, the system 300 may include a state engine 324 that may obtain and / or remove the current state of the application and / or device for the adjusted container graphical element. Additionally, the state engine may determine an updated state for the application and / or device based on processing the voice utterance and / or touch input associated with the request. For example, when the automated assistant 304 predicts that the user is requesting to modify a setting on another computing device, the state engine 326 may determine the current state of the setting on the other computing device. Based on this current state, the state engine 326 may identify a state GUI element that may characterize the status of the device and incorporate the state GUI element into the adjusted container graphical element. For example, when the state of the setting corresponds to a value within a range of numerical values, the state engine 326 may generate a state GUI element that characterizes the range of numerical values ​​and highlights the current state of the setting. This state GUI element may be incorporated into the adjusted container graphical element currently rendered in the interface of the computing device 302. In this way, as the user continues to provide additional portions of the request, the user can be informed of the current state of the settings, thereby refreshing his or her memory of any current state and assisting the user in determining the desired updated state relative to the current state.

[0048] 4A and 4B show methods 400 and 420 for controlling an automated assistant to repeatedly provide a graphical element in a display interface when a user provides a request directed to the automated assistant. Methods 400 and 420 may be implemented by one or more applications, computing devices, and / or any other application or module capable of interacting with an automated assistant. Method 400 may include an operation 402 for determining whether a user has provided a request to the automated assistant. The request may be included within a voice utterance, such as, for example, "Assistant, adjust ...", where the voice utterance may refer to a first portion of the request to be fulfilled by the automated assistant.

[0049] When the automated assistant determines that at least a portion of the request has been received, the method 400 may proceed from operation 402 to operation 404. Operation 404 may include determining whether the request corresponds to a complete request or an incomplete request. In other words, the automated assistant may determine whether the user has provided sufficient information for the automated assistant to initiate fulfillment of the request. According to the above example, when the user provides the voice utterance "Assistant, adjust ...", the automated assistant may determine that the request corresponds to an incomplete request. Based on this determination, the method 400 may proceed from operation 404 to operation 406. Otherwise, if the automated assistant determines that the request corresponds to a complete request, the method 400 may proceed from operation 404 to operation 424 via continuation element "B", as shown in and described in connection with FIG. 4A and FIG. 4B. In various implementations, the method 400 may proceed to operation 406 even when the request is determined to be a complete request in operation 404. Therefore, it should be understood that the methods 400 and 420 are illustrative and not meant to be limiting.

[0050] Operation 406 may include causing the generic container graphical element to be rendered in a display interface of the computing device. The generic container graphical element may act as a placeholder for other graphical elements to which the generic container graphical element may be dynamically adapted. For example, the generic container graphical element may be a graphical rendering of a shape having a body that includes sufficient area to allocate other graphical elements. The other graphical elements may include, but are not limited to, control elements for controlling one or more applications and / or devices, status elements for indicating a current state of one or more applications and / or devices, device identifiers for one or more applications and / or devices, media elements based on media content, and / or any other type of element that may be rendered in a display interface. Method 400 may proceed from operation 406 to operation 408, which may include determining whether additional portions of the incomplete request have been received by the automated assistant.

[0051] The additional portion of the request may be a voice utterance, such as "...temperature...". When it is determined that the additional portion of the incomplete request has been received, the method 400 may proceed from operation 408 to operation 410. When it is instead determined that another portion of the incomplete request has not been received, the method 400 may proceed from operation 408 to optional operation 424 of the method 420 via continuation element "A", as shown in Figures 4A and 4B and described in connection with Figure 4B. Optional operation 424, shown in Figure 4B, may include rendering one or more selectable suggestions in a display interface. The one or more selectable suggestions may be based on one or more portions of the incomplete request received from the user by the automated assistant. In this manner, although the user did not provide a request that was complete, the automated assistant may still provide one or more selectable suggestions that are predicted to correspond to one or more intents that the user may be attempting to communicate. Thereafter, optionally, the method 420 may return to operation 402 via continuation element "C" as shown in Figures 4A and 4B.

[0052] When it is determined that additional portions of the incomplete request have been received at operation 408, the method 400 may proceed from operation 408 to operation 410. Operation 410 may include determining that the incomplete request corresponds to a particular intent. In some examples, operation 410 may be performed after the user provides one or more additional inputs to complete the incomplete request. For example, when the user provides a first portion of the request, "Assistant, adjust ..." followed by a second portion of the request (e.g., contained within the same voice utterance or an additional voice utterance following the voice utterance), "...temperature...", the automated assistant may determine that the user is requesting to modify the temperature settings of the application and / or device. In some implementations, a particular intent may have slot values ​​for corresponding parameters associated with the particular intent. In some implementations, a request from a user may be considered incomplete based on a predicted probability, as determined by the automated assistant or other application. For example, the predicted probability may indicate the likelihood that the user is requesting that the particular intent be executed. When the predicted probability meets a probability threshold, the request from the user may be considered complete. Slot values ​​for the corresponding parameters can then be assigned to specific intents based on additional input from the user and / or data available to the automated assistant.

[0053] Method 400 may proceed from operation 410 to operation 412, which may include dynamically adapting the generic container graphical element, resulting in a tailored container graphical element specific to a particular intent. The tailored container graphical element may include one or more control elements for controlling a particular assistant action, one or more status elements indicating the status of an application and / or device, and / or other content described herein. For example, based on a portion of the request (e.g., "Assistant, adjust ..."), the automated assistant may dynamically adapt various graphical elements to the generic container graphical element for indicating the current temperature setting of a particular device (e.g., hallway thermostat). Alternatively or additionally, the automated assistant may have another graphical element assigned to the generic container graphical element for adjusting the temperature setting of the particular device. In this way, the user will be able to see the current state of the particular device, as well as options for controlling the particular device. By repeatedly assigning graphical elements to the container graphical element, the automated assistant and the computing device may preserve time and resources that would otherwise be consumed waiting for the user to provide a completed request.

[0054] Method 400 may proceed from operation 412 to operation 414, which may include determining whether an input for initializing a particular assistant action has been received. The input may be, for example, another portion of the request contained within the same voice utterance, or an additional voice utterance following the voice utterance (e.g., "...at 72 degrees..."), and / or a touch input in an area of ​​the display interface that is rendering a particular adjusted container graphical element. When it is determined that an input for initializing the fulfillment of the request has been received, method 400 proceeds from operation 414 through continuation element "B" to operation 422 of method 420, as shown in and described in connection with FIG. 4A and FIG. 4B. Otherwise, method 400 may proceed from operation 414 through continuation element "A" to optional operation 424 and / or operation 402.

[0055] Operation 422 may include initializing a fulfillment based on a request from a user. The fulfillment may correspond to executing a particular intent to fulfill the request. For example, the user may provide an additional voice utterance such as "...to 72 degrees..." based on a particular adjusted container graphical element indicating that the device's current state is 67 degrees. Thus, the user's memory may be refreshed with the information carried in the particular adjusted container graphical element. From operation 422, method 420 may proceed to optional operation 426, which may include rendering a response output based on the fulfillment. For example, the particular adjusted container graphical element may be assigned additional content based on the performance of the fulfillment. According to the previous example, the container graphical element may be assigned additional graphical content to indicate that the device's temperature setting has been successfully adjusted or modified from a current state of 67 degrees to an updated state of 72 degrees. Thereafter, method 420 may return to operation 402 via continuation element "C".

[0056] 5 is a block diagram 500 of an exemplary computer system 510. The computer system 510 typically includes at least one processor 514 that communicates with several peripheral devices via a bus subsystem 512. Such peripheral devices may include a storage subsystem 524, including, for example, a memory 525 and a file storage subsystem 526, a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input and output devices enable user interaction with the computer system 510. The network interface subsystem 516 provides an interface to external networks and is coupled to corresponding interface devices in other computer systems.

[0057] The user interface input devices 522 may include a keyboard, a pointing device such as a mouse, a trackball, a touchpad, a graphics tablet, a scanner, a touch screen integrated into a display, a voice recognition system, an audio input device such as a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and ways for inputting information into the computer system 510 or onto a communications network.

[0058] The user interface output devices 520 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or some other mechanism for producing a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and manners for outputting information from the computer system 510 to a user or to another machine or computer system.

[0059] Storage subsystem 524 stores programming and data configurations that provide the functionality of some or all of the modules described herein. For example, storage subsystem 524 may include logic for performing selected aspects of method 400 and / or implementing one or more of system 300, computing device 104, computing device 204, an automated assistant, and / or any other applications, devices, apparatus, and / or modules discussed herein.

[0060] Such software modules are typically executed by the processor 514 alone or in combination with other processors. The memory 525 used within the storage subsystem 524 may include several memories, including a main random access memory (RAM) 530 for storing instructions and data during program execution, and a read-only memory (ROM) 532 in which fixed instructions are stored. The file storage subsystem 526 may provide persistent storage for program and data files, and may include hard disk drives, floppy disk drives and associated removable media, CD-ROM drives, optical drives, or removable media cartridges. Modules implementing the functionality of some implementations may be stored within the storage subsystem 524 by the file storage subsystem 526, or within other machines accessible by the processor 514.

[0061] The bus subsystem 512 provides a mechanism that allows the various components and subsystems of the computer system 510 to communicate with each other in the intended manner. Although the bus subsystem 512 is shown generally as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0062] The computer system 510 can be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of the computer system 510 shown in Figure 5 is intended only as a specific example to illustrate some implementations. Many other configurations of the computer system 510 are possible, having more or fewer components than the computer system shown in Figure 5.

[0063] In situations where the systems described herein may collect or utilize personal information about users (or, as they are often referred to herein, "participants"), users may be provided with the opportunity to control whether a program or function collects user information (e.g., information about the user's social networks, social actions or activities, occupation, user preferences, or the user's current geographic location) or whether and / or how content that may be more relevant to the user is received from a content server. Additionally, certain data may be processed in one or more ways such that personally identifiable information is removed before it is stored or used. For example, the identity of the user may be processed such that personally identifiable information about the user cannot be determined, or, if geographic location information is obtained (such as at the city, zip code, state level), the user's geographic location may be generalized such that the user's specific geographic location cannot be determined. Thus, users may have control over how information is collected and / or used about them.

[0064] Although several implementations have been described and illustrated herein, various other means and / or structures for performing the functions and obtaining the results and / or one or more of the advantages described herein may be utilized, and each such variation and / or modification is deemed to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on the particular application or applications in which the teachings are used. Those skilled in the art will recognize and be able to ascertain, using no more than routine experimentation, many equivalents to the specific implementations described herein. Thus, the implementations described above are presented merely as examples, and it should be understood that within the scope of the appended claims and their equivalents, implementations may be practiced otherwise than as specifically described and claimed. Implementations of the present disclosure have been directed to each individual feature, system, article, material, kit, and / or method described herein. Furthermore, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent.

[0065] In some implementations, a method implemented by one or more processors is provided, comprising receiving, at a computing device, a first portion of a request from a user contained within a voice utterance. The voice utterance is directed to an automated assistant accessible via the computing device. The method further comprises determining one or more candidate intents associated with the first portion of the request based on processing the first portion of the request, and rendering a generic container graphical element at a display interface of the computing device based on the one or more candidate intents. The generic container graphical element may be dynamically adapted to any one of a plurality of heterogeneous tailored container graphical elements, each of the plurality of heterogeneous tailored container graphical elements being associated with a corresponding one of the one or more candidate intents. The method further includes receiving, at the computing device, a second portion of the request from the user that is also included within the voice utterance or included within an additional voice utterance received after the voice utterance, determining that the request corresponds to a particular intent among the one or more candidate intents based on processing the second portion of the request, and dynamically adapting a generic container graphical element rendered in the display interface to a particular tailored container graphical element among the plurality of heterogeneous tailored container graphical elements based on the particular intent.

[0066] Optionally, these and other implementations of the techniques described herein include one or more of the following features.

[0067] In some implementations, a particular tailored container graphical element may feature a slot value for a corresponding parameter associated with a particular intent, and the first and second parts of the request may not specify a slot value.

[0068] In some implementations, the method may further include, in response to receiving the first portion of the request, causing a display interface of the computing device to visually render natural language content characterizing the first portion of the request. The display interface may render the natural language content of the first portion of the request simultaneously with rendering the generic container graphical element.

[0069] In some implementations, the method may further include determining that a threshold duration has elapsed after receiving the first portion of the request. Causing the generic container graphical element to be rendered in the display interface may be performed based on the elapse of the threshold duration.

[0070] In some implementations, the particular adjusted container graphical element may include a particular geographic control element associated with a current state of one or more settings of the computing device or one or more additional computing devices in communication with the computing device. In some versions of such implementations, the method may further include receiving, at the computing device, a third portion of a request from the user also included within the voice utterance, the additional voice utterance, or a further additional voice utterance received after the voice utterance or the additional voice utterance, the third portion of the request including an updated state for the one or more settings, and causing, by the automated assistant, the one or more settings of the computing device or one or more of the additional computing devices to change from the current state to the updated state.

[0071] In some implementations, dynamically adapting the generic container graphical element rendered in the display interface to the particular tailored container graphical element may include selecting the particular tailored container graphical element from among a plurality of disparate tailored container graphical elements based on a type of slot value identified in the second speech utterance. The type of slot value may correspond to a numeric value that is limited to a range of numeric values.

[0072] In some implementations, the generic container graphical element may be rendered in a display interface of the computing device prior to receiving the second part of the request.

[0073] In some implementations, the generic container graphical element may be rendered in a display interface of the computing device while the second part of the request is received.

[0074] In some implementations, determining one or more candidate intents associated with the first portion of the request based on processing the first portion of the request may include: processing a stream of audio data generated by one or more microphones of the computing device using a streaming automatic speech recognition (ASR) model to generate an ASR output, where the stream of audio data captures the first portion of the request; processing the ASR output using a natural language understanding (NLU) model to generate the NLU output; and determining one or more candidate intents associated with the first portion of the request based on the NLU output. In some versions of such implementations, determining that the request corresponds to a particular intent of the one or more candidate intents based on processing a second portion of the request may include processing the stream of audio data using a streaming ASR model to generate an additional ASR output, where the stream of audio data also captures the second portion of the request; processing the additional ASR output using the NLU model to generate the additional NLU output; and selecting a particular intent from among the one or more candidate intents based on the additional NLU output.

[0075] In some implementations, a method implemented by one or more processors is provided that includes receiving, at a computing device, a portion of a request submitted by a user, the portion of the request being included in a voice utterance directed to an automated assistant accessible via the computing device; determining, by the automated assistant, the portion of the request being associated with modifying a current state of one or more settings of the computing device or one or more additional computing devices in communication with the computing device via the automated assistant; determining adjusted container graphical element data characterizing a current state of the one or more settings based on the current state of the one or more settings; and determining, based on the adjusted container graphical element data, a portion of the request associated with modifying a current state of one or more settings of the computing device or one or more additional computing devices in communication with the computing device via the automated assistant. causing a display interface of the computing device to render one or more adjusted container graphical elements indicating a current state of the one or more settings; and in response to causing a display interface of the computing device to render one or more adjusted container graphical elements indicating the current state of the one or more settings, receiving at the computing device an additional portion of a request submitted by the user, the additional portion of the request being included within a voice utterance or an additional voice utterance received after the voice utterance, the additional portion of the request including an updated state for the one or more settings; and causing, by the automated assistant, to change the one or more settings of the computing device or one or more of the additional computing devices from the current state to the updated state.

[0076] Optionally, these and other implementations of the techniques described herein include one or more of the following features.

[0077] In some implementations, each of the one or more adjusted container graphical elements may include a graphical icon to represent a current state of one or more settings.

[0078] In some implementations, the method may further include, in response to receiving the portion of the request, causing a display interface of the computing device to visually render natural language content that characterizes the portion of the request. The one or more coordinated graphical container elements may be rendered simultaneously as the display interface renders the natural language content.

[0079] In some implementations, the request portion may not include the current state of one or more settings.

[0080] In some implementations, causing the display interface of the computing device to render one or more adjusted container graphical elements indicative of a current state of the one or more settings based on the adjusted container graphical element data may include rendering a first adjusted container graphical element of the one or more adjusted container graphical elements indicative of a first setting of the one or more settings of a first computing device of the one or more additional computing devices separate from the computing device, and rendering a second adjusted container graphical element of the one or more adjusted container graphical elements indicative of a second setting of the one or more settings of a second computing device of the one or more additional computing devices separate from the computing device. In some versions of such implementations, the first setting of the first computing device may correspond to a volume setting of the first computing device, and the second setting of the second computing device may correspond to a volume setting of the second computing device. In additional or alternative versions of such implementations, the first setting of the first computing device may correspond to a brightness setting of the first computing device, and the second setting of the second computing device may correspond to a brightness setting of the second computing device.

[0081] In some implementations, the method may further include determining, based on processing the portion of the request, that the user has paused in providing the request. Having a display interface of the computing device render one or more adjusted container graphical elements indicative of a current state of the one or more settings may be responsive to determining that the user has paused in providing the request. In some versions of such implementations, determining, based on processing the portion of the request, that the user has paused in providing the request may include determining, based on processing the portion of the request, that the user paused after providing a particular word or phrase in providing the request. In additional or alternative versions of such implementations, having a display interface of the computing device render one or more adjusted container graphical elements indicative of a current state of the one or more settings may be responsive to determining that the user has paused for a threshold duration in providing the request. In additional or alternative versions of such implementations, determining that the user paused in providing the request based on processing the portion of the request may include processing a stream of audio data generated by one or more microphones of the computing device using a streaming automatic speech recognition (ASR) model to generate an ASR output, where the stream of audio data captures the portion of the request, processing the ASR output using a natural language understanding (NLU) model to generate an NLU output, and determining based on the NLU output that the user paused in providing the request. In additional or alternative versions of such implementations, determining that the user paused in providing the request based on processing the portion of the request may include determining audio-based characteristics associated with the portion of the request based on processing the portion of the request, and determining that the user paused in providing the request based on the audio-based characteristics associated with the portion of the request.

[0082] In some implementations, a method implemented by one or more processors is provided, the method including receiving, at a computing device, a first portion of a request from a user contained within a voice utterance. The voice utterance is directed to an automated assistant accessible via the computing device. The method further includes determining, based on processing the first portion of the request, that the portion of the request includes a specific word or phrase associated with controlling the computing device or one or more additional computing devices in communication with the computing device, and rendering a generic container graphical element in a display interface of the computing device based on the first portion of the request including the specific word or phrase. The generic container graphical element may be dynamically adapted to any one of a plurality of heterogeneous tailored container graphical elements, each of the plurality of heterogeneous tailored container graphical elements being associated with a corresponding intent determined based on processing the first portion of the voice utterance. The method further includes receiving, at the computing device, a second portion of the request from the user that is also included within the voice utterance or included within an additional voice utterance received following the voice utterance, determining that the request corresponds to a particular intent among the one or more candidate intents based on processing the second portion of the request, and dynamically adapting a generic container graphical element rendered in the display interface to a particular tailored container graphical element among the plurality of heterogeneous tailored container graphical elements based on the particular intent.

[0083] Optionally, these and other implementations of the techniques described herein include one or more of the following features.

[0084] In some implementations, determining, based on processing the first portion of the request, that the portion of the request includes a specific word or phrase associated with controlling the computing device or one or more additional computing devices in communication with the computing device may include processing a stream of audio data generated by one or more microphones of the computing device using a streaming automatic speech recognition (ASR) model to generate an ASR output, where the stream of audio data captures the first portion of the request, and determining, based on the ASR output, that the portion of the request includes a specific word or phrase associated with controlling the computing device or one or more additional computing devices.

[0085] In some versions of such an implementation, causing a generic container graphical element to be rendered on a display interface of the computing device based on a first portion of the request including a particular word or phrase may include determining that the particular word or phrase maps to a generic container graphical element in an on-device memory of the computing device, and in response to determining that the particular word or phrase maps to the generic container graphical element, causing the generic container graphical element to be rendered on the display interface of the computing device without processing the ASR output using a natural language understanding (NLU) model.

[0086] Another implementation may include a non-transitory computer-readable storage medium storing instructions executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) to perform a method such as one or more of the methods described above and / or elsewhere herein. Yet another implementation may include a system of one or more computers including one or more processors operable to execute the stored instructions to perform a method such as one or more of the methods described above and / or elsewhere herein.

[0087] It should be understood that all combinations of the foregoing concepts and additional concepts described in more detail herein are contemplated as being part of the subject matter disclosed herein, for example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein. [Explanation of symbols]

[0088] 102 users 104 Computing Devices 106 Display Interface 108 requests 110 Generic container graphical elements 112 Graphical Container Elements 120 Current Status 122 request 124 Media Content 142 Request 144 Control Elements 202 users 204 Computing Devices 206 Display Interface 208 Request 210 Generic container graphical elements 222A Graphical Container Elements 222B Graphical container elements 222C Graphical container elements 224 Device Identifier 226 Request 228 Device Identifier 230 Device Identifier 232 Natural Language Content 242 request 300 Systems 302 Computing Devices 304 Automated Assistants 306 Input Processing Engine 308 Voice Processing Engine 310 Data Parsing Engine 312 Parameter Engine 314 Output Generation Engine 316 Candidate Intent Engine 318 General-purpose container engine 320 Assistant Interface 322 Assistant Call Engine 324 State Engine 326 State Engine 330 Application Data 332 Device Data 334 Specific Applications 336 Context Data 510 Computer Systems 512 Bus Subsystem 514 processor 516 Network Interface Subsystem 520 User Interface Output Device 522 User Interface Input Devices 524 Memory Subsystem 526 File Storage Subsystem 525 Memory 526 File Storage Subsystem 530 Main Random Access Memory (RAM) 532 Read-Only Memory (ROM)

Claims

1. 1. A method implemented by one or more processors, comprising: receiving, at a computing device, a first portion of a request from a user contained within a voice utterance, the voice utterance being directed to an automated assistant accessible via the computing device; determining one or more candidate intents associated with the first portion of the request based on processing the first portion of the request; Rendering a generic container graphical element in a display interface of the computing device based on the one or more candidate intents, the generic container graphical element may be dynamically adapted to any one of a plurality of heterogeneous tailored container graphical elements; each of the plurality of heterogeneous coordinated container graphical elements being associated with a corresponding one of the one or more candidate intents; receiving, at the computing device, a second portion of the request from the user contained within the voice utterance or contained within an additional voice utterance received subsequent to the voice utterance; determining, based on processing the second portion of the request, that the request corresponds to a particular intent among the one or more candidate intents; dynamically adapting the generic container graphical element rendered in the display interface to a particular tailored container graphical element of the plurality of disparate tailored container graphical elements based on the particular intent; The method includes:

2. characterizing slot values ​​for corresponding parameters for which the particular adjusted container graphical element is associated with the particular intent; The method of claim 1 , wherein the first part of the request and the second part of the request do not specify the slot value.

3. in response to receiving the first portion of the request, causing the display interface of the computing device to visually render natural language content that characterizes the first portion of the request; The method of claim 1 or 2, wherein the display interface renders the natural language content of the first portion of the request simultaneously with rendering the generic container graphical element.

4. determining that a threshold duration has elapsed after receiving the first portion of the request; The method of claim 1 , wherein the step of causing the generic container graphical element to be rendered in the display interface is performed based on the passage of the threshold duration.

5. 5. The method of claim 1, wherein the particular adjusted container graphical element comprises a particular graphical control element associated with a current state of one or more settings of the computing device or one or more additional computing devices in communication with the computing device.

6. receiving, at the computing device, a third portion of the request from the user contained within the voice utterance, the additional voice utterance, or a further voice utterance received after the voice utterance or the additional voice utterance, the third portion of the request including an updated state for the one or more settings; causing the automated assistant to change the one or more settings of the computing device or one or more of the additional computing devices from the current state to the updated state; 6. The method of claim 5, further comprising:

7. said step of dynamically adapting said generic container graphical element rendered in said display interface to said particular tailored container graphical element comprises: selecting the particular adjusted container graphical element from among the plurality of heterogeneous adjusted container graphical elements based on a type of slot value identified in a second speech utterance; said type of slot value corresponds to a numeric value limited to a range of numeric values; 7. The method of claim 1 , comprising:

8. The method of claim 1 , wherein the generic container graphical element is rendered in the display interface of the computing device before receiving the second part of the request.

9. The method of claim 1 , wherein the generic container graphical element is rendered on the display interface of the computing device while the second part of the request is received.

10. determining one or more candidate intents associated with the first portion of the request based on processing the first portion of the request, processing a stream of audio data generated by one or more microphones of the computing device using a streaming automatic speech recognition (ASR) model to generate an ASR output, the stream of audio data capturing the first portion of the request; processing the ASR output using a natural language understanding (NLU) model to generate an NLU output; determining the one or more candidate intents associated with the first portion of the request based on the NLU output; 10. The method of any one of claims 1 to 9, comprising:

11. determining that the request corresponds to a particular intent among the one or more candidate intents based on processing the second portion of the request, processing the stream of audio data using a streaming ASR model to generate an additional ASR output, the stream of audio data also capturing the second portion of the request; processing the additional ASR output using the NLU model to generate an additional NLU output; selecting the particular intent from among the one or more candidate intents based on the additional NLU output; The method of claim 10, comprising:

12. 1. A method implemented by one or more processors, comprising: Receiving a portion of a request submitted by a user at a computing device, the portion of the request being included within a voice utterance directed to an automated assistant accessible via the computing device; determining, by the automated assistant, that the portion of the request is associated with modifying a current state of one or more settings of the computing device or one or more additional computing devices in communication with the computing device via the automated assistant; determining adjusted container graphical element data characterizing the current state of the one or more settings based on the current state of the one or more settings; causing a display interface of the computing device to render one or more adjusted container graphical elements indicative of the current state of the one or more settings based on the adjusted container graphical element data; in response to causing the display interface of the computing device to render the one or more adjusted container graphical elements indicating the current state of the one or more settings. receiving, at the computing device, an additional portion of the request submitted by the user, the additional portion of the request being included within the voice utterance or an additional voice utterance received after the voice utterance, the additional portion of the request including an updated state for the one or more settings; causing the automated assistant to change the one or more settings of the computing device or one or more of the additional computing devices from the current state to the updated state; The method includes:

13. The method of claim 12 , wherein each of the one or more adjusted container graphical elements includes a graphical icon to represent the current state of the one or more settings.

14. responsive to receiving the portion of the request, causing the display interface of the computing device to visually render natural language content that characterizes the portion of the request; The method of claim 12 or 13, wherein the one or more coordinated graphical container elements are rendered simultaneously as the display interface renders the natural language content.

15. The method of claim 12 , wherein the portion of the request does not include the current state of the one or more settings.

16. causing the display interface of the computing device to render the one or more adjusted container graphical elements indicative of the current state of the one or more settings based on the adjusted container graphical element data, causing a first adjusted container graphical element of the one or more adjusted container graphical elements to be rendered indicative of a first configuration of the one or more configurations of a first computing device of the one or more additional computing devices other than the computing device; causing a second adjusted container graphical element of the one or more adjusted container graphical elements to be rendered indicative of a second configuration of the one or more configurations of a second computing device of the one or more additional computing devices other than the computing device; 16. The method of any one of claims 12 to 15, comprising:

17. the first setting of the first computing device corresponds to a volume setting of the first computing device; The method of claim 16 , wherein the second setting of the second computing device corresponds to a volume setting of the second computing device.

18. the first setting of the first computing device corresponds to a brightness setting of the first computing device; The method of claim 16 , wherein the second setting of the second computing device corresponds to a brightness setting of the second computing device.

19. determining that the user paused in providing the request based on processing the portion of the request; 19. The method of claim 12, wherein the step of causing the display interface of the computing device to render the one or more adjusted container graphical elements indicating the current state of the one or more settings is in response to determining that the user paused in providing the request.

20. determining that the user has paused in providing the request based on processing the portion of the request, determining, based on processing the portion of the request, that the user paused after providing a particular word or phrase in providing the request; 20. The method of claim 19, comprising:

21. 20. The method of claim 19, wherein the step of causing the display interface of the computing device to render the one or more adjusted container graphical elements indicating the current state of the one or more settings is in response to determining that the user paused for a threshold duration in providing the request.

22. determining that the user has paused in providing the request based on processing the portion of the request, processing a stream of audio data generated by one or more microphones of the computing device using a streaming automatic speech recognition (ASR) model to generate an ASR output, the stream of audio data capturing the portion of the request; processing the ASR output using a natural language understanding (NLU) model to generate an NLU output; determining, based on the NLU output, that the user paused in providing the request; 20. The method of claim 19, comprising:

23. determining that the user has paused in providing the request based on processing the portion of the request, determining audio-based characteristics associated with the portion of the request based on processing the portion of the request; determining that the user paused in providing the request based on the audio-based characteristic associated with the portion of the request; 20. The method of claim 19, comprising:

24. 1. A method implemented by one or more processors, comprising: receiving, at a computing device, a first portion of a request from a user contained within a voice utterance, the voice utterance being directed to an automated assistant accessible via the computing device; determining, based on processing the first portion of the request, that the portion of the request includes a particular word or phrase associated with controlling the computing device or one or more additional computing devices in communication with the computing device; rendering a generic container graphical element in a display interface of the computing device based on the first portion of the request including the particular word or phrase; the generic container graphical element may be dynamically adapted to any one of a plurality of heterogeneous tailored container graphical elements; each of the plurality of disparate coordinated container graphical elements is associated with a corresponding intent determined based on processing the first portion of the speech utterance; receiving, at the computing device, a second portion of the request from the user contained within the voice utterance or contained within an additional voice utterance received subsequent to the voice utterance; determining, based on processing the second portion of the request, that the request corresponds to a particular intent among the one or more candidate intents; dynamically adapting the generic container graphical element rendered in the display interface to a particular tailored container graphical element of the plurality of disparate tailored container graphical elements based on the particular intent; The method includes:

25. determining, based on processing the first portion of the request, that the portion of the request includes the particular word or phrase associated with controlling the computing device or the one or more additional computing devices in communication with the computing device; processing a stream of audio data generated by one or more microphones of the computing device using a streaming automatic speech recognition (ASR) model to generate an ASR output, the stream of audio data capturing the first portion of the request; determining, based on the ASR output, that the portion of the request includes the particular word or phrase associated with controlling the computing device or the one or more additional computing devices; 25. The method of claim 24, comprising:

26. causing the generic container graphical element to be rendered at the display interface of the computing device based on the first portion of the request including the particular word or phrase, determining that the particular word or phrase is mapped to the generic container graphical element within an on-device memory of the computing device; in response to determining that the particular word or phrase is mapped to the generic container graphical element, rendering the generic container graphical element at the display interface of the computing device without processing the ASR output using a natural language understanding (NLU) model; 26. The method of claim 25, comprising:

27. At least one processor; a memory storing instructions that, when executed, cause said at least one processor to perform operations corresponding to any one of claims 1 to 26; A system comprising:

28. 27. A non-transitory computer-readable storage medium storing instructions that, when executed, cause at least one processor to perform operations corresponding to any one of claims 1 to 26.

Images