Passive disambiguation of assistant command

Abstract

To passively disambiguate assistant commands.SOLUTION: An automated assistant can initialize execution of an assistant command associated with an interpretation that is predicted to be responsive to user input, while simultaneously providing suggestions for alternative assistant command(s) associated with alternative interpretation(s) that is / are also predicted to be responsive to the user input. The alternative assistant command(s) can be selectable such that, when selected, the automated assistant can pivot from executing the assistant command to initializing execution of the selected alternative assistant command(s). Further, the alternative assistant command(s) can be partially fulfilled prior to any user selection thereof. Accordingly, implementations set forth herein can enable the automated assistant to quickly and efficiently pivot between assistant commands that are predicted to be responsive to the user input.SELECTED DRAWING: Figure 1B

Description

[Background technology]

[0001] A person may engage in human-computer interaction using interactive software applications referred to herein as "automated assistants" (also referred to as "digital agents," "chatbots," "interactive personal assistants," "intelligent personal assistants," "assistant applications," "conversational agents," etc.). For example, a person (who may be referred to as a "user" when interacting with an automated assistant) may provide commands and / or requests to the automated assistant using verbal natural language input (i.e., speech), which may in some cases be converted to text and then processed, and / or by providing textual (e.g., typed) natural language input.

[0002] In many cases, interacting with an automated assistant may create many opportunities to misinterpret ambiguous user inputs, including assistant requests and / or commands. For example, assume that a user provides a request for playback of media content (e.g., a song) available through multiple different media applications. In this example, the automated assistant may provide an output (e.g., an audible and / or visual output) that requests the user to select a specific application to be utilized for playback of the media content--rather than the automated assistant automatically selecting a specific application in response to the request and causing immediate playback of the media content. Furthermore, assuming there are multiple media content items with the same name, the automated assistant may provide additional or alternative output (e.g., an audible and / or visual output) that requests the user to select a specific media content item with the same name--rather than the automated assistant selecting a specific media content in response to the request and causing playback of the specific media content in a specific application. As a result, the interaction between the automated assistant and the user may be prolonged, thereby wasting the computational resources of the client device utilized to implement the interaction between the automated assistant and the user and wasting the user's time.

[0003] In some cases, even if an automated assistant automatically selects a particular application to be utilized for playing media content in response to a request and / or automatically selects a particular media content, there is often no efficient mechanism for the user to pivot to an alternative application or alternative media content that may also fulfill the request. For example, assume that the request for playing media content is a request for playing a song titled "Crazy" that is available for playing by a plurality of different media applications. Further assume that there are multiple songs titled "Crazy" by different artists, including at least a first artist and a second artist. In this example, if the user intended to cause the playing of "Crazy" by the second artist, but the automated assistant automatically selects and automatically causes the playing of "Crazy" by the first artist, the user may be required to provide further user input to cancel the playing of "Crazy" by the first artist and provide further additional user input (often enhanced to include the name of the second artist) to provide the playing of "Crazy" by the second artist. As a result, the interaction between the automated assistant and the user is prolonged, thereby wasting the computational resources of the client device utilized to implement the interaction between the automated assistant and the user. Summary of the Invention [Means for solving the problem]

[0004] Some implementations described herein relate to an automated assistant that automatically initiates the execution of at least a first action to fulfill an assistant command included in a verbal utterance given by a user. In addition, at the same time, the automated assistant has a selectable element related to a corresponding alternative action associated with an alternative fulfillment of the assistant command included in the verbal utterance provided for presentation to the user while the first action is being executed by the automated assistant. Thus, when the user selects the selectable element, the automated assistant terminates the execution of the first action and automatically initiates the execution of the corresponding alternative action associated with an alternative fulfillment of the assistant command included in the verbal utterance. In these and other ways described herein, the automated assistant can passively disambiguate the verbal utterance when the first action does not correspond to a particular action intended by the user, such that the user does not have to subsequently resend the verbal utterance or any clarifying verbal utterance. In other words, the automated assistant can initiate the execution of a given action predicted to be responsive to the verbal utterance and can quickly and efficiently pivot to an alternative action that is also considered to be responsive to the verbal utterance. As used herein, having the automated assistant simultaneously provide a selectable element for presentation to the user while a first operation is being performed by the automated assistant may include having the selectable element provided for presentation to the user contemporaneously with the first operation being automatically initiated by the automated assistant and / or within a threshold duration before and / or after the time the first operation is automatically initiated by the automated assistant.

[0005] In some versions of these implementations, assistant input data characterizing a plurality of interpretations predicted to respond to the verbal utterance may be generated based on the verbal utterance. In these implementations, each of the plurality of interpretations may include a corresponding intent, one or more corresponding parameters associated with the corresponding intent, and one or more corresponding slot values ​​of the one or more corresponding parameters associated with the corresponding intent. Additionally, metric data characterizing a predicted degree of correspondence between each of the plurality of interpretations and an assistant command included in the verbal utterance may be generated based on the assistant input data. In some implementations, the metric data may include a confidence level associated with processing the verbal utterance using various components, such as, for example, ASR metrics associated with an ASR output generated based on processing the verbal utterance using an automatic speech recognition (ASR) model, NLU metrics associated with an NLU output generated based on processing the ASR output using a natural language understanding (NLU) model, performance metrics associated with a performance output generated based on processing the NLU output using a performance model and / or rules, and / or other metrics associated with processing the verbal utterance. In additional or alternative implementations, the metrics data may be based on user profile data of the user who provided the verbal utterance, user profile data of other users similar to the user who provided the verbal utterance, aggregate increase in verbal utterances including requests across a particular geographic region, and / or other data. In some versions of those implementations, the metrics data may be generated using one or more machine learning models and / or heuristic processes that process these various signals to generate the metrics data. Based on the assistant input data and the metrics data, the automated assistant may automatically initialize a first action associated with a first interpretation of the plurality of interpretations to fulfill an assistant command included in the verbal utterance, and may provide selectable elements associated with other interpretations of the plurality of interpretations for presentation to the user.

[0006] For example, assume that a user provides the verbal utterance "Play the song 'Crazy'." The automated assistant can process the verbal utterance using various components (e.g., an ASR component, an NLU component, an execution component, and / or other components) to generate assistant input data characterizing multiple interpretations. In this example, each of the multiple interpretations may be associated with a musical intent based on a determination that the user intends for a song titled "Crazy" to be played in response to the verbal utterance. However, in providing the verbal utterance, the user specified a slot value for a song parameter associated with the musical intent (e.g., "Crazy") but did not specify a slot value associated with an artist parameter associated with the musical intent, a slot value for an application parameter associated with the musical intent, or a slot value for any other parameter that may be associated with the musical intent. Nonetheless, in generating assistant input data characterizing multiple interpretations predicted to be in response to the verbal utterance, the automated assistant can infer various slot values ​​to generate the multiple interpretations. For example, a first interpretation may include musical intent, song parameter slot value “Crazy”, artist parameter slot value “Artist 1”, and application parameter slot value “Application 1”, a second interpretation may include musical intent, song parameter slot value “Crazy”, artist parameter slot value “Artist 2”, and application parameter slot value “Application 1”, a third interpretation may include musical intent, song parameter slot value “Crazy”, artist parameter slot value “Artist 1”, and application parameter slot value “Application 2”, and so on.

[0007] It should be understood that while the above example is described in the context of multiple interpretations that each share the same intent (e.g., a musical intent), it is by way of example and not intended to be limiting. In contrast to the above example, assume that the user instead provided the verbal utterance "Play 'The Floor is Lava'." Similarly, the automated assistant may process the verbal utterance using various components (e.g., an ASR component, an NLU component, an execution component, and / or other components) to generate assistant input data that characterizes the multiple interpretations. However, in this example, the multiple interpretations may be associated with different intents. For example, a first interpretation could include a musical intent, a song parameter slot value "The Floor is Lava", an artist parameter slot value "Artist 1", and an application parameter slot value "Application 1", a second interpretation could include a musical intent, a song parameter slot value "The Floor is Lava", an artist parameter slot value "Artist 2", and an application parameter slot value "Application 1", a third interpretation could include a television program intent, a video parameter slot value "The Floor is Lava", and an application parameter slot value "Application 2", a fourth interpretation could include a game intent, a game parameter slot value "The Floor is Lava", and an application parameter slot value "Application 3", and so on.

[0008] Further, in these examples, the automated assistant can automatically initialize a given operation associated with a given one of the multiple interpretations to fulfill the assistant command included in the verbal utterance based on the assistant input data and based on the metrics data (e.g., ASR metrics data, NLU metrics data, and / or performance metrics data) generated for each of the multiple interpretations when processing the verbal utterance. For example, in the first example, the automated assistant can automatically initialize execution of a first operation associated with a first interpretation including a musical intent, a song parameter slot value of “Crazy”, an artist parameter slot value of “Artist 1”, and an application parameter slot value of “Application 1”, assuming that the metrics data indicates that the first interpretation most likely corresponds to the verbal utterance assistant command. Nevertheless, in this example, the automated assistant can cause selectable elements associated with the alternative interpretations to be rendered for presentation to the user while the first operation is automatically initialized and executed (e.g., while “Crazy” by “Artist 1” is played by “Application 1”). For example, while "Crazy" by "Artist 1" is played by "Application 1," corresponding selectable elements associated with a second interpretation (e.g., musical intent, song parameter slot value "Crazy," artist parameter slot value "Artist 2," and application parameter slot value "Application 1") and corresponding selectable elements associated with a third interpretation (e.g., musical intent, song parameter slot value "Crazy," artist parameter slot value "Artist 1," and application parameter slot value "Application 2") may be rendered for presentation to a user.

[0009] Thus, in response to a user's selection of one of the corresponding selectable elements (e.g., by touch selection directed to the corresponding selectable element or voice selection of the corresponding selectable element), the automated assistant can pivot away from performing a first action to automatically initialize and execute an alternative action associated with the alternative action to fulfill the assistant command contained in the verbal utterance. For example, in response to a user's selection of a corresponding selectable element associated with a second interpretation, the automated assistant can terminate execution of a first action (e.g., playing "Crazy" by "Artist 1" in "Application 1") and automatically initialize execution of a second action associated with the second interpretation (e.g., initiating playing "Crazy" by "Artist 2" in "Application 1"). As a result, the automated assistant can quickly and efficiently pivot between different interpretations of the verbal utterance without having to generate and render prompts for the user to actively disambiguate these different interpretations of the verbal utterance before initiating any actions. Rather, by enabling passive disambiguation of assistant commands contained in verbal utterances as described herein, the automated assistant can prioritize reducing latency in fulfilling assistant commands contained in verbal utterances and can prioritize faster and more efficient conclusion of a human-computer interaction between the automated assistant and the user.

[0010] Additionally, while the above examples are described with reference to assistant commands related to controlling media content, it should be understood that this is by way of example and not intended to be limiting. For example, assume that the user instead provides the verbal utterance, "Translate the light to 20 percent." Similarly, the automated assistant may process the verbal utterance using various components (e.g., an ASR component, an NLU component, an execution component, and / or other components) to generate assistant input data that characterizes multiple interpretations. In this example, assume that the verbal utterance was intended by the user to cause one or more smart lights controllable via the automated assistant to be dimmed to 20%. However, because a user may not typically use this local language to adjust the brightness settings of the user's lights, a first interpretation of the verbal utterance may include a translation intent with a slot value of "the light to 20 percent" as a term or phrase parameter to be translated from the first language to one or more second languages. Nevertheless, a second interpretation of the verbal utterance may include a translation intent with a slot value of "the light to 20 percent" for the brightness parameter. A third interpretation of the verbal utterance may include a light control intent with a brightness parameter slot value "20 percent" and a light location parameter slot value "kitchen", and a third interpretation of the verbal utterance may include a light control intent with a brightness parameter slot value "20 percent" and a light location parameter slot value "living room", and so on.As a result, the automated assistant can not only automatically initialize a given action associated with a given one of multiple interpretations to fulfill an assistant command contained in a spoken utterance (e.g., providing a translation of "the light to 20 percent" from a first language into one or more second languages), but can also provide selectable elements associated with alternative interpretations of the spoken utterance that, when selected, allow a user to adjust one or more smart lights by touch selection of the selectable element or voice selection of the selectable element.

[0011] In various implementations, processing of the additional verbal utterance may be biased toward content related to the selectable element rendered for presentation to the user, such as when the user provides a voice selection of the selectable element via the additional verbal utterance. For example, as described above, assume that the user provides the verbal utterance "Play the song 'Crazy'." Further, the automated assistant automatically initiates execution of a first action associated with a first interpretation that includes a musical intent, a slot value of the song parameter "Crazy," a slot value of the artist parameter "Artist 1," and a slot value of the application parameter "Application 1." Further, corresponding selectable elements associated with a second interpretation including musical intent, song parameter slot value “Crazy”, artist parameter slot value “Artist 2”, and application parameter slot value “Application 1” and corresponding selectable elements associated with a third interpretation including musical intent, song parameter slot value “Crazy”, artist parameter slot value “Artist 1”, and application parameter slot value “Application 2” are rendered for presentation to the user. In this example, ASR and / or NLU processing of the additional verbal utterances can be biased toward the slot values ​​associated with these alternative interpretations, or ASR and / or NLU output generated based on processing of the additional verbal utterances can be biased toward the slot values ​​associated with these alternative interpretations.

[0012] In various implementations, the selectable elements associated with the alternative interpretations may be rendered for presentation to the user for a threshold duration after the automated assistant automatically initiates execution of the assistant command to fulfill the assistant command included in the verbal utterance. In some versions of these implementations, the threshold duration may be static. For example, the selectable elements associated with the alternative interpretations may be rendered for presentation to the user for 10 seconds, 15 seconds, or any other static threshold duration after the automated assistant automatically initiates execution of the assistant command to fulfill the assistant command included in the verbal utterance. In other versions of these implementations, the threshold duration may be dynamic and may be based, for example, on one or more of the intents of one or more of the multiple interpretations or the type of action to be performed based on one or more of the multiple actions. For example, in an implementation in which the multiple actions are associated with media content, the selectable elements associated with the alternative interpretations may be rendered for presentation to the user for only 10 seconds. However, in an implementation in which the multiple actions are associated with controlling the user's smart device, the selectable elements associated with the alternative interpretations may be rendered for presentation to the user for only 30 seconds.

[0013] In various implementations, performance data for alternative actions associated with a selectable element may be pre-cached on a client device utilized by a user to interact with the automated assistant to reduce latency in executing the alternative actions. Continuing with the example above in which a user provides the verbal utterance "Play the song 'Crazy'" and the automated assistant automatically initiates execution of a first action associated with a first interpretation including a musical intent, a song parameter slot value "Crazy", an artist parameter slot value "Artist 1", and an application parameter slot value "Application 1", the automated assistant may queue "Crazy" by "Artist 2" on "Application 1" for the second interpretation and may queue "Crazy" by "Artist 1" on "Application 2" for the third interpretation. In other words, the automated assistant can, in the background of the client device, establish connections with various applications, queue various content for playback on the client device, queue various content for display on the client device, and / or otherwise pre-cache performance on the client device.

[0014] In various implementations, the selectable elements rendered for presentation to the user may be included in GUI data characterizing an assistant graphical user interface (GUI) in response to a verbal utterance. The GUI data may include, for example, one or more control elements associated with controlling a first action automatically initiated by an automated assistant, selectable elements associated with alternative actions, and / or any other data associated with how and / or when to render content for visual presentation to the user. For example, the GUI data may include prominence data indicating a corresponding area of ​​a client device's display utilized to display selectable elements associated with alternative actions and a corresponding size of a corresponding area of ​​a client device's display utilized to display selectable elements associated with alternative actions. Continuing with the example above in which the user provides the verbal utterance "Play the song 'Crazy'" and the automated assistant automatically initiates execution of a first operation associated with a first interpretation including a musical intent, a song parameter slot value of "Crazy," an artist parameter slot value of "Artist 1," and an application parameter slot value of "Application 1," the GUI data in this example may indicate that the selectable elements associated with the second interpretation should be displayed more prominently than the selectable elements associated with the third interpretation, e.g., the selectable elements associated with the second interpretation encompass a larger area of ​​the client device's display, the selectable elements associated with the second interpretation are displayed above the selectable elements associated with the third interpretation, etc. In this example, the GUI data may indicate that the selectable elements associated with the second interpretation should be displayed more prominently than the selectable elements associated with the third interpretation based on metrics data associated with the second interpretation indicating that the second interpretation is more likely to be responsive to the verbal utterance compared to the third interpretation.Also, in this example, the GUI data may indicate that only the selectable elements associated with the second interpretation should be displayed, without displaying selectable elements associated with the third interpretation. The GUI data may indicate that only the selectable elements associated with the second interpretation should be displayed, without displaying selectable elements associated with the third interpretation, based on, for example, the metric data associated with the second interpretation meets a metric threshold and the metric data associated with the third interpretation does not meet a metric threshold, the size of the display of the client device indicates that only one selectable element can be displayed, and / or other criteria.

[0015] In various implementations, a user can provide one or more inputs to cause additional selectable elements associated with the additional interpretations and not initially provided for presentation to the user to be displayed. For example, a user can swipe (e.g., swipe up, swipe down, swipe left, and / or swipe right) in a particular direction on the display of the client device or provide other touch input (e.g., long tap, hard tap) to cause additional selectable elements to be provided for presentation to the user. As another example, a user can provide additional verbal utterances including one or more particular words or phrases (e.g., "More," "Show me more," etc.) that, when detected, cause the automated assistant to provide additional selectable elements for presentation to the user.

[0016] In various implementations, when automatically initiating the execution of a given action predicted to be responsive to a verbal utterance by an automated assistant, the automated assistant may consider one or more costs associated with the given action. The one or more costs may include, for example, whether an assistant command associated with the given action utilizes financial information of a user, whether an assistant command associated with the given action causes an electronic communication (e.g., a phone call, a text message, an email message, a social media message, etc.) to be initiated and / or sent from a client device of a user to an additional client device of an additional user, whether an assistant command associated with the given action consumes a threshold amount of computational resources, whether an assistant command associated with the given action delays other processes, whether an assistant command associated with the given action affects the user or another person in some way if the given action is inaccurate, and / or other costs. For example, the costs associated with executing an action that utilizes financial information or causes a client device to initiate and / or send an electronic communication may be considered to be much greater than the costs associated with having media content provided to a user for playback.

[0017] By using the techniques described herein, various technical advantages may be realized. As one non-limiting example, the techniques described herein enable an automated assistant to passively disambiguate assistant commands contained in a verbal utterance provided by a user. For example, a user may provide a verbal utterance including a request, and the automated assistant may automatically initialize execution of an assistant command that is predicted to comply with the request based on a first interpretation of the request while simultaneously providing selectable elements associated with other interpretations of the request that are also predicted to comply with the request. Thus, when a user provides a user selection of one of the selectable elements, the automated assistant may quickly and efficiently pivot to automatically initialize execution of assistant commands associated with other interpretations. Notably, the automated assistant does not need to generate and render a prompt before initializing execution of the assistant command. As a result, latency in fulfilling a request may be reduced, computational resources of a client device may be saved, and the duration of a human-computer interaction may be shortened. Additionally, performance data associated with these other interpretations may be pre-cached on the client device such that latency in switching execution to one of these alternative assistant commands may be reduced when the user provides a user selection of one of the selectable elements.

[0018] The above description is provided as a summary of some implementations of the present disclosure. Further descriptions of those and other implementations are provided in more detail below. [Brief description of the drawings]

[0019] [Figure 1A] FIG. 13 illustrates a scene in which a user invokes an automated assistant that can suggest alternative interpretations in response to the user input while simultaneously executing a particular interpretation that is predicted to be most relevant to the user input. [Figure 1B] FIG. 13 illustrates a scene in which a user invokes an automated assistant that can suggest alternative interpretations in response to the user input while simultaneously executing a particular interpretation that is predicted to be most relevant to the user input. [Figure 1C] FIG. 13 illustrates a scene in which a user invokes an automated assistant that can suggest alternative interpretations in response to the user input while simultaneously executing a particular interpretation that is predicted to be most relevant to the user input. [Figure 1D] FIG. 13 illustrates a scene in which a user invokes an automated assistant that can suggest alternative interpretations in response to the user input while simultaneously executing a particular interpretation that is predicted to be most relevant to the user input. [Figure 2A] FIG. 1 illustrates a scene in which a user invokes an automated assistant that can respond to user input to initiate the execution of a particular interpretation while simultaneously providing selectable suggestions on a display interface of a computing device. [Figure 2B] FIG. 1 illustrates a scene in which a user invokes an automated assistant that can respond to user input to initiate the execution of a particular interpretation while simultaneously providing selectable suggestions on a display interface of a computing device. [Figure 2C] FIG. 1 illustrates a scene in which a user invokes an automated assistant that can respond to user input to initiate the execution of a particular interpretation while simultaneously providing selectable suggestions on a display interface of a computing device. [Diagram 3] FIG. 1 illustrates a system that can invoke an automated assistant to suggest alternative user interpretations in response to user input while simultaneously performing a particular interpretation that is predicted to be most relevant to the user input. [Figure 4A] FIG. 1 illustrates a method for operating an automated assistant to provide alternative interpretations in response to spoken utterances that may be interpreted differently and / or may be missing certain parameters. [Figure 4B] FIG. 1 illustrates a method for operating an automated assistant to provide alternative interpretations in response to spoken utterances that may be interpreted differently and / or may be missing certain parameters. [Diagram 5] FIG. 1 is a block diagram of an exemplary computer system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] 1A, 1B, 1C, and 1D respectively show views 100, 120, 140, and 160 in which a user 102 invokes an automated assistant that can suggest alternative interpretations in response to a user input while simultaneously executing a particular interpretation predicted to be most relevant to the user input. The suggested alternative interpretations may be offered via a display interface 106 and / or other interface of a computing device 104 for selection by the user 102 if the user 102 determines that the executing interpretation is not the one the user intended. For example, as shown in FIG. 1A, the user 102 may provide a verbal utterance 108 to the automated assistant via an audio interface of the computing device 104, such as "Assistant, 'Play Science Rocks,'" which may refer to a piece of music the user 102 would like to hear. In response to receiving the verbal utterance 108, the automated assistant can initiate execution of an assistant command associated with a particular interpretation of the verbal utterance that is predicted to have the greatest degree of correspondence with the verbal utterance 108 provided by the user 102.

[0021] As shown in FIG. 1B , in response to the verbal utterance 108, the automated assistant can cause the computing device 104 to render a graphical element 122 on the display interface 106 of the computing device 104. The graphical element 122 can operate as an interface for controlling one or more actions corresponding to the intent being performed. For example, in response to receiving the verbal utterance 108, the graphical element 122 can be rendered on the display interface 106 to indicate that the song "Science Rocks" is being played via a first music application accessible to the computing device 104. In some implementations, because the user 102 did not specify a particular music application for which the song is rendered or a particular artist of the song "Science Rocks", the automated assistant can infer one or more different slot values ​​of a corresponding parameter associated with the musical intent, such as one or more different music applications that the user may intend the automated assistant to interact with, and one or more different artists that the user may intend the automated assistant to utilize to play the song "Science Rocks". For example, the automated assistant may select slot values ​​of corresponding parameters that are predicted to have the greatest degree of correspondence with the verbal utterance 108, and the automated assistant can automatically initialize playback of the song "Science Rocks" using those slot values, such as the slot value of the application parameter First Music Application and the slot value of the artist parameter Pop Class, as shown by graphical element 122 in FIG. 1B.

[0022] In some implementations, in addition to performing this initial interpretation, the automated assistant can generate one or more other alternative interpretations that can be suggested to the user 102 via one or more selectable elements, such as the first suggestion element 124 and the second suggestion element 126. In some implementations, additional suggested interpretations can be provided to the user 102 when the initial interpretation does not have a predicted degree of correspondence (i.e., a correspondence metric) with the user request and / or spoken utterance 108 that meets a threshold degree of correspondence (i.e., a metric threshold). For example, the first suggestion element 124 can suggest that the requested song be played using a different music application (e.g., a second music application of the application parameters), and the second suggestion element 126 can suggest that the initially predicted music application play a song with the same name (e.g., “Science Rocks”) but by a different artist (e.g., “Music Group” as shown in FIG. 1B). Additionally, the first suggestion element 124 and the second suggestion element 126 may be rendered simultaneously as the automated assistant executes an assistant command associated with the initial interpretation. This may enable the automated assistant to execute an initial interpretation that has the greatest degree of correspondence with the requested command while also providing suggestions in case the automated assistant is incorrect about the initial interpretation. Notably, the first suggestion element 124 and the second suggestion element 126 may be provided for presentation to the user only for a threshold duration.

[0023] In various implementations, as described in more detail with respect to FIG. 3, the interpretation may be generated based on processing the spoken utterance 108 using various components available to the automated assistant in the computing device 104. For example, an automatic speech recognition (ASR) component may be utilized to process audio data capturing the spoken utterance 108 to generate an ASR output. The ASR output may include, for example, a speech hypothesis predicted to correspond to the spoken utterance 108, ASR metrics associated with the speech hypothesis, phonemes predicted to correspond to the spoken utterance 108, and / or other ASR output. Additionally, a natural language understanding (NLU) component may be utilized to process the ASR output to generate an NLU output. The NLU output may include, for example, one or more intents predicted to satisfy the spoken utterance 108 (e.g., a musical intent in the example of FIGS. 1A-1D ), corresponding parameters associated with each of the one or more intents, corresponding slot values ​​of the corresponding parameters, NLU metrics associated with the one or more intents, corresponding parameters, and / or corresponding slot values, and / or other NLU output. Additionally, a performance component may be utilized to process the NLU output to generate performance data. The performance data may correspond to various actions that, when executed, cause the automated assistant to execute corresponding assistant commands in an attempt to fulfill the spoken utterance, and may optionally be associated with performance metrics that indicate how likely the execution of a given assistant command is predicted to satisfy the spoken utterance. In particular, each of the interpretations described herein may include various combinations of intents, corresponding parameters, and corresponding slot values, and the automated assistant may generate assistant input data that characterizes these various combinations of intents, corresponding parameters, and corresponding slot values.

[0024] In some implementations, the automated assistant can automatically initialize the execution of an action associated with the initial interpretation based on correspondence metrics for each of the multiple interpretations. In some versions of these implementations, correspondence metrics can be generated based on ASR metrics, NLU metrics, and performance metrics associated with each of the multiple interpretations. In additional or alternative implementations, correspondence metrics can be generated based on user profile data of the user who provided the verbal utterance 108 (e.g., user preferences, a history of the user's interactions with various applications accessible at the computing device 104, a user's search history, a user's purchase history, a user's calendar information, and / or any other information about the user of the computing device 104), user profile data of other users similar to the user who provided the verbal utterance 108, an aggregate increase in verbal utterances including requests over a particular geographic region, and / or other data.

[0025] In some versions of those implementations, the metric data may be generated using one or more machine learning models and / or heuristic processes that process these various signals to generate the metric data. For example, one or more machine learning models may be trained to process these signals to determine, for each of the multiple interpretations, a correspondence metric indicative of a predicted degree of correspondence between each of the multiple interpretations and the verbal utterance 108. In some of those implementations, the one or more machine learning models may be trained on-device based on data generated locally on the computing device 104 such that the one or more machine learning models are personalized to the user 102 of the computing device 104. The one or more machine learning models may be trained based on multiple training cases, each of which may include a training case input and a training case output. The training case input may include any combination of examples of interpretations of these signals and / or verbal utterances, and the training case output may include a ground truth output indicative of a ground truth interpretation of the training case input. The training case inputs of a given training case may be applied as inputs across one or more machine learning models to generate predicted outputs including correspondence metrics for each of the example interpretations, and the predicted outputs may be compared to the ground truth outputs to generate one or more losses. Additionally, the one or more losses may be utilized to update the one or more machine learning models (e.g., by backpropagation). The one or more machine learning models may be deployed after sufficient training (e.g., based on processing a threshold amount of training cases, based on training for a threshold duration, based on the performance of the one or more machine learning models during training, etc.). As another example, for each of the multiple interpretations, one or more heuristic-based processes or rules may be utilized to process these signals and / or the multiple interpretations to determine a correspondence metric indicative of a predicted degree of correspondence between each of the multiple interpretations and the verbal utterance 108.Based on the assistant input data and the metrics data, the automated assistant can automatically initialize a first action associated with a first interpretation of the plurality of interpretations to fulfill an assistant command included in the verbal utterance and provide selectable elements associated with other interpretations of the plurality of interpretations for presentation to the user.

[0026] For example, an initial interpretation may be selected from among the multiple interpretations as the one most likely to satisfy the verbal utterance based on the initial interpretation having the greatest predicted degree of correspondence to satisfy the verbal utterance, and may be automatically initialized by the automated assistant in response to the verbal utterance. For example, as shown in FIG. 1B, the automated assistant may use a first music application to cause the first music application to render an audio output corresponding to the song "Science Rocks" by artist "Pop Class". Nevertheless, one or more other interpretations having the next greatest degree of correspondence with the verbal utterance may be the basis for the first selectable element 124 (e.g., the song "Science Rocks" by artist "Pop Class", but using the second music application rather than the first music application) and the second selectable element 126 (e.g., the song "Science Rocks" by artist "Music Class", but using the first music application rather than the first music application). In this manner, in some implementations, the user 102 may select one or more selectable suggestion elements simultaneously with or after performing an action associated with the initial interpretation that is predicted to have the highest degree of correspondence with the verbal utterance.

[0027] In some implementations, the selectable suggestions may be selected using assistant input, such as another spoken utterance and / or other input gestures to the computing device and / or any other device associated with the automated assistant. For example, as shown in FIG. 1C, the user 102 may provide another spoken utterance 142, such as "Play on the second music application." In some implementations, the automated assistant may provide an indication 128 that the audio interface of the computing device 104 remains initialized to receive input directed to one or more of the selectable suggestions (e.g., as shown in FIG. 1B). As shown in FIGS. 1C and 1D, in response to receiving the other spoken utterance 142, the automated assistant may determine that the user 102 has provided a request for the automated assistant to stop performing an action associated with the automatically initialized initial interpretation, and may initialize the performance of an alternative action associated with an alternative interpretation corresponding to the first selectable element 124. In response to receiving the other verbal utterance 142, the automated assistant can automatically initiate execution of an alternative action associated with the alternative interpretation corresponding to the first selectable element 124. Alternatively, in some implementations, the automated assistant can cause the graphical user interface 106 of the computing device 104 to render a graphical element 162 to indicate that the alternative action associated with the alternative interpretation has been performed.

[0028] In some implementations, the automated assistant can cause the automated assistant and / or other applications to bias one or more ASR and / or NLU components toward content associated with the first selectable element 124 and the second selectable element 126, such as words or phrases of one or more slot values ​​corresponding to one or more of the selectable elements. For example, in the example of FIG. 1C, the automated assistant can bias one or more processing of the spoken utterance 142 toward words such as “second application” and / or “Music Group.” In this manner, when the automated assistant leaves the audio interface initialized (as indicated by the graphical element 128), the user 202 may select to provide another spoken utterance such as “second application.” In response, processing of the audio data capturing the spoken utterance 142 can be biased toward any words associated with the first selectable element 124 and the second selectable element 126.

[0029] In some implementations, the prominence and / or area of ​​each selectable suggestion and / or graphical element relative to other selectable suggestions and / or graphical elements may be based on a predicted degree of correspondence of its respective interpretation with respect to a request from the user 102. For example, as shown in FIG. 1B , a first area of ​​the display interface 106 associated with a first selectable element 124 and a second area of ​​the display interface 106 associated with a second selectable element 126 may have the same area on the display interface 106 when their respective degrees of correspondence with the verbal utterance 108 are the same. However, when the predicted degree of correspondence associated with an alternative interpretation of the first selectable element 124 is greater than the alternative interpretation of the second selectable element 126, the first area of ​​the display interface 106 associated with the first selectable element 124 may be larger than the second area of ​​the display interface 106 associated with the second selectable element 126. In some implementations, selectable suggestions corresponding to some or all of the alternative interpretations may be omitted from the display interface 106 of the computing device 104 based on the metric.

[0030] 2A, 2B, and 2C respectively show views 200, 220, and 240 in which a user 202 invokes an automated assistant that can respond to a user input and provide selectable suggestions at a display interface 206 of a computing device 204 while initializing execution of a particular interpretation of the user input. The selectable suggestions can be generated based on the natural language content of the user input, which has multiple different interpretations. For example, as shown in FIG. 2A, the user 202 can provide a spoken utterance 208 such as "Assistant, translate to 20 percent." As an example throughout FIGS. 2A-2C, assume that the user 202 provides the spoken utterance 208 to cause the brightness level of one or more lights in the user's 202 home to be adjusted to 20%. However, because the spoken utterance 208 can have multiple different interpretations, the automated assistant can process the spoken utterance 208 to determine whether other suggestions should be rendered for presentation to the user 202.

[0031] For example, in response to receiving the verbal utterance 208, the automated assistant may cause the display interface 206 of the computing device 204 to render a first selectable element 222 and a second selectable element 224, as shown in FIG. 2B. The first selectable element 222 may correspond to a first interpretation predicted by the automated assistant or other application to be most likely to satisfy the request embodied in the verbal utterance 208. The second selectable element 224 may correspond to a second interpretation predicted to be less likely to satisfy the request than the first interpretation associated with the first selectable element 222. In some implementations, the automated assistant may cause the execution of an assistant command associated with the first interpretation in response to the verbal utterance while simultaneously rendering a second selectable element 224 that allows the user 202 to initiate execution of an alternative assistant command associated with the second interpretation.

[0032] Alternatively or additionally, the automated assistant may automatically execute an assistant command associated with the first interpretation in response to receiving the verbal utterance 208. The automatic execution of the assistant command and / or the alternative assistant command may be executed when one or more "costs" of executing the assistant command and / or the alternative assistant command meet a threshold. For example, a value of the amount of processing and / or time predicted to be consumed to execute the proposed user intent may be estimated. When the value meets a certain threshold, the automated assistant may execute the assistant command and / or the alternative assistant command in response to the verbal utterance 208. For example, as shown in FIG. 2B, the automated assistant may automatically execute an assistant command associated with the first selectable element 222, thereby causing an IoT home application to adjust the brightness of the lights in the user's home (e.g., "Kitchen Lights") from 50% to 20%. Additionally, the automated assistant may also cause a translation application to translate a portion of the verbal utterance 208 (e.g., "To 20%"). In this case, both the assistant command and the alternative assistant command may be executed because both may be executed without consuming a lot of processing and / or time, but if one of the assistant commands requires a threshold amount of processing and / or time or would otherwise adversely affect the user or additional users, then neither the assistant command nor the alternative assistant command may be executed.

[0033] In some implementations, the automated assistant may cause the display interface 206 of the computing device 204 to render a particular selectable element in response to a spoken utterance 208, but the user 202 may wish to have additional alternative interpretations for selection. For example, as shown in FIG. 2C, the user 202 may provide an input gesture 250 to the computing device 204 to cause the display interface 206 to render a third selectable element 242 and a fourth selectable element 246. The prominence of each selectable element may depend on the expected relevance of each element to the user input. For example, the area of ​​the third selectable element 242 and the area of ​​the fourth selectable element 246 may be smaller than the areas of the first selectable element 222 and the second selectable element 224, respectively. This may be based in part on a predicted degree of correspondence of the third interpretation (controllable via the third selectable element 242) and the fourth interpretation (controllable via the fourth selectable element 246) of the verbal utterance 208. In other words, the display area of ​​the third selectable element 242 and the fourth selectable element 246 may be smaller than the first selectable element 222 because the first interpretation associated with the first selectable element 222 and the second interpretation associated with the second selectable element 224 are predicted to be more likely to satisfy the requirements embodied in the verbal utterance 208 than the third and fourth interpretations.

[0034] In some implementations, the input gesture 250 may cause the automated assistant to select one or more other slot values ​​for the first interpretation and / or the second interpretation. These selected slot values ​​may be the basis for the third selectable element 242 and the fourth selectable element 246. For example, the first interpretation may include a particular slot value that identifies "Kitchen Lights" for the lighting location parameters of the first interpretation, and the third interpretation may include another slot value that identifies "Basement Lights" for the lighting location parameters of the third interpretation. Additionally, the fourth interpretation may include a different slot value that identifies "Hallway Lights" for the lighting location parameters of the fourth interpretation.

[0035] FIG. 3 illustrates a system 300 that can invoke an automated assistant 304 to suggest alternative interpretations in response to a user input while simultaneously performing a particular interpretation predicted to be most relevant to the user input. The automated assistant 304 can operate as part of an assistant application provided on one or more computing devices, such as a computing device 302 (e.g., computing device 104 of FIGS. 1A-1D, computing device 204 of FIGS. 2A-2C, and / or other computing devices such as a server device). A user can interact with the automated assistant 304 via an assistant interface 320, which can be a microphone, a camera, a touch screen display, a user interface, and / or any other device that can provide an interface between a user and an automated assistant. For example, a user can initialize the automated assistant 304 by providing verbal, textual, and / or graphical input to the assistant interface 320 to cause the automated assistant 304 to initialize one or more actions (e.g., to provide data, control a peripheral device, access an agent, generate input and / or output, etc.). Alternatively, the automated assistant 304 may be initialized based on processing of the context data 336 using one or more trained machine learning models. The context data 336 may characterize one or more features of an environment in which the automated assistant 304 is accessible and / or one or more features of a user predicted to be interacting with the automated assistant 304. The computing device 302 may include a display device, which may be a display panel including a touch interface for receiving touch inputs and / or gestures to enable a user to control applications 334 of the computing device 302 via the touch interface.In some implementations, the computing device 302 may not have a display device, thereby providing an audible user interface output without providing a graphical user interface output. Additionally, the computing device 302 may provide a user interface such as a microphone for receiving verbal natural language input from a user. In some implementations, the computing device 302 may include a touch interface or may not have a camera or other visual component, but may optionally include one or more other sensors.

[0036] The computing device 302 and / or other third party client devices can optionally communicate with a server device over a network, such as the Internet, to implement the system 300. Additionally, the computing device 302 and any other computing devices can communicate with each other over a local area network (LAN), such as a Wi-Fi network. In some implementations, the computing device 302 can offload computational tasks to a server device to conserve the computing resources of the computing device 302. For example, the server device can host the automated assistant 304 and / or the computing device 302 can send inputs received at one or more assistant interfaces 320 to the server device. However, in some additional or alternative implementations, the automated assistant 304 can be hosted locally on the computing device 302, and various processes that may be associated with the operation of the automated assistant can be executed on the computing device 302.

[0037] In various implementations, all or some aspects of the automated assistant 304 may be implemented on the computing device 302. In some of those implementations, aspects of the automated assistant 304 may be implemented by the computing device 302 and interface with a server device that may implement other aspects of the automated assistant 304. The server device may optionally serve multiple users and their associated assistant applications via multiple threads. In implementations in which all or some aspects of the automated assistant 304 are implemented by the computing device 302, the automated assistant 304 may be an application separate from (e.g., installed "on top of") the operating system of the computing device 302--or alternatively, may be implemented directly by (e.g., thought of as an operating system application but integral to the operating system) the operating system of the computing device 302.

[0038] In some implementations, the automated assistant 304 can include an input processing engine 306, which can employ multiple different modules to process input and / or output for the computing device 302 and / or the server device. For example, the input processing engine 306 can include a speech processing engine 308 that can process audio data capturing the verbal utterance received at the assistant interface 320 to identify text embodied in the verbal utterance. The audio data can be transmitted from the computing device 302 to a server device, for example, to conserve the computing resources of the computing device 302, and the server device can transmit the text embodied in the verbal utterance back to the computing device 302. Additionally or alternatively, the audio data can be processed exclusively at the computing device 302.

[0039] The process for converting audio data to text may include speech recognition algorithms that may employ neural networks and / or statistical models to identify groups of audio data that correspond to words or phrases (e.g., using an ASR model). The text converted from the audio data may be analyzed by the data analysis engine 310 and made available to the automated assistant 304 as text data that may be used to generate and / or identify command phrases, intents, actions, slot values, and / or any other content specified by the user (e.g., using an NLU model). In some implementations, the output data provided by the data analysis engine 310 may be provided to the parameter engine 312 to determine (e.g., using execution models and / or rules) whether the user has provided input that corresponds to a particular intent, action, and / or routine that may be executed by the automated assistant 304 and / or an application or agent that may be accessed by the automated assistant 304. For example, assistant data 338 may be stored on the server device and / or computing device 302 and may include data defining one or more actions that may be executed by the automated assistant 304 and parameters required to execute the action. The parameter engine 312 can generate one or more parameters for the intent, action, and / or slot value and provide the one or more parameters to the output generation engine 314. The output generation engine 314 can use the one or more parameters to communicate with the assistant interface 320 to provide output to the user and / or to communicate with one or more applications 334 to provide output to the one or more applications 334.Output to the user may include, for example, visual output that may be visually rendered for presentation to the user via a display interface of the computing device 302 or a display of an additional computing device in communication with the computing device 302, audible output that may be audibly rendered for presentation to the user via a speaker of the computing device 302 or a speaker of an additional computing device in communication with the computing device 302, smart device control commands to control one or more network-connected smart devices in communication with the computing device 304, and / or other output.

[0040] The automated assistant application may include and / or access on-device ASR, on-device NLU, and on-device execution. For example, the on-device ASR may be performed using an on-device ASR module that processes audio data (detected by a microphone) using an ASR model stored locally on the computing device 302. The on-device ASR module generates an ASR output based on processing of the audio data, such as one or more speech hypotheses corresponding to recognized text of the verbal utterance (if any) present in the audio data end-to-end, or predicted phonemes predicted to correspond to the verbal utterance, and the speech hypotheses corresponding to the recognized text may be generated based on the predicted phonemes. Also, for example, the on-device NLU module may process the speech hypotheses corresponding to the recognized text generated by the ASR module using the NLU model to generate NLU data. The NLU data may include a predicted intent corresponding to the verbal utterance and, optionally, slot values ​​of parameters associated with the intent. Also, for example, the on-device performance module can process the NLU data using the performance model and / or rules, and optionally other local data, to determine assistant commands to execute to fulfill the predicted intent of the spoken utterance. These assistant commands can include, for example, obtaining local and / or remote responses (e.g., answers) to the spoken utterance, interactions with locally installed applications to execute based on the spoken utterance, commands to send to Internet of Things (IoT) devices (directly or via corresponding remote systems) based on the spoken utterance, and / or other resolved actions to execute based on the spoken utterance. The on-device performance can then initiate local and / or remote performance / execution of the determined actions to fulfill the spoken utterance.

[0041] In various implementations, remote ASR, remote NLR, and / or remote execution may be utilized at least selectively. For example, speech hypotheses corresponding to recognized text may be at least selectively sent to a remote automated assistant component for remote NLU and / or remote execution. For example, speech hypotheses corresponding to recognized text may be optionally sent for remote execution in parallel with on-device execution or in response to failure of on-device NLU and / or on-device execution. However, on-device ASR, on-device NLU, on-device execution, and / or on-device execution may be preferred at least because of the reduced latency they offer when fulfilling verbal utterances (due to no client-server round trip being required to fulfill the verbal utterance). Additionally, on-device functionality may be the only functionality available in situations where there is no or limited network connectivity.

[0042] In some implementations, the computing device 302 may include one or more applications 334 that may be provided by a first party entity that is the same as the entity that provided the computing device 302 and / or the automated assistant 304 and / or by a third party entity that is different from the entity that provided the computing device 302 and / or the automated assistant 304. The application state engine of the automated assistant 304 and / or the computing device 302 may access application data 330 to determine one or more actions that may be performed by the one or more applications 334, as well as the state of each application of the one or more applications 334 and / or the state of each device associated with the computing device 302. The device state engine of the automated assistant 304 and / or the computing device 302 may access device data 332 to determine one or more actions that may be performed by the computing device 302 and / or one or more devices associated with the computing device 302 (e.g., one or more network-connected smart devices in communication with the computing device 302). Additionally, the application data 330 and / or any other data (e.g., device data 332) can be accessed by the automated assistant 304 to generate context data 336, which can characterize the context in which a particular application 334 and / or device is running and / or the context in which a particular user is accessing the computing device 302, the application 334, and / or any other device or module.

[0043] While one or more applications 334 are executing on the computing device 302, the device data 332 can characterize a current operational state of each application 334 executing on the computing device 302 and / or executing remotely of the computing device 302 (e.g., one or more streaming applications). Additionally, the application data 330 can characterize one or more features of the running applications 334, such as the content of one or more graphical user interfaces being rendered at the direction of the one or more applications 334. Alternatively or additionally, the application data 330 can characterize action schemas, which may be updated by the respective applications and / or by the automated assistant 304 based on the respective applications' current operational state. Alternatively or additionally, the one or more action schemas for the one or more applications 334 may remain static but be accessed by the application state engine to determine a suitable action to initialize via the automated assistant 304.

[0044] The computing device 302 may further include an assistant invocation engine 322 that may use one or more trained machine learning models to process the application data 330, the device data 332, the context data 336, and / or any other data accessible to the computing device 302. The assistant invocation engine 322 may process the data to determine whether to wait for the user to explicitly speak an invocation phrase to invoke the automated assistant 304 (and determine whether the user has spoken an invocation phrase to invoke the automated assistant 304) or--instead of requesting the user to explicitly speak an invocation phrase--to consider the data as indicating the user's intent to invoke the automated assistant. For example, the one or more trained machine learning models may be trained using training data examples based on scenarios in which a user is in an environment in which multiple devices and / or applications exhibit various operating conditions. The training data examples may be generated to capture audio data including at least the invocation phrase and / or training data characterizing contexts in which the user invokes the automated assistant and other contexts in which the user does not invoke the automated assistant. When the one or more trained machine learning models are trained with these examples of training data, the assistant invocation engine 322 can cause the automated assistant 304 to detect or limit detection of spoken invocation phrases from the user based on context and / or environmental features. Additionally or alternatively, the assistant invocation engine 322 can cause the automated assistant 304 to detect or limit detection of one or more assistant commands from the user based on context and / or environmental features without the user having to provide any invocation phrases.

[0045] In some implementations, the system 300 may include a suggestion generation engine 316 that can assist in processing input from a user to determine whether to execute a particular interpretation of a verbal utterance and / or provide one or more suggestions regarding alternative interpretations of the verbal utterance. In response to receiving an assistant input, the suggestion generation engine 316 can identify a number of different interpretations that have a degree of correspondence with the request embodied in the assistant input, such as different combinations of predicted intent and / or slot values ​​of parameters associated with the predicted intent. The degree of correspondence of a particular interpretation may be characterized by a metric that may have a value that may be compared to one or more different thresholds. For example, a particular interpretation of a verbal utterance may be identified in response to a user input, and a metric of the particular interpretation may be determined to meet a target threshold. Based on this determination, the automated assistant 304 may automatically initiate execution of an action associated with the particular interpretation and omit rendering of suggestion elements corresponding to other interpretations associated with lower value metric.

[0046] In some implementations, when a particular interpretation has the highest metric value for a particular user input, but the metric value does not meet a target threshold, the automated assistant 304 may determine to render one or more additional suggestion elements. For example, the automated assistant may still automatically initiate execution of an action associated with a particular interpretation, but may also render selectable elements corresponding to other alternative interpretations with the highest metric value. These selectable elements may be rendered on a display interface of the computing device 302 without any additional user input requesting the selectable elements. In some implementations, the prominence of one or more of these selectable elements may be based on respective metric values ​​associated with each alternative interpretation of the selectable elements. In some implementations, the metric values ​​associated with each selectable suggestion element may be compared to a suggestion threshold to determine whether to render a corresponding selectable element of the selectable elements. For example, in some implementations, alternative interpretations associated with metric values ​​that do not meet a suggestion threshold may still be rendered in response to a subsequent input from the user (e.g., a swipe gesture to reveal additional suggestions).

[0047] In some implementations, the system 300 may include a suggestion feature engine 318 that may generate assistant GUI data for rendering one or more GUI elements in response to receiving input from a user. The assistant GUI data may characterize a size of the particular selectable element based on a corresponding metric value associated with the particular selectable element, a placement of the particular selectable element on the display based on a corresponding metric value associated with the particular selectable element (e.g., a horizontal and / or vertical displacement of the particular selectable element is characterized by the corresponding placement data of the particular selectable element), display data of the particular selectable element on the display characterizing a corresponding display characteristic of the particular selectable element (e.g., a bold characteristic of the particular selectable element, a fill characteristic of the particular selectable element against a background, and / or any other display characteristic associated with visually rendering the particular selectable element), an amount of the particular selectable element displayed based on a size of the display interface of the computing device 302 and / or based on an amount of alternative explanatory interpretations having corresponding metric values ​​that meet a suggestion threshold, and / or other GUI data. Among other things, the GUI data may also characterize control elements for actions associated with the initial interpretation predicted to satisfy the verbal utterance. The control elements may be based on the type of action and may include, for example, media control elements when the action includes playing media content, slider elements for adjusting parameters of an IoT device when the action includes controlling an IoT device (e.g., temperature of a thermostat, brightness of a smart light, etc.), and / or other control elements based on the type of action.

[0048] In some implementations, the system 300 may include a fulfillment engine 326 that may determine whether to initiate execution and / or partial fulfillment of one or more user intents identified by the automated assistant 304 in response to a user input. For example, the fulfillment engine 326 may cause an action associated with a particular interpretation of the verbal utterance to be executed in response to the user input when a metric associated with the particular interpretation meets a target threshold. Alternatively or additionally, when an identified interpretation of the verbal utterance does not meet a target threshold, a particular interpretation associated with a highest metric value (e.g., a greatest degree of correspondence with the user input) may be executed by the fulfillment engine 326. In some implementations, one or more alternative interpretations having a next highest metric value (e.g., a next greatest degree of correspondence with the user input) may be at least partially fulfilled and / or executed by the fulfillment engine 326. For example, data may be retrieved by the automated assistant 304 and / or another application to facilitate at least partial fulfillment of an alternative interpretation having a next highest metric value.

[0049] In some implementations, the system 300 may include a training data engine 324 that may generate training cases for initially training one or more machine learning models described herein (e.g., utilized in generating correspondence metrics as described in connection with FIG. 1) and generate training cases for updating one or more machine learning models based on how a user interacts with the automated assistant 304. The initial training of one or more machine learning models based on a plurality of training cases is described above in connection with FIG. 1, and the plurality of training cases may be generated using the training data engine 324. Furthermore, selection or non-selection of any selectable elements associated with alternative interpretations provided for presentation to a user as described herein may be utilized to generate additional training cases for updating one or more machine learning models. For example, assume that a user does not select any of the selectable elements associated with alternative interpretations. In this example, signals and / or interpretations associated with a verbal utterance may be utilized as inputs for the training cases, and a ground truth output indicating the initially selected interpretation may be utilized to provide positive reinforcement for the selection of the initial interpretation. In contrast, assume that a user selects one of the selectable elements associated with an alternative interpretation. In this example, signals and / or interpretations associated with the verbal utterances can be used as inputs for the training examples, and ground truth outputs indicative of alternative interpretations can be used to provide positive reinforcement for the selection of the alternative interpretations (while providing negative reinforcement for the selection of the initial interpretation). Thus, the one or more machine learning models can be updated over time to enable the automated assistant 304 to improve the selection of the initial interpretation. For example, if the initial interpretation selected is associated with a first application for playing media, but the user generally selects an alternative interpretation associated with a second application for playing media, the one or more machine learning models can be updated over time to reflect that the user prefers the second application for playing media.

[0050] 4A and 4B illustrate methods 400 and 420 for operating an automated assistant to provide alternative interpretations in response to a spoken utterance that may be interpreted differently and / or may be missing certain parameters. Methods 400 and 420 may be performed using one or more applications, computing devices, and / or any other devices or modules that may interface with the automated assistant. Method 400 may include an operation 402 of determining whether an assistant input is detected by the computing device. The assistant input may be a spoken utterance that includes a request from a user to have the automated assistant execute an assistant command. In response to receiving the spoken utterance, the automated assistant may perform one or more operations (e.g., an ASR operation, an NLU operation, a perform operation, and / or other operation) to generate multiple interpretations of the spoken utterance. For example, a spoken utterance such as "Assistant, play Congolese." Although the user may intend the verbal utterance to instruct the automated assistant to play Congo music, the automated assistant, in response to receiving the verbal utterance, may automatically initialize a particular action associated with the first interpretation and further identify other actions to suggest to the user based on alternative interpretations.

[0051] Method 400 may proceed from operation 402 to operation 404 when assistant input is detected. If not, the automated assistant may continue to detect user input at operation 402. Operation 404 may include generating input data capable of identifying different interpretations. For example, in response to the aforementioned verbal utterance, the automated system may generate assistant input data characterizing different interpretations of the verbal utterance, such as playing Congolese music via a first application (e.g., playMusic("Congolese", first application)), translating the speech into a language spoken in Congo (e.g., translateSpeech("play", English, Swahili)), and / or playing Congolese music via a second application (e.g., playMusic("Congolese", second application)).

[0052] Method 400 may proceed from operation 404 to operation 406, which may include generating metrics data characterizing a predicted degree of correspondence between a request from the user contained in the verbal utterance and a plurality of different interpretations of the verbal utterance. In other words, for each interpretation, a metric may be generated to characterize the degree of correspondence between the respective interpretation and the verbal utterance. For example, to continue the previous example, an interpretation for playing music in a first application may have a greatest degree of correspondence with the user request as compared to the other interpretations. An interpretation for playing music in a second application may have a next greatest degree of correspondence, and another interpretation for translating speech may have a particular degree of correspondence that is less than the other two interpretations for playing music.

[0053] Method 400 may proceed from operation 406 to operation 408, which may include determining whether a metric of a particular interpretation meets a metric threshold. The metric threshold may be a threshold at which the automated assistant determines that the corresponding interpretation is predicted to meet the spoken utterance to such an extent (e.g., 75%, 60%, or any other value) that additional suggested interpretations may not be acknowledged by the automated assistant. Thus, when a metric corresponding to a particular interpretation meets the metric threshold, the automated assistant may initiate execution of an action associated with the interpretation without rendering other suggested interpretations. However, when the metric of any identified interpretation does not meet the metric threshold, the automated assistant may initiate execution of an action associated with the interpretation corresponding to the metric that is closest to meeting the metric threshold, and simultaneously suggest other interpretations to the user.

[0054] When the metric of a particular interpretation meets the metric threshold, the method 400 may proceed from operation 408 to operation 410. Otherwise, when the metric of any particular interpretation does not meet the metric threshold, the method 400 may proceed from operation 408 to operation 412 described below. Operation 410 may include causing the computing device to initialize execution of a particular interpretation corresponding to the metric that meets the metric threshold. For example, when the metric corresponding to the interpretation of playing music by a first application meets the metric threshold, the automated assistant may cause the computing device to initialize execution of an action associated with that particular interpretation. In such a case, the automated assistant may be objectively confident that execution of an action associated with a particular interpretation satisfies a request from the user, but the automated assistant may optionally provide the user with an action associated with an alternative interpretation.

[0055] For example, method 400 may proceed from operation 410 to operation 422 provided in method 420 of FIG. 4B via continuation element "A". Operation 422 may include determining whether an input has been received and is directed to identifying an alternative interpretation. For example, a user may perform a swipe gesture on a display interface of a computing device to cause the automated assistant to provide additional suggestions of other actions associated with other interpretations that may be performed in response to the verbal utterance. These other interpretations may have been identified, for example, in operation 404, but did not have corresponding metrics that met the metric threshold. For example, when the computing device is playing music via a first application, a user may provide an input gesture or other assistant input to the computing device (e.g., swipe a display interface of the computing device) to reveal one or more selectable elements. Each selectable element of the one or more selectable elements may correspond to an action associated with an alternative interpretation of the verbal utterance. For example, when a user swipes the display interface while a first application is playing music, a first selectable element may be rendered that corresponds to the action of playing music in a second application, as shown by the alternative interpretation.

[0056] When it is determined at operation 422 that user input is received, the method 420 may proceed from operation 422 via continuation element "B" to operation 416, described below. When none of the metric for any of the interpretations meets the metric threshold at operation 408, the method 400 may proceed from operation 408 to operation 412. Operation 412 may include identifying a particular interpretation having a greatest degree of correspondence with a request from the user contained in the verbal utterance. For example, a metric corresponding to an interpretation for playing music in a first application may not meet the metric threshold, but the metric may nevertheless have a greatest degree of correspondence with a request from the user contained in the verbal utterance. In other words, execution of the interpretation may be predicted to be most likely to meet the request, at least as compared to other interpretations identified based on the verbal utterance.

[0057] From operation 412, method 400 may proceed to operation 414, which may include having the computing device automatically initiate execution of an operation associated with the particular interpretation. Following the previous example, this operation associated with the particular interpretation may cause Congo music to be played by the first application. Thus, the computing device may render the Congo music through the first application and an audio interface of the computing device. From operation 414, method 400 may proceed to operation 416, which may include identifying one or more other interpretations that may satisfy the requirements contained in the verbal utterance. For example, operation 416 may include identifying an alternative interpretation associated with the user data generated in operation 404.

[0058] Alternatively or additionally, the automated assistant may identify one or more other alternative interpretations that may not have been identified by the input data generated in operation 404. For example, based on the action associated with the particular interpretation performed in operation 414 and / or contextual data associated with a request from the user included in the verbal utterance, the automated assistant may identify other content and / or actions associated with other interpretations to suggest to the user. For example, the automated assistant may identify actions having interpretations that may be similar to the currently performed user intent but may include other parameters or other slot values ​​of the parameters or other parameters. For example, when an action associated with a particular interpretation of playing music is being performed, other actions associated with suggested alternative interpretations may include slot values ​​based on similarity to speech characteristics (e.g., detected phonemes) of the verbal utterance during ongoing processing. For example, the automated assistant may determine that the word "Portuguese" has a certain degree of correspondence and / or similarity to the verbal word "Congolese" spoken by the user via the verbal utterance. Based on this determination, the automated assistant can generate suggestion data characterizing selectable suggestions for playing Portuguese music via the first application as alternative interpretations.

[0059] Method 400 may proceed from operation 416 to operation 418, which may include causing the computing device to render one or more selectable elements corresponding to the one or more other interpretations. The one or more selectable elements may be rendered simultaneously with a particular selectable element corresponding to a particular interpretation being performed by the automated assistant. For example, an audio control interface for controlling audio playback of the content by the first application may be rendered simultaneously with the one or more selectable elements identified in operation 416.

[0060] From operation 418, the method 400 may proceed to operation 424 via continuation element "C", as shown in method 420 of FIG. 4B. Operation 424 may include determining whether a user selection to select another selectable element is received. When it is determined that a user input to select another selectable element of the one or more selectable elements is received, the method 420 may proceed to operation 426. Operation 426 may include causing the computing device to initiate execution of an alternative action associated with a particular alternative interpretation corresponding to the selected selectable element. Alternatively or additionally, in response to the selection of the selectable element, the automated assistant may cause the alternative action to be executed and cancel and / or pause ongoing execution of the originally selected action associated with the particular interpretation (e.g., playing Congo music by the first application).

[0061] In some implementations, the method 420 may optionally proceed from operation 426 to optional operation 428. Alternatively, when no input is received from the selection of another selectable element, the method 420 may proceed from operation 424 to optional operation 428. Operation 428 may include training (or updating) one or more machine learning models in response to the selection or non-selection of one or more selectable elements. For example, when another selectable element corresponds to another interpretation that differs from the user's intent being performed, the trained machine learning model may then be trained (or updated) such that subsequent interpretations are biased toward the selected interpretation in response to receiving a subsequent instance of the request. As a result, in response to receiving a subsequent instance of the request, an alternative interpretation (e.g., play Congo music in a second application) may be initialized.

[0062] 5 is a block diagram 500 of an exemplary computer system 510. Generally, the computer system 510 includes at least one processor 514 that communicates with several peripheral devices via a bus subsystem 512. These peripheral devices may include, for example, a storage subsystem 524 including a memory 525 and a file storage subsystem 526, a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input and output devices enable user interaction with the computer system 510. The network interface subsystem 516 provides an interface to an external network and is coupled to corresponding interface devices of other computer systems.

[0063] The user interface input devices 522 may include a keyboard, a pointing device such as a mouse, a trackball, a touchpad, or a graphics tablet, a scanner, a touch screen integrated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computer system 510 or a communications network.

[0064] The user interface output devices 520 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or some other mechanism for generating a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computer system 510 to a user or to another machine or computer system.

[0065] Storage subsystem 524 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 524 may include logic for performing selected aspects of method 400 and method 420 and / or for implementing one or more of system 300, computing device 104, computing device 204, an automated assistant, and / or any other application, device, apparatus, and / or module discussed herein.

[0066] These software modules are generally executed by the processor 514 alone or in combination with other processors. The memory 525 used in the storage subsystem 524 may include several memories including a main random access memory (RAM) 530 for storing instructions and data during execution of the program, and a read-only memory (ROM) 532 in which certain instructions are stored. The file storage subsystem 526 may provide persistent storage for program and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules implementing the functionality of a particular implementation may be stored by the file storage subsystem 526 in the storage subsystem 524 or other machines that can be accessed by the processor 514.

[0067] The bus subsystem 512 provides a mechanism for allowing the various components and subsystems of the computer system 510 to communicate with each other as intended. Although the bus subsystem 512 is shown generally as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0068] The computer system 510 can be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, a client device (e.g., computing device 104 of FIGS. 1A-1D, computing device 302 of FIG. 3, and / or other client devices), or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of the computer system 510 shown in FIG. 5 is intended only as a specific example intended to illustrate some implementations. Many other configurations of the computer system 510 having more or fewer components than the computer system shown in FIG. 5 are possible.

[0069] In situations where the systems described herein may collect or utilize personal information about users (or as is often referred to herein as "participants"), the user may be given the opportunity to control whether a program or feature collects user information (e.g., information about the user's social network, social actions or activities, occupation, user preferences, or the user's current geographic location) or whether and / or how content that may be more relevant to the user should be received from a content server. Also, certain data may be processed in one or more ways before being stored or used, such that personally identifiable information is removed. For example, the user's identity may be processed such that personally identifiable information cannot be determined about the user, or if geographic location information is obtained, the user's geographic location may be generalized (such as to the level of a city, zip code, or state) so that the user's specific geographic location cannot be determined. Thus, the user may be able to control how information is collected and / or used about the user.

[0070] Although several implementations have been described and illustrated herein, various other means and / or structures for performing the functions and / or obtaining one or more of the results and / or advantages described herein may be utilized, and each such change and / or modification is deemed to be within the scope of the implementations described herein. More broadly, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on the particular application or applications in which the teachings are used. Those skilled in the art will recognize or be able to ascertain using no more than routine experimentation many equivalents to the specific implementations described herein. Thus, the implementations described above are presented merely as examples, and it will be understood that within the scope of the appended claims and their equivalents, implementations may be practiced otherwise than as specifically described and claimed. Implementations of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. Furthermore, any combination of two or more such features, systems, articles, materials, kits, and / or methods is within the scope of the disclosure, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent.

[0071] In some implementations, a method implemented by one or more processors is provided, comprising the steps of: receiving, at a computing device, a verbal utterance of a user directed to an automated assistant, the verbal utterance including an assistant command to be performed by the automated assistant; generating, based on the verbal utterance, assistant input data characterizing a plurality of interpretations predicted to be responsive to the verbal utterance, each of the interpretations including a corresponding intent, one or more corresponding parameters associated with the corresponding intent, and one or more corresponding slot values ​​for each of the one or more corresponding parameters, each of the interpretations including at least one unique corresponding slot value; and generating, based on the assistant input data, assistant input data characterizing a plurality of interpretations predicted to be responsive to the verbal utterance. The method includes generating metrics data characterizing a predicted degree of correspondence between each of the interpretations and an assistant command included in the verbal utterance, causing the automated assistant to automatically initiate execution of a first action associated with a first interpretation of the plurality of interpretations to fulfill the assistant command included in the verbal utterance based on the metrics data and the assistant input data, and rendering one or more selectable suggestion elements on a display interface of the computing device based on the metrics data and the assistant input data, each of the one or more selectable suggestion elements being associated with a corresponding alternative interpretation of the plurality of interpretations to fulfill the assistant command included in the verbal utterance. A user selection of a given selectable suggestion element of the one or more selectable suggestion elements causes the automated assistant to initiate execution of a corresponding alternative action associated with the given selectable suggestion element.

[0072] These and other implementations of the technology disclosed herein may optionally include one or more of the following features.

[0073] In some implementations, causing the automated assistant to automatically initiate execution of a first operation to fulfill an assistant command contained in the verbal utterance may cause the first application to generate an instance of particular content.

[0074] In some implementations, the method may further include biasing an automatic speech recognition (ASR) process or a natural language understanding (NLU) process toward content relevant to the one or more selectable suggestion elements in response to causing the automated assistant to automatically initiate execution of a first operation to fulfill an assistant command contained in the verbal utterance.

[0075] In some implementations, the method may further include having the automated assistant access application data to facilitate preparation of performing corresponding alternative actions associated with the one or more selectable suggestion elements based on the metric data and the assistant input data.

[0076] In some implementations, the method may further include determining, based on the verbal utterance, that one or more of the corresponding slot values ​​of one or more of the corresponding parameters associated with the plurality of interpretations were not specified by the user by the verbal utterance. The automated assistant may infer a specific slot value of one or more of the corresponding parameters associated with the first interpretation. In some versions of those implementations, the method may further include inferring an alternative specific slot value of each of the corresponding alternative interpretations based on the verbal utterance. A user selection of a given selectable suggestion element may cause an alternative operation to be initialized using the alternative specific slot value. In some further versions of those implementations, the specific slot value may identify a first application for rendering the specific content, and the alternative specific slot value may identify a different second application for rendering the alternative specific content. In some further additional or alternative versions of those implementations, the specific slot value may identify a reference of a first entity for rendering the specific content, and the alternative specific slot value may identify a reference of a different second entity for rendering the alternative specific content.

[0077] In some implementations, causing the one or more selectable suggestion elements to be rendered on a display interface of the computing device may include causing the automated assistant to automatically initiate execution of a first operation to fulfill an assistant command included in the verbal utterance, followed by causing the one or more selectable suggestion elements to be rendered on a display interface of the computing device for a threshold duration.

[0078] In some implementations, a method implemented by one or more processors is provided, comprising: receiving, at a computing device, a verbal utterance of a user directed to an automated assistant, the verbal utterance including an assistant command to be performed by the automated assistant; generating, based on the verbal utterance, metrics data identifying a first metric characterizing a degree to which a first operation is predicted to satisfy the assistant command and a second metric characterizing another degree to which a second operation is predicted to satisfy the assistant command; generating, based on the first operation and the second operation, GUI data characterizing an assistant graphical user interface (GUI) responsive to the verbal utterance; in response to receiving the verbal utterance, causing the automated assistant to automatically initialize execution of the first operation; and causing a display interface of the computing device to render the assistant GUI according to the GUI data and the metrics data. The GUI data is generated to identify a first selectable element and a second selectable element, the first selectable element being selectable to control execution of the first operation and the second selectable element being selectable to automatically initialize execution of the second operation.

[0079] These and other implementations of the technology disclosed herein may optionally include one or more of the following features.

[0080] In some implementations, when a first action is predicted to satisfy an assistant command to a greater extent than a second action is predicted to satisfy an assistant command to a different degree, a first selectable element may be positioned more prominently in the assistant GUI than a second selectable element.

[0081] In some implementations, causing the display interface to render the assistant GUI according to the GUI data and the metric data may include positioning the first selectable element adjacent to the second selectable element. When the first action is predicted to satisfy the assistant command to a greater extent than another degree the second action is predicted to satisfy the assistant command, in the assistant GUI, a first area of ​​the first selectable element may be larger than a second area of ​​the second selectable element.

[0082] In some implementations, causing the display interface to render the assistant GUI according to the GUI data and the metric data may include positioning a first selectable element adjacent to a second selectable element, where in the assistant GUI the first selectable element may be positioned according to first positioning data specific to the first selectable element and characterizing a corresponding first position of the first selectable element in the assistant GUI, and the second selectable element may be positioned according to second positioning data specific to the second selectable element and characterizing a corresponding second position of the second selectable element in the assistant GUI.

[0083] In some implementations, in the assistant GUI, a first selectable element may be displayed based on corresponding first display data that is specific to the first selectable element and characterizes a corresponding first display characteristic of the first selectable element in the assistant GUI, and a second selectable element may be displayed based on corresponding second display data that is specific to the second selectable element and characterizes a corresponding second display characteristic of the second selectable element in the assistant GUI.

[0084] In some implementations, the first operation may be a particular operation performed on a computing device and the second operation may be a particular operation performed on another computing device, and further, a user selection of the second selectable element may cause a particular operation to be initiated on the other computing device.

[0085] In some implementations, the second selectable element may be selectable by touch input at a display interface of the computing device when the first action is being performed.

[0086] In some implementations, the second selectable element may be selectable by an additional spoken utterance simultaneously with the first action being performed.

[0087] In some implementations, in the assistant GUI, a prominence of the rendering of a first selectable element relative to a second selectable element may be based on a difference between a first metric and a second metric. In some versions of those implementations, when the difference between the first metric and the second metric does not meet a suggestion threshold, the second selectable element may not be rendered in the display interface, and a particular touch gesture applied by a user to the display interface of the computing device may cause the second selectable element to be rendered in the display interface of the computing device.

[0088] In some implementations, a method implemented by one or more processors is provided that includes receiving, at a computing device, a verbal utterance of a user directed to an automated assistant, the verbal utterance including an assistant command to be performed by the automated assistant; identifying a first action that may be initiated by the automated assistant to fulfill the assistant command in response to receiving the verbal utterance; automatically initiating performance of the first action in response to the verbal utterance; identifying at least a second action that may be initiated by the automated assistant to fulfill the assistant command based on a degree to which the first action is predicted to be responsive to the verbal utterance; and causing a display interface of the computing device to render at least a second selectable element associated with the second action based on the verbal utterance, where when the second selectable element is selected, causing the automated assistant to initialize the second action instead of the first action.

[0089] These and other implementations of the technology disclosed herein may optionally include one or more of the following features.

[0090] In some implementations, identifying at least a second behavior that may be initiated by the automated assistant to fulfill the assistant command may include generating a metric characterizing the degree to which the first behavior is predicted to be responsive to the verbal utterance and determining whether the metric meets a metric threshold. The automated assistant may identify at least a second behavior to suggest to the user when the metric does not meet the metric threshold.

[0091] In some implementations, identifying at least a second action that may be initiated by the automated assistant to fulfill the assistant command may include determining whether the first action is a particular type of action. The automated assistant may determine to identify at least a second action when the first action is a particular type of action, and the identified second action may be a different type of action than the particular type of action of the first action. In some versions of those implementations, the particular type of action may include a communication action that includes communicating with a different user.

[0092] In some implementations, the method may further include terminating execution of the first action in response to receiving a user selection of the second selectable element.

[0093] In some implementations, identifying at least a second action that may be initiated by the automated assistant to fulfill the assistant command may include determining an amount of additional actions to identify for suggestion to a user via a display interface of the computing device. In some versions of those implementations, determining an amount of additional actions to identify for suggestion to a user via a display interface of the computing device may be based on a size of the display interface of the computing device. In some additional or alternative versions of those implementations, determining an amount of additional actions to identify for suggestion to a user via a display interface of the computing device may be based on corresponding metrics that characterize a corresponding degree to which each of the additional actions is predicted to be responsive to a verbal utterance.

[0094] Other implementations may include a non-transitory computer-readable storage medium that stores instructions executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) to perform methods such as one or more of the methods described above and / or elsewhere herein. Still other implementations may include one or more computer systems including one or more processors operable to execute the stored instructions to perform methods such as one or more of the methods described above and / or elsewhere herein.

[0095] It should be understood that all combinations of the above concepts, and additional concepts described in more detail herein, are contemplated as part of the subject matter disclosed herein, for example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as part of the subject matter disclosed herein. [Explanation of symbols]

[0096] 100 Views 102 users 104 Computing Devices 106 Display Interface 108 Oral Speech 120 Views 122 Graphical Elements 124 First proposed element, first selectable element 126 Second proposed element, second selectable element 128 Indications 140 Sights 142 Alternative oral utterances 160 Views 162 Graphical Elements 200 Views 202 users 204 Computing Devices 206 Display Interface 220 Views 222 First selectable element 224 Second selectable element 240 Sights 242 Third optional element 246 Fourth optional element 250 Input Gestures 300 Systems 302 Computing Devices 304 Automated Assistant 306 Input Processing Engine 308 Speech Processing Engine 310 Data Analysis Engine 312 Parameter Module 314 Output Generation Engine 316 Proposal Generation Engine 318 Suggested Feature Engine 320 Assistant Interface 322 Assistant Call Engine 324 Training Data Engine 326 Execution Engine 330 Application Data 332 Device Data 334 Applications 336 Context Data 338 Assistant Data 400 ways 420 method 510 Computer Systems 512 Bus Subsystem 514 processor 516 Network Interface Subsystem 520 User Interface Output Device 522 User Interface Input Devices 524 Storage Subsystem 525 Memory 526 File Storage Subsystem 530 Main Random Access Memory (RAM) 532 Read-Only Memory (ROM)

Claims

1. 1. A method implemented by one or more processors, comprising: receiving, at a computing device, a user's verbal utterance directed to an automated assistant, the verbal utterance including an assistant command to be performed by the automated assistant; Generating assistant input data based on the verbal utterance, the assistant input data characterizing a plurality of interpretations predicted to be responsive to the verbal utterance, each of the interpretations including a corresponding intent, one or more corresponding parameters associated with the corresponding intent, and one or more corresponding slot values ​​for each of the one or more corresponding parameters, each of the interpretations including at least one unique corresponding slot value; generating metric data characterizing a predicted degree of correspondence between each of the plurality of interpretations and the assistant command included in the verbal utterance based on the assistant input data; causing the automated assistant to automatically initiate execution of a first action associated with a first interpretation of the plurality of interpretations to fulfill the assistant command included in the verbal utterance based on the metric data and the assistant input data; causing a display interface of the computing device to render one or more selectable suggestion elements based on the metric data and the assistant input data, each of the one or more selectable suggestion elements being associated with a corresponding alternative interpretation of the plurality of interpretations for fulfilling the assistant command included in the verbal utterance; and a step in which a user selection of a given selectable suggestion element among the one or more selectable suggestion elements causes the automated assistant to initiate execution of a corresponding alternative action associated with the given selectable suggestion element.

2. 2. The method of claim 1, wherein the step of causing the automated assistant to automatically initiate execution of the first action to accomplish the assistant command contained in the verbal utterance causes a first application to generate an instance of a particular content.

3. in response to causing the automated assistant to automatically initiate performance of the first action to accomplish the assistant command included in the verbal utterance; 3. The method of claim 1 or claim 2, further comprising biasing an automatic speech recognition (ASR) process or a natural language understanding (NLU) process towards content relevant to the one or more selectable suggestion elements.

4. 4. The method of claim 1, further comprising: having the automated assistant access application data to facilitate preparation of performing the corresponding alternative action associated with the one or more selectable suggestion elements based on the metric data and the assistant input data.

5. determining, based on the verbal utterance, that one or more of the corresponding slot values ​​of one or more of the corresponding parameters associated with the plurality of interpretations were not specified by the user via the verbal utterance; 5. The method of claim 1, further comprising the step of the automated assistant inferring one or more specific slot values ​​of the corresponding parameters associated with the first interpretation.

6. inferring alternative specific slot values ​​for each of the corresponding alternative interpretations based on the verbal utterance, The method of claim 5 , further comprising: the user's selection of the given selectable suggestion element causing the alternative action to be initialized using the alternative particular slot value.

7. 7. The method of claim 6, wherein the specific slot value identifies a first application for rendering a specific content, and the alternative specific slot value identifies a different second application for rendering the alternative specific content.

8. 7. The method of claim 6, wherein the specific slot value identifies a reference to a first entity for rendering specific content, and the alternative specific slot value identifies a reference to a different second entity for rendering the alternative specific content.

9. causing the one or more selectable suggestion elements to be rendered on the display interface of the computing device, 9. The method of claim 1, further comprising causing the automated assistant to automatically initiate execution of the first operation to fulfill the assistant command included in the verbal utterance, and then causing the one or more selectable suggestion elements to be rendered on the display interface of the computing device for a threshold duration.

10. 1. A method implemented by one or more processors, comprising: receiving, at a computing device, a user's verbal utterance directed to an automated assistant, the verbal utterance including an assistant command to be performed by the automated assistant; generating, based on the verbal utterance, metric data identifying a first metric characterizing a degree to which a first action is predicted to satisfy the assistant command and a second metric characterizing a different degree to which a second action is predicted to satisfy the assistant command; Generating GUI data characterizing an assistant graphical user interface (GUI) responsive to the verbal utterance based on the first action and the second action, the GUI data is generated to identify a first selectable element and a second selectable element; the first selectable element being selectable to control execution of the first operation and the second selectable element being selectable to automatically initiate execution of the second operation; in response to receiving the verbal utterance, causing the automated assistant to automatically initiate execution of the first action; and causing a display interface of the computing device to render the assistant GUI according to the GUI data and the metric data.

11. 11. The method of claim 10, wherein the first selectable element is positioned more prominently in the assistant GUI than the second selectable element when the degree to which the first action is predicted to satisfy the assistant command is greater than the other degree to which the second action is predicted to satisfy the assistant command.

12. causing the display interface to render the assistant GUI according to the GUI data and the metric data; disposing the first selectable element adjacent to the second selectable element; 12. The method of claim 10 or 11, comprising arranging in the assistant GUI such that a first area of ​​the first selectable element is larger than a second area of ​​the second selectable element when the degree to which the first action is predicted to satisfy the assistant command is greater than the other degree to which the second action is predicted to satisfy the assistant command.

13. causing the display interface to render the assistant GUI according to the GUI data and the metric data; disposing the first selectable element adjacent to the second selectable element; 13. The method of claim 10, further comprising: arranging, in the assistant GUI, the first selectable element according to first placement data that is specific to the first selectable element and characterizes a corresponding first position of the first selectable element in the assistant GUI, and the second selectable element according to second placement data that is specific to the second selectable element and characterizes a corresponding second position of the second selectable element in the assistant GUI.

14. 14. A method according to any one of claims 10 to 13, wherein in the assistant GUI, the first selectable element is displayed based on corresponding first display data that is specific to the first selectable element and characterizes a corresponding first display characteristic of the first selectable element in the assistant GUI, and the second selectable element is displayed based on corresponding second display data that is specific to the second selectable element and characterizes a corresponding second display characteristic of the second selectable element in the assistant GUI.

15. the first operation is a specific operation executed on the computing device, and the second operation is the specific operation executed on another computing device; 15. The method of claim 10, wherein a user selection of the second selectable element causes the particular action to be initiated on the other computing device.

16. 16. The method of claim 10, wherein the second selectable element is selectable by touch input at the display interface of the computing device when the first action is being performed.

17. 17. The method of any one of claims 10 to 16, wherein the second selectable element is selectable by an additional spoken utterance simultaneously with the first action being performed.

18. 18. The method of claim 10, wherein in the assistant GUI, a prominence of a rendering of the first selectable element relative to the second selectable element is based on a difference between the first metric and the second metric.

19. when the difference between the first metric and the second metric does not meet a suggestion threshold, the second selectable element is not rendered in the display interface; and 20. The method of claim 18, wherein a particular touch gesture applied by the user to the display interface of the computing device causes the second selectable element to be rendered on the display interface of the computing device.

20. 1. A method implemented by one or more processors, comprising: receiving, at a computing device, a user's verbal utterance directed to an automated assistant, the verbal utterance including an assistant command to be performed by the automated assistant; identifying a first action that can be initiated by the automated assistant to fulfill the assistant command in response to receiving the verbal utterance; automatically initiating execution of the first action in response to the verbal utterance; identifying at least a second action that may be initiated by the automated assistant to fulfill the assistant command based on the degree to which the first action is predicted to be responsive to the verbal utterance; and causing a display interface of the computing device to render at least a second selectable element associated with the second action based on the verbal utterance, the second selectable element, when selected, causing the automated assistant to initialize the second action instead of the first action.

21. Identifying at least the second action that can be initiated by the automated assistant to fulfill the assistant command, generating a metric characterizing the degree to which the first action is predicted to be responsive to the verbal utterance; determining whether the metric satisfies a metric threshold; 21. The method of claim 20, further comprising: the automated assistant identifying, determining, at least the second action to suggest to the user when the metric does not meet the metric threshold.

22. Identifying at least the second action that can be initiated by the automated assistant to fulfill the assistant command, determining whether the first motion is a particular type of motion, determining, when the first action is the particular type of action, to identify at least the second action; 22. The method of claim 20 or claim 21, comprising determining that the identified second behavior is a different type of behavior than the identified type of behavior of the first behavior.

23. 23. The method of claim 22, wherein the particular type of activity comprises a communication activity comprising communicating with a different user.

24. in response to receiving a user selection of the second selectable element; 24. The method of any one of claims 20 to 23, further comprising terminating execution of the first operation.

25. Identifying at least the second action that can be initiated by the automated assistant to fulfill the assistant command, 25. The method of any one of claims 20 to 24, comprising determining an amount of additional actions to identify for suggesting to the user via the display interface of the computing device.

26. 26. The method of claim 25, wherein determining the amount of additional actions to identify for suggesting to the user via the display interface of the computing device is based on a size of the display interface of the computing device.

27. 26. The method of claim 25, wherein determining the amount of additional actions to identify for suggesting to the user via the display interface of the computing device is based on a corresponding metric characterizing a corresponding degree to which each of the additional actions is predicted to be responsive to the verbal utterance.

28. At least one processor; and a memory storing instructions that, when executed, cause the at least one processor to perform operations corresponding to any one of claims 1 to 27.

29. 28. A non-transitory computer-readable storage medium storing instructions that, when executed, cause at least one processor to perform operations corresponding to any one of claims 1 to 27.

Images