Providing a secondary automated assistant with relevant queries based on past interactions

Abstract

A system and method for providing audio data from an initially invoked automated assistant to a subsequently invoked automated assistant. The initially invoked automated assistant may be invoked by a user utterance followed by audio data including a query. The query is provided to a secondary automated assistant for processing. The user may then submit a query related to the first query. In response, the initially invoked automated assistant provides the query to the secondary automated assistant instead of providing the query to another secondary automated assistant based on a similarity between the first query and the subsequent query.

Description

[Technical field]

[0001] This application relates to providing a secondary automated assistant with relevant queries based on past interactions. [Background technology]

[0002] Humans may participate in human-computer dialogs with interactive software applications, referred to herein as "automated assistants" (also referred to as "digital agents," "chatbots," "interactive personal assistants," "intelligent personal assistants," "assistant applications," "conversational agents," etc.). For example, a human (who may be referred to as a "user" when interacting with an automated assistant) may provide commands and / or requests to the automated assistant using verbal natural language input (i.e., speech), which may in some cases be converted to text and then processed, and / or by providing textual (e.g., typed) natural language input. The automated assistant responds to the request by providing a responsive user interface output, which may include audible and / or visual user interface output.

[0003] As described above, many automated assistants are configured to be interacted with via verbal utterances, such as an invocation instruction followed by a verbal query. To protect user privacy and / or to conserve resources, a user must often explicitly invoke an automated assistant before the automated assistant can fully process the verbal utterance. Explicit invocation of an automated assistant typically occurs in response to a particular user interface input received at a client device. The client device includes an assistant interface that provides a user of the client device with an interface for interfacing with the automated assistant (e.g., receiving verbal and / or typed input from the user and providing an audible and / or graphical response) and interfaces with one or more additional components that implement the automated assistant (e.g., a remote server device that processes the user input and generates an appropriate response).

[0004] Some user interface inputs that can invoke an automated assistant via a client device include hardware and / or virtual buttons on the client device (e.g., tapping a hardware button, selecting a graphical interface element displayed by the client device) to invoke the automated assistant. Many automated assistants can additionally or alternatively be invoked in response to one or more verbal invocation phrases, also known as "hot words / phrases" or "trigger words / phrases." For example, verbal invocation phrases such as "Hey Assistant," "OK Assistant," and / or "Assistant" can be spoken to invoke the automated assistant.

[0005] Often, a client device including an assistant interface includes one or more locally stored models that the client device utilizes to monitor the occurrence of a verbal invocation phrase. Such a client device may utilize the locally stored models to locally process received audio data and may discard any audio data that does not include the verbal invocation phrase. However, if the local processing of the received audio data indicates the occurrence of the verbal invocation phrase, then the client device causes the audio data and / or subsequent audio data to be further processed by the automated assistant. For example, if the verbal invocation phrase is "Hey Assistant" and the user speaks "Hey, Assistant, what time is it," audio data corresponding to "what time is it" may be processed by the automated assistant based on the detection of "Hey Assistant" and utilized to provide an automated assistant response for the current time. On the other hand, if a user simply speaks "what time is it" (without first speaking the invocation phrase or providing an alternative invocation input), no response from the automated assistant is provided as a result of the absence of an invocation phrase (or other invocation input) before "what time is it." Summary of the Invention [Means for solving the problem]

[0006] Described herein are techniques for providing dialog data from a primary automated assistant to one or more secondary assistants based on a user's previous interactions with one or more secondary automated assistants. For example, various techniques are directed to a primary automated assistant receiving a dialog between a user and a primary assistant, providing a first query that is part of the dialog to one or more secondary assistants, and then providing a query related to the first query to the same secondary assistant to which the first query was provided for processing instead of providing the query to another secondary assistant. A user can first invoke a primary automated assistant and issue one or more queries that the invoked primary automated assistant can provide to the secondary assistant. The secondary assistant can process the queries and provide a response to the user via its own interface or via the primary automated assistant. At some point in the dialog between the user and the primary automated assistant, a query related to the first processed query may be received. The primary automated assistant can identify that a second received query is related to the first query and provide the second query to the same secondary automated assistant instead of to another secondary assistant, thereby facilitating continuity in communication between the user and the secondary automated assistant and preventing related queries from being processed by different secondary automated assistants.

[0007] In some implementations, a user may issue an invocation phrase such as "OK Assistant" that invokes an automated assistant (also referred to herein as the "first invoked automated assistant") but does not explicitly invoke other automated assistants with which the first invoked automated assistant may at least selectively interact in processing a query received in connection with (e.g., immediately after, immediately before) the invocation. Thus, a user may specify to utilize the first invoked automated assistant rather than individually invoking one of the other automated assistants based on providing an invocation input specific to the first invoked automated assistant. For example, a first invocation phrase (e.g., "OK Assistant A"), when detected, may exclusively invoke the first automated assistant without invoking any other automated assistants. Similarly, a second invocation phrase (e.g., "OK Assistant B"), when detected, may exclusively invoke the second automated assistant without invoking the first invoked automated assistant and / or any other automated assistants. Once invoked, the initially invoked assistant may at least selectively interact with other automated assistants (i.e., "secondary assistants") in processing input provided in connection with the call.

[0008] A user may utter an invocation phrase, such as "OK Assistant," and / or otherwise perform one or more actions, that invokes a primary automated assistant but does not explicitly invoke other automated assistants with which the primary automated assistant may at least selectively interact in processing queries received in association with (e.g., immediately following, immediately preceding) the invocation phrase. Thus, a user may specify to utilize a primary automated assistant, rather than individually invoking only one of the other automated assistants, based on providing an invocation input specific to the primary automated assistant. For example, a first invocation phrase (e.g., "OK Assistant A"), when detected, may exclusively invoke the first automated assistant without invoking the primary automated assistant and / or any other automated assistants. Similarly, a second invocation phrase (e.g., "OK Assistant B"), when detected, may exclusively invoke the second automated assistant without invoking the primary automated assistant and / or any other automated assistants. Other invocation phrases (e.g., "OK Assistant"), when detected, may invoke the primary assistant. When invoked, the primary assistant can at least selectively interact with a first automated assistant and / or a second automated assistant (i.e., a "secondary assistant") in processing input provided in connection with the invocation. In some implementations, the primary assistant can constantly interact with one or both of the first and second automated assistants and can be a "meta-assistant" or "general automated assistant" that itself lacks one or more automated assistant capabilities, such as speech recognition, natural language understanding, and / or fulfillment capabilities. In other examples, the primary automated assistant can also interact with the secondary assistant while performing its own query processing to determine a response to the query.

[0009] The primary automated assistant can provide a query to only one of the secondary assistants based on the user's historical interactions with the primary and / or secondary assistants. For example, the primary automated assistant can receive a query and provide the query to the first automated assistant. The user can be provided with a response from the first automated assistant directly from the first automated assistant or via the primary automated assistant. The primary automated assistant can then receive a query related to the first query. The primary automated assistant can determine that the new query is related to the first query and provide the second query only to the first automated assistant and not to the second automated assistant. Thus, when an intervening query is processed by the first automated assistant, the user can be provided with a consistent response to the related query instead of the related query being processed by a different secondary automated assistant.

[0010] Thus, utilization of the techniques described herein mitigates the need for a user to explicitly invoke each of multiple automated assistants based on the information the user is interested in being provided by a particular automated assistant. For example, implementations described herein mitigate the need for a user to explicitly invoke a first automated assistant to handle a particular type of query and a second automated assistant to handle other types of queries. Furthermore, by identifying portions of a dialog previously fulfilled by a first automated assistant and providing that automated assistant with additional related queries to process, consistent results may be provided without the need to invoke multiple automated assistants to process related queries. By utilizing a primary assistant to perform an initial analysis of a received query, pre-processing of the query (e.g., natural language understanding, text-to-speech processing, automatic speech recognition) may be required only once for a given query, rather than the query being processed by each secondary assistant individually. Furthermore, the techniques described herein mitigate the need for a user to extend the overall duration of the interaction via follow-up queries in order to receive a response from an automated assistant that the user is interested in processing the query. Thus, these implementations attempt to determine which automated assistant processes related queries even when the related queries are not consecutive in the dialog, providing objective improvement but may otherwise provide inconsistent or incomplete results and require additional processing.

[0011] A primary automated assistant may be invoked by an invocation phrase that invokes the primary automated assistant and not other automated assistants in the same environment. For example, the primary automated assistant may be invoked with the phrase "OK Assistant" and one or more secondary assistants may be invoked with one or more other invocation phrases (e.g., "OK Assistant A", "Hey Assistant B"). Once invoked, the primary assistant processes the query accompanying the invocation (e.g., the query preceding and / or following the invocation phrase). In some implementations, the primary assistant may identify multiple queries accompanying the invocation. For example, the primary assistant may continue to receive audio data for a period of time after the invocation phrase and process any additional utterances of the user during that period. Thus, a user may not need to invoke a primary automated assistant for each query, but instead may utter multiple queries, each of which may be followed by a response from the primary automated assistant and / or one or more other secondary assistants.

[0012] In some implementations, the received verbal utterance may be processed to determine the intent, the terms in the query, and / or a classification of the query. For example, the primary automated assistant may perform speech recognition, natural language processing, and / or one or more other techniques to convert the audio data into data that can be further processed to determine the user's intent. In some implementations, one or more secondary assistants may further process the incoming verbal utterance even if the processing secondary assistant has not been invoked by the user. For example, a user may invoke the primary automated assistant by uttering the phrase "OK Assistant" followed by a query. The primary automated assistant, along with the user and / or one or more other secondary assistants in the same environment as the primary automated assistant, may activate a microphone and capture the utterance. In some implementations, only the primary automated assistant may be invoked and provide an indication of the query (e.g., audio data, a textual representation of the audio, the user's intent) to one or more of the secondary automated assistants.

[0013] In some implementations, the primary automated assistant can continue to run on the client device along with one or more secondary assistants. For example, the device may be running the primary automated assistant as well as one or more other automated assistants that are invoked using a different invocation phrase than the primary automated assistant. In some implementations, the one or more secondary automated assistants may continue to run on a device different from the device running the primary automated assistant, and the primary automated assistant can communicate with the secondary automated device through one or more communication channels. For example, the primary automated assistant may provide a textual representation of the query via Wi-Fi, an ultrasonic signal perceptible by the secondary automated assistant via a speaker, and / or one or more other communication channels.

[0014] In some implementations, the primary automated assistant can categorize the query based on the intent of the query. The categorization can indicate to the primary automated assistant what type of response satisfies the query. For example, for a query of "Set an alarm," the primary automated assistant can perform natural language understanding on the query to determine the intent of the query (i.e., the user wants to set an alarm). The categorization of the query can include, for example, a request to answer a question, a request to run an application, and / or an intent to perform some other task that one or more of the automated assistants can perform.

[0015] When a query is received by a primary automated assistant, the primary automated assistant can determine which secondary automated assistant can best fulfill the requested intent. For example, the primary automated assistant can process the query, determine that the intent of the query is to set an alarm, and provide the intent to one or more secondary automated assistants in communication with the primary automated assistant. Also, for example, the primary automated assistant may be further configured to process some or all of the incoming queries. For example, the primary automated assistant may receive the query, process the query, determine whether it is capable of fulfilling the query, and perform one or more tasks in response to the query.

[0016] In some implementations, one or more of the secondary automated assistants may inform the primary automated assistant of tasks that they can perform. For example, the secondary assistant may provide a signal to the primary automated assistant indicating that it can access a music application, set an alarm, and respond to a question with an answer. Thus, the primary automated assistant may verify which of the set of secondary automated assistants can perform the task before providing a query and / or an indication of the intent of the query to the secondary automated assistant.

[0017] When a primary automated assistant receives a query and processes the query to determine the intent of the query, the primary automated assistant may access one or more databases to determine whether the user prefers a particular secondary automated assistant for that classification of query. For example, a user may utilize Assistant A to set an alarm in a previous interaction with an automated assistant, but may prefer Assistant B when the user is interested in receiving an answer to a question. The user's preference may be determined based on an explicit instruction by the user (e.g., setting a preference for Assistant A to set alarms) and / or based on previous usage (e.g., the user invokes Assistant B more frequently than Assistant A when asking a question).

[0018] The primary automated assistant can then provide the query to the secondary automated assistant for further processing. For example, a user may utter a query, and the primary automated assistant may determine the intent from the query and then provide the intent of the query to the secondary automated assistant, which can process (and has been indicated to the primary automated assistant that it can). The secondary automated assistant may then perform additional processing of the query and perform one or more tasks, as necessary. The tasks may include, for example, determining an answer to a question, setting an alarm, playing music via a music application, and utilizing information in the query to execute one or more other applications. In some implementations, the secondary automated assistant can provide a response (or other fulfillment) to the primary automated assistant for the completion of the task. For example, a user may utter a query including a question, and the query may be provided to the secondary automated assistant. The secondary automated assistant may process the query, generate a response, and provide the response to the primary automated assistant, which may provide an audible response to the question to the user.

[0019] In some cases, the user may utter a second query related to the previous query. For example, the user may utter the query "How tall is the president," which may be fulfilled by assistant A. Following the query, immediately after the answer is provided, or as a subsequent query with other unrelated queries in between, the user may utter "How old is he," in relation to the president mentioned in the previous query. The primary automated assistant and / or one or more of the secondary assistants may determine the answer to the follow-up query by identifying the intent of the first query (e.g., an interest in the president's height being provided) and determining that the second query is an additional question about the president mentioned in the first query. The dialogue between the primary automated assistant and the user may continue with additional queries and responses that may be related or unrelated to each other.

[0020] When a user utters a query related to a previous query, the user may be interested in having the subsequent request fulfilled by the same secondary assistant that fulfilled the first request. In some cases, however, the subsequent query and the previous query are part of a dialog that includes an intervening query that is unrelated to the previous and subsequent queries. For example, a user may invoke a primary automated assistant and utter "How tall is the president," followed by "Set an alarm for 9am" (not related to the first query), followed by "How old is he," which is related to "How tall is the president." The user may be interested in both the first and third queries being fulfilled by the same secondary automated assistant, and the intervening query may be handled by a different automated assistant (depending on the intent of the query) or may be handled in some other way apart from the related query.

[0021] The primary automated assistant can receive an utterance from a user that includes a query and identify a secondary assistant that can process the query. For example, for the query "How tall is the president," the primary automated assistant can determine that the first automated assistant is configured to process "question / answer" type queries and provide the query to the first secondary assistant. The query can be provided to the first automated assistant via one or more network protocols (e.g., Wi-Fi, Bluetooth, LAN) and / or one or more other channels. For example, the primary automated assistant can generate an ultrasonic signal that is inaudible to humans that can be captured by a microphone on the device running the first automated assistant. The signal can be, for example, a processed version of the audio data that includes the query (e.g., raw audio data processed to a higher frequency and / or a textual representation of the audio data that is transmitted through a speaker but inaudible to humans). Also, for example, the primary automated assistant and the secondary automated assistant receiving the query can continue to run on the same device and provide the query directly to the secondary assistant. In some implementations, the query may be provided to the secondary automated assistant as audio data, as a textual representation of the query and / or as a textual representation of some of the information contained within the query, as the intent of the query, and / or as other automatic speech recognition output.

[0022] The selected secondary automated assistant can provide a response, such as an audible response, to the user, and the primary automated assistant can store the user's interaction with the secondary automated assistant. For example, the primary automated assistant may store queries and responses from the secondary automated assistant, the type of response provided by the secondary automated assistant (e.g., question and answer), and an indication of the secondary automated assistant that provided the response.

[0023] Following the initial query, the primary automated assistant may receive additional queries from the user that are not related to the first query. These queries may be provided to the same secondary assistant to which the first query was provided, and / or to one or more other secondary automated assistants (or the primary automated assistant, if configured to do so). Following the intervening query, the user may speak a query related to the first query (e.g., one or more similar keywords in a query that includes the same type, similar intent, reference to a previous query), and may determine, based on stored user interactions with the secondary automated assistant, that the user previously interacted with a secondary automated assistant that responded to a similar query. Based on determining that the user previously interacted with a particular secondary automated assistant, the primary automated assistant may provide subsequent queries to the same secondary automated assistant.

[0024] In some implementations, providing the related query to the secondary automated assistant may be based on the time since the user interacted with the secondary automated assistant. For example, the primary automated assistant may store the time of interaction along with an indication of the dialog between the user and the secondary automated assistant. When the user then utters a query related to the previous query, the primary automated assistant may determine the amount of time that has elapsed since the user's last interaction with the secondary automated assistant. If the primary automated assistant determines that the last interaction was within a threshold time before the current query, the primary automated assistant may provide the new query to the same secondary automated assistant for further processing. Conversely, if the user utters a query related to the previous query, but determines that the time that has elapsed since the historical interaction between the user and the secondary automated assistant exceeds the threshold time, the primary automated assistant may determine that the same automated assistant should not be invoked to process the query, even if it is related to the previous query. However, regardless of whether the threshold time has elapsed, the primary automated assistant may still provide the query to the secondary automated assistant if it determines that the secondary automated assistant can process the new query.

[0025] In some implementations, a user may determine that a particular type of query should be processed by a different automated assistant based on the user's preference for one of the secondary assistants, regardless of whether a previous related query was processed by a different automated assistant. For example, in some cases, the primary assistant may identify a secondary automated assistant that the user previously utilized for a particular type of query and / or a particular context (e.g., the user's location, time of day, subject matter of the query). In those cases, the primary automated assistant may provide the query to the preferred secondary automated assistant, even though the primary automated assistant could provide the query to a different automated assistant based on other factors.

[0026] As an example, a primary automated assistant may identify that both secondary automated "Assistant A" and "Assistant B" can fulfill the query "Play music." In a past interaction, the user may have indicated that "Assistant A" should play music using a music application to which the user has access. A previous query of "What is the current number 1 song" may have been provided to "Assistant B." Thus, the query may be provided to either "Assistant B," which handled the related query, or "Assistant A," which the user designated as the preferred automated assistant for playing music. The primary automated assistant may provide the query to "Assistant A" using the context of previous responses from "Assistant B" based on the user's preferences, even though ongoing dialogue with "Assistant B" may avoid switching between automated assistants.

[0027] In some cases, a secondary automated assistant may not be able to provide a response to a query. For example, a query may be provided to a secondary automated assistant, and the secondary automated assistant does not respond and / or provides an indication that a response is unavailable. In those cases, a different secondary automated assistant may be provided with the query. For example, candidate secondary automated assistants may be ranked, and the highest ranked secondary assistant may be provided with the query first. If the response is not successful, the next highest ranked secondary assistant may be provided with the query.

[0028] In some implementations, the ranking of candidate automated assistants can be based on output from a machine learning model. For example, queries provided to one or more secondary assistants can be embedded in an embedding space, and the candidate secondary assistants can be ranked based on the similarity between the embedding of the previous query and the current query. Other metrics that can be utilized to rank the automated assistants can include, for example, the text and / or environmental context associated with the provided query, the time, the location, the application running on the client device when the query was provided, and / or other actions of the user and / or other actions of the computing device running the candidate secondary assistant both when the query was initially provided and in its current state.

[0029] Once the secondary automated assistant provides a response, the query and indication of the response may be stored by the primary automated assistant for further processing of queries related to those queries. For example, the secondary assistant may be provided with an "answer" query, which is a request for additional information. The same secondary assistant may then be provided with a subsequent "answer" query, even if it provided an intervening query to a different secondary automated assistant, as described above. The indication may be stored along with contextual and / or other information, such as the time the query was provided, the location of the secondary automated assistant, the state of one or more applications running on the user's computing device, and / or other signals that may be utilized to determine which secondary automated assistant to subsequently provide a subsequent query to.

[0030] The above description is provided as a summary of some implementations of the present disclosure. Further descriptions of those and other implementations are described in more detail below. [Brief description of the drawings]

[0031] [Figure 1] FIG. 1 illustrates an example environment in which implementations disclosed herein may be implemented. [Diagram 2] FIG. 1 is a block diagram of an example environment in which various methods disclosed herein may be implemented. [Figure 3A] FIG. 2 is a block diagram of another example environment in which various methods disclosed herein may be implemented. [Figure 3B] FIG. 2 is a block diagram of another example environment in which various methods disclosed herein may be implemented. [Figure 4A] FIG. 1 is a flowchart illustrating an example method according to various implementations disclosed herein. [Figure 4B] FIG. 1 is a flowchart illustrating an example method according to various implementations disclosed herein. [Figure 5A] FIG. 1 illustrates a dialog including a user, a first automated assistant, and a second automated assistant, according to various implementations disclosed herein. [Figure 5B] FIG. 1 illustrates a dialog including a user, a first automated assistant, and a second automated assistant, according to various implementations disclosed herein. [Figure 6] 1 is a flowchart illustrating an example method according to various implementations disclosed herein. [Figure 7] FIG. 1 illustrates an example architecture of a computing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] Referring to FIG. 1, an exemplary environment is provided that includes multiple automated assistants that may be invoked by a user 101. The environment includes a first standalone interactive speaker 105 having a microphone (not shown) and a second standalone interactive speaker 110 having a microphone (also not shown). The first speaker may at least partially execute a first automated assistant that may be invoked with an invocation phrase. The second speaker 110 may execute a second automated assistant that may be invoked with an invocation phrase that is either the same invocation phrase as the first automated assistant or a different phrase to allow the user to select which automated assistant to invoke based on the spoken phrase. In the exemplary environment, the user 101 speaks a verbal utterance 115 of "OK Assistant, set an alarm" near the first speaker 105 and the second speaker 110. If one of the first and / or second automated assistants is configured to be invoked by the phrase "OK Assistant," the invoked assistant may process the query that follows the invocation phrase (i.e., "set an alarm").

[0033] In some implementations, a device such as the first speaker 105 may be running multiple automated assistants. Referring to FIG. 2, an exemplary environment including multiple client devices running multiple automated assistants is shown. The system includes a first client device 205 running a first automated assistant 215 and a second automated assistant 220. Each of the first and second automated assistants can be invoked by speaking an invocation phrase (either unique to each assistant or the same phrase to invoke both assistants) near the client device 205 such that audio is captured by the microphone 225 of the client device 205. For example, the user 101 can invoke the first automated assistant 215 by speaking "OK Assistant 1" near the client device 205 and further invoke the second automated assistant 220 by speaking the phrase "OK Assistant 2" near the client device 205. Based on which invocation phrase is spoken, the user can indicate which of the multiple assistants running on the first client device 205 the user is interested in handling the verbal query. The exemplary environment further includes a second client device 210 running a third automated assistant 245. The third automated assistant 245 can be configured to be invoked using a third invocation phrase, such as "OK Assistant 3," as may be captured by the microphone 230. In some implementations, one or more of the automated assistants of FIG. 2 may be absent. Additionally, the exemplary environment may include additional automated assistants not present in FIG. 2. For example, the system may include a third device running additional automated assistants, and / or the client device 210 and / or the client device 205 may be running additional automated assistants and / or fewer automated assistants than shown.Each of the automated assistants 215, 220, and 225 can include one or more components of the automated assistants described herein. For example, the automated assistant 215 can include its own audio capture component to process incoming queries, a visual capture component to process incoming visual data, a hotword detection engine, and / or other components. In some implementations, automated assistants running on the same device, such as the automated assistants 215 and 220, can share one or more components that may be utilized by both automated assistants. For example, the automated assistants 215 and 220 can share one or more of an on-device speech recognizer, an on-device NLU engine, and / or other components.

[0034] In some implementations, one or more of the automated assistants may be invoked by a general invocation phrase, such as "OK Assistant," that does not individually invoke any of the other automated assistants separately. When a user speaks the general invocation phrase, one or more of the automated assistants may act as a primary automated assistant and coordinate responses among the other automated assistants. With reference to FIG. 3A, a primary automated assistant 305 is shown along with secondary automated assistants 310 and 315. The primary automated assistant may be invoked with the phrase "OK Assistant," or another general invocation phrase, which may indicate that the user is interested in providing a query to multiple automated assistants. Similarly, the secondary automated assistants 310 and 315 may each have one or more alternative invocation phrases that, when spoken by a user, invoke the corresponding automated assistant. For example, the secondary automated assistant 310 may be invoked with the invocation phrase "OK Assistant A," and the secondary automated assistant 315 may be invoked with the invocation phrase "OK Assistant B." In some implementations, the primary automated assistant 305 and one or more of the secondary automated assistants 310 and 315 may be running on the same client device, as shown in Figure 2 with respect to the first automated assistant 215 and the second automated assistant 220. In some implementations, one or more secondary automated assistants may be running on separate devices, as shown in Figure 2 with respect to the third automated assistant 245.

[0035] The environment shown in FIG. 3A further includes a database 320. The database 320 may be utilized to store interactions between a user and one or more automated assistants. In some implementations, the primary automated assistant 305 may receive a query from a user, such as a query uttered following an invocation phrase that invokes the primary automated assistant 305. For example, a user may utter "OK ​​Assistant, how tall is the president?" The phrase "OK Assistant" may be an invocation phrase specific to the primary automated assistant 305, and once invoked, the primary automated assistant 305 may determine the intent of the query following the invocation phrase (i.e., "how tall is the president"). In some implementations, the primary automated assistant 305 may provide the query to one or more of the secondary automated assistants 310 and 315 for further processing. Additionally, the primary automated assistant 305 may store an indication of the query in the database 320 along with the assistant to whom the query was provided for future access by the primary automated assistant 305.

[0036] In some implementations, the primary automated assistant 305 may be a "meta-assistant" that may constantly interact with one or both of the secondary automated assistants 310 and 315 and may itself lack one or more automated assistant capabilities, such as speech recognition, natural language understanding, and / or fulfillment capabilities. In other examples, the primary automated assistant may also interact with the secondary assistant while performing its own query processing to determine a response to the query. For example, as described further herein, the primary automated assistant 305 may include a query processing engine, or the primary automated assistant 305 may not be configured to process queries and provide responses to the user.

[0037] Other components of the automated assistants 305, 310, and 315 are optional and may include, for example, a local speech-to-text ("STT") engine (to convert captured audio to text), a local speech-to-text ("TTS") engine (to convert text to speech), a local natural language processor (to determine the semantic meaning of the audio and / or the text converted from the audio), and / or other local components. Because the client device on which the automated assistant runs may be relatively constrained in terms of computational resources (e.g., processor cycles, memory, batteries, etc.), the local components may have limited functionality compared to any counterparts included within any cloud-based automated assistant components running remotely in conjunction with the automated assistant.

[0038] Referring again to FIG. 2, in some implementations, one or more of the automated assistants 210, 215, and 220 may be invoked by one or more gestures that indicate a user's interest in interacting with a primary automated assistant. For example, a user may indicate intent to invoke an automated assistant by interacting with a device, such as pressing a button or touch screen, may perform a movement that is visible and that can be captured by an image capture device, such as a camera, and / or may view the device so that the image capture device can recognize the user's movement and / or positioning. When the user performs a gesture or action, the automated assistant may be invoked and begin capturing audio data following the gesture or action, as described above.

[0039] In some implementations, one automated assistant may be selected as a primary assistant, and one or more other automated assistants may be designated as secondary assistants. For example, a user may speak a common invocation phrase to multiple automated assistants in the user's vicinity. One or more components may determine which of multiple devices running automated assistants is closest to the user, and the closest automated assistant may be designated as the primary automated assistant, and the other automated assistants are designated as secondary assistants. Also, for example, when a user invokes an automated assistant, one or more components may determine which automated assistant was most frequently utilized by the user and designate that automated assistant as the primary automated assistant.

[0040] In some implementations, a user can invoke a particular automated assistant with an invocation phrase that is specific to that automated assistant, and that automated assistant can be designated as the primary automated assistant. For example, a user can speak the invocation phrase "OK Assistant 1" to invoke the first assistant 215, which is then designated as the primary automated assistant. Other automated assistants, such as the second automated assistant 220 and the third automated assistant 245, can then be invoked by the primary automated assistant, be provided with queries by the primary automated assistant, and / or receive responses from the other automated assistants as described herein.

[0041] In some implementations, one or more automated assistants, such as the first automated assistant 215 and the second automated assistant 220, may share one or more modules, such as a natural language processor and / or the results of a natural language TTS and / or STT processor. For example, referring again to FIG. 2, both the first automated assistant 215 and the second automated assistant 220 may share natural language processing, such that once the client device 205 receives audio data, the audio data may be processed into text and then the text may be provided to both automated assistants 215 and 220. Also, for example, one or more components of the client device 205 may process the audio data into text and provide a textual representation of the audio data to the third automated assistant 245, as described further below. In some implementations, the audio data may not be processed into text and instead be provided to one or more of the automated assistants as raw audio data.

[0042] With reference to Figure 3B, components of primary automated assistant 305 and secondary automated assistant 310 are shown. As previously mentioned, primary automated assistant 305 and / or one or more secondary automated assistants 310 may include additional components other than those shown in Figure 3B. Additionally or alternatively, one or more components may not be present in primary automated assistant 305 and / or one or more secondary automated assistants 310. For example, primary automated assistant 305 may include query processing engine 340 and perform as both a primary automated assistant and a secondary automated assistant.

[0043] The invocation engine 335 is operable to detect one or more verbal invocation phrases and to invoke the primary automated assistant 305 in response to detecting one of the verbal invocation phrases. For example, the invocation engine 335 can invoke the primary automated assistant 305 in response to detecting a verbal invocation phrase such as "Hey Assistant," "OK Assistant," and / or "Assistant." The invocation engine 335 can continuously process (e.g., when not in "inactive" mode) a stream of audio data frames based on an output from one or more microphones of the client device executing the primary automated assistant 305 to monitor for the occurrence of the verbal invocation phrase. While monitoring for the occurrence of the verbal invocation phrase, the invocation engine 335 discards (e.g., after temporarily storing in a buffer) any audio data frames that do not include the verbal invocation phrase. However, when the invocation engine 335 detects the occurrence of the verbal invocation phrase in the processed audio data frames, the invocation engine 335 can invoke the primary automated assistant 305.

[0044] 4A and 4B, a flow chart is shown illustrating a method of providing a query to a secondary automated assistant instead of providing a query to another automated assistant. For convenience, the operations of the method are described with reference to a system that performs the operations, such as the primary automated assistant 305 and the secondary automated assistants 310 and 315 shown in FIGS. 3A and 3B. The illustrated method system includes one or more processors and / or other components of a client device. Furthermore, although the illustrated method operations are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0045] At 405, the primary automated assistant is invoked. The primary automated assistant may share one or more characteristics with the primary automated assistant 305. For example, invoking the primary automated assistant may include the user speaking an invocation phrase specific to the primary automated assistant (e.g., "OK Assistant"), performing a gesture detected by one or more cameras of the client device running the primary automated assistant 305, pressing a button on the client device running the primary automated assistant, and / or one or more other actions by the user indicating that the user is interested in having one or more subsequent (or previous) utterances processed by the primary automated assistant. In some implementations, the primary automated assistant is a "meta-assistant" that performs only limited operations, such as determining which of multiple secondary automated assistants to provide a query to. For example, the primary automated assistant may include one or more of the components of the primary automated assistant 305 shown in FIG. 3B (e.g., query classifier 330, assistant determination module 325) without including a query processing engine for processing queries and providing responses. In some implementations, the primary automated assistant 305 also performs the operations of a secondary automated assistant and may include a query processing engine, such as the query processing engine 340 of the secondary automated assistant 310. A verbal query may be provided by a user using an invocation phrase. For example, referring to FIG. 5A, a dialog between a user 101 and multiple automated assistants is shown. To initiate the dialog, the user uttered a verbal utterance 505 that includes the invocation phrase "OK Assistant" and the query "how tall is the president."

[0046] At 410, the primary automated assistant 305 determines the intent of the query provided as a verbal utterance by the user. In some implementations, the query classifier 330 of the primary automated assistant 305 can determine a classification of one or more queries provided by the user to the primary automated assistant 305. For example, the primary automated assistant 305 can receive a verbal utterance including a query after invocation and then perform NLU on the audio data to determine the intent of the query. The query classifier 330 can then determine the classification of the query. As an example, the primary automated assistant 305 can receive audio data including a user uttering the query "how tall is the president". After NLU processing, the query classifier 330 can determine that the query is an "answer" query (i.e., a request from the user that an answer to the query be provided). Also, for example, the primary automated assistant 305 can receive audio data including a user uttering the query "set an alarm". After NLU processing, the query classifier 330 can determine that the query is a "productivity" query (i.e., a request from a user to perform a task). Other examples of query classifications can include requesting a device, such as a smart light and / or home appliance, to perform a task, requesting a third-party application to perform a task (e.g., sending an email and / or text message), and / or other classifications of queries that may be performed by an automated assistant. Additionally, the query classification categories can include additional information related to the query, such as the subject matter of the query, additional information related to the query (e.g., the current state of the device while the query is being provided), and / or other fine-grained classifications that may be utilized to identify past queries related to the current query.

[0047] Once the query is classified, the assistant determination module 325 can determine which automated assistant to provide the query to for further processing. As previously described, the database 320 can include indications of past interactions between the user and the secondary automated assistants 310 and 315. Each indication can include an identifier of the query classification and the automated assistant that responded to the query. As an example, the query classifier 330 can determine that the query "how tall is the president" is an "answer" query and provide it to the secondary automated assistant 310. Additionally, the primary automated assistant 305 can store the query and / or an indication of the query classification in the database 320 along with an indication that the query was provided to the secondary automated assistant 310.

[0048] In some implementations, a user may indicate a preference for a particular automated assistant to process queries of a particular classification. Referring again to FIG. 4A, at 415, the assistant determination module 325 may determine whether the user has indicated a preference for a particular secondary automated assistant to process queries of the classification determined at 410. If the user 101 has indicated a preference, at 420, the preferred secondary automated assistant may be provided with the query. For example, the user may indicate, either explicitly or based on past interactions with the secondary automated assistant, that all text messaging (e.g., queries classified as “messaging”) be processed by the automated assistant 310. The user's indication of preference may be stored in the database 320, and for subsequent queries classified as “messaging”, the query may be provided to the automated assistant 310 instead of another secondary automated assistant.

[0049] If the user has not indicated a preference for a particular secondary automated assistant, in decision block 425, the assistant determination module 325 determines, for a classified query, whether a secondary automated assistant has been provided with a query of the same classification as the current query within a threshold time period. In some implementations, the assistant determination module 325 can identify from the database 320 which automated assistants have previously processed queries of the same classification as the current query. For example, a query of "how many kids does he have" may be received by the primary automated assistant 305 and classified as an "answer" query by the query classifier 330. The assistant determination module 325 can determine, based on one or more indications in the database 320, that a previous "answer" query was processed by the secondary automated assistant 310 and / or that the user has indicated a preference for "answer" queries to be processed by the secondary automated assistant 310. In response, the assistant decision module 325 may decide, at 430, to provide the query "how many kids does he have" to the secondary automated assistant 310 instead of providing the query to another automated assistant.

[0050] In some implementations, one or more communication protocols may be utilized to facilitate communication between the automated assistants. For example, referring to FIG. 3B, the secondary automated assistant 310 includes an API 345 that may be utilized by the primary automated assistant 305 to provide queries to the secondary automated assistant 310. In some implementations, one or more other communication protocols may be utilized. For example, in an example where the primary automated assistant 305 and the secondary automated assistant 310 are running on separate client devices, the primary automated assistant 305 may utilize a speaker of the client device running the assistant to transmit a signal (e.g., an ultrasonic audio signal including audio data of spoken utterances and / or encoding of natural language processing of the query) that may be detected by a microphone of the client device.

[0051] In some implementations, the secondary automated assistant 310 can include a capabilities communicator 350 that can provide the primary automated assistant 305 with a classification of queries that can be processed by the secondary automated assistant 310. For example, the secondary automated assistant 310 can be configured to process "answer" queries (i.e., can include a query processing engine 340 that can respond to queries that include questions), but may not be configured to process "message" requests. Thus, the capabilities communicator 350 can provide an indication of a classification of queries that can be processed and / or an indication of a classification of queries to opt out of responding to. In some implementations, the secondary automated assistant 310 can utilize the API 390 of the primary automated assistant 305 and / or one or more other communication protocols (e.g., ultrasound signals) to indicate its capabilities to the primary automated assistant 305.

[0052] In block 445, the interaction between the user and the secondary automated assistant to which the query was provided is stored in the database 320. In some implementations, the query (and / or the classification of the query) and the time may be stored along with the automated assistant that processed the query. Thus, in some implementations, the assistant determination module 325, in determining which secondary automated assistant to provide the query to, may determine whether one of the secondary automated assistants has processed a query of the same classification within a threshold time period. As an example, a user may submit a query of the "answer" classification that is provided to the secondary automated assistant 310 for processing. An indication of the query (or its classification) may be stored by the assistant determination module 325 along with a timestamp indicating the time the query was provided to the secondary automated assistant 315. If the user subsequently provides another query that the query classifier 330 determines to be an "answer" query (e.g., the next query, or a query after one or more intervening queries), the assistant determination module 325 may determine how long it has been since the secondary automated assistant 315 last processed an "answer" query. If the secondary automated assistant 315 processes the “answer” query within the threshold time, subsequent “answer” queries may be provided to the secondary automated assistant 315 .

[0053] In some implementations, the query processing engine 340 may provide at least a portion of the query and / or information contained within the query to a third-party application to determine a response. For example, a user may submit a query of "Make me a dinner reservation tonight at 6:30," which is provided to a secondary automated assistant for further processing. The secondary automated assistant may be provided with the audio data and / or a textual representation of the audio data, perform natural language understanding on the query, and provide some of the information to a third-party application, such as a dining reservation application. In response, the third-party application may determine a response (e.g., check whether a dining reservation is available) and provide the response to the secondary automated assistant, which may generate a response based on the response from the third-party application.

[0054] In some implementations, storing the interaction may include storing the context of the query in database 320 along with the secondary automated assistant that processed the query. Additionally or alternatively, a timestamp and / or other information related to the query may be stored in database 320 along with the instruction of the interaction. For example, a query of "how tall is the president" may be received by primary automated assistant 305. The query may be provided to secondary automated assistant 310, and a classification of the query, a timestamp of when the query was provided to secondary automated assistant 310, and / or the query (e.g., audio data of the user's verbal utterance, NLU data of the query, textual representation of the query) may be stored in database 320. Additionally, the context of the query may be stored, such as an instruction that the query is related to "the president." Subsequently, the user may submit a query of "how many kids does he have," and the assistant determination module 325 may determine, based on the instructions stored in the database 320, that the user has submitted a query of the same classification (e.g., an "answer" query) within a threshold time period. The primary automated assistant 305 may provide the query (i.e., "how many kids does he have") along with the context (e.g., the previous query is related to "the president") to the secondary automated assistant 310 for further processing. In some implementations, upon receiving the query of "how many kids does he have," the secondary automated assistant 310 may separately store an indication of the context of the previous query, such that the secondary automated assistant 310 may determine that "he" refers to "the president" based on the previous context.

[0055] Referring again to FIG. 4, in some implementations, at decision block 425, the assistant determination module 325 may determine that a secondary automated assistant has not processed a query of the same classification as the current query within a threshold time period. For example, a query may be submitted that is the first query of that classification received by the primary automated assistant 305 during the current session and / or within a time period. If the assistant determination module 325 determines, based on the indications in the database 320, that the user has not indicated a preference for an automated assistant to process queries of the current classification and / or that a secondary automated assistant has not processed queries of the current classification, the assistant determination module 325 may select a secondary automated assistant to process the query based on the capabilities of the available secondary automated assistants. For example, the primary automated assistant 305 may determine one or more candidate secondary automated assistants to provide queries based on the current classification based on the capabilities of each secondary automated assistant provided by the capabilities communicator 350 as described herein. For example, an “answer” query may not have been handled by either the secondary automated assistant 310 or the secondary automated assistant 315 within a threshold time period. Assistant determination module 325 may determine, based on the capabilities communicators of each of the available secondary automated assistants, that secondary automated assistant 310 is configured to handle "answer" queries and that secondary automated assistant 315 is not configured to handle "answer" queries. Thus, instead of providing the query to secondary automated assistant 315, which is not configured to handle such queries, the query may be provided to secondary automated assistant 310.

[0056] In some implementations, the secondary automated assistant may be selected based on the user who uttered the query. For example, if the primary automated assistant cannot determine that a related query has been provided to the secondary automated assistant within a threshold amount of time, the primary automated assistant may decide to provide the query to the same secondary automated assistant that was previously provided with the query by the same user. In one example, a first user may provide a query of "Do I have anything on my calendar" to the primary automated assistant, which is provided to the secondary automated assistant for processing. Other users may subsequently provide additional queries that are processed by different secondary automated assistants. The first user may subsequently submit a query of "How tall is the president," but more than a threshold amount of time after submitting the first query. In that case, to improve conversational continuity, the same secondary automated assistant may be provided with the subsequent query based on identifying the speaker as the same speaker as the initial query.

[0057] In decision block 430, the assistant determination module 325 may determine that an automated assistant has not processed a query of the current classification within a threshold time period. In response, in block 440, the assistant determination module 325 may provide the query to a secondary automated assistant that is capable of processing a query of the current classification. For example, the assistant determination module 325 may select an automated assistant that has indicated, via the automated assistant capabilities communicator 350, that it is capable of processing a query of the current classification. Once the query is provided to a capable secondary automated assistant, in block 445, the interaction may be stored in the database 320 for later use by the assistant determination module 325.

[0058] The assistant determination module 325 may determine that multiple secondary automated assistants may be configured to process queries of a particular classification, but none have processed such queries within a threshold time period (or multiple secondary automated assistants have processed the current query). Additionally or alternatively, multiple secondary automated assistants may have processed queries of the same classification within a threshold time period. For example, secondary automated assistant 310 may have processed a query of “set an alarm” that was classified by query classifier 330 as a “productivity” query, and secondary automated assistant 315 may have processed a query of “turn off the bedroom lights” that was also classified by query classifier 330 as a “productivity” query. If multiple secondary automated assistants are identified by assistant determination module 325 to process a query, the candidate secondary automated assistants may be ranked and / or scored such that one secondary automated assistant may be selected instead of the other secondary automated assistants.

[0059] Referring to FIG. 5A, an interaction between a user 101 and two automated assistants 305 and 310 is shown. In the illustrated environment, the primary automated assistant 305 and the secondary automated assistants 310 and 315 are each shown running on separate devices. However, in some implementations, one or more of the automated assistants may be running on the same device, as shown in FIG. 2. Furthermore, the illustrated dialogue is received by the primary automated assistant 305, which does not fulfill the query. Thus, the verbal utterances of the user 101 may be detected by a speaker of the primary automated assistant 305, which provides the query to the secondary automated assistants 310 and / or 315 for further processing. Thus, the user 110 may interact with the primary automated assistant 305, but as shown, responses to the query may be provided by the secondary automated assistants 310 and 315.

[0060] In a dialog turn 505, the user 101 utters the utterance "OK Assistant, how tall is the president." As described herein, the invocation phrase "OK Assistant" can invoke the primary automated assistant 305, which further receives the verbal query "how tall is the president" via one or more microphones. The query classifier 330 of the primary automated assistant 305 can determine a classification of the query, such as a classification of "answer" indicating that the user 101 is requesting an answer to the query. Additionally, the primary automated assistant 305 can determine, based on an indication in the database 320, that the user 101 has not previously submitted a query classified as an "answer" within a threshold time period within the current dialog session. In response, the assistant determination module 325 can provide the query to one of the candidate secondary automated assistants, such as the secondary automated assistant 310. The decision to provide a query to one automated assistant instead of providing it to another automated assistant can be based, for example, on a user's preference for secondary automated assistant 310 in providing "answer" queries identified based on past interactions between the user and the secondary automated assistant, the time of day, the user's location, and / or other factors indicating that the user is interested in secondary automated assistant 310 handling the query instead of secondary automated assistant 315 handling the query. Additionally, capabilities communicator 350 of secondary automated assistant 310 can provide an indication to primary automated assistant 305 that it is capable of handling the "answer" query, and primary automated assistant 305 can provide the query to secondary automated assistant 310 based on the indicated automated assistant capabilities.

[0061] Referring to FIG. 5B, a diagram is provided that includes the dialog turn of FIG. 5A and a component that processes a query. As shown, the verbal utterance 505 is first provided to the query classifier 330, which determines a classification of an "answer" at 505A. The classification of the "answer" and / or other information related to the query (e.g., context) is provided to the assistant determination module 325, which checks in the database 320 to determine whether a secondary automated assistant has previously processed the "answer" query (505B). Because this is the first query that has been classified and processed as an "answer," the database 320 does not include an indication of a secondary automated assistant and provides an "n / a" response 505C. In response, the assistant determination module 325 determines to provide the query to the secondary automated assistant 310 (at 505D) instead of providing the query to the secondary automated assistant 315 based on one or more factors (e.g., the user's previous interaction with the secondary automated assistant, machine learning model output).

[0062] The primary automated assistant provides the query to the secondary automated assistant 310 for further processing. In response, the secondary automated assistant's query processing engine 340 determines a response 510 to the query. For example, referring again to FIG. 5A, in response to the query "how tall is the president," the secondary automated assistant 310 responds with "the president is 6 feet, 3 inches." In some implementations, the response may be provided by a speaker specific to the secondary automated assistant 310 (e.g., a speaker integral to the device running the automated assistant). In some implementations, the response may be provided by a speaker of one or more other devices, such as the primary automated assistant 305. In those cases, even if a speaker is utilized by more than one automated assistant in providing a response, a voice profile may be utilized to provide a response specific to the secondary automated assistant 310 such that a user can recognize, based on the audio characteristics of the response, that the response is generated by the secondary automated assistant 310 and not another automated assistant. Further, referring again to FIG. 5B , once a query is provided to the secondary automated assistant 310 (and / or the secondary automated assistant 310 provides a response 510), the assistant determination module 325 can store in the database 320 an interaction 505E indicative of the query, the query classification, a timestamp when the query was provided, the context of the query, and / or other information related to the query, such that when a subsequent “answer” query is received, the assistant determination module 325 can utilize the interaction to identify the previous interaction and determine which automated assistant to provide the subsequent “answer” query to.

[0063] In some implementations, the assistant determination module 325 may determine that multiple secondary automated assistants have processed a query of the current classification. In those cases, in block 450, the assistant determination module 325 may rank the candidate secondary automated assistants to determine which secondary automated assistant to provide the query to. As previously mentioned, in some implementations, a query with the current classification may not have been previously processed within a threshold time period (or may not have been processed at all). Thus, any secondary automated assistant configured to process the query may be a candidate automated assistant. In some implementations, multiple secondary automated assistants may be processing queries of the same classification as the current query, and the assistant determination module 325 may determine which of the multiple assistants to provide the query to. Additionally, in some implementations, related queries of a different classification than the current query may be processed, and one of the secondary automated assistants may be selected to process the query in preference to other secondary automated assistants based on determining that the previous query is related.

[0064] In some implementations, to determine which secondary automated assistant to provide the current query to, the current query and the previous queries may be embedded in an embedding space to determine whether one or more of the previous queries are relevant to the current query and therefore to rank the candidate secondary automated assistants. For example, if multiple secondary automated assistants have previously processed queries of the same classification as the current query, the previous queries may be embedded in the embedding space and the relevance of the current query to the previously processed queries. Thus, the secondary automated assistant that previously processed the query most relevant to the current query may be selected as the secondary automated assistant to process the current query (and / or the relevance of the previously processed queries may be a factor in determining which automated assistant to provide the current query to).

[0065] Referring again to FIG. 5A, a verbal utterance 515 is provided by the user 101. The utterance, "OK Assistant, what's on my calendar," is received by the primary automated assistant 305, classified, and provided to the secondary automated assistant 310 for further processing. In response, the secondary automated assistant 310 provides the response, "you have a meeting at 5." Referring again to FIG. 5B, the verbal utterance 515 is first provided to the query classifier 305, which determines that the query is an "answer" query 515A. The classification is provided to the assistant determination module 325, which determines, based on instructions stored in the database 320, whether the user 101 has previously interacted with an automated assistant that processed the "answer" query. After checking database 320 at 515B, a previous interaction between user 101 and secondary automated assistant 305 is identified (515C). However, assistant determination module 325 may further determine that the query is not related to the previous "answer" query based on context information identified from the previous "answer" query 505. In response, assistant determination module 325 may not select secondary automated assistant 305 as the automated assistant to handle the query based on determining that the context of verbal utterance 515 is not related to the context of the previous "answer" query (i.e., utterance 515 is not a continuation of a dialog initiated by utterance 505).The assistant determination module 325 can determine to provide a query to the secondary automated assistant 310 based on, for example, capabilities indicated by each of the automated assistants, such as identifying that the secondary automated assistant 310 is configured to access the user's calendar and has been previously utilized by the user to access the user's calendar, and / or based on an indicated user preference for the secondary automated assistant 310 to take precedence over the secondary automated assistant 305 when processing calendar-related queries. Once a query is provided to the secondary automated assistant 310, the interaction is stored (515E) in the database 320, and the secondary automated assistant 310 can provide a response. As with previous interactions stored in the database 320, interactions with the secondary automated assistant 310 can be stored with context information, timestamps, vector embeddings of the query and / or response provided by the secondary automated assistant, and / or other information related to the query and / or response for utilization in determining whether to provide a subsequent query to the secondary automated assistant. In some implementations, previous interactions can be stored for a period of time such that for a subsequent query, only interactions that occurred within a threshold time period are reviewed to determine whether the query is related to the previous query.

[0066] 5A, the user 101 provides the verbal utterance 525 of "how many kids does he have." This verbal utterance is a continuation of the dialog started in utterance 505, and at least some of the context of the previous dialog is necessary to generate a response. For example, the verbal utterance 525 includes the term "he," which refers to "the president" as indicated by the user 101 in utterance 525. In response, the secondary automated assistant 310 responds with "the president has two children."

[0067] Referring again to FIG. 5B, the spoken utterance 525 is provided to the query classifier 330, which classifies the query as an “answer” query at 525A. As with the previous query, the assistant determination module 325 checks in the database 320 to identify (at 525C) that both the secondary automated assistant 310 and the secondary automated assistant 315 have processed the “answer” query within the threshold time period (525B). However, based on the content of the current query (e.g., having the term “he” without indicating what the term refers to) and based on context information that may be stored with the interaction 505E, the assistant determination module 325 may decide to provide the query to the secondary automated assistant 310 (525D) instead of providing the query to the secondary automated assistant 315 (which has processed an “answer” query that is not related to the current query). Additionally, the interaction with the secondary automated assistant 310 is stored in the database 320 (525E) along with the query information, context, and / or other information, as described herein.

[0068] Referring again to FIG. 5A, the user then provides a verbal utterance 535 of "OK, Assistant, send Bob a message of 'Meeting is at 5'". In response, the secondary automated assistant 310 executes the request and responds to the user 101 with "I sent Bob the message via Message Application". In some implementations, the primary automated assistant, as previously described, may determine that the verbal utterance is not related to other previous queries and therefore may determine which automated assistant to provide the query based on the capabilities provided by each automated assistant, the user's preferences, and / or other factors, etc. In some implementations, the primary automated assistant 305 may determine (based on the terms in the query) that the utterance 535 is related to a meeting and that the user 101 previously interacted with the secondary automated assistant 315 in relation to the meeting in a calendar application. Thus, for continuity, the primary automated assistant 305 may provide a query to the secondary automated assistant 315 in preference to the secondary automated assistant 310 based on determining that the context of the query is more relevant to a previous query processed by the secondary automated assistant 315 than one previously processed by the secondary automated assistant 310.

[0069] 5B, the spoken utterance is first provided to the query classifier 330, which determines that the query is a "messaging" query (i.e., a request to send a message) (535A). In response, the assistant determination module 325 checks the database 320 (535B) and does not identify a previous interaction between the user 101 and the secondary automated assistant that handled the "messaging" query, but does identify (at 535C) a previous interaction between the user 101 and the secondary automated assistant 315 regarding the calendar (i.e., stored interaction 515E). Thus, the assistant determination module 325 determines that the previous interaction with the secondary automated assistant 315 is more relevant to the current query, and instead of providing the query to the secondary automated assistant 310, the assistant determination module 325 can provide the query to the secondary automated assistant 315 (535D). For example, assistant determination module 325 may utilize one or more machine learning models to determine that utterance 535 is most similar to the response (e.g., utterance 535 and response 520 include a reference to “meeting”) and embed utterances 505, 515, 525, and 535 and / or responses 510, 520, and 530 in the embedding space to process utterance 515 and provide utterance 535 to the same secondary automated assistant 315 that processed utterance 515 and provided response 520. In response, a query processing engine of secondary automated assistant 315 may determine a response, perform the requested action (i.e., send a message to Bob), and provide response 540 to user 101. Interaction 535E with the secondary automated assistant may then be stored in database 320 for subsequent use in determining when to provide a “messaging” query to the secondary automated assistant.

[0070] In some implementations, the machine learning model may receive one or more other signals as inputs and provide as output a ranking of candidate secondary automated assistants to provide a query if a previous query of the same classification has not been provided to the secondary automated assistant within a threshold time period. For example, a current context and a historical context corresponding to when a previous query was provided to the secondary automated assistant may be utilized as inputs to the machine learning model to determine which automated assistant to provide the query to. The context of the query may include, for example, images and / or interfaces currently presented to the user via one or more computing devices, the current application being accessed by the user when providing the query, the time the user provided the query, and / or one or more other user or device features that may indicate the user's intent to provide the query to one secondary automated assistant instead of providing the query to another secondary automated assistant.

[0071] 4B, the query may be provided to the highest ranked candidate secondary automated assistant in block 455. If the selected secondary automated assistant successfully processes the query in decision block 460, the interaction with the secondary automated assistant is stored in database 320 (in block 465). If the selected automated assistant does not successfully process the query (e.g., is unable to generate a response, is unable to perform the intended action), the query may be provided to the next highest ranked candidate automated assistant in block 470. This may continue until providing the query to all candidate automated assistants has failed and / or until a candidate secondary automated assistant successfully processes the query.

[0072] In some implementations, when a secondary automated assistant is first provided with a query determined to be related to a previous query provided to a different secondary automated assistant, the context of the previous query may be provided with the query. For example, for an initial query of "How tall is the president," a text context including an indication that the query was related to "the president" is stored in database 320. If a query of "how many kids does he have" is subsequently received and provided to a different secondary automated assistant than the first query, the text context of "the president" may be provided to assist the secondary automated assistant in resolving "he" in the subsequent query. In some implementations, other contextual information, such as the time of day, the state of one or more computing devices, an application currently being executed by the user, and / or other context of the previous and / or current query that may assist the selected secondary automated assistant in resolving one or more terms in the provided query, may be provided with the subsequent query.

[0073] FIG. 6 shows a flow chart illustrating another example method 600 for providing a query to a secondary automated assistant instead of providing the query to another automated assistant. For convenience, the operations of method 600 are described with reference to a system that performs the operations. The system of method 600 includes one or more processors and / or other components of a client device. Moreover, although the operations of method 600 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0074] In step 605, a call is received by the primary automated assistant. In some implementations, the call can be a verbal utterance that transitions the primary automated assistant from an inactive state (e.g., not processing incoming verbal utterances) to an active state. In some implementations, the call can include a user gesture, a user selecting a button via the client device and / or an interface of the client device, and / or one or more other actions indicating an interest by the user in activating the primary automated assistant.

[0075] In some implementations, the primary automated assistant can share one or more characteristics with the primary automated assistant 305. For example, in some implementations, the primary automated assistant can receive verbal utterances via one or more microphones and determine a secondary automated assistant to process the verbal utterances. In some implementations, the primary automated assistant can include a query processing engine that enables the primary automated assistant to generate responses to verbal queries and / or other utterances. In some implementations, the primary automated assistant can perform processing of the verbal utterances, such as ASR, TTS, NLU, and / or other processing to determine the content of the verbal utterances.

[0076] In some implementations, the call can indicate a particular secondary automated assistant to which audio data is to be provided. For example, a primary automated assistant can be called by a general call such as "OK Assistant" and further be called with a secondary call indicating a particular automated assistant such as "OK Assistant A." In this case, the general automated assistant can provide audio (or provide an indication of audio data) specifically to "Assistant A" instead of providing audio data to other secondary automated assistants. Additionally or alternatively, if the call indicates a particular automated assistant, the call can be utilized along with one or more other signals to rank the automated assistants when determining which automated assistant to provide audio data to.

[0077] In step 610, a verbal query is received by the primary assistant. In some implementations, the verbal query can include a request for the automated assistant to perform one or more actions. For example, a user may submit a verbal query of "How tall is the president" with the intent of receiving a response that includes an answer to the posed question. Also, for example, a user may submit a verbal query of "turn off the lights" with the intent of transitioning one or more smart lights from on to off.

[0078] In step 615, historical data for the user is identified. The historical data may include information related to one or more secondary automated assistants and queries processed by each of the secondary automated assistants. For example, the historical data may include a textual representation of the query, the content of the query as it was processed, the automated assistant that generated a response to the query, the response that was generated, and / or other indications of the historical interactions of the user with one or more automated assistants.

[0079] In step 620, a relationship between at least a portion of the historical interaction and the verbal query is determined. In some implementations, the relationship may be determined based on identifying previous interactions between the user and one or more of the secondary automated assistants and determining, based on the current query, which secondary automated assistant processed the previous query that is most similar to the current query. For example, the user may have submitted a previous query of "How tall is the president". The user may then submit a query of "how many kids does he have", where "he" refers to "the president". Based on the context of a second query that follows the first query (either directly or with one or more intervening queries), a relationship between the queries may be determined. Furthermore, based on instructions stored in the database regarding the previous query, the same secondary automated assistant that generated a response to the first query may be identified, such that the second query may be provided to the same automated assistant.

[0080] In step 625, the audio data including the verbal query is provided to the secondary automated assistant. In some implementations, the audio data may be provided as a transcription of the audio data. For example, a general automated assistant may perform automatic speech recognition on the audio data and provide the transcription to the secondary automated assistant. In some implementations, the audio data may be provided as audio (e.g., raw audio data and / or pre-processed audio data), which may then be processed by a particular secondary automated assistant.

[0081] 7 is a block diagram of an exemplary computing device 710 that may be optionally utilized to perform one or more aspects of the techniques described herein. The computing device 710 typically includes at least one processor 714 that communicates with a number of peripheral devices via a bus subsystem 712. These peripheral devices may include, for example, a storage subsystem 724 including a memory subsystem 725 and a file storage subsystem 726, a user interface output device 720, a user interface input device 722, and a network interface subsystem 716. The input and output devices allow a user to interact with the computing device 710. The network interface subsystem 716 provides an interface to external networks and is coupled to corresponding interface devices in other computing devices.

[0082] The user interface input devices 722 may include a keyboard, a pointing device such as a mouse, a trackball, a touchpad, or a graphics tablet, a scanner, a touch screen integrated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computing device 710 or a communications network.

[0083] The user interface output devices 720 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computing device 710 to a user or to another machine or computing device.

[0084] Storage subsystem 724 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 724 may include logic for performing selected aspects of the methods of Figures 5 and 6 and / or for implementing the various components shown in Figures 2 and 3.

[0085] These software modules are generally executed by the processor 714 alone or in combination with other processors. The memory 725 used within the storage subsystem 724 may include several memories, including a main random access memory (RAM) 730 for storing instructions and data during program execution, and a read-only memory (ROM) 732 in which fixed instructions are stored. The file storage subsystem 726 may provide persistent storage for program and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules implementing the functionality of a particular implementation may be stored by the file storage subsystem 726 within the storage subsystem 724 or within other machines accessible by the processor 714.

[0086] The bus subsystem 712 provides a mechanism for allowing the various components and subsystems of the computing device 710 to communicate with each other as intended. Although the bus subsystem 712 is shown generally as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0087] The computing device 710 may be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of the computing device 710 shown in FIG. 7 is intended only as a specific example for purposes of illustrating some implementations. Many other configurations of the computing device 710 may have more or fewer components than the computing device shown in FIG.

[0088] In some implementations, a method implemented by one or more processors is provided, comprising: receiving a call by a general automated assistant, where receiving the call causes the general automated assistant to be invoked; receiving, via the invoked general automated assistant, a verbal query captured in audio data generated by one or more microphones of the client device; identifying historical interaction data of a user, where the historical interaction data is generated based on one or more past queries of the user and one or more responses generated by a plurality of secondary automated assistants within a time period in response to the one or more past queries; and determining, based on comparing a transcription of the verbal query with the historical interaction data, that a relationship exists between the verbal query and a portion of the historical interaction data associated with a particular automated assistant of the plurality of secondary automated assistants. In response to determining that a relationship exists between the verbal query and the particular automated assistant, the method further comprises providing an indication of the audio data to the particular automated assistant instead of providing the indication of the audio data to any other automated assistant of the secondary automated assistants, where providing the indication of the audio data causes the particular automated assistant to generate a response to the verbal query.

[0089] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0090] In some implementations, the method further includes determining, based on the historical interaction data, that an additional relationship exists between the verbal query and an additional portion of the historical interaction data associated with a second particular automated assistant of the plurality of secondary automated assistants; determining, based on the historical interaction data, a second assistant time for the second assistant historical interaction of the user with the second particular automated assistant; determining, based on the historical interaction data, a first assistant time for the first assistant historical interaction of the user with the particular automated assistant; ranking the particular automated assistant and the second particular automated assistant based on the first assistant time and the second assistant time; and selecting the particular automated assistant based on the ranking, wherein providing the instruction of the audio data to the particular automated assistant instead of providing the instruction of the audio data to any other automated assistant of the secondary automated assistants is further based on selecting the particular automated assistant based on the ranking. In some of those implementations, the method further includes determining a first similarity score for the relationship and a second similarity score for the additional relationship, and ranking the particular automated assistant and the second particular automated assistant is further based on the first similarity score and the second similarity score.

[0091] In some implementations, the method further includes determining a classification of the verbal query, and providing a transcription of the verbal query to the specific automated assistant is further based on the classification. In some of those implementations, the method further includes identifying a user's indicated preference to utilize a specific automated assistant for one or more of the past queries that also have a classification based on the historical interaction data, and providing the audio data to the specific automated assistant is further based on the user's preference. In other of those implementations, the method further includes generating instructions for a general automated assistant to provide the audio data to the specific automated assistant, and storing the instructions with the classification instructions. In some of those examples, the method further includes determining a current context of the audio data, and storing the current context with instructions for a general automated assistant to provide the audio data to the specific automated assistant.

[0092] In some implementations, a particular automated assistant performs automatic speech recognition on an instruction of the audio data to generate a text representation of the audio data, generates natural language processing of the text representation, and generates a response based on the natural language processing. In some of those implementations, generating the response includes providing a result of the natural language processing to a third-party application, receiving a response from the third-party application, and generating the response based at least in part on the response from the third party.

[0093] In some implementations, the invocation indicates a particular automated assistant and the indication of the audio data is an audio representation of the audio data.

[0094] In situations where certain implementations discussed herein may collect or use personal information about users (e.g., user data extracted from other electronic communications, information about the user's social network, the user's location, the user's time, the user's biometric information, and user activity and demographic information, relationships between users, etc.), users are provided with one or more opportunities to control whether information is collected, whether the personal information is stored, whether the personal information is used, and how information about the user is collected, stored, and used. That is, the systems and methods discussed herein collect, store, and / or use a user's personal information only if they receive explicit permission to do so from the associated user.

[0095] For example, a user is provided with control over whether a program or feature collects user information about that particular user or other users associated with the program or feature. Each user from whom personal information is collected is presented with one or more options that allow control of the collection of information associated with that user, to provide permission or approval as to whether information is collected and what portions of the information should be collected. For example, a user may be provided with one or more such control options over a communications network. In addition, certain data may be processed in one or more ways before being stored or used, such that personally identifiable information is removed. As one example, a user's identification information may be processed such that personally identifiable information cannot be determined. As another example, a user's geographic location may be generalized to a larger region such that the user's specific location cannot be identified.

[0096] Although several implementations have been described and illustrated herein, various other means and / or structures for performing the functions and / or obtaining one or more of the results and / or advantages described herein may be utilized, and each such change and / or modification is considered to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are meant to be exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on the particular application in which the teachings are used. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific implementations described herein. Thus, the foregoing implementations have been presented by way of example only, and it should be understood that within the scope of the appended claims and their equivalents, the implementations may be practiced otherwise than as specifically described and claimed. Implementations of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent. [Explanation of symbols]

[0097] 101 users 105 first standalone interactive speaker, first speaker 110 Second standalone interactive speaker, second speaker 115 Oral Speech 205 first client device, client device 210 Second client device 215 First Automated Assistant, First Assistant 220 Second Automated Assistant 225 Microphone 230 Microphone 245 The Third Automated Assistant 305 Primary Automated Assistant 310 Secondary Automated Assistant 315 Secondary Automated Assistant 320 Database 325 Assistant Decision Module 330 Query Classifier 335 Call Engine 340 Query Processing Engine 345 API 350 Ability Communicator 390 API 505 Oral utterances, speech 510 Response 515 Oral speech, speech 515E Memorized Dialogue 520 Response 525 Oral utterances, speech 530 Response 535 Oral utterances, speech 535E Dialogue 540 Response 710 Computing Devices 712 Bus Subsystem 714 Processor 716 Network Interface Subsystem 720 User Interface Output Devices 722 User Interface Input Devices 724 Memory Subsystem 725 Memory Subsystem 726 File Storage Subsystem 730 Main Random Access Memory (RAM) 732 Read-Only Memory (ROM)

Claims

1. 1. A method implemented by one or more processors of a client device, the method comprising: receiving a call by a generic automated assistant, where receiving the call causes the generic automated assistant to be invoked; receiving, via the invoked generic automated assistant, a verbal query captured in audio data generated by one or more microphones of the client device; identifying historical interaction data for a user, the historical interaction data being generated based on one or more past queries of the user and one or more responses generated by a plurality of secondary automated assistants within a period of time in response to the one or more past queries; determining, based on comparing a transcription of the spoken query to the historical interaction data, that a relationship exists between the spoken query and a portion of the historical interaction data associated with a particular automated assistant among the plurality of secondary automated assistants; In response to determining that the relationship exists between the spoken query and the particular automated assistant, providing an indication of the audio data to the particular automated assistant in lieu of providing the indication to any other automated assistants of the secondary automated assistants, where providing the indication of the audio data causes the particular automated assistant to generate a response to the verbal query; Including, method.

2. Determining, based on the historical interaction data, that an additional relationship exists between the verbal query and an additional portion of the historical interaction data associated with a second particular automated assistant of the plurality of secondary automated assistants; Determining a second assistant time for a second assistant historical interaction between the user and the second particular automated assistant based on the historical interaction data; Determining a first assistant time for a first assistant historical interaction between the user and the particular automated assistant based on the historical interaction data; ranking the particular automated assistant and the second particular automated assistant based on the first assistant time and the second assistant time; selecting the particular automated assistant based on the ranking; Further comprising: providing the indication of the audio data to the particular automated assistant instead of providing the indication to any other automated assistant of the secondary automated assistants is further based on selecting the particular automated assistant based on the ranking; The method of claim 1.

3. The method further includes determining a first similarity score for the relationship and a second similarity score for the additional relationship, and the step of ranking the specific automated assistant and the second specific automated assistant is further based on the first similarity score and the second similarity score. The method of claim 2.

4. determining a classification of the spoken query, and providing the transcription of the spoken query to the specific automated assistant is further based on the classification; The method of claim 1.

5. Further comprising: identifying an indicated preference of the user to utilize the particular automated assistant for one or more of the past queries that also have the classification based on the historical interaction data; and providing the audio data to the particular automated assistant further based on the user's preference. The method of claim 4.

6. generating instructions for the general automated assistant to provide the audio data to the specific automated assistant; storing said indication together with an indication of said classification; 5. The method of claim 4, further comprising:

7. determining a current context of the audio data; storing the current context together with the instructions of the general automated assistant to provide the audio data to the specific automated assistant; 7. The method of claim 6, further comprising:

8. 2. The method of claim 1, wherein the particular automated assistant performs automatic speech recognition on the indication of the audio data to generate a text representation of the audio data, generates natural language processing of the text representation, and generates a response based on the natural language processing.

9. Producing a response is providing results of the natural language processing to a third party application; and receiving a response from the third party application; and generating the response based at least in part on the response from the third party application.

10. 10. The method of claim 1, wherein the invocation indicates the particular automated assistant and the indication of the audio data is an audio representation of the audio data.

11. 1. A system comprising one or more processors and a memory storing instructions, the instructions causing the one or more processors to, in response to execution of the instructions by the one or more processors: an act of receiving a call by a generic automated assistant, the act of receiving the call causing the generic automated assistant to be invoked; receiving, via the invoked generic automated assistant, a verbal query captured within audio data generated by one or more microphones of a client device; an operation of identifying historical interaction data for a user, the historical interaction data being generated based on one or more past queries of the user and one or more responses generated by a plurality of secondary automated assistants within a period of time in response to the one or more past queries; determining, based on comparing a transcription of the spoken query to the historical interaction data, that a relationship exists between the spoken query and a portion of the historical interaction data associated with a particular automated assistant among the plurality of secondary automated assistants; In response to determining that the relationship exists between the spoken query and the particular automated assistant, providing an indication of the audio data to the particular automated assistant in lieu of providing an indication of the audio data to any other automated assistants of the secondary automated assistants, the act of providing the indication of the audio data causing the particular automated assistant to generate a response to the verbal query; A system that executes the above.

12. The instructions cause the one or more processors to: determining, based on the historical interaction data, that an additional relationship exists between the verbal query and an additional portion of the historical interaction data associated with a second particular automated assistant of the plurality of secondary automated assistants; determining a second assistant time for a second assistant historical interaction between the user and the second particular automated assistant based on the historical interaction data; determining a first assistant time for a first assistant historical interaction between the user and the particular automated assistant based on the historical interaction data; ranking the particular automated assistant and the second particular automated assistant based on the first assistant time and the second assistant time; selecting the particular automated assistant based on the ranking; Then, providing the indication of the audio data to the particular automated assistant instead of providing the indication to any other automated assistant of the secondary automated assistants is further based on selecting the particular automated assistant based on the ranking; The system of claim 11.

13. 12. The system of claim 11, wherein the particular automated assistant performs automatic speech recognition on the indication of the audio data to generate a text representation of the audio data, generates natural language processing of the text representation, and generates a response based on the natural language processing.

14. Producing a response is providing results of the natural language processing to a third party application; and receiving a response from the third party application; and generating the response based at least in part on the response from the third party application.

15. 15. The system of claim 11, wherein the invocation indicates the particular automated assistant and the indication of the audio data is an audio representation of the audio data.

16. in response to execution of instructions by one or more processors, an act of receiving a call by a generic automated assistant, the act of receiving the call causing the generic automated assistant to be invoked; receiving, via the invoked generic automated assistant, a verbal query captured within audio data generated by one or more microphones of a client device; an operation of identifying historical interaction data for a user, the historical interaction data being generated based on one or more past queries of the user and one or more responses generated by a plurality of secondary automated assistants within a period of time in response to the one or more past queries; determining, based on comparing a transcription of the spoken query to the historical interaction data, that a relationship exists between the spoken query and a portion of the historical interaction data associated with a particular automated assistant among the plurality of secondary automated assistants; In response to determining that the relationship exists between the spoken query and the particular automated assistant, providing an indication of the audio data to the particular automated assistant in lieu of providing an indication of the audio data to any other automated assistants of the secondary automated assistants, the act of providing the indication of the audio data causing the particular automated assistant to generate a response to the verbal query; At least one computer-readable storage medium containing instructions for executing the method.

17. The instruction: Determining, based on the historical interaction data, that an additional relationship exists between the verbal query and an additional portion of the historical interaction data associated with a second particular automated assistant of the plurality of secondary automated assistants; Determining a second assistant time for a second assistant historical interaction between the user and the second particular automated assistant based on the historical interaction data; Determining a first assistant time for a first assistant historical interaction between the user and the particular automated assistant based on the historical interaction data; ranking the particular automated assistant and the second particular automated assistant based on the first assistant time and the second assistant time; selecting the particular automated assistant based on the ranking; Further comprising: Providing the indication of the audio data to the particular automated assistant instead of providing the indication to any other automated assistant of the secondary automated assistants is further based on selecting the particular automated assistant based on the ranking.

17. At least one computer-readable storage medium as recited in claim 16.

18. 17. At least one computer-readable storage medium as described in claim 16, wherein the particular automated assistant performs automatic speech recognition on the indication of the audio data to generate a text representation of the audio data, generates natural language processing of the text representation, and generates a response based on the natural language processing.

19. Producing a response is Providing the results of the natural language processing to a third-party application; and receiving a response from the third party application; and generating the response based at least in part on the response from the third party application.

20. 20. At least one computer-readable storage medium according to any one of claims 16 to 19, wherein the invocation indicates the particular automated assistant and the indication of the audio data is an audio representation of the audio data.

Images