Transfer of conversation data from the first-invoked automated assistant to the second-invoked automated assistant

Abstract

A system and method for providing interaction data from an initially invoked automated assistant to a subsequently invoked automated assistant. A first automated assistant may be invoked by a user's utterance, followed by an interaction with the user that is processed by the first automated assistant. During the interaction, a request is received to transfer the interaction data to a second automated assistant. The request may originate from the user, by the first automated assistant and / or by the second automated assistant. Once approved, the first automated assistant provides the prior interaction data to the second automated assistant. The second automated assistant performs one or more actions based on the interaction data.

Description

[Technical field]

[0001] This relates to transferring dialogue data from an initially invoked automated assistant to a subsequently invoked automated assistant. [Background technology]

[0002] Humans may participate in human-computer interactions with interactive software applications, referred to herein as "automated assistants" (also referred to as "digital agents," "chatbots," "interactive personal assistants," "intelligent personal assistants," "assistant applications," "conversational agents," etc.). For example, a human (who may be referred to as a "user" when interacting with an automated assistant) may provide commands and / or requests to the automated assistant using spoken natural language input (i.e., utterances), which may in some cases be converted to text and then processed, and / or may provide commands and / or requests by providing natural language input that is textual (e.g., typed). The automated assistant responds to the request by providing a responsive user interface output, which may include audible and / or visual user interface output,

[0003] As described above, many automated assistants are configured to interact via spoken utterances following invocation of the assistant. To protect user privacy and / or conserve resources, the user must often explicitly invoke the automated assistant before the automated assistant processes all of the spoken utterances. Explicit invocation of an automated assistant typically occurs in response to a particular user interface input being received at a client device. The client device includes an assistant interface that provides a user of the client device with an interface for interfacing with the automated assistant (e.g., receiving spoken and / or typed inputs from the user and providing audible and / or graphical responses) and with one or more additional components that implement the automated assistant (e.g., a remote server device that processes the user inputs and generates appropriate responses).

[0004] Some user interface inputs that can invoke an automated assistant via a client device include hardware and / or virtual buttons on the client device (e.g., pressing a hardware button, selecting a graphical interface element displayed by the client device) to invoke the automated assistant. Additionally or alternatively, many automated assistants can be invoked in response to one or more spoken invocation phrases, also known as "hot words / phrases" or "trigger words / phrases." For example, spoken invocation phrases such as "hey assistant," "okay assistant," and / or "assistant" can be spoken to invoke an automated assistant.

[0005] In many cases, a client device that includes an assistant interface includes one or more locally stored models that the client device uses to at least selectively monitor for the occurrence of a spoken invocation phrase. Such a client device can locally process received audio data using the locally stored models and discard any audio data that does not include the spoken invocation phrase. However, when the local processing of the received audio data indicates the occurrence of a spoken invocation phrase, the client device will cause that audio data and / or subsequent audio data to be further processed by the automated assistant. For example, if the spoken invocation phrase is "hey assistant" and the user speaks "hey assistant, what time is it," audio data corresponding to "what time is it" can be processed by the automated assistant based on the detection of "hey assistant" and used to provide the automated assistant's response of the current time. On the other hand, if the user simply speaks "What time is it?" (without first speaking the call phrase or providing an alternative call input), no response will be provided by the automated assistant as a result of "What time is it?" not being preceded by a call phrase (or other call input). Summary of the Invention [Problem to be solved by the invention]

[0006] Techniques are described herein for providing interaction data from a first automated assistant to a second automated assistant in response to a request to transfer the interaction data. For example, various techniques are directed to processing an interaction between a user and the first automated assistant by an initially invoked automated assistant, receiving a request to transfer processing of a query that is part of the interaction to a second automated assistant, and providing interaction data indicative of the interaction data to the second automated assistant for further processing. A user may initially invoke an automated assistant and issue one or more queries to which the invoked automated assistant may generate a response. At some point during the interaction between the user and the initially invoked automated assistant, a request may be received indicating that the user has an interest in additional interactions and / or an interest in a query that is being processed by the second automated assistant. The interaction and / or interaction data related to the interaction may be provided from the initially invoked automated assistant to the second assistant, and the second assistant may perform one or more actions, such as processing the query and generating a response. In these or other manners, the second assistant can obtain the interaction and / or interaction data without having to engage in a computationally laborious interaction with the user. Moreover, the initially invoked automated assistant can guide the interaction between the user and the assistant through selectively forwarding the interaction and / or interaction data to the second assistant. Furthermore, the second assistant can, in some implementations, generate responses that could not be generated by the initially invoked automated assistant alone, thereby improving the robustness of the interaction between the user and the assistant. [Means for solving the problem]

[0007] In some implementations, a user can utter an invocation phrase, such as "Okay, Assistant," that can invoke a first automated assistant (also referred to herein as the "first invoked automated assistant"), but that does not explicitly invoke other automated assistants with which the first invoked automated assistant can at least selectively interact in processing a query received in association with the invocation phrase (e.g., immediately after, immediately before, etc.). Thus, a user can specify to use the first invoked automated assistant, rather than individually invoking one of the other automated assistants, based on providing an invocation input that is specific to the first invoked automated assistant. For example, a first invocation phrase (e.g., "Okay, Assistant A"), when detected, can exclusively invoke the first automated assistant without invoking any other automated assistant. Similarly, a second invocation phrase (e.g., "Okay, Assistant B"), when detected, can exclusively invoke the second automated assistant without invoking the first invoked automated assistant and / or any other automated assistant. Once invoked, the initially invoked assistant may at least selectively interact with other automated assistants (i.e., "secondary assistants") in processing input provided in connection with the invocation.

[0008] A user can invoke an automated assistant by uttering an invocation phrase that may enable the user to interact with the initially invoked automated assistant once the automated assistant is initially invoked. The interaction may include one or more queries issued by the user and one or more responses subsequently generated by the initially invoked automated assistant. As an example, a user may initially invoke Assistant 1 by uttering, "Okay, Assistant 1, what are some good kids movies?" In this case, "Okay, Assistant 1" may be a hotword that invokes the first automated assistant, and the first automated assistant may generate and provide the response, "Movie A is a good kids movie." In response, the user may issue additional queries, either related or unrelated to the first query, to generate an interaction between the initially invoked automated assistant and the user. For example, the user may issue the query, "What are some other good movies?" to be provided with additionally generated responses related to "good kids movies" from the previous query. Also, for example, the user may then say, "I wonder what the weather's like today," and the initially invoked automated assistant may provide a response that is unrelated to the user's prior query and / or the initially invoked automated assistant's prior response.

[0009] As another example, an automated assistant may respond with "Movie A is a good kids movie," to which the user may follow up with the query "Okay, play it on a streaming service." In this case, the automated assistant may determine the intent corresponding to the query based on a prior interaction between the user and the automated assistant, which may include determining what the user meant by the term "it," such as "Movie A." In response to the follow-up query "Play it on a streaming service," the originally invoked automated assistant may provide data to "Streaming Service," a streaming media application that can stream "Movie A" to the user via a user interface.

[0010] As yet another example, once a user has invoked the initially invoked automated assistant, the user can utter the phrase "What's the weather like in San Francisco?", and the initially invoked automated assistant can generate a response to the query, such as "Today's weather in San Francisco is about 50 degrees Fahrenheit and cloudy." In some cases, the dialogue may include additional queries of the user that are related to the previous query. For example, the user may utter a follow-up query, "What's Miami like?", indicating an intent to be provided with the weather in Miami. In response, the initially invoked automated assistant can respond, "The weather in Miami is about 80 degrees Fahrenheit and sunny." The dialogue may continue between the user and the initially invoked automated assistant, both with related queries and with queries that include unrelated intent.

[0011] In some implementations, multiple automated assistants may exist in the same environment. For example, a user may invoke an automated assistant running on a device that is in proximity to a second device running a second automated assistant. Also, for example, a user may invoke an automated assistant running on a device that is also running a second automated assistant. As described above, a user may invoke a first automated assistant by uttering a hotword or phrase that activates the first automated assistant but does not invoke other automated assistants. For example, a user may utter the hotword "okay, assistant 1" to invoke a first automated assistant, without invoking a second automated assistant. Similarly, a user may utter the hotword "okay, assistant 2" to invoke a second automated assistant without invoking the first automated assistant.

[0012] In some implementations, an automated assistant can provide other automated assistants in the environment with types of queries and / or commands that the automated assistant can process. For example, an automated assistant may have access to a calendar application, and as a result, the automated assistant can provide an indication to other automated assistants that queries related to the user's calendar can be processed by the automated assistant itself. Also, for example, an automated assistant can indicate to other automated assistants via one or more signals that the automated assistant can stream media via a media streaming application. If the invoked automated assistant is unable to process a query, the other automated assistants can be provided with information regarding the capabilities of other automated assistants so that the automated assistant unable to process the query can learn of other automated assistants in the same environment that may be able to successfully process the query and provide a response to the user. For example, one or more automated assistants can broadcast an ultrasound signal indicating the types of queries and / or applications that the broadcasting automated assistant can process. Also, for example, an automated assistant may be able to learn of the capabilities of other automated assistants in its environment by providing an indication of its own capabilities via Wi-Fi, an application programming interface (API), and / or other communication channels.

[0013] In some implementations, an automated assistant may proactively broadcast its capabilities. In such a case, when the first-invoked automated assistant receives a query that it cannot handle, it may search for other automated assistants that broadcast their capabilities to handle that type of request. Proactive broadcasting of capabilities may reduce the delay in determining which, if any, second assistants to forward the interaction and / or interaction data to, which may result in reduced delay in the second assistant generating a response. In some implementations, when the first-invoked automated assistant receives a request that it cannot handle, it may broadcast a request to the other automated assistants to determine whether the request can be handled by one or more other automated assistants. For example, when an first-invoked automated assistant receives a request related to a calendar application to which it does not have access, it may broadcast a message to the other automated assistants asking whether the request can be handled by a different automated assistant. In response, an automated assistant that can handle the request may respond in the affirmative, thereby informing the first-invoked automated assistant that the request can be handled by the automated assistant.

[0014] In some cases, the first-invoked automated assistant may not be configured to generate a response to the user's query. For example, the first-invoked automated assistant may receive a query from a user, process the query, and generate a response to the query. For a follow-up query, the first-invoked automated assistant may not be able to generate a substantive response when the automated assistant is not configured to respond to the type of query issued by the user and / or due to an explicit request by the user, and may move the generation of the response to another automated assistant. In such a case, the first-invoked automated assistant may forward a portion of the prior interaction, or data representing at least a portion of the prior interaction, to the second automated assistant for further processing. Based on the intent of the query and / or the prior interaction, the second automated assistant may generate a response and further perform one or more actions based on the generated response. For example, the second automated assistant, once provided with the interaction and / or the interaction data from the prior interaction, may provide an audible response to the user, interact with an application, and / or perform other tasks based on the query.

[0015] If the first-invoked automated assistant is not configured to respond to a query, it can determine that other automated assistants may be more capable of fulfilling the query. For example, a user may utter, "Play movie A on a streaming service." If the first-invoked automated assistant may not have access to "streaming service," it can determine whether one or more other automated assistants are configured to handle such a request. For example, other automated assistants in the environment may broadcast ultrasonic signals indicative of such capabilities. For example, alternative automated assistants in the environment may have previously broadcast their capabilities to the first-invoked automated assistant, and the first-invoked automated assistant may have stored (e.g., stored locally) the capabilities to be associated with the alternative automated assistant. If an appropriate second automated assistant is identified by the first-invoked automated assistant (e.g., by accessing a stored association of capabilities to the second automated assistant), a transfer request of the interaction data may be provided to the second automated assistant. Once the request is granted by the second automated assistant, the originally invoked automated assistant can send interaction data including at least a portion of the interaction (e.g., the interaction and / or the intent of the query) to the second automated assistant. The second automated assistant can then process the query and perform one or more actions, such as launching a “streaming service” to provide movie A to the user via a user interface.

[0016] In some implementations, the first-invoked automated assistant can provide a suggestion to the user to transfer the dialogue data to the second automated assistant. For example, the user may provide a query to the first-invoked automated assistant for which the first-invoked automated assistant cannot process or otherwise provide a response that fulfills the request by the user. The first-invoked automated assistant can respond with a suggestion to the user that another automated assistant may be configured to provide a response, such as "I don't know, would you like me to check Assistant 2?" In response, the user can utter "yes," and the first-invoked automated assistant can provide the dialogue data to Assistant 2, which can then be further handled by Assistant 2. In this case, the transfer request (i.e., the utterance "yes") is provided directly by the user, and the invocation of the second automated assistant is provided by the first-invoked automated assistant.

[0017] In some implementations, forwarding suggestions can be provided to the user based on identifying a user's preference and determining that the query is of a type in which the user is interested and that can be fulfilled by a different automated assistant. For example, the initially invoked automated assistant can determine that the user previously used a different automated assistant, such as an automated assistant that has access to the user's calendar application, to handle a query related to a calendar. When the initially invoked automated assistant receives a request related to the calendar application as part of a dialogue, it can provide at least a portion of the dialogue and / or data indicative of the dialogue to the automated assistant preferred by the user to handle the request.

[0018] In some implementations, the forwarding request can be based on user preferences. For example, the user can indicate, via one or more user preferences, that all queries related to the calendar application be handled by the first automated assistant. If a query related to the calendar application is received by another automated assistant, the invoked automated assistant can treat the user's setting of a user preference related to the calendar query as a request to forward interactions for all queries related to the calendar application to the first automated assistant.

[0019] In some implementations, a user can continue a dialogue with a second automated assistant after first invoking another automated assistant. For example, a user may invoke an automated assistant with "Okay, Assistant 1" and provide the query "What's the weather like in Miami?" The first-invoked automated assistant may respond to the query with "The weather is about 75 degrees Fahrenheit and sunny," etc. The user may then utter "Okay, Assistant 2, what about San Francisco," indicating that the user is interested in the second assistant (i.e., the automated assistant invoked with "Okay, Assistant 2") to further process the dialogue. The first-invoked automated assistant may determine that the dialogue includes a reference to the second automated assistant and may provide prior dialogue data to the second automated assistant to enable the second automated assistant to determine the context and / or intent of the query directed to it. For example, the query from the previous example does not explicitly include "What's the weather like in San Francisco?", but the intent to be provided with weather information may be included in the previous interaction based on the relationship of that query to the previous query (i.e., "What's the weather like in Miami?").

[0020] In some implementations, a later-invoked automated assistant may determine that it does not have the conversational context and / or intent necessary to fulfill the query. For example, a user may invoke Assistant 1 with the query, "What movie should I watch?", to which Assistant 1 may respond, "Movie A is a good movie." The user may then utter, "Okay, Assistant 2, play it on a streaming service." Because the movie title (i.e., "Movie A") is not part of the dialogue with Assistant 2, Assistant 2 may request from Assistant 1 the context, intent, and / or portions of the previous dialogue with Assistant 1 so that it can determine what the user intended by the term "it" in the query, "play it on a streaming service." Once Assistant 1 provides the context of the previous dialogue, Assistant 2 may then determine that "it" refers to "Movie A" and launch the "streaming service" to play the requested movie.

[0021] In some implementations, speaker verification may be performed before the first-invoked automated assistant provides dialogue data to the later-invoked automated assistant. For example, a user may say, "Okay, Assistant 1, what's a good movie to buy?" and Assistant 1 may provide a response of, "Movie A has a good rating." The user may then say, "Okay, Assistant 2, buy it at the online store." Assistant 2 may request a portion of the prior dialogue to determine the context of the query directed to Assistant 2, along with audio data that may include, for example, the user saying, "Buy it at the online store." Before providing dialogue data and / or context of the prior dialogue to Assistant 2, Assistant 1 may perform verification using audio data from Assistant 2 to determine whether the speaker of the prior dialogue is the same speaker who said, "Buy it at the online store." For example, Assistant 1 may perform text-independent speaker verification using one or more on-device models that generate speaker vectors to verify the speaker associated with the prior dialogue. Alternatively or in addition, Assistant 1 can perform text-dependent speaker verification (TDSV) by processing the audio data with an on-device TDSV model to generate a speaker vector and verify that the speaker vector matches a speaker vector associated with the prior interaction. If speaker verification determines that the same speaker issued the query to Assistant 2 as in the prior interaction with Assistant 1, Assistant 1 can provide interaction data as described above. If the speaker in both interactions cannot be verified as the same speaker, Assistant 1 can provide an indication to Assistant 2 that the speaker is not the same and / or that no additional interaction data is available.

[0022] In some implementations, the confirmation may include determining whether the query directed to the later-invoked automated assistant is a continuation of a prior interaction. For example, the interaction with the initially-invoked automated assistant may include a user uttering "What's the weather like in San Francisco?" followed by a response from the initially-invoked automated assistant. The user may then utter "Okay, Assistant 2, what about Miami?", which is directed to the later-invoked automated assistant. The later-invoked automated assistant may provide at least a portion of the audio (or data indicative of the interaction and speaker) to the initially-invoked automated assistant. To confirm, the initially-invoked automated assistant may determine whether the utterance is related to one or more prior interactions with the user before providing the interaction data to the later-invoked automated assistant. For example, the later-invoked automated assistant may send audio data of the user uttering "What about Miami?", and the initially-invoked automated assistant may determine that the utterance may be related to the context of a prior interaction regarding the weather in San Francisco.

[0023] Thus, the techniques described herein reduce the need for a user to explicitly invoke a second automated assistant when the first invoked automated assistant is unable to fulfill a request or is otherwise unable to respond to a request from the user. This results in a reduction in the overall duration of a user's interaction with multiple automated assistants, with a corresponding reduction in network and / or computational resource usage. Furthermore, if the first invoked automated assistant knows of another automated assistant that would be objectively more likely to return a more complete response, the user can learn of the capabilities of the second automated assistant without the user being required to invoke the second automated assistant to determine whether a more complete response is possible. By allowing a query to be sent to the second automated assistant, the need for a user to prolong a conversation with an automated assistant is reduced by aiding the user in receiving a response from the assistant that is most likely to respond affirmatively to a query, without requiring the user to use resources required to resume a dialogue with the second automated assistant. Furthermore, by providing interaction data to the second automated assistant, it is unnecessary for the second assistant to prompt the user for information that has already been provided. This allows for context for the interaction to be provided without requiring the user to repeat information that has already been provided to the first automated assistant.

[0024] The above description is provided as an overview of some implementations of the present disclosure. Further descriptions of these and other implementations are described in more detail below. [Brief description of the drawings]

[0025] [Figure 1] 1 is an illustration of an example environment in which implementations disclosed herein may be implemented. [Diagram 2] FIG. 1 is a block diagram of an example environment in which various methods disclosed herein may be implemented. [Diagram 3] FIG. 1 is a block diagram of another example environment in which various methods disclosed herein may be implemented. [Figure 4A] 1 illustrates the transfer of interaction data from a first automated assistant to a second automated assistant. [Figure 4B] 1 illustrates the transfer of a dialogue from an initially invoked automated assistant to a later invoked automated assistant after verifying the speaker. [Figure 5A] FIG. 2 is a diagram of an interaction involving a user, a first automated assistant, and a second automated assistant, according to various implementations disclosed herein. [Figure 5B] FIG. 2 is a diagram of an interaction involving a user, a first automated assistant, and a second automated assistant, according to various implementations disclosed herein. [Figure 5C] FIG. 2 is a diagram of an interaction involving a user, a first automated assistant, and a second automated assistant, according to various implementations disclosed herein. [Figure 6] 1 is a flowchart illustrating an example method according to various implementations disclosed herein. [Figure 7] 1 is a flowchart illustrating an example method according to various implementations disclosed herein. [Figure 8] 1 is a flowchart illustrating an example method according to various implementations disclosed herein. [Figure 9] 1 illustrates an example architecture of a computing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] Referring to FIG. 1, an exemplary environment is provided that includes multiple automated assistants that can be invoked by a user 101. The environment includes a first standalone interactive speaker 105 with a microphone (not shown) and a second standalone interactive speaker 110 with a microphone (also not shown). The first speaker can be running a first automated assistant that can be invoked, at least in part, using an invocation phrase. The second speaker 110 can be running a second automated assistant that can be invoked using an invocation phrase, either the same invocation phrase as the first automated assistant or a different phrase that allows the user to select which automated assistant to invoke based on the spoken phrase. In this exemplary environment, the user 101 is speaking a spoken utterance 115 of "Okay, assistant, what's a good animated movie" in the vicinity of the first speaker 105 and the second speaker 110. If one of the first and / or second automated assistants is configured to be invoked by the phrase "Okay, assistant," the invoked assistant may process the query that follows the invocation phrase (i.e., "What are some good anime movies?").

[0027] In some implementations, a device such as a first speaker 105 may run multiple automated assistants. Referring to FIG. 2, an exemplary environment including multiple client devices running multiple automated assistants is illustrated. The system includes a first client device 105 running a first automated assistant 215 and a second automated assistant 220. The first and second automated assistants may each be invoked by speaking an invocation phrase (either unique for each assistant or the same phrase to invoke both assistants) in the vicinity of the client device 105 such that audio can be captured by the microphone 225 of the client device 105. For example, a user 101 may invoke the first automated assistant 215 by speaking "Okay, assistant 1" in the vicinity of the client device 105, and may further invoke the second automated assistant 220 by speaking the phrase "Okay, assistant 2" in the vicinity of the client device 105. Based on which invocation phrase is spoken, the user can indicate which of multiple assistants running on the client device 105 the user is interested in processing the spoken query. The exemplary environment further includes a second client device 110 running a third automated assistant 245. The third automated assistant can be configured to be invoked with a third invocation phrase, such as "Okay, assistant 3," so that it can be captured by the microphone 230. In some implementations, one or more of the automated assistants of FIG. 2 may be missing. Additionally, the exemplary environment may include additional automated assistants not present in FIG. 2. For example, the system may include a third device running an additional automated assistant and / or the client device 110 and / or the client device 105 may have additional and / or fewer automated assistants running than illustrated.

[0028] Automated assistants 215, 220, and 245 may each include one or more components of the automated assistants described herein. For example, automated assistant 215 may include its own voice capture component for processing input queries, a visual capture component for processing input visual data, a hotword detection engine, and / or other components. In some implementations, automated assistants running on the same device, such as automated assistants 215 and 220, may share one or more components that may be used by both of these automated assistants. For example, automated assistant 215 and automated assistant 220 may share one or more of an on-device speech recognizer, an on-device NLU engine, and / or other components.

[0029] In some implementations, one or more automated assistants may share one or more modules, such as a natural language processor and / or natural language results, TTS, and / or STT processor. For example, referring back to FIG. 2, both the first automated assistant 215 and the second automated assistant 220 may share pre-language processing, such that when the client device 105 receives audio data, the audio data may be processed into text once and then provided to the automated assistants 215 and 220. Also, for example, one or more components of the client device 105 may process the audio data into text and provide a textual representation of the audio data to the third automated assistant 245, as described later herein. In some implementations, the audio data may not be processed into text, but instead be provided to one or more of the automated assistants as raw audio data.

[0030] 3, an example of two automated assistants 305 and 310 is illustrated. As illustrated, the first invoked automated assistant 305 is running on a first client device 301, and the later invoked automated assistant 310 is running on a second client device 302. However, in some implementations, both automated assistants 305 and 310 may be running on the same device, as illustrated in FIG.

[0031] The initially invoked automated assistant 305 can be invoked by one or more phrases captured via microphone 320 and processed by invocation engine 315. In some implementations, the initially invoked automated assistant 305 can be invoked by a user via one or more gestures captured by a camera (not shown) and processed by invocation engine 315. Invocation engine 315 can determine whether the user is interested in using the initially invoked automated assistant 305 based on spoken utterances and / or gestures. Similarly, the later invoked automated assistant 310 includes an invocation engine 385 that can determine whether the user is interested in interacting with the later invoked automated assistant 310 when uttering a phrase by a similar phrase and / or gesture that is specific to the invocation of the later invoked automated assistant 310.

[0032] The initially invoked automated assistant 305 includes a query processing engine 330 that can process a query submitted by a user, such as by a spoken utterance captured by the microphone 320. In some implementations, the user can proceed to interact with the initially invoked automated assistant 305. For example, the user can submit a query, "What's the weather like today?", and the query processing engine 330 can generate a response, and the initially invoked automated assistant 305 can respond to the query, such as, "It's sunny today, and it's about 80 degrees Fahrenheit." The user can then submit additional queries that may be related to the earlier query or unrelated to the earlier query. For example, the user can submit a follow-up query, "What will tomorrow be like?", indicating an interest in being provided with tomorrow's weather, and the query processing engine 330 can generate a response based on the interaction that has occurred with the user. In response, the initially invoked automated assistant 305 can respond with an appropriate weather forecast. Also, a user may submit an unrelated query, for example "what are some good movies to watch" that is unrelated to the previous query, and will be provided with an appropriate response that is unrelated to the previous query.

[0033] In some implementations, a user may submit a query that the initially invoked automated assistant 305 is unable to process. For example, if the user submits the query "What's in my calendar today" to the automated assistant 305, the automated assistant 105 may not have access to the user's 101 calendar application. In response, the query processing engine 330 may provide a negative response, such as "Sorry, I don't have access to your calendar." If this occurs, the user may be required to submit the query to a later invoked automated assistant 310 that may have access to the user's calendar. The automated assistant 310 may then provide a response, such as "You have a meeting at 3 o'clock today." If the initially invoked automated assistant 305 and the later invoked automated assistant 310 do not have access to the user's calendar application, the user may be required to further invoke another automated assistant or to access the calendar application directly. Thus, in such cases, computing resources are wasted in determining which automated assistant can provide the information of interest to the user.

[0034] 4A, a diagram is provided illustrating the flow of data between a user 101, an initially invoked automated assistant 305, and a later invoked automated assistant 310. Initially, an interaction 415 can occur between the user 101 and the initially invoked automated assistant 305. For example, the user can utter one or more phrases that invoke the initially invoked automated assistant 305, such as "Okay, Assistant A." The phrase can be a phrase that invokes only the initially invoked automated assistant 305 and not the later invoked automated assistant 310.

[0035] In some cases, the first-invoked automated assistant 305 may not be configured to handle the user's 101 request. In some implementations, the first-invoked automated assistant 305 may suggest to the user to forward the request to the later-invoked automated assistant 310. For example, the user 101 may submit a query of "Play that on a streaming service" and the first-invoked automated assistant 305 may not be configured to communicate with the "streaming service" and / or may not otherwise be able to communicate with the "streaming service". However, the later-invoked automated assistant 310 may provide an indication that it is configured to be able to communicate with the "streaming service", such as an ultrasonic signal broadcast by the speaker 370 and picked up by the microphone 320. In such a case, the first-invoked automated assistant 305 may respond with a response that suggests the later-invoked automated assistant 310, such as "Sorry, I can't do that. Would you like me to ask assistant B?" In some implementations, this suggestion may not be provided to the user 101. Instead, the initially invoked automated assistant 305 may determine that the request "Play that on a streaming service" is a request to transfer the interaction to the subsequently invoked automated assistant 310. Thus, in some cases, a positive response to a transfer suggestion 420 (e.g., "Yes" in response to "Do you want me to ask Assistant B to do that") may be a transfer request 425. Additionally, in the absence of a transfer suggestion 420, a request by the user 101 to perform an action that the initially invoked automated assistant 305 is not configured to perform may be a transfer request (e.g., "Play that on a streaming service").Also, for example, a request to the automated assistant to perform an action that was previously set as a user preference may be a forwarding request if that action is requested.

[0036] Once the user 101 has requested the transfer and / or confirmed the suggestion of the transfer, the initially invoked automated assistant 305 can invoke (435) the later invoked automated assistant 310. In some implementations, the initially invoked automated assistant 305 can communicate with one or more additional automated assistants running on the same or different devices via the assistant collaboration module 325 of the initially invoked automated assistant 305, such as via the assistant collaboration module 375 of the later invoked automated assistant 310. For example, the initially invoked automated assistant 305 can be invoked by the user, and the assistant collaboration module 325 can use one or more protocols, such as Wi-Fi, Bluetooth, an ultrasonic signal broadcast by the client device 105 via speaker 350 and received by the later invoked automated assistant 310 via microphone 365, APIs 390 and 345, and / or other communication channels.

[0037] After invocation of the subsequently-invoked automated assistant 310, the initially-invoked automated assistant 305 can forward (440) at least a portion of the interaction data to the subsequently-invoked automated assistant 310. The interaction data can include audio data of the user's making request, which can then be further analyzed via ASR and / or NLU by the subsequently-invoked automated assistant 310. In some implementations, the initially-invoked automated assistant 305 can perform ASR and / or NLU, and the interaction data can include a textual representation of the interaction and / or NLU output.

[0038] In response to being provided with the interaction data, the query processing engine 340 can process the query, and the subsequently invoked automated assistant 310 can perform (430) one or more actions responsive to the interaction data provided by the initially invoked automated assistant 305. The actions may include, for example, accessing one or more applications, communicating with one or more other devices, performing a search, providing a synthesized voice output, and / or other actions that the subsequently invoked automated assistant 310 is configured to perform. Once the one or more actions are performed (430), the interaction may continue between the user and either the initially invoked automated assistant 305 and / or the subsequently invoked automated assistant 310.

[0039] As an example, referring to FIG. 5A, an interaction between a user 101 and two automated assistants 305 and 310 is illustrated. The illustrated interaction does not include a suggestion of forwarding. First, an interaction occurs between the user and the first-invoked automated assistant 305. The interaction begins with the user requesting, "Okay, assistant, what's a good animated movie?" 505, to which the first-invoked automated assistant 305 responds, "Movie A is a good animated movie" 510. In this case, the request can be handled by the first-invoked automated assistant 305. The user then requests, "Okay, assistant, play it on a streaming service" 515. The first-invoked automated assistant 305 can determine that it is not configured to access and / or utilize the "streaming service," and the later-invoked automated assistant 310 can process the request (e.g., is configured to communicate with the "streaming service"). The initially invoked automated assistant 305 can invoke the later invoked automated assistant 310 and provide dialogue data that enables the later invoked automated assistant 310 to perform the action of playing "Movie A" on a "streaming service." For example, the initially invoked automated assistant 305 can provide audio data of the dialogue (e.g., dialogue portions 505, 510, and / or 515), a textual representation of the dialogue, and / or other representation of the intent of the dialogue between the user 101 and the initially invoked automated assistant 305. In response, the later invoked automated assistant 310 provides a synthesized voice output of "Now playing on a streaming service" 520 and plays Movie A on a streaming service.In this case, the forwarding request may include a prior indication by the user, via one or more user preferences and / or past behavior, that the query for “Streaming Services” be handled by a later-invoked automated assistant 310.

[0040] As another example, referring to FIG. 5B, an interaction between a user 101 and two automated assistants 305 and 310 is illustrated. In this illustrated interaction, the user 101 is provided with a suggestion to transfer at least a portion of the interaction with the first-invoked automated assistant 305 to the later-invoked automated assistant 310. Initially, an interaction occurs between the user 101 and the first-invoked automated assistant 305. The interaction begins with the user requesting, "Okay, assistant, what's in my calendar?" 525. The first-invoked automated assistant 305 does not have access to the user's calendar application and may determine, based on instructions provided by the other automated assistants, that the automated assistant 310 has access to the user's calendar application, as described herein. The first-invoked automated assistant 305 responds, "I can't access your calendar. Do you want me to check assistant 2?" 530. This response includes an explicit suggestion to the user 101 to transfer the dialogue to "Assistant 2," which refers to the later-invoked automated assistant 310. The user 101 responds with "Yes" 535, and the originally-invoked automated assistant 305 can then invoke the assistant 310 as described above, and further transfer all or a portion of the prior dialogue (or data indicative of the dialogue) to the later-invoked automated assistant 310. The later-invoked automated assistant 310, which has access to the user's calendar application, can continue the dialogue by responding (540) with information from the user's calendar.

[0041] 4B, a diagram is provided illustrating the flow of data between the user 101, the initially invoked automated assistant 305, and the later invoked automated assistant 310. Initially, a dialogue 445 can occur between the user 101 and the initially invoked automated assistant 305. For example, the user may utter one or more phrases that invoke the initially invoked automated assistant 305, such as "Okay, Assistant A." This phrase may be a phrase that invokes only the initially invoked automated assistant 305 and not the later invoked automated assistant 310. This dialogue may continue between the initially invoked automated assistant 305 and the user 101, such as "What's a good movie to watch?" from the user 101, followed by "Movie A is a good movie to watch" from the initially invoked automated assistant 305.

[0042] At some point in the interaction with the initially invoked automated assistant 305, the user may invoke the subsequently invoked automated assistant 310 by uttering a hotword that is specific to the subsequently invoked automated assistant 310, such as "Okay, Assistant B." The hotword may be followed by an interaction 450 that is directed to the subsequently invoked automated assistant 310, such as "Okay, Assistant B, play that on a streaming service." Thus, in this case, the user 101 has specifically invoked the automated assistant 310 and initiated a direct interaction with the assistant 310.

[0043] In some cases, the later-invoked automated assistant 310 may request information about a previous interaction with the first-invoked automated assistant 305 to determine the intent and / or context of the request. For example, in interaction 445, the user may say, "What's a good movie?" to which the first-invoked automated assistant 305 may respond, "Movie A is a good movie." The user 101 may then say, "Okay, Assistant B, play it on a streaming service," as an additional interaction 450. Once invoked, the later-invoked automated assistant 310 may process the user's 101 utterance. However, at this point, the later-invoked automated assistant 310 may not have the context to resolve the meaning of "it" in the user's request. Thus, the first-invoked automated assistant 305 may provide interaction data 465 to the later-invoked automated assistant 310 to determine the context and / or intent in interaction 450.

[0044] In some implementations, when the later-invoked automated assistant 310 is invoked and processes the user's 101 utterance, the verification module 395 can verify that the speaker of the utterance (i.e., the user 101) previously participated in a dialogue with the initially-invoked automated assistant 305. In some implementations, the verification module 335 can provide speaker verification data 455 to the verification module 395, which can determine whether the speaker of the earlier dialogue with the initially-invoked automated assistant 305 is the same speaker as the speaker of dialogue 445. In some implementations, the microphone 365 can generate audio data from the user's utterance during the earlier dialogue, and the verification module 335 can process the audio data to verify whether the speaker of dialogue 445 (captured by the microphone 365) is the same as the speaker of dialogue 450 (also captured by the microphone 365). Thus, in some cases, the interaction data 445 may already have been received by the subsequently invoked automated assistant 310 and no interaction data is required to be provided by the originally invoked automated assistant 305 .

[0045] The speaker confirmation data may include, for example, all or a portion of the interaction captured by the microphone 365 of the subsequently invoked automated assistant 310, the speaker's voice profile, and / or other data that may enable the confirmation module 395 to confirm that the speaker of interaction 450 is the same speaker as interaction 445. Once the speaker confirmation data 455 is provided, the initially invoked automated assistant 305 may determine whether the same speaker participated in the interaction with the initially invoked automated assistant 305. If the initially invoked automated assistant 305 determines that an interaction with a user has occurred, the initially invoked automated assistant 305 may provide a confirmation confirmation 460 to the subsequently invoked automated assistant 310. This confirmation indicates that the speaker has engaged with the initially invoked automated assistant 305, and the initially invoked automated assistant 305 may further provide interaction data 465 of the interaction that has occurred between the user 101 and the initially invoked automated assistant 305. In this way, the later-invoked automated assistant 310 can determine the context and resolve any ambiguities in the interaction 450 that was previously captured by the microphone associated with the later-invoked automated assistant 310. The later-invoked automated assistant 310 can then perform one or more actions 470 in response to the interaction 450, such as accessing an application and / or providing a synthesized voice in response.

[0046] As an example, referring to FIG. 5C , an interaction between a user 101, an initially invoked automated assistant 305, and a later invoked automated assistant 310 is illustrated. The user 101 provides a spoken utterance of “Okay, assistant 1, what are some good movies to buy?” 545, which is directed to the initially invoked automated assistant 305. The initially invoked automated assistant 305 responds to the utterance with “Movie B has a good rating” 550. In response, the user 101 provides an utterance of “Okay, assistant 2, buy it from the online store” 555, which is directed to the later invoked automated assistant 310 and is captured by one or more microphones of the client device running the later invoked automated assistant 310. The later-invoked automated assistant 310 can provide the initially-invoked automated assistant 305 with speaker confirmation data of the user issuing the utterance 555 so that the initially-invoked automated assistant 305 can confirm that the utterance 555 was provided by the same user as the utterance 545. In some implementations, once the same speaker is confirmed, the later-invoked automated assistant 310 can be provided with the prior interaction data (e.g., the utterance 545). In some implementations, the later-invoked automated assistant 310 has captured the utterance 545 via one or more microphones of the client device on which it is running and therefore may already have the prior interaction data. For example, in some implementations, the later-invoked automated assistant 310 can continuously capture limited audio and, once invoked, can use the previously captured audio that was intended for the initially-invoked automated assistant 305 after confirming that the speaker is the same in both cases.

[0047] 6 shows a flow chart illustrating an example method 600 for processing an interaction with a first automated assistant and providing interaction data to a second automated assistant. For convenience, the operations of method 600 are described with reference to a system that performs those operations. The system of method 600 includes one or more processors and / or other components of a client device. Furthermore, although the operations of method 600 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, removed, or added.

[0048] In step 605, the initially invoked automated assistant participates in a dialogue with the user. The initially invoked automated assistant may be invoked by one or more hot words uttered by the user and processed by the invocation engine 315. Once invoked, the query processing engine 330 may process the user's utterance and determine one or more actions to perform in response to the user's utterance. For example, the initially invoked automated assistant 305 may be invoked by the phrase "Okay, assistant A" followed by the query "What are some good animated movies?" The query processing engine 330 may determine the response "Movie A is a good movie," which may be provided to the user via synthesized speech. The dialogue may continue between the user and the initially invoked automated assistant 305 such that the user may provide additional utterances, either related to the previous utterance (and / or related to the response from the initially invoked automated assistant 305) or unrelated utterances.

[0049] In step 610, the initially invoked automated assistant receives a request to transfer a portion of the interaction to a second automated assistant. A component sharing one or more characteristics with the assistant collaboration module 325 can determine whether another automated assistant, such as the subsequently invoked automated assistant 310, is configured to process the interaction data. For example, the assistant collaboration module 375 can provide an indication, such as an audible signal via the speaker 370 and / or an indication via the APIs 345 and 390, that can be received by the initially invoked automated assistant 305 indicating the type of query that the subsequently invoked automated assistant 310 can process. As an example, the subsequently invoked automated assistant 310 can be configured to communicate with the user's calendar application and can provide an ultrasonic signal that can be received by the microphone 320 to indicate to the initially invoked automated assistant 305 that a query related to the user's calendar application can be processed by the subsequently invoked automated assistant 310.

[0050] In some implementations, a user may request that interaction data be transferred to a later-invoked automated assistant. In some cases, a user may explicitly request that an interaction that began with a first-invoked automated assistant be continued with a second automated assistant. For example, a user may say, "Okay, assistant 1, play a song on assistant 2." In some cases, a user may issue a query that cannot be processed by the first-invoked automated assistant. For example, in an interaction with a first-invoked automated assistant, a user may say, "Okay, assistant, go to the online store and buy movie 1," which may be a transfer request to a different automated assistant that can fulfill this request.

[0051] In some implementations, a transfer request may be provided by a user in response to a transfer suggestion from the originally invoked automated assistant. For example, a user may say, "Okay, Assistant 1, what's on my calendar?" and the originally invoked automated assistant may not be configured to fulfill that request. The originally invoked automated assistant may respond, "I don't have access to your calendar. Do you want me to ask Assistant 2?" The user may then issue a transfer request, such as "Yes," and the originally invoked automated assistant 305 may invoke the later invoked automated assistant 310 to transfer the interaction data.

[0052] In some implementations, the later-invoked automated assistant 310 may request the transfer of dialogue data. For example, in some cases, the user 101 may invoke the initially-invoked automated assistant 305 to participate in a dialogue, such as the dialogue illustrated in FIG. 5C. At some point in the dialogue, the user stops interacting with the initially-invoked automated assistant 305 and instead invokes the later-invoked automated assistant with dialogue 555 (i.e., "Okay, Assistant 2, buy it at the online store"). At this point, the later-invoked automated assistant 310 may not have access to the previous conversation because the later-invoked automated assistant 310 was not invoked during dialogue turns 545 and 550. In such a case, the later-invoked automated assistant 310 may submit a request to the initially-invoked automated assistant 305 to transfer the dialogue (or data indicative of the dialogue) to the later-invoked automated assistant 310.

[0053] In some implementations, the later-invoked automated assistant 310 can provide voice confirmation data to the first-invoked automated assistant 305 along with the request for transfer of the interaction data to verify that the speaker of the interaction with the first-invoked automated assistant 305 is the same speaker who invoked (and subsequently interacted with) the later-invoked automated assistant 310. For example, with reference to FIG. 3, the verification module 335 can provide data indicative of the speaker of the call and / or utterance to the later-invoked automated assistant 310 via one or more communication channels. The verification module 395 can determine that the speaker of the earlier interaction is the same user 101 as the later utterance, for example, based on voice profile data and / or audio data of the utterance directed to the later-invoked automated assistant 310. Upon verification, in step 615, the interaction data is provided by the first-invoked automated assistant 305 to the second (i.e., later-invoked) automated assistant 310.

[0054] In some implementations, the dialogue data can be audio data of a prior dialogue, for which the query processing engine 340 can determine the intent and / or context for the subsequent utterance. In some implementations, the intent and / or context can be determined by the query processing engine 330 of the initially invoked automated assistant 305, and this intent and / or context can be provided as dialogue data to the subsequently invoked automated assistant 310. In those implementations, the query processing engine 340 can determine one or more actions to perform based on the subsequent utterance and the prior dialogue.

[0055] In step 620, the second automated assistant performs one or more actions in response to being provided with the interaction data. The actions may include, for example, the later-invoked automated assistant fulfilling a request and / or providing a response based on an utterance that is resolved based on the intent and / or context of the prior interaction. In some implementations, the actions may include the later-invoked automated assistant interacting with one or more other applications. For example, referring back to the interaction of FIG. 5C, the actions performed by the later-invoked automated assistant 310 may include interacting with an online store application and / or website. Other actions may include the later-invoked automated assistant 310 responding with responses that it is configured to fulfill, interacting with a calendar application, interacting with a streaming service, and / or interacting with other applications.

[0056] 7 shows a flowchart illustrating an example method 700 for providing authorization to a second automated assistant to process interaction data from an interaction between a user and an initially invoked automated assistant. In step 705, a spoken query is received by the first automated assistant. The first automated assistant can share one or more characteristics with the initially invoked automated assistant 305. In some implementations, step 705 can share one or more characteristics with step 605 of FIG. 6.

[0057] In step 710, the first automated assistant identifies a prior interaction between the user and the first (invoked) automated assistant. This interaction may include one or more utterances of the user and one or more responses of the first automated assistant. For example, referring to FIG. 5B, the prior interaction may include the user uttering an utterance 525 along with a response 530 of the first automated assistant 305. The prior interaction data may be determined by a component that shares one or more characteristics with the query processing engine 330.

[0058] In step 715, approval of the second automated assistant can be provided to the first automated assistant 305. For example, approval can include user 101 utterance 535, which is included in the transfer suggestion 530. Also, for example, with reference to FIG. 5A, approval of the transfer can be implied by an utterance of user 101, such as utterance 515 indicating approval by user 101 to continue the same dialogue with the second automated assistant 310.

[0059] In step 720, an instruction for the query is provided to the second automated assistant. In some implementations, step 720 can share one or more characteristics with step 615 of FIG. 6. For example, the instruction can include a previous interaction between the first automated assistant and the user, the intent and / or context of the interaction, and / or other data that enables the second automated assistant to resolve one or more intents and / or determine the context for a subsequent utterance directed to the second automated assistant.

[0060] In step 725, one or more actions may be performed by the second automated assistant in response to the user's utterance directed to the second automated assistant. Step 725 may share one or more characteristics with step 620 of Figure 6. For example, the one or more actions may include the second automated assistant providing a response to which it is configured to respond and / or interacting with one or more applications.

[0061] FIG. 8 shows a flowchart illustrating an example method 800 for verifying that the speaker of a first-invoked interaction with an automated assistant is the same as the speaker of a later-invoked request for processing of interaction data by an automated assistant.

[0062] In step 805, a user is determined who provided a spoken utterance requesting that a later-invoked automated assistant use prior dialogue data. For example, referring to FIG. 5C, utterance 555 is an utterance that invokes an automated assistant 310 and requests that dialogue data be transferred to the later-invoked automated assistant 310. In some implementations, the request to transfer dialogue data may be directed to the first automated assistant 305, such as "Okay, assistant, check if assistant 2 can handle this request." In some implementations, the request to transfer dialogue data may be directed to the later-invoked automated assistant 310, as illustrated in FIG. 5C. In some implementations, a request by a user may be responsive to a suggestion to transfer dialogue data, as illustrated in FIG. 5B.

[0063] In step 810, audio data capturing the spoken utterance is determined. The audio data can be captured by a component that shares one or more characteristics with the microphone 365 and / or the microphone 320. For example, the microphone 320 can capture the spoken utterance and the assistant collaboration module 325 can forward the audio data to the later-invoked automated assistant 310. Also, for example, the microphone 365 can directly capture audio data that, in addition to being a forwarding request, can be used to invoke the later-invoked automated assistant 310 via the call engine 385.

[0064] In step 815, one or more components determine whether the audio data was provided by the same user as the previous interaction. The one or more components can share one or more characteristics with the verification module 335 and / or the verification module 395. For example, the audio data can be captured by the microphone 365 and sent to the initially invoked automated assistant 305, and the verification module 395 can determine whether the audio data was provided by the same user as the previous interaction. Also, for example, the initially invoked automated assistant 305 can provide verification data to the subsequently invoked automated assistant 310, and the verification module 335 can determine whether the speaker of the spoken utterance (captured by the microphone 365) is the same speaker as the previous interaction.

[0065] In step 820, the prior interaction data is provided to the subsequently invoked automated assistant. The prior interaction data may include audio data of the prior interaction, the intent and / or context of the prior interaction, and / or other data that may be utilized by the subsequently invoked automated assistant to determine the intent and / or context for the user's subsequent utterances. In some implementations, if the check in step 815 indicates that the speaker is not the same as the prior interaction, one or more components may prevent the interaction data from being forwarded to the subsequently invoked automated assistant 310.

[0066] 9 is a block diagram of an example computing device 910 that may be optionally used to perform one or more aspects of the techniques described herein. The computing device 910 typically includes at least one processor 914 that communicates with a number of peripheral devices via a bus subsystem 912. These peripheral devices may include a storage subsystem 924, including, for example, a memory subsystem 925 and a file storage subsystem 926, a user interface output device 920, a user interface input device 922, and a network interface subsystem 916. The input and output devices enable a user to interact with the computing device 910. The network interface subsystem 916 provides an interface to external networks and is coupled to corresponding interface devices in other computing devices.

[0067] The user interface input devices 922 may include pointing devices such as a mouse, trackball, touchpad, or graphics tablet, a keyboard, a scanner, a touch screen integrated into a display, a voice recognition system, an audio input device such as a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computing device 910 or a communications network.

[0068] The user interface output devices 920 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computing device 910 to a user or to other machines or computing devices.

[0069] Storage subsystem 924 stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, storage subsystem 924 may include logic for performing selected aspects of the methods of Figures 6-8 and / or for implementing the various components illustrated in Figures 2-4.

[0070] These software modules are generally executed by the processor 914, either alone or in combination with other processors. The memory 925 used in the storage subsystem 924 may include several memories, including a main random access memory (RAM) 930 for storage of instructions and data during program execution, and a read-only memory (ROM) 932 in which fixed instructions are stored. The file storage subsystem 926 may provide persistent storage for program and data files, and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules implementing the functionality of some implementations may be stored by the file storage subsystem 926 in the storage subsystem 924, or in other machines accessible by the processor 914.

[0071] The bus subsystem 912 provides a mechanism for allowing the various components and subsystems of the computing device 910 to communicate with each other as intended. Although the bus subsystem 912 is shown generally as a single bus, alternative implementations of the bus subsystem could use multiple buses.

[0072] The computing device 910 may be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Because the nature of computers and networks is ever-changing, the description of the computing device 910 shown in Figure 9 is intended only as a specific example intended to illustrate some implementations. Many other configurations of the computing device 910 are possible, having more or fewer components than the computing device shown in Figure 9.

[0073] In some implementations, a method implemented by one or more processors is provided, the method including: receiving, during an interaction between the user and the initially invoked automated assistant, by the initially invoked automated assistant via an interface of the first automated assistant of a first client device in an environment with the user, a spoken utterance of the user directed to the initially invoked automated assistant as part of the interaction; processing the spoken utterance by the initially invoked automated assistant to generate a response to the spoken utterance; causing the initially invoked automated assistant to render the response to the spoken utterance by the first client device as part of the interaction; and receiving a request to transfer interaction data generated by the first automated assistant as part of the interaction to a second automated assistant after rendering the response by the initially invoked automated assistant. In response to receiving the request, the method further includes providing, by the first automated assistant, the interaction data to the second automated assistant via a communication channel with the second automated assistant, the providing the interaction data causing the second automated assistant to perform one or more actions based on the interaction data.

[0074] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0075] In some implementations, the transfer request is an additional spoken utterance of the user directed to the originally invoked automated assistant.

[0076] In some implementations, the request to transfer is a request from the second automated assistant to process the spoken utterance. In some of these implementations, the request to transfer includes audio data indicating the spoken utterance. In some of these implementations, the method further includes, by the first invoked automated assistant, verifying that the user is the speaker indicated by the audio data based on processing the audio data with a speaker verification model, and providing the dialogue data to the second automated assistant is contingent on verifying that the user is the speaker indicated by the audio data.

[0077] In some implementations, the second automated assistant is running on a second device remote from the first device, and the one or more actions include generating an audio output of the second assistant that is provided to the user via the second client device.

[0078] In some implementations, the request to be processed includes instructions for one or more actions to be performed by a second automated assistant, and the originally invoked assistant is unable to perform the one or more actions.

[0079] In some implementations, another method implemented by one or more processors is provided, the method including: receiving, by an initially invoked automated assistant client of an initially invoked automated assistant, a spoken query of a user captured in audio data generated by one or more microphones of a first client device in the user's environment, the first automated assistant client being installed on the first client device; identifying a user intent indicated by one or more terms of the interaction based on a prior interaction between the user and the initially invoked automated assistant indicated by the audio data; providing authorization to process the query based on the user intent to a second automated assistant client installed on a second client device in the environment; and providing, by the initially invoked automated assistant client, an indication of the query and the user intent to the second automated assistant client via a communication channel with the second automated assistant client, where the second automated assistant client performs one or more actions by providing the indication and the user intent.

[0080] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0081] In some implementations, the communication channel includes an ultrasonic communication channel, and providing the query instruction and the user intent to the second automated assistant client includes causing an ultrasonic signal incorporating the query instruction and the user intent to be rendered via one or more speakers of the first client device.

[0082] In some implementations, the method further includes receiving a second spoken query from the user requesting the second automated assistant client to process the query, and the step of providing approval is responsive to receiving the second spoken query.

[0083] In some implementations, the method further includes determining that the first automated assistant is unable to process the user intent, providing a suggestion to the user to provide approval, and receiving approval from the user, wherein the approval is provided to the second automated assistant client in response to receiving the approval from the user.

[0084] In some implementations, the method further includes performing automatic speech recognition on the audio data to capture the spoken query to generate a text query, the indication of the query being the text query.

[0085] In some implementations, another method implemented by one or more processors is provided, the method including determining, at the first invoked automated assistant, that the user has provided a spoken utterance that requests that the later invoked automated assistant use interaction data from the user's previous interaction with the first invoked automated assistant. In response to determining that the user has provided the spoken utterance, the method further includes processing, by the first invoked automated assistant, audio data capturing the spoken utterance, and determining, by the first invoked automated assistant, based on the processing, whether the audio data was provided by the same user involved in the previous interaction. When it is determined that the user is the same user, the method further includes providing the previous interaction data to the later invoked automated assistant. When it is determined that the user is not the same user, the method includes avoiding providing the interaction data.

[0086] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0087] In some implementations, the audio data is captured by a microphone of the first device running the originally invoked automated assistant.

[0088] In some implementations, the audio data is captured by a microphone of a second device that runs the later-invoked automated assistant.

[0089] In some implementations, the spoken utterance is directed to the automated assistant that was originally invoked.

[0090] In some implementations, the spoken utterance is directed to an automated assistant that is later invoked.

[0091] In some implementations, the prior interaction data is provided via a non-human audible signal generated by a speaker of a first client device running the initially invoked automated assistant and received by one or more microphones of a second client device running the later invoked automated assistant.

[0092] In some implementations, the prior interaction data is provided via an application programming interface between the initially invoked automated assistant and the subsequently invoked automated assistant.

[0093] In some implementations, the prior interaction data includes a textual representation of the interaction between the user and the originally invoked automated assistant.

[0094] In some implementations, the prior interaction data includes audio data that captures the interaction between the user and the originally invoked automated assistant.

[0095] In situations where certain implementations contemplated herein may collect or use personal information about a user (e.g., user data extracted from other electronic communications, information about the user's social network, the user's location, the user's time, the user's biometric information, and the user's activity and demographic information, relationships between multiple users, etc.), the user is provided with one or more opportunities to control whether information is collected, whether the personal information is stored, whether the personal information is used, and how the information is collected, stored, and used about the user. That is, the systems and methods contemplated herein collect, store, and / or use a user's personal information only upon receiving explicit authorization to do so from the associated user.

[0096] For example, a user is provided with control over whether a program or feature collects user information about that particular user or other users associated with the program or feature. Each user about whom personal information is collected is presented with one or more options that allow the user to control the collection of information related to that user to provide permission or approval as to whether and what portions of the information are collected. For example, a user may be provided with one or more such control options over a communications network. Additionally, certain data may be processed in one or more ways before being stored or used such that personally identifiable information is deleted. As one example, a user's identity may be processed such that personally identifiable information cannot be determined. As another example, a user's geographic location may be generalized to a larger area such that a user's specific location cannot be determined.

[0097] Although several implementations have been described and illustrated herein, various other means and / or structures for performing the functions described herein and / or obtaining one or more of the results and / or effects described herein may be used, and each such variation and / or modification is considered to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on the particular application or applications in which these teachings are used. Those skilled in the art will recognize or be able to ascertain, using no more than routine experimentation, many equivalents to the specific implementations described herein. Thus, it should be understood that the above implementations are presented by way of example, and that within the scope of the appended claims and their equivalents, implementations may be realized other than as specifically described and claimed. Implementations of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. Furthermore, any combination of two or more of these features, systems, articles, materials, kits, and / or methods is not mutually inconsistent and is within the scope of the present disclosure. [Explanation of symbols]

[0098] 105 client devices 110 Client Devices 215 The First Automated Assistant 220 Second Automated Assistant 225 Microphone 230 Microphone 245 The Third Automated Assistant 301 Client Device 302 Client Device 305 First Called Automated Assistant 310 Automated assistant called later 315 Call Engine 320 Microphone 325 Assistant Collaboration Module 330 Query Processing Engine 335 Confirmation Module 340 Query Processing Engine 345 API 350 Speaker 365 Microphone 370 Speaker 375 Assistant Collaboration Module 385 Call Engine 390 API 395 Confirmation Module 415 Dialogue 420 Transfer Suggestion 425 Forward Request 430 Performing an action 435 Call 440 Dialogue Data 445 Dialogue 450 Dialogue 455 Speaker confirmation data 460 Confirmation Confirmation 465 Conversation Data 470 Performing an action 914 Processor 916 Network Interface 920 User Interface Output Device 922 User Interface Input Devices 924 Storage Subsystem 925 Memory Subsystem 926 File Storage Subsystem 930 RAM 932 ROM

Claims

1. 1. A method implemented by one or more processors, comprising: During the interaction between the user and the first invoked automated assistant, Receiving, by the initially invoked automated assistant, a spoken utterance of the user directed to the initially invoked automated assistant as part of the interaction via an interface of a first automated assistant on a client device in an environment with the user; processing the spoken utterance by the initially invoked automated assistant to generate a response to the spoken utterance; causing a response to the spoken utterance by the initially invoked automated assistant to be rendered by the client device as part of the interaction; receiving a request to transfer dialogue data generated by the first automated assistant as part of the dialogue to a second automated assistant after having the response rendered by the initially invoked automated assistant, the request to transfer being an additional spoken utterance of the user directed to the initially invoked automated assistant or a request from the second automated assistant to process the spoken utterance; In response to receiving said request, providing, by the first invoked automated assistant, the interaction data to the second automated assistant via a communication channel with the second automated assistant; Including, The method of claim 1, wherein the step of providing interaction data causes the second automated assistant to perform one or more actions based on the interaction data.

2. The method of claim 1 , wherein the request to forward includes audio data indicative of the spoken utterance.

3. and further comprising verifying, by the initially invoked automated assistant, that the user is the speaker indicated by the audio data based on processing the audio data with a speaker verification model; 3. The method of claim 2, wherein providing the dialogue data to the second automated assistant is contingent on verifying that the user is the speaker indicated by the audio data.

4. 4. The method of claim 1, wherein the second automated assistant is running on another client device separate from the first client device, and the one or more actions include generating an audio output of the second assistant that is provided to the user via the other client device.

5. 5. The method of claim 1, wherein the interaction data includes instructions for the one or more actions to be performed by the second automated assistant, and the initially invoked assistant is unable to perform the one or more actions.

6. receiving, by a first invoked automated assistant client, a spoken query of the user captured in audio data generated by one or more microphones of a first client device in the user's environment, the first automated assistant client being installed on the first client device; Based on a previous interaction between the user and the initially invoked automated assistant indicated by the audio data, identifying a user intent indicated by one or more terms of the interaction; Providing authorization to process the query based on the user intent to a second automated assistant client installed on a second client device in the environment; Providing, by the first invoked automated assistant client, an indication of the query and the user intent to the second automated assistant client via a communication channel with the second automated assistant client; Including, The computer-implemented method, wherein providing the instructions and the user intent causes the second automated assistant client to perform one or more actions.

7. The communication channel includes an ultrasonic communication channel, and the step of providing the indication of the query and the user intent to the second automated assistant client includes:

7. The method of claim 6, comprising causing an ultrasound signal incorporating the indication of the query and the user intent to be rendered via one or more speakers of the first client device.

8. 8. The method of claim 6 or 7, further comprising receiving a second spoken query from the user requesting the second automated assistant client to process the query, and wherein the step of providing approval is in response to receiving the second spoken query.

9. Determining that the first automated assistant client is unable to process the user intent; In response to determining that the first automated assistant client is unable to process the user intent, providing a suggestion to the user to provide the approval; receiving the approval from the user, the approval being provided to the second automated assistant client in response to receiving the approval from the user; 9. The method of any one of claims 6 to 8, further comprising:

10. 10. The method of claim 6, further comprising performing automatic speech recognition on the audio data to capture the spoken query to generate a text query, the indication of the query being the text query.

11. determining, at a first invoked automated assistant, that the user has provided a spoken utterance that requests that the subsequently invoked automated assistant use prior interaction data from a prior interaction of the user with the first invoked automated assistant; in response to determining that the user has provided the spoken utterance, processing audio data capturing the spoken utterance by the initially invoked automated assistant; determining, by the initially invoked automated assistant, based on the processing, whether the audio data was provided by the same user as was involved in the previous interaction; When it is determined that the user is the same user, providing the prior interaction data to the subsequently invoked automated assistant; When it is determined that the user is not the same user, avoiding providing the dialogue data; 4. A computer-implemented method comprising:

12. The method of claim 11 , wherein the audio data is captured by a microphone of a first device running the initially invoked automated assistant.

13. The method of claim 11 , wherein the audio data is captured by a microphone of a second device running the subsequently invoked automated assistant.

14. 14. The method of claim 11, wherein the spoken utterance is directed to the initially invoked automated assistant.

15. 14. The method of claim 11, wherein the spoken utterance is directed to the subsequently invoked automated assistant.

16. 16. The method of claim 11, wherein the prior interaction data is provided via a non-human audible signal generated by a speaker of a first client device running the initially invoked automated assistant and received by one or more microphones of a second client device running the later invoked automated assistant.

17. 16. The method according to claim 11, wherein the previous interaction data is provided via an application programming interface between the initially invoked automated assistant and the subsequently invoked automated assistant.

18. 18. The method of claim 11, wherein the previous interaction data comprises a textual representation of an interaction between the user and the initially invoked automated assistant.

19. A computer program product which, when executed by one or more processors of a computing system, causes the computing system to perform the method of any one of claims 1 to 18.

20. One or more computing devices configured to perform the method of any one of claims 1 to 18.

Images