Transferring conversation data from the initially invoked automation assistant to subsequent invoked automation assistants.
By selectively transferring dialogue data among automated assistants, the problems of resource waste and dialogue prolongation in existing technologies are solved, enabling more efficient query processing and response.
Patent Information
- Application Number
- CN202180088756.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2021-11-22
- Filing Date
- 2021-12-16
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2041-12-16
AI Technical Summary
In existing technologies, automated assistants need to explicitly invoke and process all audio data when handling user queries, resulting in wasted resources and prolonged conversations. In particular, when the query exceeds its capabilities, the user needs to restart the conversation to invoke a more suitable assistant.
By detecting the user's calling phrase through the initial automated assistant, dialogue data is selectively transferred to a more suitable second automated assistant for processing, reducing the use of computing and network resources and providing a more complete response.
It reduces the need for users to explicitly invoke a second assistant, shortens conversation duration, reduces resource consumption, and improves the completeness and efficiency of responses.
Smart Images

Figure CN116670638B_ABST
Abstract
Description
Background Technology
[0001] Humans can engage in human-computer dialogue using interactive software applications referred to herein as “automated assistants” (also known as “digital agents,” “chatbots,” “interactive personal assistants,” “intelligent personal assistants,” “assistant applications,” “conversational agents,” etc.). For example, humans (who may be referred to as “users” when interacting with an automated assistant) can provide commands and / or requests to the automated assistant using spoken natural language input (i.e., utterances), which in some cases can be converted into text and then processed, and / or by providing text-based (e.g., typed) natural language input. The automated assistant responds to requests by providing responsive user interface outputs, which can include auditory and / or visual user interface outputs.
[0002] As described above, many automated assistants are configured to interact via spoken words after being invoked. To protect user privacy and / or conserve resources, users must often explicitly invoke the automated assistant before it can fully process the spoken words. Explicit invocation of an automated assistant typically occurs in response to some user interface input received at a client device. The client device includes an assistant interface that provides the user of the client device with an interface for interacting with the automated assistant (e.g., receiving spoken and / or typed input from the user and providing auditory and / or graphical responses), and interacts with one or more additional components implementing the automated assistant (e.g., a remote server device that processes user input and generates appropriate responses).
[0003] Some user interface inputs that enable the invocation of an automation assistant via a client device include hardware and / or virtual buttons on the client device for invoking the automation assistant (e.g., tapping a hardware button, selecting a graphical interface element displayed by the client device). Many automation assistants can be additionally or alternatively invoked in response to one or more verbal invocation phrases, also known as "hot words / phrases" or "trigger words / phrases." For example, one can invoke an automation assistant by saying verbal invocation phrases such as "Hey Assistant," "OK Assistant," and / or "Assistant."
[0004] Typically, a client device including an assistant interface includes one or more locally stored models that the client device utilizes to at least selectively detect the occurrence of spoken call phrases. Such a client device is capable of locally processing received audio data using the locally stored models and discarding any audio data that does not include a spoken call phrase. However, when local processing of the received audio data indicates the occurrence of a spoken call phrase, the client device then allows the audio data and / or subsequent audio data to be further processed by the automation assistant. For example, if the spoken call phrase is "Hey, Assistant," and the user says "Hey, Assistant, what time is it," then the audio data corresponding to "what time is it" can be processed by the automation assistant based on the detection of "Hey, Assistant," and used to provide an automation assistant response for the current time. On the other hand, if the user simply says "what time is it" (without first saying the call phrase or providing alternative call input), the automation assistant will not provide any response because there is no call phrase (or other call input) preceding "what time is it." Summary of the Invention
[0005] This document describes techniques for providing dialogue data from a first automated assistant to a second automated assistant in response to a request to transfer dialogue data. For example, various techniques involve: the initially invoked automated assistant handling the dialogue between the user and the initially invoked automated assistant; receiving a request to transfer processing of a query as part of the dialogue to the second automated assistant; and providing dialogue data indicative of the dialogue data to the second automated assistant for further processing. The user can initially invoke the automated assistant and utter one or more queries, and the invoked automated assistant can generate a response to the one or more queries. At some point in the dialogue between the user and the initially invoked automated assistant, a request can be received indicating that the user is interested in additional dialogue and / or in the query being processed by the second automated assistant. Dialogue and / or dialogue data related to the dialogue can be provided by the initially invoked automated assistant to the second assistant, which can perform one or more actions, such as processing the query to generate a response. In these and other ways, the second assistant can obtain the dialogue and / or dialogue data without having to engage in computationally intensive dialogue with the user. Furthermore, the initially invoked automated assistant can guide the user-assistant dialogue by selectively transferring dialogue and / or dialogue data to the second assistant. Furthermore, in some implementations, the second assistant can generate responses that might not be generated independently by the initially invoked automation assistant—thus increasing the robustness of the user assistant dialogue.
[0006] In some implementations, a user can utter a call phrase, such as “OK Assistant,” which invokes a first automation assistant (also referred to herein as the “initial invoked automation assistant”), but does not explicitly invoke other automation assistants. The initially invoked automation assistant can interact with, at least selectively, the other automation assistants when processing received queries associated with the call phrase (e.g., immediately following or preceding it). Therefore, based on providing call input specific to the initially invoked automation assistant, a user can specify to utilize the initially invoked automation assistant instead of individually invoking one of the other automation assistants. For example, the first call phrase (e.g., “OK Assistant A”) can specifically invoke the first automation assistant without invoking any other automation assistants when detected. Similarly, a second call phrase (e.g., “OK Assistant B”) can specifically invoke the second automation assistant without invoking the initially invoked automation assistant and / or any other automation assistants when detected. The initially invoked assistant, when invoked, can interact with, at least selectively, the other automation assistants (i.e., “assistant assistants”) when processing input provided in association with the call.
[0007] Users can invoke an automation assistant by speaking a phrase that, once initially invoked, allows the user to engage in a conversation with the initially invoked assistant. The conversation can consist of one or more queries spoken by the user, followed by one or more responses generated by the initially invoked automation assistant. For example, a user can initially invoke Assistant 1 by saying, “OK Assistant 1, what is a good movie for kids?” In this instance, “OK Assistant 1” can be the hot word invoking the first automation assistant, which can generate and provide the response, “Movie A is a good movie for kids.” In response, the user can speak additional queries, related or unrelated to the first query, to generate a conversation between the initially invoked automation assistant and the user. For example, the user could say, “What is another good movie?” to be provided with additional generated responses related to the previous query, “good movies for kids.” Furthermore, for example, a user can then say "What is the weather today?", and the initially invoked automation assistant can provide a response that is unrelated to the user's previous query and / or the initial response of the initially invoked automation assistant.
[0008] As another example, in response to an automation assistant saying "Movie A is a good movie for kids," a user could then ask, "OK, play it on Streaming Service." In this instance, the automation assistant can determine the intent of the query, which can include determining what the user means by using the term "it" in phrases like "Movie A" based on previous conversations between the user and the automation assistant. In response to the subsequent query "Play it on Streaming Service," the initially invoked automation assistant can provide data to the streaming application "Streaming Service," which can then stream "Movie A" to the user via the user interface.
[0009] As yet another example, once the initially invoked automation assistant is called, the user can say the phrase "What is the weather in San Francisco?", and the initially invoked automation assistant can generate a response to that query, such as "The weather today in San Francisco is 50 degrees and cloudy." In some instances, the dialogue may include additional queries from the user related to the previous query. For example, the user could say a follow-up query, "What about in Miami?", indicating an intent to be provided with information about the weather in Miami. In response, the initially invoked automation assistant could reply, "The weather in Miami is 80 degrees and sunny." The dialogue can continue between the user and the initially invoked automation assistant, with both relevant and irrelevant queries.
[0010] In some implementations, multiple automation assistants can coexist in the same environment. For example, a user can invoke an automation assistant running on a device near a second device that is also executing a second automation assistant. Furthermore, for example, a user can invoke an automation assistant running on a device that is also executing a second automation assistant. As previously mentioned, a user can invoke the initially invoked automation assistant by saying a hotword or phrase that activates the initial automation assistant without invoking other automation assistants. For example, a user can say the hotword "OK Assistant1" to invoke the first automation assistant without invoking the second. Similarly, a user can say the hotword "OK Assistant 2" to invoke the second automation assistant without invoking the first.
[0011] In some implementations, the automation assistant can provide indications to other automation assistants in the environment regarding the types of queries and / or commands it can handle. For example, an automation assistant may have access to a calendar application and thus can provide indications to other automation assistants that queries related to the user's calendar can be handled by that automation assistant. Furthermore, for example, the automation assistant can indicate to other automation assistants via one or more signals that it can stream media via a media streaming application. Information about the capabilities of other automation assistants can be provided to them so that, in instances where the invoked automation assistant cannot handle a query, it knows of other automation assistants in the same environment that might be able to successfully handle the query and provide a response to the user. For example, one or more automation assistants can broadcast ultrasonic signals indicating the types of queries and / or application types that the broadcast automation assistant can handle. Additionally, for example, the automation assistant may provide indications of its capabilities via Wi-Fi, an application programming interface (API), and / or other communication channels that allow the automation assistant to learn about the capabilities of other automation assistants in the environment.
[0012] In some implementations, the automation assistant can preemptively broadcast its capabilities. In those instances, when the initially invoked automation assistant receives a query it cannot handle, it can search for other automation assistants that are broadcasting capabilities to handle that type of request. Preemptive broadcasting of capabilities can reduce latency in determining which secondary assistant (if any) to transfer the conversation and / or conversation data to, and thus can lead to reduced latency in the secondary assistant's response generation. In some implementations, when the initially invoked automation assistant receives a request it cannot handle, it can broadcast a request to other automation assistants to determine whether the request can be handled by one or more other automation assistants. For example, the initially invoked automation assistant might receive a request related to a calendar application it does not have access to, and can broadcast a message to other automation assistants asking whether the request can be handled by a different automation assistant. In response, an automation assistant capable of handling the request can respond affirmatively, thereby informing the initially invoked automation assistant that the request can be handled by that automation assistant.
[0013] In some instances, the initially invoked automation assistant may not be configured to generate a response to a user query. For example, the initially invoked automation assistant might receive a query from the user and process it to generate a response to that query. In subsequent queries, the initially invoked automation assistant may not generate a substantial response, such as when it is not configured to respond to the type of query spoken by the user and / or when the user explicitly requests that the generated response be passed to a different automation assistant. In those instances, the initially invoked automation assistant may be able to pass data from a portion of the previous conversation, or at least a portion of the previous conversation, to a second automation assistant for further processing. Based on the intent of the query and / or the previous conversation, the second automation assistant may generate a response and further perform one or more actions based on the generated response. For example, once the second automation assistant has been provided with the conversation and / or conversation data from the previous conversation, it may be able to provide an audible response to the user, interact with the application, and / or perform other tasks based on the query.
[0014] In instances where the initially invoked automation assistant is not configured to respond to a query, the automation assistant can determine that another automation assistant might be more capable of completing the query. For example, a user might state the query "Play Movie A on Streaming Service". In instances where the initially invoked automation assistant may not have access to "Streaming Service", the initially invoked automation assistant can determine whether one or more other automation assistants are configured to handle such a request. For example, other automation assistants in the environment might broadcast ultrasonic signals indicating their capabilities. For example, an alternative automation assistant in the environment might have previously broadcast its capabilities to the initially invoked automation assistant, and that automation assistant stores those capabilities associated with the alternative automation assistant (e.g., locally stored). If a suitable second automation assistant is identified by the initially invoked automation assistant (e.g., by accessing the stored associations with the second automation assistant's capabilities), a request to transfer dialogue data can be made to the second automation assistant. Once the request has been granted by the second automation assistant, the initially invoked automation assistant can send the dialogue, including at least a portion of the dialogue (e.g., the intent of the dialogue and / or the query), to the second automation assistant. Then, the second automation assistant can process the query and perform one or more actions, such as initiating a "streaming service" to serve movie A to the user via the user interface.
[0015] In some implementations, the initially invoked automation assistant can suggest to the user that the conversation data be transferred to a second automation assistant. For example, the user can provide the initially invoked automation assistant with a query that it cannot process or otherwise provide a response to satisfy the user's request. The initially invoked automation assistant can respond to the user with a suggestion that indicates another automation assistant can be configured to provide a response, such as "I don't know, would you like me to check with Assistant 2?" In response, the user can say "Yes," and the initially invoked automation assistant can provide the conversation data to Assistant 2, who can then further process the data. In this example, the transfer request (i.e., the "Yes" utterance) is provided directly by the user and the invocation of the second automation assistant is provided by the initially invoked automation assistant.
[0016] In some implementations, referral suggestions can be provided to the user based on identifying user preferences and determining that the query is the type the user is interested in having completed by different automated assistants. For example, the initially invoked automated assistant can determine that the user has previously used different automated assistants to handle calendar-related queries, such as automated assistants with access to the user's calendar application. When the initially invoked automated assistant receives a calendar application-related request as part of a conversation, it can provide at least a portion of the conversation and / or data instructing the conversation to the automated assistant the user prefers to handle the request.
[0017] In some implementations, the transfer request can be based on user preferences. For example, a user can instruct, via one or more user preferences, that all queries related to the calendar application be handled by a first automation assistant. In another instance where an automation assistant receives a query related to the calendar application, the invoked automation assistant can interpret the user's settings of user preferences related to calendar queries as a request to transfer the conversation for all calendar application-related queries to the first automation assistant.
[0018] In some implementations, a user can continue the conversation with a second automation assistant after initially invoking a different automation assistant. For example, a user might invoke an automation assistant with "OK Assistant 1" and provide the query "What is the weather like in Miami?" The initially invoked automation assistant could respond to the query with something like, "The weather is 75 and sunny." The user could then say, "OK Assistant 2, how about in San Francisco," indicating interest in the second assistant (i.e., the automation assistant invoked with "OK Assistant 2") to further the conversation. The initially invoked automation assistant can determine that the conversation includes a reference to the second automation assistant and provide the second automation assistant with previous conversation data to allow the second automation assistant to determine the context and / or intent of its query. For example, the query from the previous example might not explicitly include "What is the weather in San Francisco," but based on the relationship between the query and the previous query, the intent to provide weather information might be included in the previous conversation (i.e., "What is the weather like in Miami").
[0019] In some implementations, the subsequently invoked automation assistant can determine that it does not have the context and / or intent of the session necessary to complete the query. For example, a user might invoke Assistant 1 and say the query “What movie should I watch?” Assistant 1 could respond with “Movie A is a good movie.” The user could then say, “OK Assistant 2, play it on StreamingService.” Because the movie title (i.e., “Movie A”) is not part of the conversation with Assistant 2, Assistant 2 can request the context, intent, and / or parts of the previous conversation with Assistant 1 from Assistant 1, allowing Assistant 2 to determine the user’s intent by the term “it” in the query “Play it on Streaming Service.” Once Assistant 1 provides the context of the previous conversation, Assistant 2 can determine that “it” refers to “Movie A” and initiate the “Streaming Service” to play the requested movie.
[0020] In some implementations, speaker verification can be performed before the initially invoked automation assistant provides dialogue data to subsequently invoked automation assistants. For example, a user might say, “OK Assistant 1, what is a good movie to buy?” and Assistant 1 could respond, “Movie A is highly rated.” The user could then say, “OK Assistant 2, buy it on Online Store.” Assistant 2 can request a portion of the previous dialogue to determine the context of the query against Assistant 2, and can include, for example, audio data such as the user’s utterance, “buy it on Online Store.” Before providing dialogue data and / or the context of the previous dialogue to Assistant 2, Assistant 1 can perform verification using the audio data from Assistant 2 to determine if the speaker in the previous dialogue is the same speaker who said “buy it on Online Store.” For example, Assistant 1 can perform text-independent speaker verification using models on one or more devices to generate speaker vectors to verify the speaker associated with the previous dialogue. Alternatively or additionally, Assistant 1 can perform TDSV by processing audio data using a text-related speaker verification (TDSV) model on the device to generate speaker vectors and verify that the speaker vectors match speaker vectors associated with previous conversations. In instances where speaker verification determines that the speaker uttering the query to Assistant 2 is the same speaker as the speaker in Assistant 1's previous conversation, Assistant 1 can provide the conversation data as described above. In instances where speaker verification cannot identify an instantiation of the same speaker in both conversations, Assistant 1 can provide Assistant 2 with indications that the speaker is different and / or additional conversation data is unavailable.
[0021] In some implementations, verification may include determining whether a query to a subsequently invoked automation assistant is a continuation of a previous conversation. For example, a conversation with the initially invoked automation assistant might include the user saying “What is the weather in San Francisco?” followed by a response from the initially invoked automation assistant. The user might then say “OK Assistant 2, what about in Miami,” addressed to the subsequently invoked automation assistant. The subsequently invoked automation assistant is able to provide at least a portion of audio (or data indicating the conversation and speaker) to the initially invoked automation assistant. For verification, the initially invoked automation assistant can determine whether the utterance is related to one or more of the user's previous conversations before providing the conversation data to the subsequently invoked automation assistant. For example, the subsequently invoked automation assistant could send audio data of the user saying “What about in Miami,” and the initially invoked automation assistant could determine that the utterance might be related to the context of a previous conversation about the San Francisco weather.
[0022] Therefore, the techniques described herein mitigate the need for users to explicitly invoke a second automation assistant when the initially invoked automation assistant cannot fulfill the request or respond to the user's request. This results in a shorter total duration of the user's conversation with multiple automation assistants and a corresponding reduction in the use of network and / or computing resources. Furthermore, in instances where the initially invoked automation assistant knows of another automation assistant more likely to return an objectively more complete response, the user can be informed of the capabilities of the second automation assistant without having to invoke it to determine if a more complete response is possible. By assisting the user in receiving a response from the assistant most likely to respond definitively without requiring the user to expend the resources needed to restart the conversation with the second automation assistant, queries are allowed to be smoothly passed to the second automation assistant, mitigating the user's need to prolong the session with the automation assistant. Moreover, by providing conversation data to the second automation assistant, the need for the second assistant to prompt the user with information that has already been provided is avoided. Therefore, the context of the conversation can be provided without requiring the user to repeat information already provided to the first automation assistant.
[0023] The above description serves as an overview of some embodiments of this disclosure. Further descriptions of those embodiments and other embodiments follow in more detail. Attached Figure Description
[0024] Figure 1 This is an illustration of an example environment in which the implementation methods disclosed herein can be carried out.
[0025] Figure 2 It is a block diagram of an example environment in which the various methods disclosed in this paper can be implemented.
[0026] Figure 3 This is a block diagram of another example environment in which the various methods disclosed herein can be implemented.
[0027] Figure 4A This diagram illustrates the transfer of conversational data from the first automation assistant to the second automation assistant.
[0028] Figure 4B This diagram illustrates how, after verifying the speaker, the conversation is transferred from the initially invoked automation assistant to a subsequently invoked automation assistant.
[0029] Figure 5A , Figure 5B and Figure 5C A dialogue involving a user, a first automation assistant, and a second automation assistant, according to various embodiments disclosed herein, is described.
[0030] Figure 6 , Figure 7 and Figure 8 Flowcharts are depicted, each illustrating an example method according to various implementations disclosed herein.
[0031] Figure 9 The illustration shows an example architecture of a computing device. Detailed Implementation
[0032] refer to Figure 1An example environment is provided, which includes multiple automated assistants that can be invoked by user 101. The environment includes a first independent interactive speaker 105 with a microphone (not depicted) and a second independent interactive speaker 110 with a microphone (also not depicted). The first speaker may be at least partially executing a first automated assistant that can be invoked using a calling phrase. The second speaker 110 may be executing a second automated assistant that can be invoked using a calling phrase, which may be the same as or a different phrase than the first automated assistant, allowing the user to select which automated assistant to invoke based on the spoken phrase. In the example environment, user 101 is speaking the spoken phrase “OK Assistant, What’s a good animated movie” near the first speaker 105 and the second speaker 110. If one of the first and / or second automated assistants is configured to be invoked by the phrase “OK Assistant,” the invoked assistant can handle the query following the calling phrase (i.e., “What’s a good animated movie”).
[0033] In some implementations, a device such as the first speaker 105 may be performing multiple automated assistants. (See reference...) Figure 2 An example environment is illustrated, comprising multiple client devices executing multiple automated assistants. The system includes a first client device 105 executing a first automated assistant 215 and a second automated assistant 220. Each of the first and second automated assistants can be invoked by speaking an invocation phrase (which is unique for each assistant or the same phrase invoking both assistants) near the client device 105, allowing audio to be captured by the microphone 225 of the client device 105. For example, user 101 can invoke the first automated assistant 215 by speaking “OK Assistant 1” near the client device 105, and further invoke the second automated assistant 220 by speaking the phrase “OK Assistant 2” near the client device 105. Based on the spoken invocation phrase, the user can indicate which of the multiple assistants executing on the client device 105 is of interest to handle the verbal query. The example environment further includes a second client device 110 executing a third automated assistant 245. The third automated assistant can be configured to be invoked using a third invocation phrase—such as “OK Assistant3”—allowing it to be captured by the microphone 230. In some embodiments, Figure 2 One or more of the automation assistants may be absent. Furthermore, the example environment may include... Figure 2Additional automated assistants not present in the diagram. For example, the system may include a third device that performs additional automated assistants and / or include client device 110 and / or client device 105 that may be performing additional automated assistants and / or fewer automated assistants than those shown in the diagram.
[0034] Each of automation assistants 215, 220, and 245 can include one or more components of the automation assistant described herein. For example, automation assistant 215 may include its own voice capture component to process incoming queries, a visual capture component to process incoming visual data, a hot word detection engine, and / or other components. In some implementations, automation assistants executing on the same device, such as automation assistants 215 and 220, can share one or more components that can be used by both automation assistants. For example, automation assistants 215 and 220 may share one or more of the on-device speech recognizer, on-device NLU engine, and / or other components.
[0035] In some implementations, one or more automation assistants may share the results of one or more modules, such as a natural language processor, and / or a natural language, TTS, and / or STT processor. For example, refer again... Figure 2 Both the first automation assistant 215 and the second automation assistant 220 can share natural language processing, such that when the client device 105 receives audio data, the audio data is processed once into text, and the text can then be provided to both automation assistants 215 and 220. Furthermore, for example, one or more components of the client device 105 can process the audio data into text and provide the text representation of the audio data to a third automation assistant 245, as further described herein. In some embodiments, the audio data may not be processed into text and may instead be provided to one or more automation assistants as raw audio data.
[0036] refer to Figure 3 The illustration shows an example of two automation assistants, 305 and 310. As shown, the initially invoked automation assistant 305 executes on a first client device 301, and the subsequently invoked automation assistant 310 executes on a second client device 302. However, in some implementations, automation assistants 305 and 310 can both execute on the same device, such as... Figure 2 As shown.
[0037] The initially invoked automation assistant 305 can be invoked by one or more phrases captured via microphone 320 and processed by invocation engine 315. In some embodiments, the initially invoked automation assistant 305 can be invoked by a user via one or more gestures captured by a camera (not shown) and processed by invocation engine 315. Invocation engine 315 can determine whether the user is interested in using the initially invoked automation assistant 305 based on spoken words and / or gestures. Similarly, the subsequently invoked automation assistant 310 includes invocation engine 385, which can determine whether the user is interested in interacting with the subsequently invoked automation assistant 310 by similar phrases and / or gestures when uttering the phrases, which are unique to the invocation of the subsequently invoked automation assistant 310.
[0038] The initially invoked automation assistant 305 includes a query processing engine 330 capable of processing queries submitted by the user, such as those captured by a microphone 320. In some implementations, the user can converse with the initially invoked automation assistant 305. For example, the user can submit the query "What is the weather today?", the query processing engine 330 can generate a response, and the initially invoked automation assistant 305 can respond to the query with, for example, "It is sunny and 80 degrees today." The user can then submit additional queries that may be related to or unrelated to the previous query. For example, the user can submit a follow-up query, "What will it be tomorrow?", indicating an interest in being provided with tomorrow's weather, and the query processing engine 330 can generate a response based on the conversation that has already taken place with the user. In response, the initially invoked automation assistant 305 can provide an appropriate weather forecast. Furthermore, for example, users can submit unrelated queries such as "What's a good movie to watch?" and be provided with appropriate responses unrelated to their previous queries.
[0039] In some implementations, a user may submit a query that the initially invoked automation assistant 305 cannot process. For example, a user may submit a query to automation assistant 105 asking "What's on my calendar today?", which may not have access to user 101's calendar application. In response, query processing engine 330 can provide a negative response, such as "I'm sorry, I don't have access to your calendar." In instances where this occurs, the user may need to submit a query to a subsequently invoked automation assistant 310, which may have access to the user's calendar. If so, automation assistant 110 can provide a response such as "You have a meeting at three today." In instances where neither the initially invoked automation assistant 305 nor the subsequently invoked automation assistant 310 has access to the user's calendar application, the user may further need to invoke a different automation assistant or directly access the calendar application. Therefore, in those instances, computing resources were wasted in order to allow users to determine which automation assistant could provide the information they were interested in.
[0040] refer to Figure 4A A diagram illustrating the data flow between user 101, the initially invoked automation assistant 305, and the subsequently invoked automation assistant 310 is provided. Initially, a dialogue 415 can occur between user 101 and the initially invoked automation assistant 305. For example, the user can say one or more phrases to invoke the initially invoked automation assistant 305, such as "OK Assistant A". This phrase can be a phrase that only invokes the initially invoked automation assistant 305 and does not invoke the subsequently invoked automation assistant 310.
[0041] In some instances, the initially invoked automation assistant 305 may not be configured to handle user 101's request. In some implementations, the initially invoked automation assistant 305 may suggest that the user transfer the request to the subsequently invoked automation assistant 310. For example, user 101 may submit a query for "Play that on Streaming Service," and the initially invoked automation assistant 305 may not be configured and / or otherwise enabled to communicate with the "Streaming Service." However, the subsequently invoked automation assistant 310 may provide indications, such as ultrasonic signals broadcast by speaker 370 and captured by microphone 320, indicating that it is configured to communicate with the "Streaming Service." In those instances, the initially invoked automation assistant 305 may respond with a response such as "I'm sorry, I can't do that. Would you like me to ask Assistant B?" referring to the subsequently invoked automation assistant 310. In some implementations, suggestions may not be provided to user 101. Instead, the initially invoked automation assistant 305 may determine that the request "Play that on Streaming Service" is a request to transfer the conversation to the subsequently invoked automation assistant 310. Therefore, in some instances, a positive response to the transfer suggestion 420 (e.g., a "Yes" response to "Would you like me to have Assistant B do that?") could be a transfer request 425. Furthermore, in instances without a transfer suggestion 420, a request from user 101 to perform an action that was not configured to be performed by the initially invoked automation assistant 305 could be a transfer request (e.g., "Play that on Streaming Service"). Additionally, for example, a request for the automation assistant to perform an action that has previously been set as a user preference could be a transfer request in the instance where that action was requested.
[0042] Once user 101 has requested a transfer and / or confirmed the transfer proposal, the initially invoked automation assistant 305 can invoke the subsequently invoked automation assistant 310. In some implementations, the initially invoked automation assistant 305 can communicate via its assistant coordination module 325 with one or more additional automation assistants—such as the assistant coordination module 375 of the subsequently invoked automation assistant 310—performing on the same or different devices. For example, the initially invoked automation assistant 305 can be invoked by the user and the assistant coordination module 325 can utilize one or more protocols, such as Wi-Fi, Bluetooth, ultrasonic signals broadcast by the client device 105 via speaker 350 and received by the subsequently invoked automation assistant 302 via microphone 365, APIs 390 and 345, and / or other communication channels.
[0043] After the subsequently invoked automation assistant 310 is invoked, the initially invoked automation assistant 305 is able to transfer at least a portion of the dialogue data 440 to the subsequently invoked automation assistant 310. The dialogue data may include audio data of the user speaking a request, which can then be further analyzed by the subsequently invoked automation assistant 310 via ASR and / or NLU. In some implementations, the initially invoked automation assistant 305 may perform ASR and / or NLU, and the dialogue data may include a text representation of the dialogue and / or NLU output.
[0044] In response to the provided dialogue data, the query processing engine 340 can process the query, and the subsequently invoked automation assistant 310 can perform one or more actions 430 in response to the dialogue data provided by the initially invoked automation assistant 305. Actions can include, for example, accessing one or more applications, communicating with one or more other devices, performing a search, providing synthesized speech output, and / or other actions that the subsequently invoked automation assistant 310 is configured to perform. Once one or more actions 430 are performed, the dialogue can continue between the user and the initially invoked automation assistant 305 and / or the subsequently invoked automation assistant 310.
[0045] As an example, refer to Figure 5AThe diagram illustrates a dialogue between user 101 and two automation assistants 305 and 310. The dialogue does not include transfer suggestions. Initially, the dialogue occurs between the user and the initially invoked automation assistant 305. The dialogue begins with the user requesting “OK Assistant, What’s a good animated movie?” 505, and the initially invoked automation assistant 305 responds with “Movie A is a good animated movie” 510. In this instance, the request can be handled by the initially invoked automation assistant 305. Next, the user requests “OK Assistant, play it on Streaming Service” 515. The initially invoked automation assistant 305 is able to determine that it is not configured to access and / or utilize the “Streaming Service”, and further determines that the subsequently invoked automation assistant 310 is capable of handling the request (e.g., configured to communicate with the “Streaming Service”). The initially invoked automation assistant 305 can invoke the subsequently invoked automation assistant 310 and provide dialogue data that allows the subsequently invoked automation assistant 310 to perform the action of playing "Movie A" on the "Streaming Service". For example, the initially invoked automation assistant 305 can provide audio data of the dialogue between user 101 and the initially invoked automation assistant 305 (e.g., dialogue segments 505, 510, and / or 515), a text representation of the dialogue, and / or other representations of the intent of the dialogue. In response, the subsequently invoked automation assistant 310 provides a synthesized speech output of "Playing now on Streaming Service" 520, and plays movie A on the streaming service. In this example, the transfer request can include the user's previous instruction via one or more user preferences and / or past behaviors that the "Streaming Service" query be handled by the subsequently invoked automation assistant 310.
[0046] As another example, see Figure 5BThe diagram illustrates a dialogue between user 101 and two automation assistants 305 and 310. In this illustrated dialogue, user 101 is offered a suggestion to transfer at least a portion of the dialogue with the initially invoked automation assistant 305 to the subsequently invoked automation assistant 310. Initially, the dialogue occurs between user 101 and the initially invoked automation assistant 305. The dialogue begins with the user requesting, “OK Assistant, what’s on my calendar?” 525. The initially invoked automation assistant 305 may not have access to the user’s calendar application, and it may be determined that automation assistant 310 has access to the user’s calendar application based on indications provided by other automation assistants as described herein. The initially invoked automation assistant 305 responds with, “I don’t have access to your calendar. Would you like me to check with Assistant 2?” 530. This response includes an explicit suggestion to user 101 to transfer the dialogue to “Assistant 2,” which is a reference to the subsequently invoked automation assistant 310. User 101 responds with "Yes" 535, and the initially invoked automation assistant 305 is able to invoke assistant 310, as previously described, and further transfer all or part of the previous conversation (or data indicating the conversation) to the subsequently invoked automation assistant 310. The subsequently invoked automation assistant 310—which has access to the user's calendar application—is able to continue the conversation by responding with information from the user's calendar 540.
[0047] refer to Figure 4B A diagram illustrating the data flow between user 101, the initially invoked automation assistant 305, and the subsequently invoked automation assistant 310 is provided. Initially, a dialogue 445 can occur between user 101 and the initially invoked automation assistant 305. For example, the user can say one or more phrases to invoke the initially invoked automation assistant 305, such as "OK Assistant A". This phrase can be a phrase that only invokes the initially invoked automation assistant 305 and does not invoke the subsequently invoked automation assistant 310. The dialogue can continue between the initially invoked automation assistant 305 and user 101, such as "What is a good movie to watch?" from user 101, followed by a response from the initially invoked automation assistant 305, "Movie A is a good movie to watch".
[0048] At some point in the conversation with the initially invoked automation assistant 305, the user can invoke the subsequently invoked automation assistant 310 by uttering a unique hotword, such as "OK Assistant B". The hotword can then lead to a conversation 450 with the subsequently invoked automation assistant 310, such as "OK Assistant B, play it on Streaming Service". Therefore, in this example, user 101 has explicitly invoked automation assistant 310 and directly begun a conversation with assistant 310.
[0049] In some instances, the subsequently invoked automation assistant 310 may need information about the previous conversation with the initially invoked automation assistant 305 in order to determine the intent and / or context of the request. For example, in conversation 445, a user might say “What’s a good movie?” The initially invoked automation assistant 305 could respond with “Movie A is a good movie.” User 101 could then say “OK Assistant B, play it on StreamingService” as an additional conversation 450. Once invoked, the subsequently invoked automation assistant 310 is then able to process user 101’s utterance. However, at that moment, the subsequently invoked automation assistant 310 may not have the context to resolve the meaning of “it” in the user’s request. Therefore, the initially invoked automation assistant can provide conversation data 465 to the subsequently invoked automation assistant 310 to determine the context and / or intent in conversation 450.
[0050] In some implementations, when the subsequently invoked automation assistant 310 has been invoked and is processing the utterances of user 101, the verification module 395 is able to verify that the speaker of the utterances (i.e., user 101) has previously participated in a conversation with the initially invoked automation assistant 305. In some implementations, the verification module 335 is able to provide speaker verification data 455 to the verification module 395, which is then able to determine whether the speaker of the previous conversation with the initially invoked automation assistant 305 is the same speaker as the speaker of conversation 445. In some implementations, the microphone 365 is able to generate audio data based on the user's utterances during the previous conversation, and the verification module 335 is able to process the audio data to verify that the speaker of conversation 445 (which has been captured by the microphone 365) is the same as the speaker of conversation 450 (also captured by the microphone 365). Therefore, in some instances, the conversation data 445 has already been received by the subsequently invoked automation assistant 310 and the conversation data does not need to be provided by the initially invoked automation assistant 305.
[0051] Speaker verification data may include, for example, all or part of the dialogue captured by the microphone 365 of the subsequently invoked automation assistant 310, the speaker's voice profile, and / or other data that allows the verification module 395 to verify that the speaker of dialogue 450 is the same speaker as in dialogue 445. Once speaker verification data 455 has been provided, the initially invoked automation assistant 305 is able to determine whether the same speaker has already participated in a dialogue with the initially invoked automation assistant 305. If the initially invoked automation assistant 305 determines that a dialogue with the user has occurred, the initially invoked automation assistant 305 is able to provide verification confirmation 460 to the subsequently invoked automation assistant 310. This confirmation indicates that the speaker has previously participated in the initially invoked automation assistant 305 and the initially invoked automation assistant 305 is able to further provide dialogue data 465 of the dialogue that occurred between the user 101 and the initially invoked automation assistant 305. Therefore, the subsequently invoked automation assistant 310 is able to determine the context of the dialogue and resolve any ambiguity in the dialogue 450 previously captured by the microphone associated with the subsequently invoked automation assistant 310. The subsequently invoked automation assistant 310 can then respond to the dialogue 450 to perform one or more actions 470, such as accessing an application and / or providing synthesized speech as a response.
[0052] As an example, refer to Figure 5CThe diagram illustrates a dialogue between user 101, the initially invoked automation assistant 305, and the subsequently invoked automation assistant 310. User 101 provides the spoken phrase 545, “OK Assistant 1, What's a good animated movie to buy?”, which is directed to the initially invoked automation assistant 305. The initially invoked automation assistant 305 responds to this phrase with “Movie B is highly rated” 550. In response, user 101 provides the phrase “OK Assistant 2, buy it on the Online Store” 555, which is directed to the subsequently invoked automation assistant 310 and captured by one or more microphones of the client device executing the subsequently invoked automation assistant 310. The subsequently invoked automation assistant 310 is able to provide the initially invoked automation assistant 305 with speaker verification data of the user who uttered phrase 555, enabling the initially invoked automation assistant 305 to verify that phrase 555 was provided by the same user as phrase 545. In some implementations, once the same speaker has been verified, the subsequently invoked automation assistant 310 can be provided with prior dialogue data (e.g., utterance 545). In some implementations, the subsequently invoked automation assistant 310 may already have prior dialogue data, having captured utterance 545 via one or more microphones of the client device executing the assistant. For example, in some implementations, the subsequently invoked automation assistant 310 may continuously capture a limited amount of audio, and once invoked, utilize the previously captured audio directed to the initially invoked automation assistant 305 after verifying that the speaker is the same in both instances.
[0053] Figure 6 A flowchart illustrating an example method 600 for processing a dialogue with a first automation assistant and providing dialogue data to a second automation assistant is provided. For convenience, the operation of method 600 is described with reference to a system performing the operation. This system of method 600 includes one or more processors and / or other components of the client device. Furthermore, although the operations of method 600 are shown in a specific order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.
[0054] At step 605, the initially invoked automation assistant engages in a dialogue with the user. The initially invoked automation assistant can be invoked by one or more hot words spoken by the user and processed by the invocation engine 315. Once invoked, the query processing engine 330 is able to process the user's utterance and determine one or more actions to be performed in response to the user's utterance. For example, the initially invoked automation assistant 305 can be invoked with the phrase "OK Assistant A" followed by the query "What's a good animated movie". The query processing engine 330 can determine the response "Movie A is a good movie", which can be provided to the user via synthesized speech. The dialogue can continue between the user and the initially invoked automation assistant 305, allowing the user to provide additional utterances, which may be related to (and / or related to) the response from the initially invoked automation assistant 305 or unrelated.
[0055] At step 610, the initially invoked automation assistant receives a request to transfer a portion of the conversation to a second automation assistant. Components sharing one or more features with the assistant coordination module 325 are capable of determining whether another automation assistant—such as the subsequently invoked automation assistant 310—is configured to handle the conversation data. For example, the assistant coordination module 375 can provide indications, such as audible signals via speaker 370 and / or via APIs 345 and 390, which can be received by the initially invoked automation assistant 305, indicating the type of query that can be handled by the subsequently invoked automation assistant 310. As an example, the subsequently invoked automation assistant 310 can be configured to communicate with the user's calendar application and can provide ultrasonic signals that can be received by microphone 320 to indicate to the initially invoked automation assistant 305 that queries related to the user's calendar application can be handled by the subsequently invoked automation assistant 310.
[0056] In some implementations, a user can request that conversation data be transferred to a subsequently invoked automation assistant. In some instances, a user can explicitly request that a conversation initiated by the initially invoked automation assistant be continued by a second automation assistant. For example, a user could say, "OK Assistant 1, Play Song on Assistant 2." In some instances, a user can state a query that the initially invoked automation assistant cannot handle. For example, in a conversation with the initially invoked automation assistant, a user could say, "OK Assistant, buy Movie1 on Online Store," which could be a request to transfer the conversation to a different automation assistant capable of fulfilling that request.
[0057] In some implementations, the transfer request can be provided by the user in response to a transfer suggestion from the initially invoked automation assistant. For example, the user might say, “OK Assistant 1, what’s on my calendar?” and the initially invoked automation assistant might not be configured to fulfill that request. The initially invoked automation assistant could respond with, “I can’t access your calendar, would you like me to ask Assistant 2?” The user can then say a transfer request, such as “Yes,” and the initially invoked automation assistant 305 can invoke the subsequently invoked automation assistant 310 to transfer the conversation data.
[0058] In some implementations, the subsequently invoked automation assistant 310 may request the transfer of dialogue data. For example, in some instances, user 101 may invoke the initially invoked automation assistant 305 and participate in the dialogue, such as... Figure 5C The dialogue is shown below. At some point in the dialogue, the user stops interacting with the initially invoked automation assistant 305 and instead invokes a subsequently invoked automation assistant with dialogue 555 (i.e., "OK Assistant 2, buy it on Online Store"). At this point, the subsequently invoked automation assistant 310 may not have access to the previous dialogue because it was not invoked when dialogue rounds 545 and 550 occurred. In those instances, the subsequently invoked automation assistant 310 is able to submit a request to the initially invoked automation assistant 305 to transfer the dialogue (or data instructing the dialogue) to the subsequently invoked automation assistant 310.
[0059] In some implementations, the subsequently invoked automation assistant 310 can provide the initially invoked automation assistant 305 with a request for transfer dialogue data and voice verification data to verify that the speaker in the dialogue with the initially invoked automation assistant 305 is the same speaker as the speaker who invoked (and subsequently interacted with) the subsequently invoked automation assistant 310. For example, see reference... Figure 3The verification module 335 can provide data indicating the speaker of the call and / or utterance to the subsequently invoked automation assistant 310 via one or more communication channels. The verification module 395 can determine, based on, for example, voice profile data and / or audio data of the utterance directed to the subsequently invoked automation assistant 310, that the speaker of the previous dialogue is the same user 101 as the one in the subsequent utterance. Once verified, at step 615, the dialogue data is provided by the initially invoked automation assistant 305 to the second (or subsequently invoked) automation assistant 310.
[0060] In some implementations, the dialogue data may be audio data from a previous dialogue, from which the query processing engine 340 can determine the intent and / or context of subsequent utterances. In some implementations, the intent and / or context may be determined by the query processing engine 330 of the initially invoked automation assistant 305, and this intent and / or context may be provided as dialogue data to the subsequently invoked automation assistant 310. In those implementations, the query processing engine 340 can determine one or more actions to be performed based on subsequent utterances and the previous dialogue.
[0061] At step 620, the second automated assistant performs one or more actions in response to being provided with dialogue data. Actions may include, for example, the subsequently invoked automated assistant fulfilling a request and / or providing a response based on the intent and / or context-resolved utterances of the previous dialogue. In some implementations, actions may include the subsequently invoked automated assistant interacting with one or more other applications. For example, refer again... Figure 5C The dialogue, and the actions performed by the subsequently invoked automation assistant 310, can include interacting with online store applications and / or websites. Other actions may include the subsequently invoked automation assistant 310 responding with a completed response, interacting with calendar applications, interacting with streaming services, and / or interacting with other applications.
[0062] Figure 7 A flowchart illustrating an example method 700 for authorizing a second automation assistant to process dialogue data from a conversation between a user and the initially invoked automation assistant is shown. At step 705, a verbal query is received by the first automation assistant. The first automation assistant is capable of sharing one or more features with the initially invoked automation assistant 305. In some embodiments, step 705 is capable of... Figure 6 Step 605 shares one or more features.
[0063] At step 710, the first automation assistant identifies a previous conversation between the user and the first (invoked) automation assistant. The conversation can include one or more utterances from the user and one or more responses from the first automation assistant. For example, refer to... Figure 5B The previous dialogue can include the user who spoke 525 and the response 530 of the first automation assistant 305. The previous dialogue data can be determined by a component that shares one or more characteristics with the query processing engine 330.
[0064] At step 715, authorization for the second automation assistant can be provided to the first automation assistant 305. For example, the authorization can include user 101's utterance 535, which is included in the transfer suggestion 530. Additionally, for example and reference... Figure 5A The transfer of authorization can be implied by the words of user 101, such as words 515 instructing user 101 to authorize the continuation of the same conversation with the second automation assistant 310.
[0065] At step 720, the query instruction is provided to the second automation assistant. In some embodiments, step 720 can be combined with... Figure 6 Step 615 shares one or more features. For example, it indicates the ability to include the conversation that previously occurred between the first automation assistant and the user, as well as the intent and / or context of the conversation, and / or other data that allows the second automation assistant to parse one or more intents and / or determine the context of subsequent utterances directed to the second automation assistant.
[0066] At step 725, the second automation assistant is able to perform one or more actions in response to user speech directed to the second automation assistant. Step 725 is able to... Figure 6 Step 620 shares one or more features. For example, one or more actions may include: providing a second automation assistant that is configured to respond to the response; and / or a second automation assistant that interacts with one or more applications.
[0067] Figure 8 A flowchart of an example method 800 is shown, which is used to verify that the speaker of the conversation with the initially invoked automation assistant is the same speaker of the request for the dialogue data to be processed by the subsequently invoked automation assistant.
[0068] At step 805, the user who provided the request to subsequently invoke the automated assistant using the verbal utterance from previous conversational data is identified. For example, refer to Figure 5CThe utterance 555 is a utterance that invokes the automation assistant 310 and requests that dialogue data be transferred to the subsequently invoked automation assistant 310. In some implementations, the request to transfer dialogue data can be directed to the first automation assistant 305, such as "OK Assistant, see if Assistant 2 can handle this request." In some implementations, the request to transfer dialogue data can be directed to the subsequently invoked automation assistant 310, such as... Figure 5C As shown. In some implementations, the user's request may be in response to a suggestion to transfer dialogue data, such as... Figure 5B As shown.
[0069] At step 810, audio data of the captured spoken words is determined. The audio data can be captured by a component that shares one or more characteristics with microphone 365 and / or microphone 320. For example, microphone 320 can capture spoken words, and assistant coordination module 325 can transfer the audio data to the subsequently invoked automation assistant 310. Alternatively, for example, microphone 365 can directly capture audio data that, in addition to being a transfer request, can also invoke the subsequently invoked automation assistant 310 via invocation engine 385.
[0070] At step 815, one or more components determine whether the audio data was provided by the same user as in the previous conversation. These components may share one or more features with verification module 335 and / or verification module 395. For example, the audio data may be captured by microphone 365 and sent to the initially invoked automation assistant 305, and verification module 395 may determine whether the audio data was provided by the same user as in the previous conversation. Furthermore, for example, the initially invoked automation assistant 305 may provide verification data to the subsequently invoked automation assistant 310, and verification module 335 may determine whether the speaker of the spoken words (captured by microphone 365) is the same as the speaker in the previous conversation.
[0071] At step 820, previous dialogue data is provided to the subsequently invoked automation assistant. The previous dialogue data may include audio data of the previous dialogue, the intent and / or context of the previous dialogue, and / or other data that the subsequently invoked automation assistant can use to determine the intent and / or context of the user's subsequent utterances. In some embodiments, where the verification in step 815 indicates that the speaker is different from the speaker in the previous dialogue, one or more components may prevent the dialogue data from being transferred to the subsequently invoked automation assistant 315.
[0072] Figure 9This is a block diagram of an example computing device 910 that can be optionally used to perform one or more aspects of the techniques described herein. The computing device 910 typically includes at least one processor 914 that communicates with a plurality of peripheral devices via a bus subsystem 912. These peripheral devices may include a storage subsystem 924, including, for example, a memory subsystem 925 and a file storage subsystem 926, a user interface output device 920, a user interface input device 922, and a network interface subsystem 916. The input and output devices allow users to interact with the computing device 910. The network interface subsystem 916 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.
[0073] User interface input device 922 may include a keyboard, a pointing device such as a mouse, trackball, touchpad, or drawing tablet, a scanner, a touchscreen integrated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. Generally, the term "input device" is used to include all possible types of devices and methods of inputting information into computing device 910 or into a communication network.
[0074] User interface output device 920 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. Generally, the term "output device" is used to encompass all possible types of devices and the manner in which information is output from computing device 910 to the user or to another machine or computing device.
[0075] Storage subsystem 924 provides the functional programming and data structures for some or all of the modules described herein. For example, storage subsystem 924 may include methods for performing... Figure 6-8 The logic of the selected aspect of the method, and / or its implementation. Figure 2 The various components depicted in -4.
[0076] These software modules are typically executed by processor 914 alone or in conjunction with other processors. The memory 925 used in storage subsystem 924 can include multiple memories, including main random access memory (RAM) 930 for storing instructions and data during program execution and read-only memory (ROM) 932 for storing fixed instructions. File storage subsystem 926 provides persistent storage for program and data files and may include hard disk drives, floppy disk drives, and associated removable media, CD-ROM drives, optical drives, or removable media cartridges. Functional modules implementing certain embodiments may be stored by file storage subsystem 926 within storage subsystem 924 or in other machines accessible to processor 914.
[0077] The bus subsystem 912 provides a mechanism for allowing various components and subsystems of the computing device 910 to communicate with each other as intended. Although the bus subsystem 912 is schematically shown as a single bus, alternative implementations of the bus subsystem may use multiple buses.
[0078] The computing device 910 can be any type of workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Figure 9 The description of the computing device 910 depicted herein is intended only as a specific example for illustrating some implementations. Many other configurations of the computing device 910 may have... Figure 9 The computing device depicted in the image has more or fewer components.
[0079] In some implementations, a method implemented by one or more processors is provided, the method comprising, during a dialogue between a user and an initially invoked automation assistant: receiving, by the initially invoked automation assistant and via a first automation assistant interface on a client device in an environment having the user, spoken words of the user directed to the initially invoked automation assistant as part of a dialogue; processing the spoken words by the initially invoked automation assistant to generate a response to the spoken words; making the response to the spoken words rendered by the client device as part of the dialogue by the initially invoked automation assistant; and receiving, by the initially invoked automation assistant and after making the response rendered, a request to transfer dialogue data generated by the first automation assistant as part of the dialogue to a second automation assistant. In response to receiving the request, the method further comprises providing the dialogue data by the first automation assistant and via a communication channel with the second automation assistant, wherein providing the dialogue data causes the second automation assistant to perform one or more actions based on the dialogue data.
[0080] These and other implementations of the technology disclosed herein can include one or more of the following features.
[0081] In some implementations, the request to be transferred is an additional verbal message from the user who originally invoked the automation assistant.
[0082] In some implementations, the request to be transferred is a request from a second automated assistant to process spoken utterances. In some of those implementations, the request to be transferred includes audio data indicating spoken utterances. In some of those implementations, the method further includes: the initially invoked automated assistant verifying, based on processing the audio data using a speaker verification model, that the user is the speaker indicated by the audio data; wherein providing dialogue data to the second automated assistant depends on verifying that the user is the speaker indicated by the audio data.
[0083] In some implementations, the second automation assistant operates on a second device separate from the first device, and one or more actions include generating second assistant audio output to the user via a second client device.
[0084] In some implementations, the request to be processed includes instructions for one or more actions to be performed by a second automation assistant, wherein the originally invoked assistant cannot perform the one or more actions.
[0085] In some implementations, another method implemented by one or more processors is provided, and includes: receiving a verbal query from an initially invoked automation assistant client, the user's verbal query being captured in audio data generated by one or more microphones of a first client device in the user's environment, the first automation assistant client being installed on the first client device; identifying a user intent indicated by one or more terms in the dialogue based on a previous conversation between the user and the initially invoked automation assistant indicated by the audio data; authorizing a second automation assistant client installed on a second client device in the environment to process the query based on the user intent; and providing the indication of the query and the user intent to the second automation assistant client by the initially invoked automation assistant client and via a communication channel with the second automation assistant client, wherein providing the indication and the user intent causes the second automation assistant client to perform one or more actions.
[0086] These and other implementations of the technology disclosed herein can include one or more of the following features.
[0087] In some implementations, the communication channel includes an ultrasonic communication channel, and the step of providing the query indication and user intent to the second automation assistant client includes rendering an ultrasonic signal containing the query indication and user intent via one or more speakers of the first client device.
[0088] In some implementations, the method further includes receiving a second verbal query from a user requesting a second automated assistant client to process the query, wherein authorization is provided in response to receiving the second verbal query.
[0089] In some implementations, the method further includes determining that a first automation assistant cannot process a user intent, offering a suggestion to the user to provide authorization, and receiving authorization from the user, wherein authorization is provided to a second automation assistant client in response to receiving authorization from the user.
[0090] In some implementations, the method further includes performing automatic speech recognition on the audio data of the captured spoken query to generate a text query, wherein the indication of the query is a text query.
[0091] In some implementations, another method implemented by one or more processors is provided, and includes determining at the initially invoked automation assistant that a user has provided a spoken utterance requesting a subsequently invoked automation assistant to utilize dialogue data from a previous conversation between the user and the initially invoked automation assistant. In response to determining that the user has provided the spoken utterance, the method further includes processing audio data of the captured spoken utterance by the initially invoked automation assistant, and determining, based on the processing, whether the audio data was provided by the same user who participated in the previous conversation. When it is determined that the user is the same user, the method further includes providing the previous dialogue data to the subsequently invoked automation assistant. When it is determined that the user is not the same user, the method includes preventing the provision of dialogue data.
[0092] These and other implementations of the technology disclosed herein can include one or more of the following features.
[0093] In some implementations, the audio data is captured by the microphone of the first device that performs the initial call to the automation assistant.
[0094] In some implementations, the audio data is captured by the microphone of a second device that subsequently invokes the automation assistant.
[0095] In some implementations, spoken words are directed to the initially invoked automation assistant.
[0096] In some implementations, spoken words are directed to an automated assistant that is subsequently invoked.
[0097] In some implementations, prior conversation data is provided via a non-human audible signal generated by the speaker of a first client device that initiated the automated assistant and received by one or more microphones of a second client device that subsequently initiated the automated assistant.
[0098] In some implementations, previous dialogue data is provided via an application programming interface between the initially invoked automation assistant and subsequent invoked automation assistants.
[0099] In some implementations, prior dialogue data includes a textual representation of the dialogue between the user and the initially invoked automation assistant.
[0100] In some implementations, the prior conversation data includes audio data of the conversation between the user and the initially invoked automation assistant.
[0101] In situations where certain implementations discussed herein may collect or use personal information about users (e.g., user data extracted from other electronic communications, information about user social networks, user location, user time, user biometric information and user activity, and demographic information, relationships between users, etc.), users are provided with one or more opportunities to control whether information is collected, whether personal information is stored, whether personal information is used, and how information is collected, stored, and used regarding users. That is, the systems and methods discussed herein only collect, store, and / or use user personal information upon receiving explicit authorization from the relevant user.
[0102] For example, providing users with control over whether a program or feature collects user information about that specific user or other users associated with that program or feature. Presenting one or more options to each user whose personal information is to be collected allows control over the collection of information related to that user, granting permission or authorization on whether to collect information and which parts of the information to collect. For example, it is possible to provide users with one or more such control options via a communication network. Furthermore, some data may be processed in one or more ways before storage or use, resulting in the removal of personally identifiable information. As an example, a user's identity may be processed to make personally identifiable information impossible to determine. As another example, a user's geographic location may be generalized to a larger area, making it impossible to determine the user's specific location.
[0103] While several embodiments have been described and illustrated herein, various other means and / or structures described herein for performing functions and / or obtaining results and / or one or more advantages may be utilized, and each of these variations and / or modifications is considered to be within the scope of the embodiments described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications using the teachings. Those skilled in the art will recognize or be able to determine many equivalents to the specific embodiments described herein using no more than conventional experimentation. Therefore, it should be understood that the foregoing embodiments are presented by way of example only, and embodiments may be practiced in ways different from those specifically described and claimed within the scope of the appended claims and their equivalents. Embodiments of this disclosure relate to each individual feature, system, article, material, kit, and / or method described herein. Furthermore, any combination of two or more such features, systems, articles, materials, kits, and / or methods, provided that such class of features, systems, articles, materials, kits, and / or methods are not inconsistent with each other, is included within the scope of this disclosure.
Claims
1. A method for transferring dialogue data, implemented by one or more processors, the method comprising: During the conversation between the user and the initially invoked automated assistant: The user's spoken words, directed to the initially invoked automation assistant and received via a first automation assistant interface on a client device in an environment with the user, are received as part of the dialogue. The spoken words are processed by the initially invoked automation assistant to generate a response to the spoken words; The automated assistant, which was initially invoked, causes the response to the spoken words to be rendered by the client device as part of the conversation; The first automated assistant, after rendering the response, receives a request to transfer dialogue data generated by the first automated assistant as part of the dialogue to a subsequently invoked automated assistant, wherein the request to transfer dialogue data generated by the first automated assistant as part of the dialogue to the subsequently invoked automated assistant is a request from the subsequently invoked automated assistant to the first invoked automated assistant and is provided by the subsequently invoked automated assistant for processing the spoken utterance based on a call initiated by the user of the subsequently invoked automated assistant; In response to receiving the request to transfer the dialogue data generated by the first automation assistant as part of the dialogue to a subsequently invoked automation assistant: The dialogue data is provided by the initially invoked automation assistant and to the subsequently invoked automation assistant via a communication channel. The provided dialogue data enables the subsequently invoked automation assistant to perform one or more actions based on the dialogue data.
2. The method according to claim 1, wherein, The request to transfer the dialogue data generated by the first automation assistant as part of the dialogue to a subsequently invoked automation assistant is additional verbal utterances from the user directed to the initially invoked automation assistant.
3. The method according to claim 1, wherein, The subsequently invoked automated assistant operates on a separate client device, detached from the client device, and wherein the one or more actions include generating an assistant audio output for the subsequent invocation provided to the user via the other client device.
4. The method according to claim 1, wherein, The request for processing the spoken words includes an instruction for one or more actions to be performed by the subsequently invoked automation assistant, wherein the initially invoked automation assistant cannot perform the one or more actions.
5. The method according to claim 1, wherein, The request to transfer the dialogue data generated by the first automation assistant as part of the dialogue to a subsequently invoked automation assistant includes audio data indicating the spoken words.
6. The method of claim 5, further comprising: The automated assistant, which was initially invoked, verifies that the user is the speaker indicated by the audio data by processing the audio data using a speaker verification model. The provision of the dialogue data to the subsequently invoked automation assistant depends on verifying that the user is the speaker indicated by the audio data.
7. A computer-implemented method for transferring dialogue data, comprising: The user's verbal query is received by the initial invoking automation assistant client, which is installed on the first client device. The user's verbal query is captured in audio data generated by one or more microphones of the first client device in the user's environment. Based on the previous conversation between the user and the automated assistant that was initially invoked, as indicated by the audio data, identify the user's intent indicated by one or more terms in the conversation; The automation assistant initially invoked receives instructions from a second automation assistant on a second client device in the environment, indicating that the second automation assistant can carry out user intents indicated by one or more terms in the dialogue; Authorization is granted to a second automation assistant client on the second client device installed in the environment to process the user's verbal query based on the user's intent, wherein authorization is granted to the second automation assistant client based on the second automation assistant's ability to implement the user's intent indicated by one or more terms in the dialogue; and The user's verbal query and intent are provided to the second automation assistant client by the initially invoked automation assistant client and via a communication channel with the second automation assistant client. The instructions and user intent are provided to cause the second automation assistant client to perform one or more actions.
8. The method according to claim 7, wherein, The communication channel includes an ultrasonic communication channel, and wherein providing the user's verbal query and the user's intent to the second automation assistant client includes: The ultrasonic signal containing the user's verbal query and the user's intent is rendered via one or more speakers of the first client device.
9. The method of claim 7, further comprising: The second automated assistant client receives a request from the user to process a second verbal query from the user, wherein the authorization is provided in response to receiving the second verbal query.
10. The method of claim 7, further comprising: It was determined that the initially invoked automation assistant was unable to process the user's intent; Provide suggestions to the user to grant the authorization; as well as Receive the authorization from the user, wherein the authorization is provided to the second automation assistant client in response to receiving the authorization from the user.
11. The method according to any one of claims 7 to 10, further comprising: Automatic speech recognition is performed on the audio data capturing the user's verbal query to generate a text query, wherein the indication of the user's verbal query is the text query.
12. A non-volatile computer-readable storage medium storing instructions that, when executed by one or more processors of a computing system, cause the computing system to perform the method according to any one of claims 1 to 11.
13. One or more computing devices, the one or more computing devices being configured to perform the method according to any one of claims 1 to 11.
Citation Information
Patent Citations
Cross-device handoffs
CN110741431A
Transferring an automated assistant routine between client devices during execution of the routine
US20200265835A1
