Selection of multiple automated assistants based on invocation properties

Abstract

A system and method for determining which automated assistant to invoke instead of invoking other automated assistants based on an invocation input common to multiple automated assistants. The invocation input is processed to determine one or more invocation features, such as prosodic features or invocation phrases, that can be utilized to determine which of multiple candidate automated assistants to invoke. Additionally, additional features of the client are processed that indicate which of multiple invokable automated assistants to invoke. When an automated assistant is invoked, additional speech data and / or features of the additional speech data are provided to the invoked automated assistant for further processing.

Description

[Background technology]

[0001] Humans can use conversational software applications, referred to herein as "automated assistants" (also referred to as "digital agents," "chatbots," "interactive personal assistants," "intelligent personal assistants," "assistant applications," "conversational agents," etc.), to facilitate human-computer interactions. For example, a human (who may be referred to as a "user" when interacting with an automated assistant) can provide commands and / or requests to the automated assistant using natural language input via voice (i.e., speech) and / or by providing natural language input via text (e.g., typed), which in some cases may be converted to text and then processed. The automated assistant responds to requests by providing responsive user interface output, which may include audio and / or visual user interface output.

[0002] As mentioned above, many automated assistants are configured to interact via voice utterances, such as a voiced query following an invocation instruction. To protect user privacy and / or conserve resources, a user is often required to explicitly invoke the automated assistant before the automated assistant can fully process the utterance. Explicit invocation of an automated assistant typically occurs in response to a particular user interface input received at a client device. The client device includes an assistant interface that provides a user of the client device with an interface for interfacing with the automated assistant (e.g., receiving voice and / or typed input from the user and providing voice and / or graphical responses, etc.) and interfaces with one or more additional components that implement the automated assistant (e.g., a remote server device that processes the user input and generates an appropriate response, etc.).

[0003] Some user interface inputs that may invoke an automated assistant via a client device include hardware and / or virtual buttons on the client device (e.g., tapping a hardware button, selecting a graphical interface element displayed by the client device) to invoke the automated assistant. Many automated assistants may additionally or alternatively be invoked in response to one or more voice invocation phrases (also called "hot words / phrases" or "trigger words / phrases"). For example, an automated assistant may be invoked by speaking a voice invocation phrase such as "Hey, Assistant," "Okay, Assistant," and / or "Assistant." Assistants may also be invoked based on one or more gestures of the user, such as, for example, pressing a button on the device and / or making a particular movement to allow the device's camera to capture the movement.

[0004] In many cases, a client device that includes an assistant interface includes one or more locally stored models that the client device uses to monitor for the occurrence of a voice invocation phrase. Such a client device may utilize the locally stored models to locally process received audio data and discard audio data that does not include the voice invocation phrase. However, if the local processing of the received audio data indicates the occurrence of the voice invocation phrase, the client device causes that audio data and / or subsequent audio data to be further processed by the automated assistant. For example, if the voice invocation phrase is "Hey, Assistant" and the user speaks "Hey, Assistant, what time is it?", the audio data corresponding to "What time is it?" is processed by the automated assistant based on the detection of "Hey, Assistant" and utilized to provide an automated assistant response for the current time. On the other hand, if the user simply speaks "What time is it?" (without first speaking an invocation phrase or providing an alternative invocation input), no response from the automated assistant is provided because the invocation phrase (or other invocation input) does not precede "What time is it?". Summary of the Invention

[0005] Described herein are techniques for selecting a particular automated assistant to be invoked to process a request from a plurality of candidate automated assistants when a call is received that can selectively invoke any of a plurality of candidate automated assistants. In other words, according to the techniques disclosed herein, in some circumstances, a particular automated assistant may be exclusively invoked when a call is received, while in other circumstances, an alternative automated assistant from the plurality of candidate automated assistants may be exclusively invoked when a call is received. For example, various techniques are directed to utilizing a general automated assistant that can receive a call from a user to determine which of a plurality of secondary automated assistants to invoke based on characteristics of the call and / or based on characteristics of audio data provided with the call. For example, a user can utter a general invocation phrase such as "Okay, assistant" to invoke both a first automated assistant and a second automated assistant. The general automated assistant can determine whether to invoke the first automated assistant to provide a query instead of invoking the second automated assistant to provide a query based on one or more characteristics of the call, characteristics of a query provided by the user, and / or additional features other than speech recognition of the voice input (e.g., the invocation voice input and / or the query, etc.).

[0006] As an example, a user may have a client device with multiple automated assistants (e.g., client applications for at least multiple automated assistants, etc.) installed on the device. A first automated assistant and a second automated assistant running on the device may both be invoked with the invocation phrase "Okay, assistant." The invocation input of "Okay, assistant" may be processed to determine whether to invoke the first automated assistant or the second automated assistant based on one or more features of the invocation input (e.g., prosodic features, vocabulary use, etc.). Also, for example, one or more other features other than the speech recognition function (e.g., other applications run by the client device, the location of the client device, a classification of where the device is located, the presence of other users near the client device, etc.) may be processed to determine whether to invoke the first automated assistant or the second automated assistant.

[0007] As another example, a user may have a client device with a first instantiation of an automated assistant installed on the device and a second instantiation of the same automated assistant installed on the same device. For example, a user may have a "home" automated assistant to use when at home and a "work" automated assistant instantiation to use when at work. An invocation input of "Okay, Assistant" may invoke both the "home" assistant and the "work" assistant, and the invocation input may be processed to determine whether to invoke the "home" instantiation or the "work" instantiation based on one or more features described herein. The location may be a geographic location and / or a classification of the location of the client device, such as a "private" location versus a "public" location.

[0008] Thus, the techniques described herein may be utilized to mitigate the occurrence of a user accidentally invoking the wrong automated assistant(s) first when attempting to invoke an intended automated assistant. Mitigating the occurrence of an unintended automated assistant being inadvertently invoked by a user reduces computational and / or network resource utilization because the computational and / or network resource(s) that would have been utilized if the unintended automated assistant had been invoked are not utilized. Additionally, accommodating a single invocation available to multiple automated assistants may reduce the amount and / or duration of input(s) from a user to invoke a particular automated assistant, thereby reducing processing time and memory resources. Thus, a user may selectively invoke one of multiple automated assistants using a single invocation without requiring additional input from the user to specify which of the multiple automated assistants is intended to be invoked. Additionally, accommodating a single invocation available to multiple automated assistants may reduce or eliminate the need to load and / or utilize multiple invocation models, each used to monitor the invocation(s) of each respective automated assistant. For example, continually loading and continually using a first hotword detection model for a first automated assistant and a second hotword detection model for a second automated assistant may consume more processor, memory resources, and power than loading and using a general hotword detection model. Also, having multiple hotword detection models monitoring multiple phrases, for example, may increase the incidence of false positives of hotwords, unnecessarily consuming resources. Additionally, digital signal processors (DSPs) in client devices are often used to process audio data with hotword detection models. DSPs may have limited memory and processor resources and may not be able to load and / or run multiple hotword detection models in parallel (or at least may be prevented from doing so without running other process(es) on the DSP).

[0009] In some implementations, both the first and second assistants can be configured to be invoked by the same invocation. In some implementations, the invocation can be a voice utterance from the user indicating which of the multiple automated assistants the user is interested in providing a voice query to. For example, both the first and second automated assistants can be invoked with the invocation phrase "Okay, assistant." In some implementations, the invocation can be an action from the user. For example, both the first and second automated assistants can be invoked by the user performing a particular gesture, pressing a button on the client device, looking at the client device, and / or performing one or more other actions that can be captured by one or more cameras on the client device.

[0010] In some implementations, the call is received and / or detected by a general automated assistant that is not configured to generate a response to the query. For example, the general automated assistant may be a "meta-assistant" configured to receive the call input, process the call input to determine which of multiple secondary automated assistants to call based on one or more features, and call the intended secondary automated assistant. In this manner, the call input and / or additional voice input before and after the call input can be processed before providing the call input and / or additional voice data to the secondary automated assistant for further processing.

[0011] In some implementations, one or more call features of the call input can be processed to determine whether to call the first automated assistant or the second automated assistant. For example, one or more prosodic features of the voice call input determined from voice data detected by one or more microphones of the client device can be utilized to determine whether to call the first automated assistant or the second automated assistant. The call input can be processed to determine prosodic features of the voice input (e.g., tone, speech rate, etc.) to call the first automated assistant instead of calling the second automated assistant based on determining that the prosodic features detected in the voice data are more likely to be present in the call to the first automated assistant than in the call to the second automated assistant.

[0012] In some implementations, in addition to or instead of the features of the call input, one or more additional features detected by the client device can be processed to determine whether to call the first automated assistant or the second automated assistant. The additional features processed to determine whether to call the first automated assistant or the second automated assistant can be added to the automatic speech recognition features of the call input. For example, the additional features may include one or more terms used by the user in a query preceding or following the call input that is more likely to be utilized when calling (and / or providing a query for) an automated assistant for "home" than when calling (and / or providing a query for) an automated assistant for "work."

[0013] In some implementations, the one or more additional features may include one or more applications that are running (or have recently been accessed and / or run) by the client device when the invocation input is detected. For example, the client device may be running a calendar application that includes calendar information for a user's "work" profile. When an invocation input that can invoke both a "work" automated assistant and a "home" automated assistant is detected, the "work" automated assistant may be invoked based on the likelihood that the user is currently engaged in work activities. Also, for example, the client device may be running games, a web browser, and / or other applications that the user is likely to access in a non-work environment, and instead of invoking the "work" profile of the automated assistant, a non-work profile of the automated assistant may be invoked.

[0014] In some implementations, the one or more additional features may include information related to the location of the client device when the call input was detected. In some implementations, the location may be the physical location of the client device, such as the geographic location of the device. In some implementations, the location may be a classification of locations, such as "public" locations and "private" locations. In some implementations, the location may be a specific area within a geographic location, such as a "living room" and a "home office." The classification of locations may be set by the user as a user preference and / or may be determined based on the user's past interactions with the client device (e.g., a location where a user frequently uses a "work" calendar may be classified as a "work" or "private" location).

[0015] In some implementations, the one or more additional characteristics may include characteristics of a visual input captured by one or more cameras of the client device when the call input is detected. For example, the visual input received during the call input may indicate the presence of another person near the client device. In response, instead of invoking a "private" automated assistant, a "public" automated assistant may be invoked. Also, for example, the visual input may indicate that the user is at home, and instead of invoking a "work" automated assistant, a "home" automated assistant may be invoked.

[0016] In some implementations, an indication of the automated assistant that was invoked by the invocation input may be provided to the user. For example, an indication may be provided to the user via an interface of the client device indicating that the automated assistant for "work" has been invoked. Also, for example, instead of invoking the second automated assistant, the user may be notified by voice that the first automated assistant has been invoked. For example, if the automated assistant for "work" has been invoked instead of invoking the automated assistant for "home", a voice indication may be provided to the user that "the assistant for work is being invoked". Also, for example, one or more voice prompts and / or responses from the invoked automated assistant may be provided using different voice profiles (e.g., a synthesized male voice for the automated assistant for "home" and a synthesized female voice for the automated assistant for "work") to allow the user to distinguish between multiple automated assistants and / or profiles.

[0017] In some implementations, one or more machine learning models may be utilized to process the features of the call input and / or one or more additional features. The call input, audio data received with the call input, and / or visual data received with the call input may be utilized to generate one or more vectors in the embedding space that indicate location information, prosodic features of the input, and / or other features described herein. The vectors may be processed using a machine learning model that may generate as an output a probability (and / or other indication) of whether to invoke the first automated assistant instead of the second automated assistant.

[0018] In some implementations, the user may provide feedback (e.g., additional audio feedback and / or visual feedback, etc.) indicating whether the invoked automated assistant was the automated assistant that was intended to be invoked. In response to negative feedback, the non-invoked automated assistant may be invoked, and one or more features processed to invoke the first automated assistant may be provided to the machine learning model as learning data for further training based on the negative feedback. For example, the user may utter the invocation phrase "Okay, assistant," which may invoke both the first automated assistant and the second automated assistant. Based on the invocation features and / or the additional features, the first automated assistant may be invoked. The user may have intended to invoke the second automated assistant instead, and may respond to the invoked automated assistant with "Please use the second automated assistant instead." In response, the second automated assistant may be invoked, and the initially provided invocation input may be utilized to further train the associated machine learning model such that a subsequent invocation with the same features is less likely to invoke the first automated assistant.

[0019] In some implementations, the call input can be processed and additional voice data captured immediately before or after the call input can be provided to the called automated assistant without further processing. For example, the client device can detect voice data including the call phrase and call a first automated assistant based on the call characteristics and / or additional characteristics described herein. To minimize delays in processing additional input (e.g., queries preceding and / or following the call), voice data preceding and / or following the call input can be provided directly to the called automated assistant. The called automated assistant can perform natural language processing, automatic speech recognition, STT, and / or other processing, etc. to process the voice data and generate a response and / or perform an action based on the provided input.

[0020] In some implementations, the call input may be processed and additional voice data captured immediately before or after the call input may be further processed to determine one or more characteristics before the voice data is provided to the invoked automated assistant. For example, a generic automated assistant may receive the call input and / or additional voice input preceding and / or following the call input. A first automated assistant may be invoked based on the characteristics of the call and / or other characteristics. The generic automated assistant may perform, for example, automatic speech recognition, natural language processing, and / or other processing of the call input and / or additional input to the input from the user, and the detected characteristics and / or processed input may be provided to the invoked automated assistant along with (or instead of) the voice data.

[0021] The above description is provided as a summary of some embodiments of the present disclosure. The details of these and other embodiments are described in more detail below. [Brief description of the drawings]

[0022] [Figure 1] FIG. 1 illustrates an example of an environment in which embodiments disclosed herein can be implemented. [Diagram 2] FIG. 1 is a block diagram of an example environment in which various methods disclosed herein may be implemented. [Figure 3A] FIG. 1 is a block diagram illustrating an embodiment of multiple automated assistants. [Figure 3B] FIG. 1 is a block diagram illustrating an embodiment of multiple automated assistants. [Figure 4] FIG. 1 is a block diagram illustrating components of a general automated assistant capable of implementing embodiments disclosed herein. [Diagram 5] 1 is a flowchart illustrating an example method according to various embodiments disclosed herein. [Figure 6] FIG. 1 illustrates an example of an architecture for a computing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] Referring to FIG. 1, an example environment including multiple automated assistants that may be invoked by a user 101 is shown. The environment includes a first standalone interactive speaker 105 with a microphone (not shown) and a camera (also not shown) and a second standalone interactive speaker 110 with a microphone (not shown) and a camera (also not shown). The first speaker may be at least partially running a first automated assistant that may be invoked with an invocation phrase. The second speaker 110 may be running a second automated assistant that may be invoked with an invocation phrase that is either the same invocation phrase as the first automated assistant or a different phrase to allow the user to select which automated assistant to invoke based on the uttered phrase. In the example environment, the user 101 is uttering a voice utterance 115 of "Okay, assistant, what's on my calendar?" near the first speaker 105 and the second speaker 110. If either the first and / or second automated assistant is configured to be invoked by the phrase "Okay, assistant," the invoked assistant can process the query that follows the invocation phrase (i.e., "What's on my calendar?"). In some implementations, one or both of the automated assistants 105 and 110 may be capable of being invoked by a user performing one or more actions that can be captured by the camera of the automated assistant. For example, the automated assistant 105 can be invoked by a user looking in the direction of the automated assistant 105, waving in the direction of the automated assistant 105, and / or performing one or more other actions that can be captured by the camera of the automated assistant 105.

[0024] In some implementations, a device such as the first speaker 105 may run multiple automated assistants. Referring to FIG. 2, an example of an environment including multiple client devices running multiple automated assistants is shown. The system includes a first client device 105 running a first automated assistant 215 and a second automated assistant 220. Each of the first and second automated assistants can be invoked by speaking an invocation phrase (either a phrase specific to each assistant or the same phrase to invoke both assistants) in the vicinity of the client device 105 so that the first and second automated assistants can perform an action whose audio is captured by the microphone 225 of the client device 105 and / or captured by the camera 235 of the client device 105. For example, the user 101 can speak "Okay, assistant 1" in the vicinity of the client device 105 to invoke the first automated assistant 215 and also speak the phrase "Okay, assistant 2" in the vicinity of the client device 105 to invoke the second automated assistant 220. Additionally, the user 101 may perform a first action to invoke the first automated assistant 215 and a second action to invoke the second automated assistant 220. Based on which invocation phrase is spoken and / or which action is performed, the user may indicate which of multiple assistants running on the client device 105 the user is interested in handling the voice query. The example environment further includes a second client device 110 running a third automated assistant 245. The third automated assistant may be configured to be invoked using a third invocation phrase, such as "Okay, assistant 3," which may be captured by the microphone 230. Additionally, the third automated assistant 245 may be configured to be invoked using a third gesture and / or action, which may be captured by the camera 250. In some implementations, one or more of the automated assistants of FIG. 2 may not be present.Additionally, the example environment may include additional automated assistants not present in Figure 2. For example, the system may include a third device running additional automated assistants, and / or client device 110 and / or client device 105 may be running additional automated assistants and / or fewer automated assistants than are shown.

[0025] In some implementations, one or more automated assistants may be invoked based on the constraints of the device on which the automated assistant is running. For example, the first client device 205 may include a camera to capture the user's gestures, while the second client device 210 may include a microphone (but not a camera), and thus may be able to identify only voice invocation. When the user performs a gesture, the gesture may be identified by the first client device 205 and may invoke at least one of the first automated assistant 215 and / or the second automated assistant 220. When the user speaks an invocation phrase, only the automated assistant on the client device equipped with a microphone may be invoked. Thus, when both the first automated assistant 215 and the third automated assistant 245 can be invoked with the same invocation input, the user may indicate a preference for one of the invokable automated assistants over the other based on the type of invocation input detected by the one or more client devices 205 and 210.

[0026] Each of automated assistants 215, 220, and 245 can include one or more components of the automated assistants described herein. For example, automated assistant 215 may include its own voice capture component for processing incoming queries, a visual capture component for processing incoming visual data, a hotword detection engine, and / or other components. In some implementations, automated assistants running on the same device, such as automated assistants 215 and 220, can share one or more components that may be utilized by both automated assistants. For example, automated assistant 315 and automated assistant 320 can share one or more of an on-device voice recognizer, an on-device NLU engine, and / or other components.

[0027] In some implementations, more than one automated assistant may be invoked by the same invocation phrase, such as "okay, assistant," that is not specific to a single automated assistant. When a user utters the invocation phrase and / or provides other invocation inputs (e.g., gestures that may invoke more than one automated assistant), one or more automated assistants may act as a general automated assistant and determine which of the automated assistants that may be invoked based on the invocation input. With reference to FIG. 3A, a general automated assistant 305 is shown along with two additional automated assistants 310 and 320. The general automated assistant 305 is configured to process invocation inputs, such as utterances that include the phrase "okay, assistant," or other invocation inputs, that may indicate that the user is interested in providing a query to one of multiple automated assistants that may be invoked by the invocation input. As described herein, the general automated assistant 305 may not include all the functionality of an automated assistant. For example, the general automated assistant 305 may not include a query processing engine and / or functionality to perform actions other than processing the invocation input to determine which of multiple automated assistants to invoke. In some implementations, the general automated assistant 305 may include functionality of other automated assistants and may determine, in response to an invocation input, whether to invoke itself or another automated assistant configured to be invoked by the same invocation input. For example, both the general automated assistant 305 and the first automated assistant 310 may be configured to be invoked in response to detection of a voice invocation phrase, such as "Hey, assistant," "Okay, assistant," and / or "Assistant." The general automated assistant 305 may continuously process a stream of audio data frames based on output from one or more microphones 320 of the client device 301 (e.g., when not in "inactive" mode) and monitor for occurrences of the voice invocation phrase.While monitoring for occurrences of the voice invocation phrase, the generic automated assistant 305 discards (e.g., after temporarily storing in a buffer) audio data frames that do not contain the voice invocation phrase. However, when the generic automated assistant 305 detects an occurrence of the voice invocation phrase in a processed audio data frame, the generic automated assistant 305 can determine whether the invocation input is intended for the generic automated assistant 305 or for one or more other automated assistants 310 and 320 that can be invoked with the same invocation input.

[0028] The automated assistants 305 and 310 may include multiple components for processing queries when invoked, such as a local speech-to-text ("STT") engine (to convert captured speech to text), a local text-to-speech ("TTS") engine (to convert text to speech), a local natural language processor (to determine the meaning of the speech and / or text converted from the speech), and / or other local components. Because the client devices on which the automated assistants run may be relatively constrained in terms of computing resources (e.g., processor cycles, memory, batteries, etc.), the local components may be limited in functionality compared to corresponding components included in the cloud-based automated assistant components running remotely in conjunction with the automated assistant(s).

[0029] In some implementations, one or more automated assistants may be invoked by one or more gestures that indicate a user's interest in interacting with a primary automated assistant. For example, a user may interact with a device, such as pressing a button or touch screen, to indicate intent to invoke an automated assistant, perform a visible movement that is captured by an image capture device, such as a camera, and / or look at the device so that the image capture device can recognize the user's movements and / or location. When the user performs a gesture or action, an automated assistant is invoked and may initiate capture of audio data following the gesture or action, as described above. Additionally, as described above, multiple automated assistants may be invoked by the same invocation input, and a particular gesture may result in a common invocation to multiple automated assistants.

[0030] In some implementations, one or more of the automated assistants 305 and 310 may share one or more modules, such as a natural language processor and / or the results of a natural language, TTS, and / or STT processor. For example, referring back to FIG. 2, both the first automated assistant 215 and the second automated assistant 220 may share natural language processing, and when the client device 105 receives the voice data, the voice data may be processed once into text and then provided to both automated assistants 215 and 220. Also, for example, one or more components of the client device 105 may process the voice data into text and provide a textual representation of the voice data to the third automated assistant 245, as described further herein. In some implementations, the voice data may not be processed into text, but instead be provided to one or more automated assistants as raw voice data.

[0031] In some implementations, the user can voice a query indicating that the user is interested in receiving a response to the query from the primary automated assistant after uttering the invocation phrase. In some implementations, the user can voice a query before or during the invocation phrase, such as "Assistant, what's the weather like" or "Assistant, what's the weather like today, what's the weather like tomorrow." The generic automated assistant 305 can process the invocation input (e.g., "Assistant") and other captured voice data (e.g., "What's the weather like") to determine which automated assistant to invoke based on features described further herein.

[0032] Referring to FIG. 3B, two instantiations of automated assistant 325 are shown, each with a different profile for the same user. The user can configure the two instantiations of automated assistant 325 such that both respond to the same user voice and can be invoked with the same invocation phrase. However, depending on which instantiation of automated assistant 325 performs an action, different results may be provided to the user. For example, a user may have a work calendar and a home calendar, each of which operates independently and handles appointments and / or other calendar functions for a specific purpose. If the user is interested in information provided by the automated assistant instantiation 325 for "work," information from the user's profile 335 for "work" may be provided to the user. Similarly, when the user is interacting with the "home" instantiation of automated assistant 325, information from the user's profile 330 for "home" may be provided to the user. In some implementations, both automated assistants 325 have the same generic invocation input that can invoke the automated assistant 325 generically but does not specify between the instantiations. For example, one or both of the automated assistants 325 may be invoked with the call input "Okay, assistant" without specifying whether the call is intended to be instantiated in the home profile 330 or the work profile 335. Thus, similar to the general automated assistant 305 of FIG. 3A , one or both instantiations of the automated assistant 325 can be configured to determine which profiling to use when it detects a general invocation phrase.

[0033] 4, components of a generic automated assistant 305 in which embodiments described herein may be implemented are shown. Although the present specification describes an environment in which a generic automated assistant processes an invocation input and determines which automated assistant to invoke, the components described with respect to generic automated assistant 305 may be present in an instantiation of automated assistant 325 and may be used to selectively determine whether to invoke an automated assistant 325 using home profile 330 as opposed to an automated assistant 325 using work profile 335.

[0034] The call input analysis engine 410 can process the call input to determine one or more call characteristics that can be utilized to determine which automated assistant to invoke. In some implementations, the call characteristics can be determined based on a general call input that can invoke multiple automated assistants. For example, with reference to FIG. 3A, the generic automated assistant 305 can process a call input of a user uttering the call phrase "Okay, assistant," which can invoke both the first automated assistant 310 and the second automated assistant 320. Also, for example, the generic automated assistant 305 can process a call input of a user performing a gesture captured by one or more cameras, which can invoke both the first automated assistant 310 and the second automated assistant 320.

[0035] In some implementations, the one or more invocation features may include one or more prosodic features of the speech input that includes the invocation input. The prosodic features may include, for example, a speaker's tone, speech rate, inflection, volume, and / or other features of human speech that may indicate whether the user intends to invoke the first automated assistant instead of invoking the second automated assistant. For example, the user may utilize the first automated assistant 310 for non-work purposes and may speak in a more relaxed manner (e.g., slower, more friendly, louder) when speaking a general invocation phrase. Conversely, the user may utilize the second automated assistant 310 for work purposes and may speak in a more formal manner (e.g., quieter, with less inflection, faster) when speaking a general invocation phrase. Thus, based on processing the user speaking the invocation phrase, invocation features can be determined that the invocation decision engine 430 can use to determine which automated assistant to invoke.

[0036] The additional input analysis engine 420 can determine one or more additional features that can be utilized to determine which automated assistant to invoke. In some implementations, the additional features can be based on the location associated with the client device running the general purpose automated assistant 305. For example, a user may be interested in utilizing a particular automated assistant at work and another automated assistant at home. If both automated assistants can be invoked using the same invocation input, the user's location can be an indicator of whether to invoke the first automated assistant (e.g., the automated assistant for work) instead of the second automated assistant (e.g., the automated assistant for home).

[0037] In some implementations, the location may be based on the geographic location of the client device running the general-purpose automated assistant 305. For example, the additional input analysis engine 420 can identify the current location of the client device running the automated assistant based on GPS and determine whether the user has previously specified that the location is a location of a particular classification. Also, for example, the additional input analysis engine 420 can identify the current location of the client device running the automated assistant based on WiFi, signal strength of wireless communication signals, and / or other indications of the device's location. In some implementations, one or more locations can be associated with a type of location, such as "airport" and / or "restaurant." In some implementations, one or more locations can be associated with an area within the identified geographic location, such as a room in a house and / or a particular office in an office building.

[0038] In some implementations, the location may be based on a classification of the location where the client device running the generic automated assistant 305 is located. For example, if the user is in a location tagged as an "airport" location, the additional input analysis engine 420 may determine that the location is a "public" location based on the type of location. Also, for example, the additional input analysis engine 420 may determine that the user is in a location that the user previously indicated was a "home" location, and the additional input analysis engine 420 may determine that the location is classified as a "private" location.

[0039] In some implementations, the additional features can be determined based on additional speech data preceding and / or following the invocation input. For example, the additional features can include prosodic features of the user speaking the query preceding and / or following the invocation input. Also, for example, the additional input analysis engine 420 can determine whether the user's speech is more closely related to the user's intent to invoke a first automated assistant instead of invoking a second automated assistant based on the word usage, vocabulary selection, and / or other terms included in the speech data. For example, the user may use more formal vocabulary when uttering a query with the intent to utilize a "private" automated assistant, and the additional input analysis engine 420 can process the speech input from the user to determine whether the user's vocabulary selection is more "formal" or more "casual."

[0040] In some implementations, additional features can be determined based on background and / or other audio data other than the user-initiated query and / or call. For example, if the audio data preceding and / or following the call input includes background noise (e.g., other speakers, etc.), the additional feature can be determined to indicate that the user is likely to be in a public location. Also, for example, if the audio data preceding and / or following the call input includes noise from a television and / or radio, the additional feature can be determined to indicate that the user is likely to be in a private environment.

[0041] In some implementations, the additional features may include features determined based on visual input received proximate to detection of the invocation input. For example, the client device running the general purpose automated assistant 305 may include a camera that can capture visual input while (or near) the user is providing the invocation input. The additional input analysis engine 420 can determine, based on the visual input, one or more features of the visual input that can indicate whether the user is interested in accessing one of the invokable automated assistants over another automated assistant.

[0042] In some implementations, the visual input characteristics may include identifying whether an additional user is near the user when the user makes the call input. For example, when the user makes the call input, the additional input analysis engine 420 can determine whether the user is alone or whether there are additional people near the user based on the captured video. In some implementations, the presence of others may indicate that the user intends to access a "public" automated assistant instead of accessing a "private" automated assistant.

[0043] In some implementations, an indication of the invoked automated assistant may be provided to the user when the invocation input is received. In some implementations, the indication may be a visual indication, such as an icon or message displayed on the interface of the user's client device. In some implementations, the indication may be an auditory indication, such as a synthesized voice indicating the name of the invoked automated assistant and / or a sound (such as a beep of a particular frequency) indicating that one automated assistant has been invoked instead of invoking another automated assistant. In some implementations, the indication may be a variation of the synthesized voice provided to the user by the automated assistant. For example, in the case where multiple automated assistants can be invoked, a first automated assistant may have a synthesized male voice when invoked and a second automated assistant may have a synthesized female voice when invoked so that the user can determine which automated assistant has been invoked.

[0044] The invocation decision engine 430 can determine whether to invoke the first automated assistant instead of invoking the second automated assistant based on the features of the processed invocation input and / or the additional input. The invocation decision engine 430 can receive the features of the invocation and / or the features of the additional input from the invocation input analysis engine 410 and the additional input analysis engine 420 and determine whether to invoke the first automated assistant instead of invoking the second automated assistant based on the features. In some implementations, the invocation decision engine 430 can utilize one or more machine learning models to determine which automated assistant to invoke. For example, the invocation decision engine 430 can provide one or more vectors representing the features of the invocation and the additional features in the embedding space to the machine learning model. The machine learning model can provide as output the probability that the first automated assistant is invoked and the probability that the second automated assistant is invoked.

[0045] In some implementations, when an automated assistant is invoked, additional voice data and / or other data may be provided to the invoked automated assistant. For example, when invoked, the general purpose automated assistant 305 can provide the user's voice utterance preceding and / or following the invocation input. In some implementations, the general purpose automated assistant 305 can communicate with the invoked automated assistant via one or more communication protocols, such as API 440. Also, for example, the general purpose automated assistant 305 can communicate via a speaker (e.g., an ultrasound signal containing voice data) that the invoked automated assistant receives via a microphone.

[0046] In some implementations, the generic automated assistant 305 can provide voice data of the user's utterance. For example, when the generic automated assistant 305 determines that a first automated assistant needs to be invoked instead of a second automated assistant, the voice data of the user's utterance of a query may be provided directly to the invoked automated assistant. In some implementations, the generic automated assistant 305 may process the voice data including the user's utterance before providing the voice data and / or additional data to the invoked automated assistant. For example, the generic automated assistant 305 can utilize STT, natural language processing, and / or automatic speech recognition to process at least a portion of the voice data. The generic automated assistant 305 can provide the processed information in addition to or in lieu of the voice data to further reduce the delay in the invoked automated assistant generating a response to the user.

[0047] In some implementations, once an automated assistant is invoked, the user can provide feedback. For example, based on the features described herein, the generic automated assistant 305 may determine to invoke the first automated assistant instead of invoking the second automated assistant. The first automated assistant may then be invoked and provided with the user's voice query. Additionally, an indication may be provided to the user indicating that the first automated assistant has been invoked. In response, the user may provide an utterance such as "No, I was talking to assistant 2," "I was talking to another assistant," and / or other negative feedback indicating that the wrong automated assistant has been invoked. In response, the generic automated assistant 305 may invoke the intended automated assistant (and / or, if the user has not specified an intended automated assistant, the automated assistant most likely to be invoked next) and provide the intended automated assistant with the user's voice query. Additionally, one or more of the invocation features and / or additional features used to initially determine to invoke the first automated assistant, along with supervised outputs generated based on the negative feedback, may be provided as training data for training the machine learning model used by the invocation decision engine 430. For example, training examples can be generated that include feature(s) as input and that include as supervised output an indication that Assistant 2 should be invoked based on the feature(s). The training examples can be used to train a machine learning model. In some implementations, positive feedback from a user can be additionally or alternatively utilized to generate training data for training a machine learning model. For example, if Assistant 1 is invoked based on processing of feature(s) using a machine learning model and the user continues to interact with Assistant 1 (implicit positive feedback) and / or there is explicit positive feedback regarding the invocation of Assistant 1, training examples can be generated that include feature(s) and that include as supervised output an indication that Assistant 1 should be invoked.

[0048] 5 is a flow chart illustrating an example method 500 for selectively determining which automated assistant to invoke. For convenience, the operations of method 500 are described with reference to a system that performs the operations. The system of method 500 includes one or more processors and / or other components of a client device. Additionally, while the operations of method 500 are illustrated in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0049] In step 505, an invocation input is detected. In some implementations, the invocation input may be a voice input from a user. For example, the invocation input may be a user uttering a particular phrase that, when uttered, may invoke both the first automated assistant and the second automated assistant. In some implementations, the invocation input may be a user performing one or more actions captured by a camera of a device running one or more automated assistants. For example, a user may wave their hand towards a client device running both a first instantiation and a second instantiation of an automated assistant that may be invoked using the same gesture.

[0050] In step 510, the invocation input is processed to determine one or more features of the invocation input that can be used to determine whether to invoke the first automated assistant instead of invoking the second automated assistant. The invocation features may include, for example, prosodic features of the user uttering the invocation phrase. For example, the user may speak with a particular tone, speed, and / or intonation when intending to invoke the first automated assistant, and with a different tone, speed, and / or intonation when intending to invoke the second automated assistant. Also, for example, if the invocation input is a gesture displayed via a camera of a client device running one or more automated assistants, visual input features that can indicate a particular automated assistant that the user is interested in invoking (such as the presence of another user) can be identified. In some implementations, the features of the invocation input may be determined by a component that shares one or more characteristics with the invocation input analysis engine 410.

[0051] In step 515, the additional input is processed to determine additional features indicative of whether the user is interested in invoking the first automated assistant instead of invoking the second automated assistant. The additional features can be determined by a component that shares one or more characteristics with the additional input analysis engine 420. The additional features can include, for example, a location and / or location classification where the user's client device is located, a visual input that indicates the presence of one or more other users when the invocation input is provided, the vocabulary and / or terminology used by the user when providing additional speech (e.g., a query) preceding and / or following the invocation input, and / or other features that can indicate the user's intent to invoke the first automated assistant instead of invoking a second automated assistant that can be invoked with the same general invocation input, etc.

[0052] In step 520, the output from steps 510 and 515 is processed to determine whether to invoke the first automated assistant or the second automated assistant based on the invocation and the additional features. In some implementations, the determination is performed by a component that shares one or more characteristics with the invocation decision engine 430. For example, the invocation decision engine 430 can utilize one or more machine learning models that receive the invocation and the additional feature vector as input and provide as output the probability that the user intends to invoke the first and second automated assistants. Based on the output, in step 525, the invocation decision engine 430 can invoke the first automated assistant or the second automated assistant instead of invoking the other automated assistant. When invoked, the invoked automated assistant may be provided with the user's voice utterance that preceded and / or followed the invocation input.

[0053] In some implementations, when a voice query is provided to the invoked automated assistant, the automated assistant can generate a response to the query. A second automated assistant may be invoked based on user feedback (e.g., if the user indicates that the wrong automated assistant was invoked). User feedback can be used to further train a machine learning model that can be used to determine whether to invoke the first automated assistant and / or the second automated assistant.

[0054] 6 is a block diagram of an exemplary computing device 610 that may optionally be utilized to perform one or more aspects of the techniques described herein. The computing device 610 typically includes at least one processor 614 that communicates with a number of peripherals via a bus subsystem 612. These peripherals may include a storage subsystem 624, such as a memory subsystem 625 and a file storage subsystem 626, a user interface output device 620, a user interface input device 622, and a network interface subsystem 616. The input and output devices allow a user to interact with the computing device 610. The network interface subsystem 616 provides an interface to external networks and is coupled to corresponding interface devices in other computing devices.

[0055] The user interface input devices 622 may include a keyboard, a pointing device such as a mouse, a trackball, a touchpad, a graphics tablet, a scanner, a touch screen integrated into a display, a voice input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all types of devices and methods for inputting information into the computing device 610 or a communications network.

[0056] The user interface output devices 620 may include a display subsystem, a printer, a fax machine, or non-visual display devices such as an audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide non-visual display devices, such as via an audio output device. In general, use of the term "output device" is intended to include any type of device and method for outputting information from the computing device 610 to a user or to another machine or computing device.

[0057] Storage subsystem 624 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 624 may include logic to perform selected aspects of the method of FIG. 5 and / or implement various components shown in FIGS. 2, 3, and 4.

[0058] These software modules typically execute on the processor 614 alone or in combination with other processors. The memory 625 used by the storage subsystem 624 may include multiple memories, such as a main random access memory (RAM) 630 for storing instructions and data during program execution, and a read-only memory (ROM) 632 in which fixed instructions are stored. The file storage subsystem 626 may provide persistent storage of program files and data files, and may include a hard disk drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules implementing the functionality of a particular embodiment may be stored by the file storage subsystem 626 in the storage subsystem 624 or on other machines accessible by the processor(s) 614.

[0059] Bus subsystem 612 provides a mechanism that allows the various components and subsystems of computing device 610 to communicate with each other as intended. Although bus subsystem 612 is illustrated generally as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0060] Computing device 610 may be of many different types, such as a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computing device 610 shown in Figure 6 is intended only as a specific example to illustrate some implementations. Many other configurations of computing device 610 are possible, including more or fewer components than the computing device shown in Figure 6.

[0061] In some embodiments, a method implemented by one or more processors is provided, the method including detecting, at a client device, a call input that at least selectively invokes a first automated assistant and a second automated assistant, determining whether the call input is directed to the first automated assistant or the second automated assistant, where the determination is based on processing of at least one of one or more call features of the call input other than any features based on speech recognition of a voice input received in association with the call input and one or more additional features detected by the client device other than the call features, and in response to determining that the call input is directed to the first automated assistant, invoking the first automated assistant instead of invoking the second automated assistant.

[0062] These and other implementations of the technology disclosed herein may include one or more of the following features.

[0063] In some implementations, the one or more call features include one or more prosodic features determined from speech data comprising the call input.

[0064] In some embodiments, determining whether the invocation input is directed to the first automated assistant or the second automated assistant includes identifying, independent of voice recognition, one or more terms included in the voice data including the invocation input and determining whether the one or more terms indicate an intent of the user to invoke the first automated assistant.

[0065] In some implementations, the one or more additional features include one or more prosodic features determined from audio data detected by one or more microphones of the client device capturing speech preceding and / or following the call input.

[0066] In some implementations, the one or more additional characteristics include one or more applications being executed on the client device within a threshold time after the call input is detected.

[0067] In some implementations, the one or more additional characteristics include the location of the client device when the call input is detected.

[0068] In some implementations, the one or more additional characteristics include an activity the user is performing when the call input is detected.

[0069] In some implementations, the one or more additional features may include features of a visual input based on visual data captured by one or more cameras of the client device when the call input is detected.

[0070] In some implementations, processing the call input includes processing, by the client device, one or more of the call features and the additional features using a machine learning model stored locally on the client device. In some of these implementations, the method further includes receiving feedback from the user in response to the invocation of the first automated assistant indicating whether the call input was intended to invoke the first automated assistant, and training the machine learning model based on the feedback.

[0071] In some implementations, the method further includes, in response to determining that the invocation input is directed to the first automated assistant, providing, at the client device, an indication that the first automated assistant has been invoked. In some of these implementations, the method further includes receiving a user input in response to the invocation of the first automated assistant, determining based on processing the user input that the user input indicates that the invocation is not directed to the first automated assistant, and in response to determining that the user input indicates that the invocation is not directed to the first automated assistant, invoking a second automated assistant.

[0072] In some embodiments, the indication includes a visual indication provided by a display of the client device. In some of these embodiments, the indication includes an audible indication provided by a speaker of the client device.

[0073] In some embodiments, the method further includes providing speech data preceding and / or following the invocation input to the first automated assistant in response to the invocation of the first automated assistant. In some of these embodiments, the speech data is provided without providing additional speech-based data based on additional processing of the speech data. In other of these embodiments, the method further includes processing the speech data to identify one or more features of the speech data and providing the one or more features along with the speech data to the first automated assistant in response to the invocation of the first automated assistant.

[0074] In situations where certain implementations described herein may collect or use personal information about users (e.g., data about users extracted from other electronic communications, information about the user's social network, the user's location, the user's time, the user's biometric information, user activity and demographic information, relationships between users, etc.), users are provided with one or more opportunities to control whether information is collected, whether the personal information is stored, whether the personal information is used, and how information about the user is collected, stored, and used. That is, the systems and methods described herein collect, store, and / or use a user's personal information only with explicit permission from the relevant user.

[0075] For example, a user may be given control over whether the program or feature collects user information about that particular user, or about other users associated with the program or feature. Each user about whom personal information is collected is presented with one or more options to control the collection of information related to that user and to give permission or approval as to whether and what portions of the information are collected. For example, a user may be provided with one or more such control options over a communications network. Additionally, certain data may be processed in one or more ways such that personally identifiable information is removed before it is stored or used. As one example, a user's identity may be treated in such a way that personally identifiable information cannot be identified. As another example, a user's geographic location may be generalized to a larger area, making it impossible to identify the user's specific location.

[0076] Although several embodiments have been described and illustrated herein, various other means and / or structures may be utilized to perform the functions and / or obtain the results and / or obtain one or more advantages described herein, and each such variation and / or modification is deemed to be within the scope of the embodiments described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and the actual parameters, dimensions, materials, and / or configurations will vary depending on the particular application(s) for which the teachings are used. Those skilled in the art will recognize or be able to ascertain using no more than routine experimentation many equivalents to the specific embodiments described herein. Thus, it should be understood that the foregoing embodiments have been presented by way of example only, and that, within the scope of the appended claims and equivalents thereof, embodiments may be practiced other than as specifically described and claimed. The embodiments of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. Furthermore, any combination of two or more such features, systems, articles, materials, kits, and / or methods is within the scope of the disclosure, unless such features, systems, articles, materials, kits, and / or methods are mutually inconsistent.

Claims

1. 1. A method implemented by one or more processors, comprising: Detecting an invocation input at the client device selectively invoking at least a first automated assistant and a second automated assistant; determining whether the call input is directed to the first automated assistant or the second automated assistant, the determining including: one or more call features of the call input other than any features based on speech recognition of a speech input received in association with the call input; said determining being based on processing of at least one of the characteristics of the call and one or more additional characteristics detected by the client device; In response to determining that the call input is directed to the first automated assistant, calling the first automated assistant instead of calling the second automated assistant; A method comprising:

2. The method of claim 1 , wherein the one or more call features include one or more prosodic features determined from speech data that includes the call input.

3. determining whether the call input is directed to the first automated assistant or the second automated assistant; identifying, independent of the speech recognition, one or more terms contained in the speech data comprising the call input; determining that the one or more terms indicate an intent of a user to invoke the first automated assistant; The method of claim 1 or claim 2, comprising:

4. 4. The method of claim 1, wherein the one or more additional features include one or more prosodic features determined from audio data detected by one or more microphones of the client device capturing speech preceding or following the call input.

5. The method of claim 1 , wherein the one or more additional features include one or more applications that are executed on the client device within a threshold time after the call input is detected.

6. The method of claim 1 , wherein the one or more additional characteristics include a location of the client device when the call input is detected.

7. The method of claim 1 , wherein the one or more additional characteristics include an activity the user is performing when the call input is detected.

8. 8. The method of claim 1, wherein the one or more additional features include one or more visual features based on visual data captured by one or more cameras of the client device when the call input is detected.

9. 2. The method of claim 1, wherein processing the call input includes processing one or more of the features of the call and the additional features by the client device using a machine learning model stored locally on the client device.

10. receiving feedback from the user in response to the invocation of the first automated assistant, the feedback indicating whether the invocation input was intended to invoke the first automated assistant; training the machine learning model based on the feedback; and The method of claim 9 further comprising:

11. 11. The method of claim 1, further comprising: providing, at the client device, an indication that the first automated assistant has been invoked in response to determining that the invocation input is directed to the first automated assistant.

12. receiving a user input in response to the invocation of the first automated assistant; determining, based on processing the user input, that the user input indicates that the call is not intended for the first automated assistant; In response to determining that the user input indicates that the call is not directed to the first automated assistant, Invoking the second automated assistant. The method of claim 11 further comprising:

13. The method of claim 11 or claim 12, wherein the indication comprises a visual indication provided by a display of the client device.

14. The method of claim 11 or claim 12, wherein the indication comprises an audio indication provided by a speaker of the client device.

15. 15. The method of claim 1, further comprising providing, in response to an invocation of the first automated assistant, voice data preceding or following the invocation input to the first automated assistant.

16. The method of claim 15 , wherein the audio data is provided without providing additional audio-based data based on additional processing of the audio data.

17. processing the audio data to identify one or more characteristics of the audio data; providing said one or more features together with said voice data to said first automated assistant in response to an invocation of said first automated assistant; The method of claim 15 further comprising:

18. 1. A client device, comprising: One or more microphones; A memory for storing instructions; Executing the instructions, Detecting an invocation input selectively invoking at least a first automated assistant and a second automated assistant; determining whether the call input is directed to the first automated assistant or the second automated assistant, the determining including: one or more call features of the call input other than any features based on speech recognition of a speech input received in association with the call input; said determining being based on processing of at least one of the characteristics of the call and one or more additional characteristics detected by the client device; In response to determining that the call input is directed to the first automated assistant, one or more processors that, instead of invoking the second automated assistant, invoke the first automated assistant; Including, a client device.

19. 20. The assistant device of claim 18, wherein the one or more call features include one or more prosodic features determined from speech data including the call input.

20. In determining whether the call input is directed to the first automated assistant or the second automated assistant, the one or more processors: Identifying, independent of the speech recognition, one or more terms contained in the speech data comprising the call input; 20. The assistant device of claim 18 or 19, wherein the one or more terms are determined to indicate a user's intent to invoke the first automated assistant.

21. At least one processor; A memory storing instructions which, when executed, cause said at least one processor to perform a method according to any one of claims 1 to 17. Including, the system.

22. 18. A computer readable storage medium storing instructions that, when executed, cause one or more processors to perform any of the operations of claims 1 to 17.

Images