Multiple concurrent voice assistants
By allowing multiple automation assistants to run concurrently and control the microphone and speakers, the problem of difficulty in switching automation assistants is solved, improving user experience and resource utilization efficiency.
Patent Information
- Application Number
- CN202280094841.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-15
- Filing Date
- 2022-12-09
- Publication Date
- 2024-12-06
AI Technical Summary
In existing automation assistant systems, it is difficult for users to switch and use multiple automation assistants without changing configuration settings, resulting in waste of resources and poor user experience, and it is difficult to determine which automation assistant is active.
Concurrent operation and switching of multiple automation assistants by allowing two or more automation assistants to listen to hot words simultaneously and control the microphone and speakers when one automation assistant initiates a session, blocking other assistants from accessing.
Improves user experience, avoids resource waste, allows users to use any supported automation assistant without changing configuration, ensures efficient utilization of resources, and displays active assistants through visual indications.
Smart Images

Figure CN119096227A_ABST
Abstract
Description
Background Art
[0001] Humans may engage in human-computer conversations using interactive software applications, which are referred to herein as "automated assistants" (also referred to as "digital agents," "interactive personal assistants," "intelligent personal assistants," "assistant applications," "conversational agents," and the like). For example, humans (who may be referred to as "users" when they interact with automated assistants) may provide commands and / or requests to automated assistants using verbal natural language input (i.e., utterances), which in some cases may be converted to text and then processed by providing textual (e.g., typed) natural language input and / or by touch and / or non-verbal physical movements (e.g., hand gestures, eye gaze, facial movements, and the like). The automated assistant responds to the request by providing responsive user interface outputs (e.g., auditory and / or visual user interface outputs), controlling one or more intelligent devices, and / or controlling one or more functions of a device implementing the automated assistant (e.g., controlling other applications of the device).
[0002] As mentioned above, many automated assistants are configured to be interacted with via spoken utterances. To maintain user privacy and / or conserve resources, the automated assistant avoids performing one or more automated assistant functions based on all spoken utterances present in audio data detected (at least in part) via a microphone of a client device implementing the automated assistant. Instead, certain processing based on spoken utterances occurs only in response to determining that certain conditions exist.
[0003] For example, many client devices including automated assistants and / or interfaces with automated assistants include hot word detection models. When the microphone of such client device is not disabled, the client device can use the hot word detection model to continuously process the audio data detected via the microphone to generate a predicted output indicating whether there are one or more hot words (including multi-word phrases), such as "Hey Assistant", "OK Assistant" and / or "Assistant". When the predicted output indicates the presence of a hot word, any audio data following (and optionally determined to include voice activity) within a threshold amount of time can be processed by one or more devices such as a speech recognition component, a voice activity detection component, and / or a remote automated assistant component. Further, a natural language understanding engine can be used to process the recognized text (from the speech recognition component), and / or an action can be performed based on the natural language understanding engine output. The action can include, for example, generating and providing a response and / or controlling one or more applications and / or smart devices. However, when the predicted output indicates that the hot word does not exist, the corresponding audio data will be discarded without any further processing, thereby saving resources and protecting user privacy.
[0004] Different automated assistants may be invoked using different hot words and may support different capabilities. For example, a first automated assistant may be invoked by the hot word "Hey Assistant" and may be able to handle a variety of general tasks, including messaging tasks, search tasks, phone call tasks, etc. A second automated assistant may be invoked by the hot word "Computer" and may also be able to handle a variety of general tasks, including messaging tasks, search tasks, phone call tasks, etc. A third automated assistant may be invoked by the hot word "Hello Car" and may be able to handle a variety of vehicle-related tasks, such as changing climate control settings and changing entertainment system settings.
[0005] A client device that implements two or more different automated assistants (such as the three automated assistants in the example above) may provide a configuration option that allows selection of a particular automated assistant. However, the user may not be able to utilize an automated assistant other than the selected automated assistant. Changing the selected automated assistant may require the user to locate and enter a settings menu, locate an option to select a different automated assistant, and specify a new automated assistant in order to utilize the new automated assistant. Additionally, the user may not be aware of which automated assistant is active, and may attempt to perform tasks that are not supported by the active automated assistant, which may result in a negative user experience and may waste network and / or computing resources as the automated assistant attempts to handle the unsupported task and fails to fulfill the request. Summary of the invention
[0006] Some implementations disclosed herein relate to supporting multiple concurrent voice assistants. As described in more detail herein, two or more automated assistants can listen for multiple hot words. A user may be able to select a specific automated assistant to handle a specific request by speaking a hot word associated with the specific automated assistant. For example, a first automated assistant may listen for a first hot word, and a second automated assistant may listen for a second hot word. When one of the automated assistants initiates a conversation, control of the microphone and speaker may be provided to the automated assistant that initiated the conversation, and other automated assistants may be blocked from accessing the microphone, speaker, and output audio data of the automated assistant that initiated the conversation.
[0007] In one example, the onboard computer may support a first automated assistant that may be invoked by the hotword "Hey Assistant" and may be capable of handling a variety of general tasks, including messaging tasks, search tasks, phone call tasks, etc. The onboard computer may also support a second automated assistant that may be invoked by the hotword "Computer" and may also be capable of handling a variety of general tasks, including messaging tasks, search tasks, phone call tasks, etc. The onboard computer may also support a third automated assistant that may be invoked by the hotword "Hello Car" and may be capable of handling a variety of vehicle-related tasks, such as changing climate control settings and changing entertainment system settings.
[0008] In this example, without changing any configuration settings, a user may be able to invoke a first session with a first automated assistant on the onboard computer by saying “Hey Assistant, tell Sarah I will arrive in ten minutes” to send a text message, and then after the first session ends, invoke a second session with a third automated assistant on the onboard computer by saying “Hello Car, lower cabin temperature” to change climate control settings. In some implementations, a visualization may be provided that indicates which automated assistant is active (e.g., triggered by a hot word) and / or receiving audio data from a microphone. By allowing a user to utilize any supported automated assistant without changing configuration settings, failures when a user attempts to utilize an automated assistant other than the selected automated assistant may be avoided, resulting in an improved user experience and more efficient utilization of network and / or computing resources.
[0009] In this example, during a first session with a first automated assistant, a third automated assistant may generate a notification "Vehicle maintenance due in 500 miles." However, due to the lower priority associated with the vehicle maintenance notification, the onboard computer may prevent the third automated assistant from interrupting the first session to present the notification on a display of the onboard computer. This notification may be temporarily suppressed until the first session ends. Once the first session has ended, the vehicle maintenance notification may be presented.
[0010] Continuing with the example, during a second session with a third automated assistant, the first automated assistant can generate a navigation notification “Turn right in 200 feet.” Due to the higher priority associated with the navigation notification, the second session can be interrupted to present the navigation notification.
[0011] In some implementations, when the accessibility service is active, two or more automated assistants can concurrently listen for multiple hot words. When one of the automated assistants initiates a session, the automated assistant initiating the session can be provided with control of the microphone and speaker, and other automated assistants can be blocked from accessing the microphone, speaker, and output audio data of the automated assistant initiating the session. The accessibility service can remain active during the automated assistant session, and the automated assistants can support accessibility-related tasks.
[0012] In various implementations, a method implemented by one or more processors may include: concurrently providing access to one or more microphones to a first automated assistant and a second automated assistant; receiving an indication from the first automated assistant that the first automated assistant has initiated a first conversation; in response to receiving the indication from the first automated assistant that the first automated assistant has initiated the first conversation: continuing to provide access to the one or more microphones to the first automated assistant; discontinuing providing access to the one or more microphones to the second automated assistant; and preventing the second automated assistant from accessing one or more portions of an output audio data stream provided for rendering via one or more speakers, the one or more portions including output audio data of the first automated assistant; receiving an indication from the first automated assistant that the first conversation has ended; and in response to receiving the indication from the first automated assistant that the first conversation has ended: continuing to provide access to the one or more microphones to the first automated assistant; resuming providing access to the one or more microphones to the second automated assistant; and resuming providing the output audio data stream to the second automated assistant. The second automated assistant may use the output audio data stream in noise cancellation.
[0013] In some implementations, preventing the second automated assistant from accessing one or more portions of the output audio data stream includes preventing the second automated assistant from accessing the output audio data stream while the first automated assistant provides auditory output during the first conversation.
[0014] In some implementations, the method can further include: determining that a telephone call has been initiated; in response to determining that the telephone call has been initiated, discontinuing providing access to the one or more microphones to the first automated assistant and the second automated assistant; determining that the telephone call has ended; and in response to determining that the telephone call has ended, resuming providing access to the one or more microphones to the first automated assistant and the second automated assistant.
[0015] In some implementations, the method may further include: after receiving an indication from the first automated assistant that the first automated assistant has initiated the first conversation and before receiving an indication from the first automated assistant that the first conversation has ended: identifying a notification to be output from the second automated assistant; determining a priority of the notification; and in response to the priority of the notification satisfying a threshold, before receiving an indication from the first automated assistant that the first conversation has ended, outputting the notification.
[0016] In some implementations, the method may further include: buffering one or more portions of the output audio data stream that include the output audio data of the first automated assistant in a buffer; pausing presentation of the one or more portions of the output audio data stream that include the output audio data of the first automated assistant while outputting a notification from the second automated assistant; and resuming presentation of the one or more portions of the output audio data stream that include the output audio data of the first automated assistant from the buffer after outputting the notification from the second automated assistant.
[0017] In some implementations, the method may further include: after receiving an indication from the first automated assistant that the first automated assistant has initiated the first conversation and before receiving an indication from the first automated assistant that the first conversation has ended: identifying a notification to be output from the second automated assistant; determining a priority of the notification; and in response to the priority of the notification not satisfying a threshold, suppressing the notification until an indication is received from the first automated assistant that the first conversation has ended.
[0018] In some implementations, the method can further include: buffering the notification in a buffer; and in response to receiving an indication from the first automated assistant that the first conversation has ended, outputting the notification from the buffer.
[0019] In some implementations, the method may further include: receiving user interface input via a graphical user interface, the user interface input being a request to initiate a second conversation on a second automated assistant; in response to receiving the user interface input: causing the second automated assistant to initiate the second conversation; continuing to provide access to one or more microphones to the second automated assistant; discontinuing providing access to the one or more microphones to the first automated assistant; and preventing the first automated assistant from accessing output audio data of the second automated assistant.
[0020] In some implementations, the method can further include: after receiving an indication from the first automated assistant that the first automated assistant has initiated the first conversation and before receiving an indication from the first automated assistant that the first conversation has ended: displaying a visual indication on the graphical user interface that access to the one or more microphones is being provided to the first automated assistant.
[0021] In some implementations, the method can further include, after receiving an indication from the first automated assistant that the first automated assistant has initiated the first conversation: providing an audible indication that access to the one or more microphones is being provided to the first automated assistant.
[0022] In some implementations, the method can further include, in response to receiving an indication from the first automated assistant that the first automated assistant has initiated the first session: reassigning a physical button that, when activated, is configured to initiate a session with the second automated assistant instead of initiating the session with the first automated assistant.
[0023] In some additional or alternative implementations, a computer program product may include one or more computer-readable storage media having program instructions stored together on the one or more computer-readable storage media. The program instructions may be executable to: receive first audio data capturing a first spoken utterance of a user via one or more microphones; concurrently provide the first audio data to a first hotword detector of a first automated assistant and a second hotword detector of a second automated assistant; receive a first confidence score and a second confidence score, the first confidence score being based on a probability of the first hotword being present in the first audio data determined by the first hotword detector of the first automated assistant, the second confidence score being based on a probability of the second hotword being present in the first audio data determined by the second hotword detector of the second automated assistant; based on the first confidence score and the second confidence score: provide the first automated assistant with second audio data received via the one or more microphones, the second audio data capturing a second spoken utterance of the user following the first spoken utterance of the user; provide an indication that audio from the one or more microphones is being provided to the first automated assistant; and prevent the second automated assistant from obtaining the second audio data.
[0024] In some implementations, a first hot word detector processes the first audio data using one or more machine learning models of the first hot word detector to generate a first predicted output, the first predicted output indicating a probability of the first hot word being present in the first audio data; and a second hot word detector processes the first audio data using one or more machine learning models of the second hot word detector to generate a second predicted output, the second predicted output indicating a probability of the second hot word being present in the first audio data.
[0025] In some implementations, the first confidence score is higher than the second confidence score; and the first confidence score satisfies a threshold. In some implementations, the indication that audio from the one or more microphones is being provided to the first automated assistant is a visual indication displayed on a graphical user interface, or an audible indication.
[0026] In some additional or alternative implementations, a system may include a processor, a computer-readable memory, one or more computer-readable storage media, and program instructions stored together on the one or more computer-readable storage media. The program instructions may be executable to: concurrently provide access to one or more microphones to a first automated assistant and a second automated assistant; receive an indication from the first automated assistant that the first automated assistant has initiated a first conversation; in response to receiving an indication from the first automated assistant that the first automated assistant has initiated the first conversation: continue to provide access to the one or more microphones to the first automated assistant; discontinue providing access to the one or more microphones to the second automated assistant; and prevent the second automated assistant from accessing one or more portions of an output audio data stream provided for rendering via one or more speakers, the one or more portions including the output audio data of the first automated assistant; receive an indication from the first automated assistant that the first conversation has ended; and in response to receiving an indication from the first automated assistant that the first conversation has ended: continue to provide access to the one or more microphones to the first automated assistant; resume providing access to the one or more microphones to the second automated assistant; and resume providing the output audio data stream to the second automated assistant, wherein the second automated assistant uses the output audio data stream in noise cancellation.
[0027] The above description is provided as an overview of some implementations of the present disclosure. Further descriptions of those implementations and other implementations are described in more detail below.
[0028] Various implementations may include a non-transitory computer-readable storage medium storing instructions that are executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), and / or a tensor processing unit (TPU)) to perform a method such as one or more of the methods described herein. Other implementations may include an automated assistant client device (e.g., a client device that includes at least an automated assistant interface for interfacing with a cloud-based automated assistant component) that includes a processor that is operable to execute the stored instructions to perform a method such as one or more of the methods described herein. Still other implementations may include a system of one or more servers that include one or more processors that are operable to execute the stored instructions to perform a method such as one or more of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 Example environments are schematically depicted in which selected aspects of the present disclosure may be implemented according to various implementations.
[0030] Figure 2 , Figure 3 , Figure 4 and Figure 5 A flow chart illustrating an example method for concurrent voice assistants is depicted.
[0031] Figure 6 An example architecture for a computing device is depicted. DETAILED DESCRIPTION
[0032] Figure 1 An example environment 100 is schematically depicted in which selected aspects of the present disclosure may be implemented, according to various implementations. Figure 1 Any computing device depicted in or elsewhere in the figures may include logic such as one or more microprocessors (e.g., a central processing unit or “CPU,” a graphics processing unit or “GPU”) that execute computer-readable instructions stored in a memory, or other types of logic such as an application specific integrated circuit (“ASIC”), a field programmable gate array (“FPGA”), etc. Figure 1 Some of the systems depicted in , such as cloud-based automated assistant component 180 , may be implemented using one or more server computing devices forming what is sometimes referred to as a “cloud infrastructure,” but this is not required.
[0033] Figure 1The client device 110 in FIG. 1 is illustrated as having one or more microphones 111, one or more speakers 112, one or more cameras and / or other visual components 113, and a display 114 (e.g., a touch-sensitive display). In some implementations, one or more of these components may be omitted. The client device 110 may further include a pressure sensor, a proximity sensor, an accelerometer, a magnetometer, and / or other sensors for generating other sensor data in addition to the audio data captured by the one or more microphones 111.
[0034] The client device 110 at least selectively executes two or more automated assistant clients, such as a first automated assistant client 140-1, ..., and an nth automated assistant client 140-n. Figure 1 In the example of , each of the automated assistant clients 140-1, ..., 140-n includes an on-device hotword detection engine 122-1, ..., 122-n, an on-device speech recognizer 142-1, ..., 142-n, an on-device natural language understanding (NLU) engine 144-1, ..., 144-n, and an on-device fulfillment engine 146-1, ..., 146-n. Each of the automated assistant clients 140-1, ..., 140-n further includes a speech capture engine 147-1, ..., 147-n and a visual capture engine 148-1, ..., 148-n. Each of the automated assistant clients 140-1, ..., 140-n may include additional and / or alternative engines, such as a voice activity detector (VAD) engine, an endpoint detector engine, and / or other engines.
[0035] One or more cloud-based automated assistant components 180 may optionally be implemented on one or more computing systems (collectively referred to as "cloud" computing systems) that are communicatively coupled to client devices 110 via one or more local area networks and / or wide area networks (e.g., the Internet), indicated generally at 190. Cloud-based automated assistant components 180 may be implemented, for example, via a cluster of high-performance servers.
[0036] In various implementations, instances of automated assistant clients 140-1, ..., 140-n, through their interaction with one or more cloud-based automated assistant components 180, may form what, from a user's perspective, appears to be a logical instance of an automated assistant with which the user may engage in human-computer interaction (e.g., verbal interaction, gesture-based interaction, and / or touch-based interaction).
[0037] The client device 110 may be, for example, a computing device of a user's vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system, etc.). In other implementations, the client device 110 may be: a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a standalone interactive speaker, a smart home appliance such as a smart TV (or a standard TV equipped with a networked dongle with automated assistant capabilities), and / or a user's wearable device including a computing device (e.g., a user's watch with a computing device, a user's glasses with a computing device, a virtual or augmented reality computing device). Additional and / or alternative client devices may be provided.
[0038] The one or more visual components 113 can take various forms, such as a monographic camera, a stereo camera, a LIDAR component (or other laser-based component), a radar component, etc. The one or more visual components 113 can be used, for example, by the visual capture engines 148-1, ..., 148-n to capture visual frames (e.g., image frames, laser-based visual frames) of the environment in which the client device 110 is deployed. In some implementations, such visual frames can be used to determine whether a user is near the client device 110 and / or the distance of the user (e.g., the user's face) relative to the client device 110. Such determinations can be used, for example, to determine whether to activate a user. Figure 1 Machine learning engines and / or other engines on the various devices depicted.
[0039] Speech capture engines 147-1, ..., 147-n may be configured to capture speech of a user and / or other audio data captured via microphone 111. Further, client device 110 may include pressure sensors, proximity sensors, accelerometers, magnetometers, and / or other sensors for generating other sensor data in addition to audio data captured via microphone 111. Such audio data and other sensor data may be utilized by hot word detection engines 122-1, ..., 122-n, and / or other engines to determine whether to activate one or more currently dormant automated assistant functions, avoid activating one or more currently dormant automated assistant functions, and / or shut down one or more currently active automated assistant functions. In some implementations, hot word detection engines 122-1, ..., 122-n process audio data and / or other sensor data using a machine learning model to generate a predicted output.
[0040] Automated assistant clients 140-1, ..., 140-n can make a decision whether to activate a currently dormant automated assistant function, avoid activating a currently dormant automated assistant function, and / or shut down a currently active automated assistant function based on determining whether the predicted output meets a threshold. Automated assistant functions may include: speech recognition, which is used to generate recognized text using an on-device speech recognizer 142-1, ..., 142-n; natural language understanding (NLU), which is used to generate NLU output using an on-device NLU engine 144-1, ..., 144-n; generating a response based on the recognized text and / or NLU output; transmission of audio data to a remote server; and / or transmission of recognized text to a remote server. For example, assume that the predicted output is a probability (e.g., 0.80 or 0.90) and the threshold is a threshold probability (e.g., 0.85). If automated assistant client 140 - 1 , . . . , 140 - n determines that the predicted output (eg, 0.90) satisfies a threshold (eg, 0.85), automated assistant client 140 - 1 , . . . , 140 - n may activate the currently dormant automated assistant function.
[0041] The automated assistant functionality may include an on-device speech recognizer 142-1, ..., 142-n, an on-device NLU engine 144-1, ..., 144-n, an on-device fulfillment engine 146-1, ..., 146-n, and additional and / or alternative engines. For example, the on-device speech recognizer 142-1, ..., 142-n may utilize an on-device speech recognition model to process audio data capturing spoken utterances to generate recognized text corresponding to spoken utterances. The on-device NLU engine 144-1, ..., 144-n may optionally utilize an on-device NLU model to perform on-device natural language understanding on the recognized text to generate NLU data. The NLU data may include, for example, an intent corresponding to a spoken utterance and optionally parameters (e.g., slot values) for the intent. Further, the on-device fulfillment engine 146-1, ..., 146-n may optionally utilize an on-device fulfillment model to generate fulfillment data based on the NLU data. This fulfillment data may define local and / or remote responses (e.g., replies) to the spoken utterance, interactions with locally installed applications to be performed based on the spoken utterance, commands to be transmitted to an Internet of Things (IoT) device based on the spoken utterance (directly or via a corresponding remote system), and / or other parsing actions to be performed based on the spoken utterance. The fulfillment data is then provided for local and / or remote implementation / execution of the determined actions to parse the spoken utterance. Execution may include, for example, rendering local and / or remote responses (e.g., visually and / or audibly (optionally utilizing a local text-to-speech module)), interacting with locally installed applications, transmitting commands to IoT devices, and / or other actions.
[0042] Display 114 can be utilized to display recognized text from speech recognizers 142-1, ..., 142-n on the device and / or one or more results from execution. Display 114 can further be one of the user interface output components through which the visual portion of a response from one of automated assistant clients 140-1, ..., 140-n is rendered.
[0043] In some implementations, the cloud-based automated assistant component 180 may include a remote ASR engine 181 that performs speech recognition, a remote NLU engine 182 that performs natural language understanding, and / or a remote fulfillment engine 183 that generates fulfillment. A remote execution module may also be optionally included, which performs remote execution based on fulfillment data determined locally or remotely. Additional and / or alternative remote engines may be included. As described herein, in various implementations, on-device speech processing, on-device NLU, on-device fulfillment, and / or on-device execution may be prioritized at least due to the latency and / or network usage reduction they provide when parsing spoken utterances (since parsing spoken utterances does not require client-server round trips). However, one or more cloud-based automated assistant components 180 may be at least selectively utilized. For example, such components may be used in parallel with on-device components, and when local components fail, outputs from such components may be utilized. For example, on-device fulfillment engine 146-1, ..., 146-n may fail in certain situations (e.g., due to relatively limited resources of client device 110), and in such situations, remote fulfillment engine 183 may utilize the more robust resources of the cloud to generate fulfillment data. Remote fulfillment engine 183 may operate in parallel with on-device fulfillment engine 146-1, ..., 146-n, and the results of the remote fulfillment engine may be utilized when on-device fulfillment fails, or the remote fulfillment engine may be invoked in response to determining the failure of on-device fulfillment engine 146-1, ..., 146-n.
[0044] In various implementations, an NLU engine (on-device and / or remote) may generate NLU data that includes one or more annotations of the recognized text and one or more (e.g., all) of the terms of the natural language input. In some implementations, the NLU engine is configured to identify and annotate various types of grammatical information in the natural language input. For example, the NLU engine may include a morphological module that may separate individual words into morphemes and / or annotate the morphemes (e.g., with their categories). The NLU engine may also include a part of speech tagger that is configured to annotate the terms with the grammatical roles of the terms. Further, for example, in some implementations, the NLU engine may additionally and / or alternatively include a dependency parser that is configured to determine syntactic relationships between terms in the natural language input.
[0045] In some implementations, the NLU engine may additionally and / or alternatively include an entity tagger configured to annotate entity references in one or more segments (such as references to people (including, for example, literary characters, celebrities, public figures, etc.), organizations, locations (real and fictional), etc.). In some implementations, the NLU engine may additionally and / or alternatively include a coreference resolver (not depicted) configured to group or "cluster" references to the same entity based on one or more contextual cues. In some implementations, one or more components of the NLU engine may rely on annotations from one or more other components of the NLU engine.
[0046] In some implementations, the NLU engine may also include an intent matcher configured to determine the intent of a user participating in an interaction with one of automated assistant clients 140-1, ..., 140-n. The intent matcher may use various techniques to determine the intent of the user. In some implementations, the intent matcher may access one or more local and / or remote data structures that include, for example, multiple mappings between grammars and response intents. For example, grammars included in the mappings may be selected and / or learned over time, and may represent common intents of users. For example, a grammar "play <artist>(play <artist>)" can be mapped to an intent that invokes a response action that causes <artist>music to play on client device 110. Another grammar "[weather|forecast] today" may be able to match user queries such as "what's the weather today" and "what's the forecast for today?" In addition to or in lieu of grammars, in some implementations, the intent matcher may employ one or more trained machine learning models, alone or in combination with one or more grammars. These trained machine learning models may be trained to recognize intents, for example, by embedding recognized text from spoken utterances into a reduced dimensionality space, and then determining which other embeddings (and therefore, intents) are closest, for example, using techniques such as Euclidean distance, cosine similarity, etc. As described above, <artist>As seen in the example grammars, some grammars have slots that can be filled with slot values (or "parameters") (e.g., <artist>). Slot values can be determined in a variety of ways. Users will typically actively provide slot values. For example, for the syntax "Order me a <topping>pizza", the user might utter the phrase "Order me a sausage pizza", in which case the slot <toppings> is automatically populated. Other slot values can be inferred based on, for example, the user's location, the currently rendered content, user preferences, and / or other cues.
[0047] A fulfillment engine (local and / or remote) may be configured to receive the predicted / estimated intent output by the NLU engine and any associated slot values, and fulfill (or "parse") the intent. In various implementations, the fulfillment (or "parse") of the user intent may cause various fulfillment information (also referred to as fulfillment data) to be generated / obtained, for example, by the fulfillment engine. This may include determining local and / or remote responses (e.g., replies) to the spoken utterance, interactions with locally installed applications to be performed based on the spoken utterance, commands to be transmitted to an Internet of Things (IoT) device based on the spoken utterance (directly or via a corresponding remote system), and / or other parsed actions to be performed based on the spoken utterance. The on-device fulfillment may then initiate local and / or remote performance / execution of the determined actions to parse the spoken utterance.
[0048] Still reference Figure 1 , the client device 110 includes an automated assistant manager 115 that manages automated assistant clients 140-1, ..., 140-n on the client device 110. In some implementations, the automated assistant manager 115 can selectively enable or disable individual automated assistant clients 140-1, ..., 140-n on the client device 110. In some implementations, the automated assistant manager 115 can install or uninstall individual automated assistant clients 140-1, ..., 140-n on the client device 110. The automated assistant manager 115 can provide a user interface that can be utilized by a user of the client device 110 to perform various management functions with respect to the automated assistant clients 140-1, ..., 140-n on the client device 110, such as the aforementioned selective enabling, disabling, installing, or uninstalling of individual automated assistant clients 140-1, ..., 140-n on the client device 110.
[0049] The automated assistant manager 115 can selectively grant or deny access to one or more microphones 111 by various automated assistant clients 140-1, ..., 140-n. The automated assistant manager 115 can also selectively grant or deny access to one or more cameras and / or other visual components 113 and / or other input or output devices of the client device 110 by various automated assistant clients 140-1, ..., 140-n. For example, the automated assistant manager 115 can manage access to one or more speakers 112 by various automated assistant clients 140-1, ..., 140-n to prevent one of the various automated assistant clients 140-1, ..., 140-n from interrupting the audio output of another of the various automated assistant clients 140-1, ..., 140-n.
[0050] Figure 2 A flow chart illustrating an example method 200 for concurrent voice assistants is depicted. For convenience, the operations of method 200 are described with reference to a system that performs the operations. This system of method 200 includes one or more processors and / or other components of a client device. In addition, although the operations of method 200 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.
[0051] At block 205, the system concurrently provides access to the one or more microphones to the first automated assistant and the second automated assistant. In some implementations, automated assistant manager 115 of client device 110 concurrently provides automated assistant clients 140-1, ..., 140-n of client device 110 with access to one or more microphones 111 of client device 110. For example, automated assistant manager 115 can concurrently provide automated assistant clients 140-1, ..., 140-n with access to an input audio data stream including audio detected by one or more microphones 111.
[0052] At block 210, the system receives an indication from the first automated assistant that the first automated assistant has initiated a first session. In some implementations, automated assistant manager 115 of client device 110 receives an indication from a first automated assistant client (e.g., automated assistant client 140-1) that the first automated assistant client has initiated a first session.
[0053] In one example, hotword detection engine 122-1 of first automated assistant client 140-1 receives an input audio data stream that includes audio detected by one or more microphones 111 to which access is provided at block 205. The input audio data stream can be utilized by hotword detection engine 122-1, which can process the audio and / or other sensor data using a machine learning model to generate a predicted output. First automated assistant client 140-1 can make a decision whether to initiate a currently dormant automated assistant function, for example, in a first session based on determining whether the predicted output satisfies a threshold.
[0054] Still reference Figure 2 At block 215, the system performs the functions of blocks 220, 225, and 230 in response to receiving an indication from the first automated assistant that the first automated assistant has initiated the first session. In some implementations, automated assistant manager 115 of client device 110 performs the functions of blocks 220, 225, and 230 in response to receiving an indication at block 210 that the first automated assistant has initiated the first session.
[0055] Still referring to block 215, in some implementations, after receiving an indication from the first automated assistant that the first automated assistant has initiated the first session at block 210, and before receiving an indication from the first automated assistant that the first session has ended at block 265, the system can display a visual indication on a graphical user interface (e.g., on display 114 of client device 110) that access to the one or more microphones is being provided to the first automated assistant. The visual indication can be an icon or other graphic, and can identify the first automated assistant (e.g., by including a logo, name, abbreviation, or other visual representation of the first automated assistant).
[0056] Still referring to block 215, in some implementations, after receiving an indication from the first automated assistant that the first automated assistant has initiated the first conversation at block 210, the system can provide an audible indication that access to the one or more microphones is being provided to the first automated assistant. The audible indication can be a chime, a spoken word or phrase, or other sound that can identify the first automated assistant.
[0057] Still referring to box 215, in some implementations, in response to receiving an indication from the first automated assistant that the first automated assistant has initiated a first session, the system can reassign a physical button that, when activated, is configured to initiate a session with a second automated assistant instead of initiating a session with the first automated assistant. For example, a vehicle in which the client device 110 is installed can include a steering wheel button that initiates an automated assistant session. The steering wheel button can be automatically configured by the automated assistant manager 115 of the client device 110 to initiate an automated assistant session using a specific automated assistant client that was recently invoked (e.g., the first automated assistant client 140-1 that initiated the first session at box 210). This can modify a previous configuration (e.g., a previous configuration to initiate an automated assistant session using a second automated assistant client). The new configuration of the physical button can be maintained for a predetermined time (e.g., 10 minutes). Alternatively, the new configuration of the physical button can be maintained until a specific event has occurred, such as a user invoking a different automated assistant.
[0058] In other implementations, there may be multiple physical buttons, each of which, when activated, initiates a session with a specific automated assistant corresponding to the specific physical button. The system may provide configuration options that allow the user to specify a mapping between a specific physical button and a specific automated assistant. In still other implementations, the system may initiate a session with a specific automated assistant in response to various gestures. For example, the various gestures may include a long or short press on a specific area of the screen, a button on the screen, or a physical button; a single tap or double tap on a specific area of the screen, a button on the screen, or a physical button; some other physical input; etc.
[0059] Still reference Figure 2 At block 220 , the system continues to provide access to the one or more microphones to the first automated assistant. In some implementations, automated assistant manager 115 of client device 110 continues to provide first automated assistant client 140 - 1 with access to the one or more microphones 111 of client device 110 .
[0060] In one example, first automated assistant client 140-1 continues to receive an input audio data stream including audio detected by one or more microphones 111. An on-device speech recognizer 142-1 of automated assistant client 140-1 can utilize an on-device speech recognition model to process audio data of the input audio data stream that captures the spoken utterance to generate recognized text corresponding to the spoken utterance. An on-device NLU engine 144-1 of automated assistant client 140-1 can optionally utilize an on-device NLU model to perform on-device natural language understanding on the recognized text to generate NLU data. The NLU data can include, for example, an intent corresponding to the spoken utterance and optionally parameters (e.g., slot values) for the intent. Further, an on-device fulfillment engine 146-1 of automated assistant client 140-1 can optionally utilize an on-device fulfillment model to generate fulfillment data based on the NLU data. This fulfillment data may define local and / or remote responses (e.g., replies) to the spoken utterance, interactions with locally installed applications to be performed based on the spoken utterance, commands to be transmitted to an Internet of Things (IoT) device based on the spoken utterance (directly or via a corresponding remote system), and / or other parsing actions to be performed based on the spoken utterance. The fulfillment data is then provided for local and / or remote implementation / execution of the determined actions to parse the spoken utterance. Execution may include, for example, rendering local and / or remote responses (e.g., visually and / or audibly (optionally utilizing a local text-to-speech module)), interacting with locally installed applications, transmitting commands to IoT devices, and / or other actions.
[0061] Still reference Figure 2 At block 225, the system suspends providing access to the one or more microphones to the second automated assistant. In some implementations, automated assistant manager 115 of client device 110 suspends providing access to the one or more microphones 111 of client device 110 to the second automated assistant client (e.g., automated assistant client 140-n). In one example, automated assistant client 140-n is blocked from receiving an input audio data stream including audio detected by one or more microphones 111.
[0062] At block 230, the system blocks the second automated assistant from accessing one or more portions of the output audio data stream provided for rendering via one or more speakers, the one or more portions including the output audio data of the first automated assistant. In some implementations, the automated assistant manager 115 of the client device 110 blocks the second automated assistant client (e.g., automated assistant client 140-n) from accessing one or more portions of the output audio data stream provided for rendering via one or more speakers 112 of the client device 110. The one or more portions of the output audio data stream that the automated assistant client 140-n is blocked from accessing can include the output audio data of the first automated assistant client 140-1, such output audio data being an auditory response to the user's spoken utterance generated by the on-device fulfillment engine 146-1 of the first automated assistant client 140-1 and rendered by utilizing the text-to-speech module of the first automated assistant client 140-1.
[0063] At block 235, the system determines whether a notification has been identified to be output from the second automated assistant. In some implementations, automated assistant manager 115 of client device 110 determines whether a notification has been identified to be output (e.g., via one or more speakers 112 of client device 110) from a second automated assistant client (e.g., automated assistant client 140-n).
[0064] In one example, automated assistant client 140-n can provide automated assistant manager 115 with a notification to be output via one or more speakers 112 of client device 110. The notification can be a navigation notification from a navigation application (e.g., "Turn right in 200 feet"), a new message notification from a messaging application, an incoming phone call notification from a phone application, etc.
[0065] Still referring to block 235, in response to determining that a notification to be output from the second automated assistant has been identified, the flow proceeds to block 240. On the other hand, in response to determining that a notification to be output from the second automated assistant has not been identified, the flow proceeds to block 260.
[0066] Still reference Figure 2 , at block 240, the system determines a priority for the notification. In some implementations, the automated assistant manager 115 of the client device 110 determines a priority for the notification received at block 235. The priority may be numeric (e.g., 1 to 5), category-based (e.g., low / medium / high), or may be based on any other measure. In one example, an alert that may require immediate action by a user (e.g., a navigation notification) is assigned a relatively high priority, and an alert that is less likely to require immediate action by a user (e.g., a notification of an incoming email message) is assigned a relatively low priority.
[0067] At block 245, the system determines whether the priority of the notification satisfies the threshold. In some implementations, automated assistant manager 115 of client device 110 determines whether the priority of the notification received at block 240 satisfies the threshold. In one example, at block 240, the system may determine whether the notification received at block 235 has a high priority, a medium priority, or a low priority. In this example, if the priority of the notification is high, the threshold may be satisfied, and if the priority of the notification is medium or low, the threshold may not be satisfied.
[0068] Still referring to block 245, in response to determining that the priority of the notification does not satisfy the threshold, flow proceeds to block 250. On the other hand, in response to determining that the priority of the notification satisfies the threshold, flow proceeds to block 255.
[0069] At block 250, in response to the priority of the notification not satisfying the threshold, the system suppresses the notification until an indication is received from the first automated assistant that the first session has ended. In some implementations, at block 250, in response to the priority of the notification identified at block 235 not satisfying the threshold at block 245, the automated assistant manager 115 of the client device 110 suppresses the notification until an indication is received from the first automated assistant client 140-1 at block 260 that the first session has ended. In some implementations, suppressing the notification includes avoiding presentation of the notification on the client device 110, for example, by avoiding rendering of an audio element of the notification via one or more speakers 112 of the client device 110. Suppressing the notification may also include avoiding rendering of a visual element of the notification via the display 114 of the client device 110.
[0070] Still referring to block 250, in some implementations, automated assistant manager 115 of client device 110 buffers the notification being suppressed in a buffer. In some implementations, in response to receiving an indication from the first automated assistant (e.g., first automated assistant client 140-1) that the first session has ended, automated assistant manager 115 of client device 110 can output the notification from the buffer, e.g., by rendering an audio element of the notification via one or more speakers 112 of client device 110 and / or rendering a visual element of the notification via display 114 of client device 110.
[0071] Still reference Figure 2 At block 255, in response to the priority of the notification satisfying the threshold, the system outputs the notification before receiving an indication from the first automated assistant that the first session has ended. In some implementations, at block 255, in response to the priority of the notification identified at block 235 satisfying the threshold at block 245, the automated assistant manager 115 of the client device 110 outputs the notification before receiving an indication from the first automated assistant client 140-1 that the first session has ended at block 260. In some implementations, outputting the notification includes presenting the notification on the client device, for example, by rendering an audio element of the notification via one or more speakers 112 of the client device 110. Outputting the notification may also include rendering a visual element of the notification via a display 114 of the client device 110.
[0072] Still referring to block 255, in some implementations, automated assistant manager 115 of client device 110 can buffer one or more portions of the output audio data stream that include the output audio data of the first automated assistant in a buffer. Automated assistant manager 115 can pause presentation of the one or more portions of the output audio data stream that include the output audio data of the first automated assistant while outputting the notification from the second automated assistant. In some implementations, after outputting the notification from the second automated assistant, automated assistant manager 115 can resume presentation of the one or more portions of the output audio data stream that include the output audio data of the first automated assistant from the buffer.
[0073] Still reference Figure 2 At block 260, the system receives an indication from the first automated assistant that the first session has ended. In some implementations, automated assistant manager 115 of client device 110 receives an indication from first automated assistant client 140-1 that the first session initiated at block 210 has ended. For example, first automated assistant client 140-1 can close one or more currently active automated assistant functions based on completing the performance / execution of the determined action to parse the spoken utterance, and in response to closing the one or more currently active automated assistant functions, send an indication to automated assistant manager 115 that the first session has ended.
[0074] At block 265, the system performs the functions of blocks 270, 275, and 280 in response to receiving an indication from the first automated assistant that the first session has ended. In some implementations, automated assistant manager 115 of client device 110 performs the functions of blocks 270, 275, and 280.
[0075] At block 270, the system continues to provide the first automated assistant with access to the one or more microphones. In some implementations, the automated assistant manager 115 of the client device 110 continues to provide the first automated assistant client 140-1 with access to the one or more microphones 111 of the client device 110. In one example, the first automated assistant client 140-1 continues to receive an input audio data stream including audio detected by the one or more microphones 111.
[0076] At block 275, the system resumes providing access to the one or more microphones to the second automated assistant. In some implementations, automated assistant manager 115 of client device 110 resumes providing access to the one or more microphones 111 of client device 110 to the second automated assistant client (e.g., automated assistant client 140-n). In one example, automated assistant client 140-n is permitted to receive an input audio data stream including audio detected by one or more microphones 111.
[0077] At block 280, the system resumes providing the output audio data stream to the second automated assistant. In some implementations, automated assistant manager 115 of client device 110 resumes providing the output audio data stream to the second automated assistant client (e.g., automated assistant client 140-n). In some implementations, the second automated assistant uses the output audio data stream in noise cancellation (e.g., acoustic echo cancellation).
[0078] Figure 3 A flow chart illustrating an example method 300 for concurrent voice assistants is depicted. For convenience, the operations of method 300 are described with reference to a system that performs the operations. This system of method 300 includes one or more processors and / or other components of a client device. In addition, while the operations of method 300 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added. The operations of method 300 may be performed in any order. Figure 2 The method 200 may be performed at any point during the method 200 (eg, if a telephone call is initiated).
[0079] At block 305, the system determines that a phone call has been initiated. In some implementations, automated assistant manager 115 of client device 110 determines that a phone call has been initiated (eg, an outgoing call has been placed, or an incoming call has been answered).
[0080] At block 310, in response to determining that a telephone call has been initiated, the system suspends providing access to the one or more microphones to the first automated assistant and the second automated assistant. In some implementations, the automated assistant manager 115 of the client device 110 suspends providing access to the one or more microphones 111 of the client device 110 to the first automated assistant client 140-1 and the second automated assistant client (e.g., automated assistant client 140-n). In one example, automated assistant client 140-1 and automated assistant client 140-n are blocked from receiving an input audio data stream including audio detected by the one or more microphones 111.
[0081] At block 315, the system determines that the telephone call has ended. In some implementations, automated assistant manager 115 of client device 110 determines that the telephone call has ended.
[0082] At block 320, in response to determining that the telephone call has ended, the system resumes providing access to the one or more microphones to the first automated assistant and the second automated assistant. In some implementations, the automated assistant manager 115 of the client device 110 resumes providing access to the one or more microphones 111 of the client device 110 to the first automated assistant client 140-1 and the second automated assistant client (e.g., automated assistant client 140-n). In one example, automated assistant client 140-1 and automated assistant client 140-n are permitted to receive an input audio data stream including audio detected by the one or more microphones 111.
[0083] Figure 4 A flow chart illustrating an example method 400 for concurrent voice assistants is depicted. For convenience, the operations of method 400 are described with reference to a system in which the operations are performed. This system of method 400 includes one or more processors and / or other components of a client device. In addition, while the operations of method 400 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added. The operations of method 400 may be performed, for example, in Figure 2 The operations of block 280 of method 200 are performed afterwards.
[0084] At block 405, the system receives a user interface input via a graphical user interface, the user interface input being a request to initiate a second session on a second automated assistant. In some implementations, automated assistant manager 115 of client device 110 receives a user interface input via a graphical user interface displayed on display 114 of client device 110, the user interface input being a request to initiate a second session on a second automated assistant client (e.g., automated assistant client 140-n). For example, a user may activate a button displayed on display 114 of client device 110 that invokes a particular automated assistant (e.g., automated assistant client 140-n). Automated assistant manager 115 may interpret this user interface input as a request to initiate a second session on automated assistant client 140-n.
[0085] At block 410, the system performs the functions of blocks 415, 420, 425, and 430 in response to receiving the user interface input. In some implementations, automated assistant manager 115 of client device 110 performs the functions of blocks 415, 420, 425, and 430 in response to receiving the user interface input at block 405.
[0086] At block 415, the system causes the second automated assistant to initiate a second session. In some implementations, the automated assistant manager 115 of the client device 110 causes the second automated assistant client (e.g., automated assistant client 140-n) to initiate the second session. For example, the automated assistant client 140-n may start a currently dormant automated assistant function. The automated assistant function may include: speech recognition to generate recognized text using an on-device speech recognizer 142-n; natural language understanding (NLU) to generate NLU output using an on-device NLU engine 144-n; generating a response based on the recognized text and / or NLU output; transmission of audio data to a remote server; and / or transmission of recognized text to a remote server.
[0087] At block 420, the system continues to provide access to the one or more microphones to the second automated assistant. In some implementations, automated assistant manager 115 of client device 110 continues to provide access to the one or more microphones 111 of client device 110 to the second automated assistant client (e.g., automated assistant client 140-n).
[0088] In one example, automated assistant client 140-n continues to receive an input audio data stream including audio detected by one or more microphones 111. An on-device speech recognizer 142-n of automated assistant client 140-n can utilize an on-device speech recognition model to process audio data of the input audio data stream that captures the spoken utterance to generate recognized text corresponding to the spoken utterance. An on-device NLU engine 144-n of automated assistant client 140-n can optionally utilize an on-device NLU model to perform on-device natural language understanding on the recognized text to generate NLU data. The NLU data can include, for example, an intent corresponding to the spoken utterance and optionally parameters (e.g., slot values) for the intent. Further, an on-device fulfillment engine 146-n of automated assistant client 140-n can optionally utilize an on-device fulfillment model to generate fulfillment data based on the NLU data. This fulfillment data may define local and / or remote responses (e.g., replies) to the spoken utterance, interactions with locally installed applications to be performed based on the spoken utterance, commands to be transmitted to an Internet of Things (IoT) device based on the spoken utterance (directly or via a corresponding remote system), and / or other parsing actions to be performed based on the spoken utterance. The fulfillment data is then provided for local and / or remote implementation / execution of the determined actions to parse the spoken utterance. Execution may include, for example, rendering local and / or remote responses (e.g., visually and / or audibly (optionally utilizing a local text-to-speech module)), interacting with locally installed applications, transmitting commands to IoT devices, and / or other actions.
[0089] At block 425, the system suspends providing access to the one or more microphones to the first automated assistant. In some implementations, automated assistant manager 115 of client device 110 suspends providing first automated assistant client 140-1 with access to the one or more microphones 111 of client device 110. In one example, automated assistant client 140-1 is blocked from receiving an input audio data stream including audio detected by one or more microphones 111.
[0090] At block 430, the system blocks the first automated assistant from accessing the output audio data of the second automated assistant. In some implementations, the automated assistant manager 115 of the client device 110 blocks the first automated assistant client 140-1 from accessing the output audio data of the automated assistant client 140-n provided for rendering via the one or more speakers 112 of the client device 110. The output audio data that the automated assistant client 140-1 is blocked from accessing may include the output audio data of the automated assistant client 140-n, such output audio data being an auditory response to the spoken utterance of the user, the response generated by the on-device fulfillment engine 146-n of the automated assistant client 140-n and rendered by utilizing the text-to-speech module of the automated assistant client 140-n.
[0091] Figure 5 A flow chart illustrating an example method 500 for concurrent voice assistants is depicted. For convenience, the operations of method 500 are described with reference to a system that performs the operations. This system of method 500 includes one or more processors and / or other components of a client device. In addition, although the operations of method 500 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.
[0092] At block 505 , the system receives first audio data capturing a first spoken utterance of a user via one or more microphones. In some implementations, automated assistant manager 115 of client device 110 receives first audio data capturing a first spoken utterance of a user via one or more microphones 111 of client device 110 .
[0093] At block 510, the system concurrently provides the first audio data to a first hotword detector of the first automated assistant and a second hotword detector of the second automated assistant. In some implementations, automated assistant manager 115 of client device 110 concurrently provides the first audio data received at block 505 to hotword detection engine 122-1 of first automated assistant client 140-1 and hotword detection engine 122-n of automated assistant client 140-n.
[0094] In some implementations, the first hot word detector processes the first audio data using one or more machine learning models of the first hot word detector to generate a first predicted output indicating a probability of the first hot word being present in the first audio data. In some implementations, the second hot word detector processes the first audio data using one or more machine learning models of the second hot word detector to generate a second predicted output indicating a probability of the second hot word being present in the first audio data.
[0095] At block 515, the system receives a first confidence score based on a probability of the first hot word being present in the first audio data determined by the first hot word detector of the first automated assistant, and a second confidence score based on a probability of the second hot word being present in the first audio data determined by the second hot word detector of the second automated assistant. In some embodiments, automated assistant manager 115 of client device 110 receives a first confidence score based on a probability of the first hot word being present in the first audio data provided at block 510 determined by hot word detection engine 122-1 of first automated assistant client 140-1, and a second confidence score based on a probability of the second hot word being present in the first audio data provided at block 510 determined by hot word detection engine 122-n of automated assistant client 140-n.
[0096] At block 520, based on the first confidence score and the second confidence score, the system performs the functions of blocks 525, 530, and 535. In some implementations, based on the first confidence score and the second confidence score, automated assistant manager 115 of client device 110 performs the functions of blocks 525, 530, and 535. In some implementations, the functions of blocks 525, 530, and 535 are performed based on the first confidence score being higher than the second confidence score and the first confidence score satisfying a threshold.
[0097] At block 525, the system provides second audio data received via the one or more microphones that captures the user's second spoken utterance following the user's first spoken utterance to the first automated assistant. In some implementations, automated assistant manager 115 of client device 110 provides second audio data received via the one or more microphones that captures the user's second spoken utterance following the user's first spoken utterance received at block 505 to first automated assistant client 140-1.
[0098] At block 530, the system provides an indication that audio from the one or more microphones is being provided to the first automated assistant. In some implementations, automated assistant manager 115 of client device 110 provides an indication that audio from one or more microphones 111 of client device 110 is being provided to first automated assistant client 140-1. In some implementations, the indication that audio from the one or more microphones is being provided to the first automated assistant is a visual indication displayed on a graphical user interface, or an audible indication.
[0099] For example, the indication may be a visual indication that audio from one or more microphones 111 is being provided to first automated assistant client 140-1. The visual indication may be an icon or other graphic and may identify first automated assistant client 140-1 (e.g., by including a logo, name, abbreviation, or other visual representation of the first automated assistant). In another example, the indication may be an auditory indication that audio from one or more microphones 111 is being provided to first automated assistant client 140-1. The auditory indication may be a chime, a spoken word or phrase, or other sound that may identify first automated assistant client 140-1.
[0100] At block 535 , the system blocks the second automated assistant from obtaining the second audio data. In some implementations, automated assistant manager 115 of client device 110 can block automated assistant client 140 - n from obtaining the second audio data from block 525 .
[0101] Figure 6 610 is a block diagram of an example computing device 610 that can optionally be utilized to implement one or more aspects of the techniques described herein. In some implementations, one or more of a client device, a cloud-based automated assistant component, and / or other components can include one or more components of the example computing device 610.
[0102] The computing device 610 typically includes at least one processor 614 that communicates with a plurality of peripheral devices via a bus subsystem 612. These peripheral devices may include a storage subsystem 624 including, for example, a memory subsystem 625 and a file storage subsystem 626, a user interface output device 620, a user interface input device 622, and a network interface subsystem 616. The input and output devices allow for user interaction with the computing device 610. The network interface subsystem 616 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.
[0103] The user interface input device 622 may include a keyboard, a pointing device such as a mouse, trackball, touchpad or graphics tablet, a scanner, a touch screen integrated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways to input information into the computing device 610 or onto a communication network.
[0104] The user interface output device 620 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for producing a visible image. The display subsystem may also provide a non-visual display such as via an audio output device. Generally speaking, the use of the term "output device" is intended to include devices and methods of all possible types of information output from the computing device 610 to a user or to another machine or computing device.
[0105] The storage subsystem 624 stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 624 may include selected aspects of the methods disclosed herein and implementations thereof. Figure 1 The logic of the various components depicted in .
[0106] These software modules are typically executed by the processor 614 alone or in combination with other processors. The memory subsystem 625 included in the storage subsystem 624 may include multiple memories including a main random access memory (RAM) 630 for storing instructions and data during program execution and a read-only memory (ROM) 632 in which fixed instructions are stored. The file storage subsystem 626 may provide persistent storage for program and data files, and may include a hard drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain implementations may be stored by the file storage subsystem 626 in the storage subsystem 624, or in other machines accessible to the processor 614.
[0107] The bus subsystem 612 provides a mechanism for the various components and subsystems of the computing device 610 to communicate with each other as intended. Although the bus subsystem 612 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.
[0108] The computing device 610 can be of different types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, there are Figure 6 The description of the computing device 610 depicted in FIG. 6 is intended only as a specific example for the purpose of illustrating some implementations. Many other configurations of the computing device 610 are possible, with the configurations having similar Figure 6 More or fewer components than the computing devices depicted.
[0109] In situations where the systems described herein collect or otherwise monitor personal information about a user or can make use of personal information and / or monitored information, the user may be provided with the opportunity to control whether a program or feature collects user information (e.g., verbal utterances), or to control whether and / or how content that may be more relevant to the user is received from a content server. In addition, certain data may be processed in one or more ways before it is stored or used so that personally identifiable information is deleted. For example, the identity of the user may be processed so that personally identifiable information of the user cannot be determined, or where geographic location information is obtained, the geographic location of the user may be generalized (such as to a city, zip code, or state level) so that the specific geographic location of the user cannot be determined. Thus, the user may have control over how information is collected about the user and / or how the information is used.
[0110] Although several implementations have been described and illustrated herein, a variety of other means and / or structures for implementing functions and / or obtaining results and / or one or more of the advantages described herein may be utilized, and each of such changes and / or modifications is considered to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications for which this teaching is used. Those skilled in the art will recognize or confirm many equivalents of the implementations described herein using only routine experiments. Therefore, it should be understood that the aforementioned implementations are presented only by way of example, and it should be understood that within the scope of the attached claims and their equivalents, implementations may be practiced in a manner other than specifically described and claimed. The implementations disclosed herein relate to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure, if such features, systems, articles, materials, kits, and / or methods do not contradict each other.< / topping> < / artist> < / artist> < / artist> < / artist>
Claims
1. A method implemented by one or more processors, the method comprising: concurrently providing access to the one or more microphones to the first automated assistant and the second automated assistant; receiving, from the first automated assistant, an indication that the first automated assistant has initiated a first conversation; In response to receiving the indication from the first automated assistant that the first automated assistant has initiated the first conversation: continuing to provide the first automated assistant with access to the one or more microphones; discontinuing providing access to the one or more microphones to the second automated assistant; as well as preventing the second automated assistant from accessing one or more portions of an output audio data stream provided for rendering via one or more speakers, the one or more portions comprising the output audio data of the first automated assistant; receiving an indication from the first automated assistant that the first conversation has ended; as well as In response to receiving the indication from the first automated assistant that the first conversation has ended: continuing to provide the first automated assistant with access to the one or more microphones; resuming providing access to the one or more microphones to the second automated assistant; as well as Resume providing the output audio data stream to the second automated assistant, wherein the second automated assistant uses the output audio data stream in noise cancellation.
2. The method of claim 1 , wherein preventing the second automated assistant from accessing the one or more portions of the output audio data stream comprises: While the first automated assistant is providing auditory output during the first conversation, the second automated assistant is blocked from accessing the output audio data stream.
3. The method according to claim 1 or 2, further comprising: Confirm that the phone call has been initiated; responsive to determining that the telephone call has been initiated, discontinuing providing access to the one or more microphones to the first automated assistant and the second automated assistant; determining that the telephone call has ended; as well as In response to determining that the telephone call has ended, resuming providing access to the one or more microphones to the first automated assistant and the second automated assistant.
4. The method according to any one of the preceding claims, further comprising: After receiving the indication from the first automated assistant that the first automated assistant has initiated the first conversation, and before receiving the indication from the first automated assistant that the first conversation has ended: identifying a notification to be output from the second automated assistant; determining a priority for said notification; as well as In response to the priority of the notification satisfying a threshold, the notification is outputted before the indication is received from the first automated assistant that the first conversation has ended.
5. The method according to claim 4, further comprising: buffering the one or more portions of the output audio data stream including the output audio data of the first automated assistant in a buffer; pausing presentation of the one or more portions of the output audio data stream that include the output audio data of the first automated assistant while outputting the notification from the second automated assistant; as well as After outputting the notification from the second automated assistant, resuming rendering of the one or more portions of the output audio data stream including the output audio data of the first automated assistant from the buffer.
6. The method according to any one of the preceding claims 1 to 3, further comprising: After receiving the indication from the first automated assistant that the first automated assistant has initiated the first conversation, and before receiving the indication from the first automated assistant that the first conversation has ended: identifying a notification to be output from the second automated assistant; determining a priority for said notification; as well as In response to the priority of the notification satisfying a threshold, suppressing the notification until the indication is received from the first automated assistant that the first conversation has ended.
7. The method according to claim 6, further comprising: buffering the notification in a buffer; In response to receiving the indication from the first automated assistant that the first conversation has ended, outputting the notification from the buffer.
8. The method according to any one of the preceding claims, further comprising: receiving, via a graphical user interface, a user interface input, the user interface input being a request to initiate a second conversation on the second automated assistant; In response to receiving the user interface input: causing the second automated assistant to initiate the second conversation; continuing to provide the second automated assistant with access to the one or more microphones; discontinuing providing the first automated assistant with access to the one or more microphones; as well as The first automated assistant is prevented from accessing output audio data of the second automated assistant.
9. The method according to any one of the preceding claims, further comprising: After receiving the indication from the first automated assistant that the first automated assistant has initiated the first conversation, and before receiving the indication from the first automated assistant that the first conversation has ended: A visual indication is displayed on a graphical user interface that the first automated assistant is being provided access to the one or more microphones.
10. The method according to any one of the preceding claims, further comprising: Upon receiving the indication from the first automated assistant that the first automated assistant has initiated the first conversation: An audible indication is provided that access to the one or more microphones is being provided to the first automated assistant.
11. The method according to any one of the preceding claims, further comprising: In response to receiving the indication from the first automated assistant that the first automated assistant has initiated the first conversation: Reassigning a physical button that, when activated, is configured to initiate a session with the second automated assistant instead of initiating a session with the first automated assistant.
12. A method implemented by one or more processors, the method comprising: receiving, via one or more microphones, first audio data capturing a first spoken utterance of a user; concurrently providing the first audio data to a first hotword detector of a first automated assistant and a second hotword detector of a second automated assistant; Receiving a first confidence score based on a probability of a first hot word being present in the first audio data determined by the first hot word detector of the first automated assistant, and a second confidence score based on a probability of a second hot word being present in the first audio data determined by the second hot word detector of the second automated assistant; Based on the first confidence score and the second confidence score: providing, to the first automated assistant, second audio data received via the one or more microphones, the second audio data capturing a second spoken utterance of the user following the first spoken utterance of the user; providing an indication that audio from the one or more microphones is being provided to the first automated assistant; as well as The second automated assistant is prevented from obtaining the second audio data.
13. The method according to claim 12, wherein: the first hotword detector processes the first audio data using one or more machine learning models of the first hotword detector to generate a first predicted output, the first predicted output indicating the probability that the first hotword is present in the first audio data; and The second hot word detector processes the first audio data using one or more machine learning models of the second hot word detector to generate a second predicted output indicating the probability that the second hot word is present in the first audio data.
14. The method according to claim 12 or 13, wherein: The first confidence score is higher than the second confidence score; and The first confidence score satisfies a threshold.
15. The method of any one of claims 12 to 14, wherein the indication that audio from the one or more microphones is being provided to the first automated assistant is a visual indication displayed on a graphical user interface, or an audible indication.
16. A computer program product comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 15.
17. A computer-readable storage medium comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 15.
18. A system comprising a processor, a computer readable memory, one or more computer readable storage media, and program instructions stored together on the one or more computer readable storage media, the program instructions being executable to perform the method according to any one of claims 1 to 15.
