Detecting and handling failures in automated voice assistant

Abstract

The invention relates to detecting and handling failures in an automated voice assistant. A method includes executing, in an inactive state, a first automated assistant at least in part on a computing device operated by a user; when in the inactive state, determining, by the first automated assistant, that a second automated assistant fails to fulfill the request of the user; in response to determining that the second automated assistant fails to fulfill the request of the user, the first automated assistant processes cached audio data that captures a spoken utterance of the user that includes the request that the second automated assistant fails to fulfill, or processes a feature of the cached audio data, to determine to fulfill a response to the request of the user; and providing, by the first automated assistant, the response to the user to fulfill the request of the user.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 202080101563.3, with a filing date of December 15, 2020. Background Art

[0003] A human being is able to engage in a human-computer conversation with an interactive software application referred to herein as an "automated assistant" (also referred to as a "digital agent," "interactive personal assistant," "intelligent personal assistant," "assistant application," "conversational agent," etc.). For example, a human being (who may be referred to as a "user" when they interact with an automated assistant) may provide commands and / or requests to the automated assistant using spoken natural language input (i.e., utterances), which in some cases may be converted to text and then processed by providing textual (e.g., typed) natural language input, and / or by touch-free and / or speech-free physical movement(s) (e.g., gesture(s), eye gaze, facial movement, etc.). The automated assistant responds to the request by providing responsive user interface output (e.g., audible and / or visual user interface output), controlling one or more smart devices, and / or controlling one or more functions of a device implementing the automated assistant (e.g., controlling other application(s) of the device).

[0004] As mentioned above, many automated assistants are configured to interact via spoken utterances. To protect user privacy and / or conserve resources, the automated assistant avoids performing one or more automated assistant functions based on all spoken utterances present in audio data detected via microphone(s) of a client device implementing (at least in part) the automated assistant. Instead, certain processing based on spoken utterances occurs only in response to determining that certain condition(s) exist.

[0005] For example, many client devices including automated assistants and / or interfaced with automated assistants include a hotword detection model. When the microphone(s) of such client device are not deactivated, using the hotword detection model, the client device can continuously process audio data via the microphone(s) to generate a prediction output indicating whether there are one or more hotwords (including multi-word phrases), such as "Hey Assistant", "Good Assistant" and / or "Assistant". When the prediction output indicates the presence of a hotword, any audio data that follows (and is optionally determined to include voice activity) within a threshold amount of time can be processed by one or more on-device and / or remote automated assistant components, such as (multiple) speech recognition components, (multiple) voice activity detection components, etc. Further, the recognized text (from (multiple) speech recognition components) can be processed using (multiple) natural language understanding engines and / or (multiple) actions can be performed based on the natural language understanding engine output. The (multiple) actions can include, for example, generating and providing responses and / or controlling one or more applications and / or (multiple) smart devices. However, when the prediction output indicates that no hot words exist, the corresponding audio data will be discarded without any further processing, thereby saving resources and protecting user privacy.

[0006] As automated assistants become more common, it is increasingly common to have multiple different automated assistants running on the same client device or across different client devices that are close to each other (e.g., in the same room). In some cases, multiple automated assistants may be pre-installed on some client devices, or alternatively a user may be provided with the option of installing one or more new additional automated assistants that can specialize in specific areas or perform certain tasks.

[0007] In situations where multiple automated assistants are running on the same client device and / or across different client devices that are in close proximity to each other, there may be situations where a user directs a request to a first automated assistant (e.g., by using a hotword associated with the first automated assistant), but the first automated assistant fails to handle the request or returns a suboptimal, inaccurate, or incomplete result in response to the request. However, if the user directs the request to a second automated assistant (e.g., by using a hotword associated with the second automated assistant), the second automated assistant may properly handle the request. Summary of the invention

[0008] Some embodiments disclosed herein relate to improving device performance and capabilities, and improving the user experience provided by an automated assistant running on a device by detecting and handling failures of other automated assistants. As described in more detail herein, in response to detecting that another automated assistant failed to fulfill a user's request or detecting a suboptimal or inaccurate result provided by the other automated assistant in response to the request, in some embodiments, the automated assistant proposes to the user a handling of the request that the other automated assistant failed to fulfill (or failed to fulfill optimally / accurately), and, if requested, provides a response that fulfills the user's request.

[0009] In some embodiments, the automated assistant may have a general understanding of the user's interactions with other automated assistants (e.g., running on the same client device and / or another client device located nearby), and in response to one of the other automated assistants failing to fulfill the user's request, offer to handle the user's request. In some embodiments, the offer to handle the user's request may be conditional on the automated assistant determining that the automated assistant is sufficiently likely to provide a response that fulfills the user's request. In other embodiments (e.g., in response to the user specifying through settings that the automated assistant should automatically respond to a failure), in response to one of the other automated assistants failing to fulfill the user's request, the automated assistant may automatically provide a response that fulfills the user's request.

[0010] In an example, a user may direct a request to a first automated assistant, such as "Ok Assistant 1, where can I buy socks near me?" The first automated assistant may respond, "Sorry, I don't know any stores nearby." In this example, the first automated assistant fails to fulfill the user's request. If the user initially directed the request to a second automated assistant, the second automated assistant may respond with "Here is a list of stores near you that sell socks. Do you need directions to one of them?"

[0011] Continuing with the example, the second automated assistant detects that the first automated assistant has failed to fulfill the user's request. In response to detecting the failure, the second automated assistant can automatically provide a response to fulfill the user's request, such as "I can help—here is a list of nearby stores." Alternatively, instead of automatically providing a response to fulfill the user's request, the second automated assistant can automatically provide an indication of the availability of the response to fulfill the user's request, such as by using a light or display of a client device on which the second automated assistant is executing, or by playing a sound (e.g., a ringtone) on a speaker of the client device on which the second automated assistant is executing.

[0012] In some implementations, the user invokes the first automated assistant via a hotword or through another mechanism on the device, such as by asking, "Ok Assistant 1, what are the best places to eat nearby?" The first automated assistant can perform query processing, for example, by running a DSP-driven hotword detector, then passing the audio for speech recognition and running a transcription of the audio through query interpretation and fulfillment.

[0013] In some embodiments, when a query is issued to a first automated assistant, the user utterance can be cached locally for further processing on a device on which a second automated assistant is running. In some embodiments, only user input is cached, while in other embodiments, the response of the first automated assistant is cached in addition to the user input. If both the first automated assistant and the second automated assistant are running on the same device, the caching can be performed at the meta-assistant software layer running on the device. In the event that the first automated assistant and the second automated assistant are not running on the same device, the second automated assistant can detect queries directed to the first automated assistant (e.g., using a hot word model that detects the first automated assistant's hot word(s) or using an always-on ASR).

[0014] In some implementations, if both automated assistants are on the same device, the meta-assistant can implement failover from a first automated assistant to a second automated assistant. On demand (i.e., in the event of a query failure) or in parallel with the first automated assistant, the meta-assistant can provide the second automated assistant with access to the audio including the user input and / or other features derived from the audio, such as an automatic speech recognition (ASR) transcription. The meta-assistant can also provide the second automated assistant with access to the first automated assistant's responses.

[0015] In some embodiments, if the two automated assistants are located on different devices, there may be no direct communication interface available between the second automated assistant and the shared software stack or meta-assistant. In this case, detection and reaction may occur independently on the software stack of the second automated assistant. In some embodiments, the second automated assistant can determine when to start processing the query (e.g., preemptively, before observing whether the first automated assistant has failed, so that it can intervene with low latency). The second automated assistant can independently determine that the user triggered the first automated assistant for the query, such as by listening for the same (multiple) hot words or having an always-on speech recognition system that can detect user-assistant interactions. The second automated assistant can also use speaker identification to distinguish the user's voice from the voice of the first automated assistant, and identify failed fulfillment of the user's query in its software stack rather than in the shared software stack.

[0016] In some embodiments, a shared interface may be provided where the primary processing stack knows whether the fulfillment of the user query was successful after the first automated assistant completes the handling. If successful, then the second automated assistant does not need to take any further action. On the other hand, if it was not successful, then the meta-assistant can provide the cached audio (or other cached results, such as explanations) to the second automated assistant. In some embodiments, the detection of whether the query was successful can also be done automatically by listening to the provided response and inferring whether it was a mishandled query. For example, the stack can utilize speaker identification for text-to-speech audio from the first automated assistant, extract the corresponding audio response, and process it through a general ASR system, and then a final NLU-based classification system (e.g., neural network-based or heuristic-based) interprets answers such as "Sorry, there's nothing I can do" as a failure of the first automated assistant.

[0017] In other embodiments, the second automated assistant can process the user's utterance in parallel with the first automated assistant and prepare answers to the user's query in case the first automated assistant fails to successfully respond to the query or returns suboptimal or inaccurate results in response to the query. The second automated assistant can provide answers to the query when the second automated assistant determines that the first automated assistant fails to successfully respond to the query or returns suboptimal or inaccurate results in response to the request. In some embodiments, the second automated assistant can intervene and provide answers before determining that the first automated assistant fails to successfully respond to the query or returns suboptimal or inaccurate results in response to the request (e.g., in response to the meta-assistant determining that the second automated assistant should take over for the first automated assistant and respond to the query).

[0018] In some embodiments, the second automated assistant can passively detect user input and answers provided by the first automated assistant to enable the second automated assistant to intervene and provide answers in a multi-turn conversation where failures may occur after multiple interactions. In some embodiments, the second automated assistant can also respond when it is able to supplement the response to the query provided by the first automated assistant.

[0019] In various embodiments, a method implemented by one or more processors may include: executing a first automated assistant at least partially on a computing device operated by a user in an inactive state; while in the inactive state, determining, by the first automated assistant, that a second automated assistant has failed to fulfill a request of the user; in response to determining that the second automated assistant has failed to fulfill the request of the user, the first automated assistant processing cached audio data that captures user spoken utterances including the request that the second automated assistant has failed to fulfill, or processing features of the cached audio data to determine a response that fulfills the user request; and providing, by the first automated assistant, to the user, the response that fulfills the user request.

[0020] In some embodiments, determining that the second automated assistant failed to fulfill the user's request may include: receiving audio data that captures an initial response; and using a speaker identification on the audio data that captures the initial response to determine that the initial response was provided by the second automated assistant. In some embodiments, determining that the second automated assistant failed to fulfill the user's request may also include using a hot word detection model to process the audio data that captures the initial response to determine that the initial response did not fulfill the user's request. In some embodiments, determining that the second automated assistant failed to fulfill the user's request may also include: using automatic speech recognition to process the audio data that captures the initial response to generate text; and using natural language processing technology to process the text to determine that the initial response did not fulfill the user's request.

[0021] In some implementations, the cached audio data further captures an initial response provided by the second automated assistant to the user. In some implementations, the second automated assistant is executed on a computing device, and the cached audio data is received by the first automated assistant via a meta-assistant executing on the computing device. In some implementations, the second automated assistant is executed on another computing device, and the cached audio data is received by the first automated assistant via one or more microphones of the computing device.

[0022] In some implementations, the first automated assistant causes a response fulfilling the user request to be provided on another computing device. In some implementations, the first automated assistant causes a response fulfilling the user request to be displayed on a display of the computing device.

[0023] In some additional or alternative embodiments, a computer program product may include one or more computer-readable storage media on which program instructions are collectively stored. The program instructions may be executable to: execute a first automated assistant at least partially on a computing device operated by a user in an inactive state; while in the inactive state, determine, by the first automated assistant, that a second automated assistant has failed to fulfill a request of the user; in response to determining that the second automated assistant has failed to fulfill the request of the user, process, by the first automated assistant, cached audio data that captures spoken utterances of the user including the request that the second automated assistant has failed to fulfill, or process features of the cached audio data to determine a response to fulfill the user's request; and provide, by the first automated assistant, an indication of the availability of the response to fulfill the user's request to the user.

[0024] In some implementations, the availability indication is a visual indication provided by the computing device. In some implementations, the program instructions are further executable to: receive, by the first automated assistant, a request to fulfill a response to a user request; and in response to receiving the request to fulfill a response to the user request, provide, by the first automated assistant, a response to fulfill the user request to the user.

[0025] In some additional or alternative embodiments, a system may include a processor, a computer-readable memory, one or more computer-readable storage media, and program instructions collectively stored on the one or more computer-readable storage media. The program instructions may be executable to: execute a first automated assistant at least partially on a computing device operated by a user in an inactive state; while in the inactive state, by the first automated assistant, determine that a second automated assistant failed to fulfill a request of the user; in response to determining that the second automated assistant failed to fulfill the request of the user, provide the user with an indication that the first automated assistant is available to fulfill the request of the user; in response to receiving an instruction to fulfill the request from the user, process, by the first automated assistant, cached audio data that captures spoken utterances of the user including the request that the second automated assistant failed to fulfill, or process features of the cached audio data to determine a response to fulfill the request of the user; and provide, by the first automated assistant, the response to fulfill the request of the user to the user.

[0026] The above description is provided as an overview of some embodiments of the present disclosure. Further descriptions of these and other embodiments are described in more detail below.

[0027] Various embodiments can include a non-transitory computer-readable storage medium storing instructions executable by one or more processors (e.g., central processing unit(s) (CPU(s), graphics processing unit(s) (GPU(s), digital signal processor(s) (DSP(s), and / or tensor processing unit(s) (TPU(s))) to perform methods such as one or more methods described herein. Other embodiments can include an automated assistant client device (e.g., a client device including at least an automated assistant interface for interfacing with cloud-based automated assistant component(s)) including processor(s) operable to execute stored instructions to perform methods such as one or more methods described herein. Other embodiments can include a system of one or more servers including one or more processors operable to execute stored instructions to perform methods such as one or more methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1A and1B Depicted are example process flows demonstrating aspects of the present disclosure, according to various implementations.

[0029] Figure 2 Depicted include Figure 1A and 1B 1 and 10. Block diagram of an example environment in which various components and implementations disclosed herein may be implemented.

[0030] Figure 3 A flow chart illustrating an example method of detecting and handling failures in other automated assistants is depicted.

[0031] Figure 4 A flow chart illustrating an example method of detecting and handling failures in other automated assistants is depicted.

[0032] Figure 5 A flow chart illustrating an example method of detecting and handling failures in other automated assistants is depicted.

[0033] Figure 6 An example architecture for a computing device is depicted. DETAILED DESCRIPTION

[0034] Figure 1A and 1B An example process flow illustrating various aspects of the present disclosure is depicted. The client device 110 Figure 1A is shown in and includes the Figure 1A Components within the box representing the client device 110. The machine learning engine 122A can receive audio data 101 corresponding to spoken speech detected via one or more microphones of the client device 110 and / or other sensor data 102 corresponding to speechless physical movement(s) detected via one or more non-microphone sensor components of the client device 110 (e.g., gesture(s) and / or movement(s), body posture(s) and / or body movement(s), eye gaze, facial movement, mouth movement, etc.). The one or more non-microphone sensors can include camera(s) or other visual sensor(s), proximity sensor(s), pressure sensor(s), accelerometer(s), magnetometer(s), and / or other sensor(s). The machine learning engine 122A processes the audio data 101 and / or other sensor data 102 using the machine learning model 152A to generate a prediction output 103. As described herein, the machine learning engine 122A may be a hotword detection engine 122B or an alternative engine, such as a voice activity detector (VAD) engine, an endpoint detector engine, and / or other engine(s).

[0035] In some embodiments, when the machine learning engine 122A generates a prediction output 103, it can be stored locally on the client device in the on-device storage 111, and optionally associated with the corresponding audio data 101 and / or other sensor data 102. In some versions of these embodiments, the prediction output can be retrieved by the gradient engine 126 for use in generating a set of gradients 106 at a later time (e.g., based on comparing the prediction output to the true value output), such as when one or more conditions described herein are met. The on-device storage 111 can include, for example, a read-only memory (ROM) and / or a random access memory (RAM). In other embodiments, the prediction output 103 can be provided to the gradient engine 126 in real time.

[0036] Client device 110 can decide whether to initiate currently dormant automated assistant function(s) (e.g., Figure 2 ), avoid using the assistant activation engine 124 to initiate the currently dormant (multiple) automated assistant functions and / or close the currently active (multiple) automated assistant functions. The automated assistant functions can include: speech recognition for generating recognized text, natural language understanding (NLU) for generating NLU output, generating responses based on recognized text and / or NLU output, transmitting audio data to a remote server, and / or transmitting recognized text to a remote server. For example, assuming that the predicted output 103 is a probability (e.g., 0.80 or 0.90) and the threshold at box 182 is a threshold probability (e.g., 0.85), if the client device 110 determines that the predicted output 103 (e.g., 0.90) satisfies the threshold at box 182 (e.g., 0.85), then the assistant activation engine 124 can initiate the currently dormant (multiple) automated assistant functions.

[0037] In some embodiments, and as Figure 1B As depicted, the machine learning engine 122A can be a hot word detection engine 122B. It is worth noting that various automated assistant functions (multiple) are currently dormant (i.e., as indicated by dashed lines), such as the on-device speech recognizer 142, the on-device NLU engine 144, and / or the on-device fulfillment engine 146. Further, assume that the predicted output 103 generated using the hot word detection model 152B and based on the audio data 101 satisfies the threshold at box 182, and the voice activity detector 128 detects user speech directed to the client device 110.

[0038] In some versions of these embodiments, assistant activation engine 124 activates on-device speech recognizer 142, on-device NLU engine 144, and / or on-device fulfillment engine 146 as currently dormant automated assistant functionality(ies). For example, on-device speech recognizer 142 can process audio data 101 using on-device speech recognition model 142A for a spoken utterance including the hotword "good assistant" and additional commands and / or phrases following the hotword "good assistant" to generate recognition text 143A, on-device NLU engine 144 can process recognition text 143A using on-device NLU model 144A to generate NLU data 145A, on-device fulfillment engine 146 can process NLU data 145A using on-device fulfillment model 146A to generate fulfillment data 147A, and client device 110 can use fulfillment data 147A in performing 150 one or more actions in response to audio data 101.

[0039] In other versions of these embodiments, assistant activation engine 124 activates a unique on-device fulfillment engine 146, without activating on-device speech recognizer 142 and on-device NLU engine 144, to process various commands, such as “no,” “stop,” “cancel,” and / or other commands that can be processed without on-device speech recognizer 142 and on-device NLU engine 144. For example, on-device fulfillment engine 146 processes audio data 101 using on-device fulfillment model 146A to generate fulfillment data 147A, and client device 110 can use fulfillment data 147A in performing 150 one or more actions in response to audio data 101. Moreover, in versions of these embodiments, the assistant activation engine 124 is initially capable of activating currently dormant automation function(s) to verify that the decision made at box 182 is correct (e.g., the audio data 101 does in fact include the hot word "good assistant") by initially only activating the on-device speech recognizer 142 to determine that the audio data 101 includes the hot word "good assistant"), and / or the assistant activation engine 124 is capable of transmitting the audio data 101 to one or more servers (e.g., the remote server 160) to verify that the decision made at box 182 is correct (e.g., the audio data 101 does in fact include the hot word "good assistant").

[0040] Return to Figure 1AIf the client device 110 determines that the predicted output 103 (e.g., 0.80) fails to satisfy the threshold value (e.g., 0.85) at block 182, then the assistant activation engine 124 can avoid initiating currently dormant automated assistant function(s) and / or shut down any currently active automated assistant function(s). Further, if the client device 110 determines that the predicted output 103 (e.g., 0.80) fails to satisfy the threshold value (e.g., 0.85) at block 182, then the client device 110 can determine whether further user interface input is received at block 184. For example, the further user interface input can be an additional spoken utterance including the hotword, additional speechless physical movement(s) acting as a proxy for the hotword, actuation of an explicit automated assistant invocation button (e.g., a hardware button or a software button), a sensed "squeeze" of the client device 110 (e.g., when the client device 110 is squeezed with at least a threshold amount of force, the automated assistant is invoked), and / or other explicit automated assistant invocations. If the client device 110 determines that no further user interface input is received at block 184 , the client device 110 can stop identifying the correction and end at block 190 .

[0041] However, if the client device 110 determines that further user interface input is received at block 184, then the system can determine whether the further user interface input received at block 184 includes correction(s) at block 186 that contradicts the decision (e.g., user-mediated or user-provided correction) made at block 182. If the client device 110 determines that the further user interface input received at block 184 does not include the correction at block 186, then the client device 110 can stop identifying corrections and end at block 190. However, if the client device 110 determines that the further user interface input received at block 184 includes the correction at block 186 that contradicts the initial decision made at block 182, then the client device 110 can determine the true value output 105.

[0042] In some embodiments, the gradient engine 126 can generate the gradient 106 based on the predicted output 103 and the true value output 105. For example, the gradient engine 126 can generate the gradient 106 based on comparing the predicted output 103 with the true value output 105. In some versions of these embodiments, the client device 110 locally stores the predicted output 103 and the corresponding true value output 105 in the device storage 111, and when one or more conditions are met, the gradient engine 126 retrieves the predicted output 103 and the corresponding true value output 105 to generate the gradient 106. For example, the one or more conditions can include that the client device is charging, the client device has at least a threshold state of charge, the temperature of the client device (based on one or more on-device temperature sensors) is less than a threshold, and / or the client device is not held by a user. In other versions of these embodiments, the client device 110 provides the predicted output 103 and the true value output 105 to the gradient engine 126 in real time, and the gradient engine 126 generates the gradient 106 in real time.

[0043] Moreover, the gradient engine 126 can provide the generated gradients 106 to the on-device machine learning training engine 132A. Upon receiving the gradients 106, the on-device machine learning training engine 132A uses the gradients 106 to update the on-device machine learning model 152A. For example, the on-device machine learning training engine 132A can utilize back propagation and / or other techniques to update the on-device machine learning model 152A. It is noted that in some embodiments, the on-device machine learning training engine 132A can utilize batch processing techniques to update the on-device machine learning model 152A based on the gradients 106 and additional gradients determined locally at the client device 110 based on additional corrections.

[0044] Further, the client device 110 can transmit the generated gradients 106 to the remote system 160. When the remote system 160 receives the gradients 106, the remote training engine 162 of the remote system 160 uses the gradients 106 and the additional gradients 107 from the additional client devices 170 to update the global weights of the global speech recognition model 152A1. The additional gradients 107 from the additional client devices 170 can each be generated based on the same or similar techniques as described above with respect to the gradients 106 (but based on locally identified corrections that are specific to those client devices).

[0045] In response to one or more conditions being met, the update distribution engine 164 can provide the updated global weights and / or the updated global speech recognition model itself to the client device 110 and / or (multiple) other client devices, as indicated by 108. One or more conditions can include, for example, a threshold duration and / or the number of trainings since the updated weights and / or the updated speech recognition model were last provided. One or more conditions can additionally or alternatively include, for example, a measured improvement in the updated speech recognition model and / or a threshold duration passed since the updated weights and / or the updated speech recognition model were last provided. When the updated weights are provided to the client device 110, the client device 110 can replace the weights of the machine learning model 152A on the device with the updated weights. When the updated global speech recognition model is provided to the client device 110, the client device 110 can replace the machine learning model 152A on the device with the updated global speech recognition model.

[0046] In some implementations, the on-device machine learning model 152A is transmitted (e.g., by the remote system 160 or other component(s)) for storage and use at the client device 110 based on the geographic region and / or other characteristics of the client device 110 and / or the user of the client device 110. For example, the on-device machine learning model 152A can be one of N available machine learning models for a given language, but can be trained based on corrections specific to a particular geographic region, and provided to the client devices 110 based on the client devices 110 being primarily located in the particular geographic region.

[0047] Now go to Figure 2 , the client device 110 is illustrated in an embodiment, wherein Figure 1A and 1B The machine learning engine is included as part of (or in communication with) one or more automated assistant clients 240 (e.g., the first automated assistant, the second automated assistant, and the meta-assistant) on various devices. The corresponding machine learning model is also illustrated as Figure 1A and 1B For simplicity, the machine learning engine interfaces from Figure 1A and 1B The other components are not in Figure 2 Medium picture. Figure 2 Picture shows Figure 1A and 1B 2 is an example of how various on-device machine learning engines and their corresponding machine learning models can be used by (multiple) automated assistant client(s) 240 to perform various actions.

[0048] Figure 2Client device 110 in FIG. 1 is illustrated with one or more microphones 211, one or more speakers 212, one or more cameras and / or other visual components 213, and display(s) 214 (e.g., a touch-sensitive display). Client device 110 may also include pressure sensor(s), proximity sensor(s), accelerometer(s), magnetometer(s), and / or other sensor(s) for generating other sensor data that supplements the audio data captured by the one or more microphones 211. Client device 110 selectively executes at least automated assistant client 240. Automated assistant client 240 may be configured to perform a plurality of operations in a plurality of ways. Figure 2 Examples of the automated assistant client 240 include an on-device hotword detection engine 122B, an on-device speech recognizer 142, an on-device natural language understanding (NLU) engine 144, and an on-device fulfillment engine 146. The automated assistant client 240 also includes a speech capture engine 242 and a visual capture engine 244. The automated assistant client 140 can include additional and / or alternative engines, such as a voice activity detector (VAD) engine, an endpoint detector engine, and / or (multiple) other engines. In some embodiments, one or more instances of the automated assistant client 240 may be omitted. Figure 2 One or more elements shown.

[0049] One or more cloud-based automated assistant components 280 can optionally be implemented on one or more computing systems (collectively referred to as "cloud" computing systems) that are communicatively coupled to client devices 110 via one or more local area networks and / or wide area networks (e.g., the Internet), generally indicated at 290. For example, cloud-based automated assistant components 280 can be implemented via a cluster of high-performance servers.

[0050] In various embodiments, from the user's perspective, an instance of automated assistant client 240 interacting with one or more cloud-based automated assistant components 280 may form what appears to be a logical instance of automated assistant 295 with which the user may engage in human-computer interaction (e.g., verbal interaction, gesture-based interaction, and / or touch-based interaction).

[0051] For example, client device 110 can be: a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device of a user's vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a stand-alone interactive speaker, a smart home appliance such as a smart TV (or a standard TV equipped with a networked dongle with automated assistant capabilities), and / or a wearable device of a user including a computing device (e.g., a watch of a user with a computing device, glasses of a user with a computing device, a virtual or augmented reality computing device). Additional and / or alternative client devices may be provided.

[0052] The one or more vision components 213 can take various forms, such as a thematic camera, a stereo camera, a LIDAR component (or other laser-based components), a radar component, etc. The one or more vision components 213 can be used, for example, by the vision capture engine 242 to capture vision frames (e.g., image frames, laser-based vision frames) of the environment in which the client device 110 is deployed. In some embodiments, such vision frame(s) can be used to determine whether a user is nearby the client device 110 and / or a user (e.g., a user's face) relative to the distance of the client device 110. For example, such determination(s) can be used to determine whether to activate a user. Figure 2 The various on-device machine learning engines and / or other engines depicted.

[0053] The speech capture engine 242 can be configured to capture the user's speech and / or other audio data captured via the microphone(s) 211. Further, the client device 110 can include pressure sensor(s), proximity sensor(s), accelerometer(s), magnetometer(s), and / or other sensor(s) for generating other sensor data that are in addition to the audio data captured via the microphone(s) 211. As described herein, such audio data and other sensor data can be used by the hotword detection engine 122B and / or other engines to determine whether to initiate one or more currently dormant automated assistant functions, avoid initiating one or more currently dormant automated assistant functions, and / or shut down one or more currently active automated assistant functions. The automated assistant functions can include an on-device speech recognizer 142, an on-device NLU engine 144, an on-device fulfillment engine 146, and additional and / or alternative engines. For example, the on-device speech recognizer 142 can process audio data capturing spoken utterances using an on-device speech recognition model 142A to generate recognized text 143A corresponding to the spoken utterances. The on-device NLU engine 144 optionally uses the on-device NLU model 144A to perform on-device natural language understanding on the recognized text 143A to generate NLU data 145A. For example, the NLU data 145A can include (multiple) intents corresponding to the spoken utterance and (multiple) parameters (e.g., slot values) optionally used for the (multiple) intents. Further, the on-device fulfillment engine 146 optionally uses the on-device fulfillment model 146A to generate fulfillment data 147A based on the NLU data 145A. The fulfillment data 147A can define local and / or remote responses (e.g., answers) to the spoken utterance, (multiple) interactions performed with (multiple) applications installed locally based on the spoken utterance, (multiple) commands transmitted to (multiple) Internet of Things (IoT) devices (directly or via corresponding (multiple) remote systems) based on the spoken utterance, and / or (multiple) other parsing actions performed based on the spoken utterance. The fulfillment data 147A is then provided for performing / executing the determined (multiple) actions locally and / or remotely to parse the spoken utterance. Execution can include, for example, rendering local and / or remote responses (e.g., visually and / or audibly (optionally using a local text-to-speech module)), interacting with locally installed applications, transmitting command(s) to IOT device(s), and / or other action(s).

[0054] Display(s) 214 can be used to display recognized text 143A and / or further recognized text 143B from speech recognizer 122 on the device, and / or one or more results from execution 150. Display(s) 214 can also be one of the user interface output component(s) through which the visual portion(s) of the response from automated assistant client 240 are rendered.

[0055] In some embodiments, (multiple) cloud-based automated assistant components 280 can include a remote ASR engine 281 that performs speech recognition, a remote NLU engine 282 that performs natural language understanding, and / or a remote fulfillment engine 283 that generates fulfillment. A remote execution module can also be optionally included, which performs remote execution based on fulfillment data determined locally or remotely. Additional and / or alternative remote engines can be included. As described herein, in various embodiments, on-device speech processing, on-device NLU, on-device fulfillment, and / or on-device execution can be prioritized at least because of the latency and / or network usage reduction they provide when parsing spoken utterances (due to the fact that (multiple) client servers are not required to round-trip to parse spoken utterances). However, one or more cloud-based automated assistant components 280 can be used at least selectively. For example, (multiple) such components can be used in parallel with (multiple) on-device components, and the output from (multiple) such components is used when (multiple) local components fail. For example, in some circumstances (e.g., due to relatively limited resources of client device 110), on-device fulfillment engine 146 may fail, and remote fulfillment engine 283 can leverage the more robust resources of the cloud to generate fulfillment data in such circumstances. Remote fulfillment engine 283 can operate in parallel with on-device fulfillment engine 146 and use its results when on-device fulfillment fails, or can be invoked in response to determining a failure of on-device fulfillment engine 146.

[0056] In various embodiments, an NLU engine (on-device and / or remote) can generate NLU data including one or more annotations of recognized text and one or more (e.g., all) terms of a natural language input. In some embodiments, the NLU engine is configured to identify and annotate various types of grammatical information in the natural language input. For example, the NLU engine may include a morphological module that can separate individual words into morphemes and / or annotate morphemes, for example, with the categories of individual words. The NLU engine may also include a portion of a speech tagger that is configured to annotate terms with the grammatical roles of the terms. Moreover, for example, in some embodiments, the NLU engine may additionally and / or alternatively include a dependency analyzer that is configured to determine syntactic relationships between terms in the natural language input.

[0057] In some implementations, the NLU engine may additionally and / or alternatively include an entity tagger configured to annotate entity references in one or more segments, such as references to people (including, for example, literary characters, celebrities, public figures, etc.), organizations, locations (real and fictional), etc. In some implementations, the NLU engine may additionally and / or alternatively include a coreference resolver (not depicted) configured to group or "cluster" references to the same entity based on one or more contextual clues. In some implementations, one or more components of the NLU engine may rely on annotations from one or more other components of the NLU engine.

[0058] The NLU engine may also include an intent matcher, which is configured to determine the intent of the user to participate in the interaction with the automated assistant 295. The intent matcher can use various techniques to determine the user's intent. In some embodiments, the intent matcher can access one or more local and / or remote data structures, which include, for example, multiple mappings between grammars and response intents. For example, the grammars included in the mapping can be selected and / or learned over time, and can represent the common intent of the user. For example, a grammar "play <artist>" may be mapped to a response action intent that calls a response action that causes the music of <artist> to be played on the client device 110. Another grammar "[weather|forecast] today" may match user queries, such as "What's the weather like today" and "What's the weather forecast like today?" In addition to or in place of grammars, in some embodiments, the intent matcher can also use one or more trained machine learning models alone or in combination with one or more grammars. These trained machine learning models can be trained to identify intents, such as by embedding the recognized text from the spoken utterance into a reduced-dimensional space, and then determining which other embeddings (and therefore intents) are closest, such as using techniques such as Euclidean distance, cosine similarity, etc. As seen in the "play <artist>" example grammar above, some grammars have slots (e.g., <artist>) that can be filled with slot values ​​(or "parameters"). Slot values ​​can be determined in a variety of ways. Often, the user actively provides slot values. For example, for the grammar "I order <toppings> pizza," the user might utter the phrase "I order sausage pizza," in which case the slot <toppings> is automatically filled. Other slot value(s) can be inferred based on, for example, the user's location, the currently rendered content, user preferences, and / or other clue(s).

[0059] The fulfillment engine (local and / or remote) can be configured to receive the predicted / estimated intent output by the NLU engine, as well as any associated slot values ​​and fulfill (or "parse") the intent. In various embodiments, the fulfillment (or "parse") of the user's intent may result in various fulfillment information (also referred to as fulfillment data), for example, generated / obtained by the fulfillment engine. This can include determining local and / or remote responses (e.g., answers) to the spoken utterance, (multiple) interactions performed with locally installed (multiple) applications based on the spoken utterance, (multiple) commands transmitted to (multiple) Internet of Things (IoT) devices (directly or via corresponding (multiple) remote systems) based on the spoken utterance, and / or (multiple) other parsed actions performed based on the spoken utterance. The on-device fulfillment can then initiate local and / or remote execution / execution of the determined (multiple) actions to parse the spoken utterance.

[0060] Figure 3 A flow chart illustrating an example method 300 for detecting and handling failures in other automated assistants is depicted. For convenience, the operations of method 300 are described with reference to a system that performs the operations. The system of method 300 includes one or more processors and / or other components of a client device. Moreover, although the operations of method 300 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0061] At block 310 , the system executes a first automated assistant at least in part on a computing device (eg, a client device) operated by a user in an inactive state.

[0062] At box 320, while in the inactive state, the first automated assistant determines whether the second automated assistant has failed to fulfill the user's request. In some embodiments, the failure of the second automated assistant to fulfill the user's request detected at box 320 may include a response from the second automated assistant indicating that the user's request cannot be fulfilled (e.g., "Sorry," "I can't do that," etc.). In other embodiments, the failure detected at box 320 may also include a response from the first automated assistant determining that the result provided by the second automated assistant in response to the user's request is suboptimal, inaccurate, or incomplete. In some embodiments, the second automated assistant may be running on a client device with the first automated assistant. In other embodiments, the second automated assistant may be running on another client device that is near the client device on which the first automated assistant is running (e.g., in the same room).

[0063] If at the iteration of block 320, the first automated assistant determines that the second automated assistant fulfilled the user's request, the system continues to block 330, and the process ends. On the other hand, if at the iteration of block 320, the first automated assistant determines that the second automated assistant failed to fulfill the user's request, the system continues to block 340.

[0064] Still referring to block 320, in some implementations, when determining whether the second automated assistant failed to fulfill the user's request, the first automated assistant receives audio data that captures the initial response, and then uses a speaker identification on the audio data that captures the initial response to determine whether the initial response was provided by the second automated assistant (e.g., the initial response was spoken in a voice known to be associated with the second automated assistant). If the first automated assistant determines that the initial response was not provided by the second automated assistant, the system continues to block 330, and the process ends. On the other hand, if the first automated assistant determines that the initial response was provided by the second automated assistant, the first automated assistant determines whether the initial response indicates that the second automated assistant failed to fulfill the user's request.

[0065] In some embodiments, the first automated assistant processes audio data that captures the initial response using a hotword detection model to determine whether the initial response fulfills the user's request (e.g., by detecting failure hotwords, such as "sorry," "I can't do that," etc.). In other embodiments, the first automated assistant processes the audio data that captures the initial response using automatic speech recognition to generate text, and then processes the text using natural language processing techniques to determine whether the initial response fulfills the user's request. In some embodiments, natural language processing techniques are used to identify suboptimal, inaccurate, or incomplete results, and based on identifying the suboptimal, inaccurate, or incomplete results, the first automated assistant can determine that the initial response fails to fulfill the user's request.

[0066] At block 340, in response to determining that the second automated assistant failed to fulfill the user's request, the first automated assistant processes the cached audio data that captures the user's spoken utterance including the request that the second automated assistant failed to fulfill, and / or processes features derived from the audio data (e.g., ASR transcription) to determine a response that fulfills the user's request. In some embodiments, at block 340, if the first automated assistant cannot determine a response that fulfills the user's request, the system continues to block 330 and the process ends. In some embodiments, the cached audio data further captures an initial response provided by the second automated assistant to the user.

[0067] In some implementations, the second automated assistant executes on the same computing device as the first automated assistant, and the cached audio data and / or features derived from the audio data (e.g., ASR transcription) are received by the first automated assistant via a meta-assistant executing on the computing device. In other implementations, the second automated assistant executes on another computing device, and the cached audio data is received by the first automated assistant via one or more microphones of the computing device. In other implementations, the second automated assistant executes on another computing device, and the first automated assistant receives the cached audio data and / or features derived from the audio data (e.g., ASR transcription) via an application programming interface (API).

[0068] At block 350, the first automated assistant provides the response fulfilling the user request (determined at block 340) to the user. In some implementations, the first automated assistant causes the response fulfilling the user request to be provided on the computing device on which the first automated assistant is executing (e.g., via a speaker or by displaying the response on a display of the computing device). In other implementations, the first automated assistant causes the response fulfilling the user request to be provided on another computing device (e.g., via a speaker or display).

[0069] Figure 4 A flow chart illustrating an example method 400 for detecting and handling failures in other automated assistants is depicted. For convenience, the operations of method 400 are described with reference to a system that performs the operations. The system of method 400 includes one or more processors and / or other components of a client device. Moreover, although the operations of method 400 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0070] At block 410 , the system executes a first automated assistant at least in part on a computing device (eg, a client device) operated by a user in an inactive state.

[0071] In box 420, while in the inactive state, the first automated assistant determines whether the second automated assistant has failed to fulfill the user's request. In some embodiments, the failure of the second automated assistant to fulfill the user's request detected at box 420 can include a response from the second automated assistant indicating that the user's request cannot be fulfilled (e.g., "Sorry," "I can't do that," etc.). In other embodiments, the failure detected at box 420 can also include a response from the first automated assistant determining that the result provided by the second automated assistant in response to the user's request is suboptimal, inaccurate, or incomplete. In some embodiments, the second automated assistant may be running on a client device with the first automated assistant. In other embodiments, the second automated assistant may be running on another client device that is near the client device on which the first automated assistant is running (e.g., in the same room).

[0072] At the iteration of block 420, if the first automated assistant determines that the second automated assistant fulfilled the user's request, the system continues to block 430 and the process ends. On the other hand, at the iteration of block 420, if the first automated assistant determines that the second automated assistant failed to fulfill the user's request, the system continues to block 440.

[0073] Still referring to block 420, in some embodiments, when determining whether the second automated assistant failed to fulfill the user's request, the first automated assistant receives audio data that captures the initial response, and then uses the speaker identification on the audio data that captures the initial response to determine whether the initial response was provided by the second automated assistant. If the first automated assistant determines that the initial response was not provided by the second automated assistant, the system continues to block 430, and the process ends. On the other hand, if the first automated assistant determines that the initial response was provided by the second automated assistant, the first automated assistant determines whether the initial response indicates that the second automated assistant failed to fulfill the user's request.

[0074] In some implementations, the first automated assistant processes the audio data capturing the initial response using a hotword detection model to determine whether the initial response fulfills the user's request (e.g., by detecting failure hotwords, such as "sorry," "I can't do that," etc.). In other implementations, the first automated assistant processes the audio data capturing the initial response using automatic speech recognition to generate text, and then processes the text using natural language processing techniques to determine whether the initial response fulfills the user's request.

[0075] At block 440, in response to determining that the second automated assistant failed to fulfill the user's request, the first automated assistant processes the cached audio data that captures the user's spoken utterance including the request that the second automated assistant failed to fulfill, and / or processes features derived from the audio data (e.g., ASR transcription) to determine a response that fulfills the user's request. In some implementations, at block 440, if the first automated assistant cannot determine a response that fulfills the user's request, the system continues to block 430 and the process ends. In some implementations, the cached audio data further captures an initial response provided by the second automated assistant to the user.

[0076] In some implementations, the second automated assistant executes on the same computing device as the first automated assistant, and the cached audio data and / or features derived from the audio data (e.g., ASR transcription) are received by the first automated assistant via a meta-assistant executing on the computing device. In other implementations, the second automated assistant executes on another computing device, and the cached audio data is received by the first automated assistant via one or more microphones of the computing device. In other implementations, the second automated assistant executes on another computing device, and the first automated assistant receives the cached audio data and / or features derived from the audio data (e.g., ASR transcription) via an application programming interface (API).

[0077] At block 450, the first automated assistant provides the user with an indication of availability to fulfill the response to the user request. In some implementations, the indication of availability is a visual indication (e.g., an indication on a display or light) and / or an audio indication (e.g., a ring tone) provided by the computing device on which the first automated assistant is executing.

[0078] At block 460, the first automated assistant determines whether a request to fulfill a response to the user request has been received (e.g., from a user). At an iteration of block 460, if the first automated assistant determines that a request to fulfill a response to the user request has not been received, the system continues to block 430, and the process ends. On the other hand, at an iteration of block 460, if the first automated assistant determines that a request to fulfill a response to the user request has been received, the system continues to block 470.

[0079] At block 470, in response to receiving the request for a response to fulfill the user request, the first automated assistant provides the response to fulfill the user request (determined at block 440). In some implementations, the first automated assistant causes the response to fulfill the user request to be provided on a computing device on which the first automated assistant is executing (e.g., via a speaker or by displaying the response on a display of the computing device). In other implementations, the first automated assistant causes the response to fulfill the user request to be provided on another computing device (e.g., via a speaker or display).

[0080] Figure 5 A flowchart of an example method 500 illustrating detection and handling of failures in other automated assistants is depicted. For convenience, the operations of method 500 are described with reference to a system that performs the operations. The system of method 500 includes one or more processors and / or other components of a client device. Moreover, although the operations of method 500 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0081] At block 510 , the system executes a first automated assistant at least in part on a computing device (eg, a client device) operated by a user in an inactive state.

[0082] At box 520, while in the inactive state, the first automated assistant determines whether the second automated assistant has failed to fulfill the user's request. In some embodiments, the failure of the second automated assistant to fulfill the user's request detected at box 520 may include a response from the second automated assistant indicating that the user's request cannot be fulfilled (e.g., "Sorry," "I can't do that," etc.). In other embodiments, the failure detected at box 520 may also include a response from the first automated assistant determining that the result provided by the second automated assistant in response to the user's request is suboptimal, inaccurate, or incomplete. In some embodiments, the second automated assistant may be running on a client device with the first automated assistant. In other embodiments, the second automated assistant may be running on another client device that is near the client device on which the first automated assistant is running (e.g., in the same room).

[0083] At the iteration of block 520, if the first automated assistant determines that the second automated assistant fulfilled the user's request, the system continues to block 530 and the process ends. On the other hand, at the iteration of block 520, if the first automated assistant determines that the second automated assistant failed to fulfill the user's request, the system continues to block 540.

[0084] Still referring to block 520, in some embodiments, when determining whether the second automated assistant failed to fulfill the user's request, the first automated assistant receives audio data that captures the initial response, and then uses the speaker identification on the audio data that captures the initial response to determine whether the initial response was provided by the second automated assistant. If the first automated assistant determines that the initial response was not provided by the second automated assistant, the system continues to block 530, and the process ends. On the other hand, if the first automated assistant determines that the initial response was provided by the second automated assistant, the first automated assistant determines whether the initial response indicates that the second automated assistant failed to fulfill the user's request.

[0085] In some implementations, the first automated assistant processes the audio data capturing the initial response using a hotword detection model to determine whether the initial response fulfills the user's request (e.g., by detecting failure hotwords, such as "sorry," "I can't do that," etc.). In other implementations, the first automated assistant processes the audio data capturing the initial response using automatic speech recognition to generate text, and then processes the text using natural language processing techniques to determine whether the initial response fulfills the user's request.

[0086] At block 540, in response to determining that the second automated assistant has failed to fulfill the user's request, the first automated assistant provides an indication to the user that the first automated assistant is available to fulfill the user's request. In some implementations, the indication is a visual indication (e.g., an indication on a display or light) and / or an audio indication (e.g., a ring tone) provided by a computing device on which the first automated assistant is executing.

[0087] At block 550, the first automated assistant determines whether instructions to fulfill the request have been received from the user. At an iteration of block 550, if the first automated assistant determines that instructions to fulfill the request have not been received from the user, the system continues to block 530, and the process ends. On the other hand, at an iteration of block 550, if the first automated assistant determines that instructions to fulfill the request have been received from the user, the system continues to block 560.

[0088] At block 560, in response to receiving an instruction from the user to fulfill the request, the first automated assistant processes the cached audio data that captures the user's spoken utterance including the request that the second automated assistant failed to fulfill, and / or processes features derived from the audio data (e.g., ASR transcription) to determine a response that fulfills the user's request. In some implementations, the processed cached audio data further captures an initial response provided by the second automated assistant to the user.

[0089] In some implementations, the second automated assistant executes on the same computing device as the first automated assistant, and the cached audio data and / or features derived from the audio data (e.g., ASR transcription) are received by the first automated assistant via a meta-assistant executing on the computing device. In other implementations, the second automated assistant executes on another computing device, and the cached audio data is received by the first automated assistant via one or more microphones of the computing device. In other implementations, the second automated assistant executes on another computing device, and the first automated assistant receives the cached audio data and / or features derived from the audio data (e.g., ASR transcription) via an application programming interface (API).

[0090] At block 570, the first automated assistant provides the response to the user that fulfills the user request (determined at block 560) to the user. In some implementations, the first automated assistant causes the response to fulfill the user request to be provided on the computing device on which the first automated assistant is executing (e.g., via a speaker or by displaying the response on a display of the computing device). In other implementations, the first automated assistant causes the response to fulfill the user request to be provided on another computing device (e.g., via a speaker or display).

[0091] Figure 6 610 is a block diagram of an example computing device 610 that can optionally be used to perform one or more aspects of the techniques described herein. In some implementations, one or more of the client device, cloud-based automated assistant component(s), and / or other components(s) can include one or more components of the example computing device 610.

[0092] The computing device 610 typically includes at least one processor 614 that communicates with a number of peripheral devices via a bus subsystem 612. These peripheral devices may include a storage subsystem 624 (including, for example, a memory subsystem 625 and a file storage subsystem 626), a user interface output device 620, a user interface input device 622, and a network interface subsystem 616. The input and output devices allow a user to interact with the computing device 610. The network interface subsystem 616 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0093] User interface input devices 622 may include a keyboard, a pointing device (such as a mouse, trackball, touchpad, or drawing tablet), a scanner, a touch screen incorporated into a display, an audio input device (such as a voice recognition system, a microphone), and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways to input information into computing device 610 or onto a communication network.

[0094] User interface output device 620 may include a display subsystem, a printer, a fax machine, or a non-visual display (such as an audio output device). The display subsystem may include a cathode ray tube (CRT), a flat panel device (such as a liquid crystal display (LCD)), a projection device, or some other mechanism for creating visual images. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, the use of the term "output device" is intended to include all possible types of devices and ways to output information from computing device 610 to a user or to another machine or computing device.

[0095] The storage subsystem 624 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 624 may include selected aspects of the methods disclosed herein and implementations thereof. Figure 1A and 1B The logic of the various components depicted.

[0096] These software modules are typically executed by the processor 614 alone or in combination with other processors. The memory subsystem 625 included in the storage subsystem 624 can include several memories, including a main random access memory (RAM) 630 for storing instructions and data during program execution and a read-only memory (ROM) 632 in which fixed instructions are stored. The file storage subsystem 626 can provide persistent storage for program and data files, and can include a hard drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules implementing the functionality of a particular embodiment can be stored by the file storage subsystem 626 in the storage subsystem 624, or in other machines accessible by the processor(s) 614.

[0097] The bus subsystem 612 provides a mechanism for the various components and subsystems of the computing device 610 to communicate with each other as intended. Although the bus subsystem 612 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0098] The computing device 610 can be of different types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, for the purposes of illustrating some embodiments, Figure 6 The description of computing device 610 depicted in FIG. 6 is intended only as a specific example. Figure 6 Many other configurations are possible with computing device 610 having more or fewer components than the computing device depicted in FIG.

[0099] In the case where the system described herein collects or otherwise monitors personal information about a user or can make use of personal information and / or monitored information, the user may be provided with the opportunity to control whether a program or feature collects user information (e.g., information about the user's social network, social actions or activities, occupation, the user's preferences, or the user's current geographic location) or to control whether and / or how to receive content that may be more relevant to the user from a content server. Moreover, before specific data is stored or used, the specific data may be disposed of in one or more ways so that personally identifiable information is removed. For example, the user's identity may be disposed of so that the user's personally identifiable information cannot be determined, or the user's geographic location (where geographic location information (such as city, zip code, or state level) is obtained) may be generalized so that the user's specific geographic location cannot be determined. Therefore, the user can control how information about the user is collected and / or used.

[0100] Although several embodiments have been described and illustrated herein, various other components and / or structures for performing functions and / or obtaining results and / or one or more of the advantages described herein may be utilized, and each of such changes and / or modifications is considered to be within the scope of the embodiments described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications for which the teachings are used. Those skilled in the art will recognize or be able to determine many equivalents of the specific embodiments described herein using only routine experiments. Therefore, it is to be understood that the aforementioned embodiments are presented only by way of example, and within the scope of the appended claims and their equivalents, the embodiments may be practiced in a manner different from that specifically described and claimed. The embodiments of the present disclosure relate to each individual feature, system, article, material, kit, and / or method described herein. In addition, if such features, systems, articles, materials, kits, and / or methods do not contradict each other, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure.

Claims

1. A method implemented by one or more processors, the method comprising: determining, by a first automated assistant executing at least in part on a computing device operated by a user, that a second automated assistant provided inaccurate results in response to a request of the user directed to the second automated assistant; In response to determining that the second automated assistant provided the inaccurate result in response to the request of the user directed to the second automated assistant, the first automated assistant automatically determines a response to fulfill the request of the user; as well as The response that fulfills the request of the user is provided to the user by the first automated assistant.

2. The method according to claim 1, wherein: Automatically determining by the first automated assistant a response to fulfill the request of the user includes automatically processing by the first automated assistant cached audio data that captures spoken utterances of the user that include the request of the user, or automatically processing by the first automated assistant features of the cached audio data.

3. The method according to claim 2, wherein: The cached audio data further captures an initial response of the second automated assistant to the request, the initial response being provided by the second automated assistant to the user.

4. The method according to claim 2, wherein: the second automated assistant executes on the computing device, and The buffered audio data is received by the first automated assistant via a meta-assistant executing on the computing device.

5. The method according to claim 2, wherein: The second automated assistant executes on another computing device, and The buffered audio data is received by the first automated assistant via one or more microphones of the computing device.

6. The method according to claim 1, wherein: Determining that the second automated assistant provided the inaccurate result in response to the request of the user includes: receiving audio data capturing an initial response of the second automated assistant to the request; and A speaker identification is used on the audio data capturing the initial response to determine that the initial response was provided by the second automated assistant.

7. The method according to claim 6, wherein: Determining that the second automated assistant provided the inaccurate result in response to the request of the user further includes processing the audio data capturing the initial response using a hotword detection model to determine that the initial response is the inaccurate result.

8. The method according to claim 6, wherein: Determining that the second automated assistant provided the inaccurate result in response to the request of the user includes: processing the audio data capturing the initial response using automatic speech recognition to generate text; and The text is processed using natural language processing techniques to determine that the initial response is a result of the inaccuracy.

9. The method according to claim 1, wherein: The first automated assistant causes the response fulfilling the request of the user to be provided on another computing device.

10. The method according to claim 1, wherein: The first automated assistant causes the response fulfilling the request of the user to be displayed on a display of the computing device.

11. A computer program product comprising one or more non-transitory computer-readable storage media having program instructions centrally stored on the one or more non-transitory computer-readable storage media, the program instructions being executable to: determining, by a first automated assistant executing at least in part on a computing device operated by a user, that a second automated assistant provided inaccurate results in response to a request of the user directed to the second automated assistant; In response to determining that the second automated assistant provided the inaccurate result in response to the request of the user directed to the second automated assistant, automatically determining, by the first automated assistant, a response to fulfill the request of the user; as well as An indication of the availability of the response to fulfill the request of the user is provided to the user by the first automated assistant.

12. The computer program product according to claim 11, wherein: Automatically determining by the first automated assistant a response to fulfill the request of the user includes automatically processing by the first automated assistant cached audio data that captures spoken utterances of the user that include the request of the user, or automatically processing by the first automated assistant features of the cached audio data.

13. The computer program product of claim 12, wherein: The cached audio data further captures an initial response of the second automated assistant to the request, the initial response being provided by the second automated assistant to the user.

14. The computer program product of claim 12, wherein: the second automated assistant executes on the computing device, and The buffered audio data is received by the first automated assistant via a meta-assistant executing on the computing device.

15. The computer program product of claim 12, wherein: The second automated assistant executes on another computing device, and The buffered audio data is received by the first automated assistant via one or more microphones of the computing device.

16. The computer program product of claim 11, wherein: Determining that the second automated assistant provided the inaccurate result in response to the request of the user includes: receiving audio data capturing an initial response of the second automated assistant to the request; using a speaker identification on the audio data capturing the initial response to determine that the initial response was provided by the second automated assistant; and The audio data capturing the initial response is processed using a hotword detection model to determine that the initial response is the inaccurate result.

17. The computer program product of claim 11, wherein: The indication of availability is a visual indication provided by the computing device.

18. The computer program product of claim 17, wherein: The program instructions are also executable to: receiving, by the first automated assistant, a request to fulfill the response to the request of the user; and In response to receiving the request to fulfill the response to the user's request, providing, by the first automated assistant, the response to fulfill the user's request to the user.

19. A system comprising: A processor, a computer-readable memory, one or more computer-readable storage media, and program instructions centrally stored on the one or more computer-readable storage media, the program instructions being executable to: determining, by a first automated assistant executing at least in part on a computing device operated by a user, that a second automated assistant provided inaccurate results in response to a request of the user directed to the second automated assistant; In response to determining that the second automated assistant provided the inaccurate result in response to the request of the user directed to the second automated assistant, providing an indication to the user that the first automated assistant is available to fulfill the request of the user; In response to receiving an instruction from the user to fulfill the request, determining, by the first automated assistant, a response to fulfill the request of the user; as well as The response that fulfills the request of the user is provided to the user by the first automated assistant.

20. The system of claim 19, wherein: Automatically determining by the first automated assistant a response to fulfill the request of the user includes automatically processing by the first automated assistant cached audio data that captures spoken utterances of the user that include the request of the user, or automatically processing by the first automated assistant features of the cached audio data.

Images