Failure detection and handling in automated voice assistants

Abstract

To provide a method for detecting and handling failures in automated voice assistants, etc.SOLUTION: A method includes the steps of: executing a first automated assistant in an inactive state; determining, by the first automated assistant while in the inactive state, that a second automated assistant has failed to fulfill a request of the user; processing, according to the determination that the second automated assistant has failed to fulfill the request of the user, cached audio data capturing the verbal utterance of the user that includes the request that the second automated assistant has failed to fulfill, or features of the cached audio data by the first automated assistant to determine a response that fulfills the request of the user; and providing the response that fulfills the request of the user to the user by the first automated assistant.SELECTED DRAWING: Figure 3

Description

[Background technology]

[0001] Humans may engage in human-computer interaction using interactive software applications (also known as “digital agents,” “interactive personal assistants,” “intelligent personal assistants,” “assistant applications,” “conversational agents,” etc.) referred to herein as “automated assistants.” For example, a person (who may be referred to as a “user” when interacting with an automated assistant) may give commands and / or requests to an automated assistant by giving textual (e.g., typed) natural language input, and / or by touch and / or utterance-free physical movement (e.g., hand gestures, gaze, facial movements, etc.) using oral natural language input (i.e., utterances), which may in some cases be converted to text and then processed. The automated assistant responds to requests by giving responsive user interface output (e.g., audible and / or visual user interface output), controlling one or more smart devices, and / or controlling one or more functions of a device implementing the automated assistant (e.g., controlling other applications of the device).

[0002] As mentioned above, many automated assistants are configured to interact through spoken language. To protect user privacy and / or conserve resources, automated assistants refrain from performing one or more of their functions based on all spoken language present in the audio data detected by the microphone of the client device implementing the automated assistant (at least partially). Rather, specific processing based on spoken language is performed only in response to a determination that certain conditions exist.

[0003] For example, many client devices that include and / or interface with automated assistants include a hotword detection model. When the microphone of such a client device is not deactivated, the client device can use the hotword detection model to continuously process audio data detected by the microphone to generate a predicted output indicating whether one or more hotwords (including multi-word phrases) such as "Hey, Assistant", "OK, Assistant", and / or "Assistant" are present. When the predicted output indicates the presence of a hotword, all subsequent audio data (and optionally determined to include voice activity) within a threshold amount of time can be processed by one or more on-device and / or remote automated assistant components such as a speech recognition component, a voice activity detection component, etc. Further, the recognized text (from the speech recognition component) can be processed using a natural language understanding engine and / or actions can be performed based on the output of the natural language understanding engine. Actions can include, for example, generating and providing a response and / or controlling one or more applications and / or smart devices. However, when the predicted output indicates the absence of a hotword, the corresponding audio data is discarded without any further processing, thereby conserving resources and protecting the user's privacy.

[0004] As automated assistants become more widespread, it is becoming increasingly common to run multiple different automated assistants on the same client device, or on different client devices located close to each other (for example, in the same room). In some situations, multiple automated assistants may be pre-installed on several client devices, or alternatively, users may be offered the option to install one or more new, additional automated assistants that may be specialized to perform specific areas or tasks.

[0005] In a situation where multiple automated assistants are running on the same client device and / or on different client devices that are close to each other, a user might make a request to the first automated assistant (for example, by using a hotword associated with the first automated assistant), but the first automated assistant might fail to process the request or return a suboptimal, inaccurate, or incomplete result in response to the request. However, if the user had made a request to the second automated assistant (for example, by using a hotword associated with the second automated assistant), the second automated assistant could have processed the request correctly. [Overview of the Initiative] [Means for solving the problem]

[0006] Some implementations disclosed herein aim to improve the performance and capabilities of a device and the user experience provided by an automated assistant running on the device by detecting and handling failures in other automated assistants. As described in more detail herein, in some implementations, upon detection of failure by another automated assistant to fulfill a user's request, or detection of a suboptimal or inaccurate result provided by another automated assistant in response to a request, the automated assistant may suggest to the user that the other automated assistant handle the request that it failed to fulfill (or failed to fulfill optimally / accurately) and, if requested, provide a response that fulfills the user's request.

[0007] In some implementations, an automated assistant may be aware of the user's interactions with other automated assistants (e.g., running on the same client device and / or another nearby client device) and may suggest processing the user's request in response to a failure by one of the other automated assistants to fulfill the user's request. In some implementations, the suggestion to process the user's request may be conditional on the automated assistant determining that it has a sufficiently high probability of being able to provide a response that satisfies the user's request. In other implementations (e.g., in response to a user specification through settings that the automated assistant should automatically respond to failures), an automated assistant may automatically provide a response that satisfies the user's request in response to a failure by one of the other automated assistants to fulfill the user's request.

[0008] In the example, the user might say to the first automated assistant, "Okay, Assistant 1. Are there any places nearby that sell socks?" The first automated assistant might respond, "Sorry, I don't know what stores are around you." In this example, the first automated assistant failed to fulfill the user's request. If the user had first made the request to the second automated assistant, the second automated assistant might have responded, "Here is a list of stores near you that sell socks. Shall I take you to one of them?"

[0009] Continuing with the example, the second automated assistant detects that the first automated assistant has failed to fulfill the user's request. In response to the detection of the failure, the second automated assistant may automatically provide a response that fulfills the user's request, such as, "I can help. Here is a list of nearby stores." Alternatively, instead of automatically providing a response that fulfills the user's request, the second automated assistant may automatically provide an indication of the availability of a response that fulfills the user's request, for example, by using the lights or display of the client device on which the second automated assistant is running, or by playing a sound (e.g., a chime) through the speaker of the client device on which the second automated assistant is running.

[0010] In some implementations, the user invokes a first automated assistant by a hotword or by another mechanism on the device, for example, by asking, "Okay, Assistant 1. Where's the best place to eat nearby?" The first automated assistant can then process the query, for example, by running a hotword detector with a DSP, then passing the speech for speech recognition, and performing speech transcription through the interpretation and fulfillment of the query.

[0011] In some implementations, when a query is issued to the first automated assistant, the user's utterance may be cached locally for further processing on the device on which the second automated assistant is running. In some implementations, only user input is cached, while in other embodiments, the user input and the response of the first automated assistant are cached. If both the first and second automated assistants are running on the same device, caching may be performed at the layer of meta-assistant software running on the device. If the first and second automated assistants are not running on the same device, the second automated assistant may detect queries directed to the first automated assistant (for example, using a hotword model to detect the hotwords of the first automated assistant, or using an always-on ASR).

[0012] In some implementations, if two automated assistants are located on the same device, the meta-assistant may enable failover from the first automated assistant to the second automated assistant. The meta-assistant may provide the second automated assistant with access to speech and / or other speech-derived features, such as speech and / or Automated Speech Recognition (ASR) transcriptions, including user input, on demand (i.e., in case of query failure) or in parallel with the first automated assistant. The meta-assistant may also provide the second automated assistant with access to the responses of the first automated assistant.

[0013] In some implementations, if the two automated assistants are located on different devices, there may be no direct communication interface available between the second automated assistant and the shared software stack or meta-assistant. In this case, detection and response may occur independently on the software stack for the second automated assistant. In some implementations, the second automated assistant may decide when to begin processing a query (for example, proactively before observing whether the first automated assistant has failed, so that intervention can occur with low latency). The second automated assistant may independently determine that the user has triggered the first automated assistant for a query, for example, by hearing the same hotword or by having an always-on speech recognition system that can detect user-assistant interaction. The second automated assistant may use speaker identification to distinguish between the user's voice and the voice of the first automated assistant and identify failed fulfillment of the user's query within the second automated assistant's software stack rather than within the shared software stack.

[0014] In some implementations, a shared interface may be provided that allows the main processing stack to know whether the user's query was successfully fulfilled after the first automated assistant has finished processing the user's query. If the user's query was successfully fulfilled, the second automated assistant does not need to take any further action. On the other hand, if the user's query was not successfully fulfilled, the meta-assistant may provide the second automated assistant with cached audio (or other cached results such as interpretations). In some implementations, detection of query success or failure can also be done automatically by listening to the provided response and inferring whether it is a mishandled query. For example, the stack could utilize speaker identification of the text-to-speech audio from the first automated assistant, extract the corresponding audio response, and process it with a general-purpose ASR system, and then with a final NLU-based classification system (e.g., neural network-based or heuristic-based), which would interpret a response such as "Sorry, I can't help you with that" as a failure by the first automated assistant.

[0015] In other implementations, a second automated assistant may process user utterances in parallel with the first automated assistant and prepare answers to user inquiries in case the first automated assistant fails to respond to the inquiry properly or returns suboptimal or inaccurate results in response to the inquiry. The second automated assistant may provide answers to inquiries at the point when it determines that the first automated assistant has failed to respond to the inquiry properly or will return suboptimal or inaccurate results in response to the request. In some implementations, the second automated assistant may intervene and provide answers to inquiries before it determines that the first automated assistant has failed to respond to the inquiry properly or will return suboptimal or inaccurate results in response to the request (for example, in response to a meta-assistant's decision that the second automated assistant should take over from the first automated assistant and respond to the inquiry).

[0016] In some implementations, a second automated assistant may passively detect both user input and the answers provided by the first automated assistant, enabling it to intervene in multi-turn conversations where failure may occur after several interactions and provide answers. In some implementations, the second automated assistant may also respond when it can complement the response to the query provided by the first automated assistant.

[0017] In various implementations, a method performed by one or more processors may include, at least in part, the steps of: running a first automated assistant in an inactive state on a computing device operated by a user; determining, while in an inactive state, that a second automated assistant has failed to satisfy a user's request; in response to the determination that the second automated assistant has failed to satisfy the user's request, the first automated assistant processes cached audio data or features of cached audio data that capture the user's spoken utterance containing the request that the second automated assistant failed to satisfy, in order to determine a response that satisfies the user's request; and providing the user with a response that satisfies the user's request through the first automated assistant.

[0018] In some implementations, the step of determining that a second automated assistant has failed to satisfy the user's request may include receiving audio data capturing the initial response and using speaker identification on the audio data capturing the initial response to determine that the initial response was provided by the second automated assistant. In some implementations, the step of determining that a second automated assistant has failed to satisfy the user's request may further include processing the audio data capturing the initial response using a hotword detection model to determine that the initial response does not satisfy the user's request. In some implementations, the step of determining that a second automated assistant has failed to satisfy the user's request may further include processing the audio data capturing the initial response using automatic speech recognition to generate text and processing the text using natural language processing techniques to determine that the initial response does not satisfy the user's request.

[0019] In some implementations, cached audio data further captures the initial response provided to the user by a second automated assistant. In some implementations, the second automated assistant runs on a computing device, and the cached audio data is received by the first automated assistant via a meta-assistant running on the computing device. In some implementations, the second automated assistant runs on another computing device, and the cached audio data is received by the first automated assistant via one or more microphones on the computing device.

[0020] In some implementations, the first automated assistant provides a response that satisfies the user's request on another computing device. In some implementations, the first automated assistant displays the response that satisfies the user's request on the computing device's display.

[0021] In some additional or alternative implementations, a computer program product may include one or more computer-readable storage media having program instructions stored together in one or more computer-readable storage media. The program instructions may be executable to at least partially execute a first automated assistant in an inactive state on a computing device operated by a user, and while inactive, the first automated assistant may determine that a second automated assistant has failed to satisfy the user's request, and in response to the determination that the second automated assistant has failed to satisfy the user's request, the first automated assistant may process cached audio data or features of cached audio data that capture the user's spoken utterance containing the request that the second automated assistant failed to satisfy to determine a response that satisfies the user's request, and the first automated assistant may be executable to provide the user with an indication of the availability of a response that satisfies the user's request.

[0022] In some implementations, availability instructions are visual instructions provided by the computing device. In some implementations, program instructions can be further executed by a first automated assistant to provide the user with a response that satisfies the user's request, upon receiving the request for a response that satisfies the user's request.

[0023] In some additional or alternative implementations, the system may include a processor, computer-readable memory, one or more computer-readable storage media, and program instructions stored together in one or more computer-readable storage media. The program instructions may be executable to at least partially execute a first automated assistant in an inactive state on a computing device operated by a user, to determine, while inactive, that a second automated assistant has failed to satisfy the user's request, to provide the user with an instruction that the first automated assistant is available to satisfy the user's request in response to the determination that the second automated assistant has failed to satisfy the user's request, and to process cached audio data or features of cached audio data that capture the user's oral utterances including the request that the second automated assistant failed to satisfy, in response to the user receiving an instruction from the user that satisfies the request, to determine a response that satisfies the user's request, and to provide the user with a response that satisfies the user's request by the first automated assistant.

[0024] The above description is given as an overview of some implementations of this disclosure. Further descriptions of those and other implementations are provided below in more detail.

[0025] Various implementations can include a non - transient computer - readable storage medium storing instructions executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), and / or a tensor processing unit (TPU)) to perform one or more of the methods described herein. Other implementations can include a client device of an automated assistant (e.g., a client device including an automated assistant interface for interfacing with at least a cloud - based automated assistant component) operable to execute instructions stored to perform one or more of the methods described herein. Still other implementations can include a system of one or more servers including one or more processors operable to execute instructions stored to perform one or more of the methods described herein.

Brief Description of the Drawings

[0026] [Figure 1A] A diagram showing an exemplary process flow that illustrates aspects of the present disclosure by an implementation. [Figure 1B] A diagram showing an exemplary process flow that illustrates aspects of the present disclosure by an implementation. [Figure 2] A block diagram of an exemplary environment that includes various components of FIGS. 1A and 1B and in which implementations disclosed herein may be implemented. [Figure 3] A flowchart showing an exemplary method of detecting and handling failures in other automated assistants. [Figure 4] A flowchart showing an exemplary method of detecting and handling failures in other automated assistants. [Figure 5] A flowchart showing an exemplary method of detecting and handling failures in other automated assistants. [Figure 6]This diagram illustrates an exemplary architecture of a computing device. [Modes for carrying out the invention]

[0027] Figures 1A and 1B illustrate exemplary process flows illustrating various aspects of the present disclosure. The client device 110 includes components shown in Figure 1A and contained within the rectangle in Figure 1A representing the client device 110. The machine learning engine 122A may receive audio data 101 corresponding to oral utterances detected by one or more microphones of the client device 110, and / or other sensor data 102 corresponding to non-speech body movements (e.g., hand gestures and / or movements, body gestures and / or body movements, gaze, facial movements, mouth movements, etc.) detected by one or more non-microphone sensor components of the client device 110. One or more non-microphone sensors may include cameras or other visual sensors, proximity sensors, pressure sensors, accelerometers, magnetometers, and / or other sensors. The machine learning engine 122A uses a machine learning model 152A to process the audio data 101 and / or other sensor data 102 to produce a predicted output 103. As described herein, the machine learning engine 122A may be an alternative engine such as a hotword detection engine 122B, or a voice activity detector (VAD) engine, an endpoint detector engine, and / or other engines.

[0028] In some implementations, when the machine learning engine 122A generates a predicted output 103, the predicted output 103 may optionally be stored in the client device's local on-device storage 111, associated with the corresponding audio data 101 and / or other sensor data 102. In some versions of those implementations, the predicted output may be retrieved by the gradient engine 126 for later use when generating a set of gradients 106 (for example, based on comparing the predicted output with a ground truth output), such as when one or more of the conditions described herein are met. The on-device storage 111 may include, for example, read-only memory (ROM) and / or random-access memory (RAM). In other embodiments, the predicted output 103 may be provided to the gradient engine 126 in real time.

[0029] Based on the determination in block 182 that the predicted output 103 meets a threshold, the client device 110 can use the assistant actuation engine 124 to determine whether to start a currently dormant automated assistant function (e.g., automated assistant 295 in Figure 2), whether to refrain from starting a currently dormant automated assistant function, and / or whether to stop a currently active automated assistant function. The functions of the automated assistant may include speech recognition for generating recognized text, NLU for generating natural language understanding (NLU) output, generating responses based on recognized text and / or NLU output, sending audio data to a remote server, and / or sending recognized text to a remote server. For example, assuming that the predicted output 103 is a probability (e.g., 0.80 or 0.90) and the threshold in block 182 is a probability of the threshold (e.g., 0.85), if the client device 110 in block 182 determines that the predicted output 103 (e.g., 0.90) satisfies the threshold (e.g., 0.85), the assistant actuation engine 124 can start the currently dormant automated assistant function.

[0030] In some implementations, as shown in Figure 1B, the machine learning engine 122A can be the hotword detection engine 122B. In particular, various automated assistant functions such as the on-device speech recognition engine 142, the on-device NLU engine 144, and / or the on-device fulfillment engine 146 are currently dormant (i.e., as indicated by the dashed lines). Furthermore, we assume that using the hotword detection model 152B, the predicted output 103 generated based on the audio data 101 satisfies a threshold in block 182, and the speech activity detector 128 detects user speech directed to the client device 110.

[0031] In some versions of these implementations, the assistant actuation engine 124 activates the on-device speech recognition engine 142, the on-device NLU engine 144, and / or the on-device performance engine 146 as functions of the currently dormant automated assistant. For example, the on-device speech recognition engine 142 may use the on-device speech recognition model 142A to process audio data 101 with respect to oral utterances including the hotword "OK, Assistant" and additional commands and / or phrases following the hotword "OK, Assistant" to generate recognized text 143A; the on-device NLU engine 144 may use the on-device NLU model 144A to process the recognized text 143A to generate NLU data 145A; the on-device performance engine 146 may use the on-device performance model 146A to process the NLU data 145A to generate performance data 147A; and the client device 110 may use the performance data 147A to perform one or more actions 150 in response to the audio data 101.

[0032] In other versions of these implementations, the assistant actuation engine 124 activates only the on-device performance engine 146, without activating the on-device speech recognition engine 142 and the on-device NLU engine 144, to process a variety of commands, such as "no," "stop," "cancel," and / or other commands that can be processed without the on-device speech recognition engine 142 and the on-device NLU engine 144. For example, the on-device performance engine 146 processes the audio data 101 using the on-device performance model 146A to generate performance data 147A, which the client device 110 can then use to perform one or more actions 150 in response to the audio data 101. Furthermore, in these implementation versions, the assistant actuation engine 124 may first activate currently dormant automated functions to confirm that the decision made in block 182 was correct (for example, that the audio data 101 really does contain the hotword "OK, assistant") by first activating only the on-device speech recognition unit 142 to determine that the audio data 101 contains the hotword "OK, assistant", and / or the assistant actuation engine 124 may send the audio data 101 to one or more servers (for example, remote server 160) to confirm that the decision made in block 182 was correct (for example, that the audio data 101 really does contain the hotword "OK, assistant").

[0033] Returning to Figure 1A, if in block 182 the client device 110 determines that the predicted output 103 (e.g., 0.80) cannot meet the threshold (e.g., 0.85), the assistant actuation engine 124 may refrain from starting any currently dormant automated assistant functions and / or suspend all currently active automated assistant functions. Furthermore, if in block 182 the client device 110 determines that the predicted output 103 (e.g., 0.80) cannot meet the threshold (e.g., 0.85), the client device 110 may determine in block 184 whether further user interface input is received. For example, further user interface input could be additional verbal utterances including a hotword, body movements not including additional utterances acting in place of a hotword, activation of an explicit automated assistant call button (e.g., a hardware or software button), a sensed "gripping" of the client device 110 (e.g., gripping the client device 110 with at least a threshold amount of force when calling an automated assistant), and / or other explicit automated assistant calls. If the client device 110 determines in block 184 that there is no further user interface input received, the client device 110 may stop identifying corrections and terminate in block 190.

[0034] However, if the client device 110 determines in block 184 that there is further user interface input received, the system can determine in block 186 whether the further user interface input received in block 184 includes a correction that contradicts the decision made in block 182 (for example, a correction made by or provided by the user). If the client device 110 determines in block 186 that the further user interface input received in block 184 does not include a correction, the client device 110 can stop identifying corrections and terminate in block 190. However, if the client device 110 determines in block 186 that the further user interface input received in block 184 includes a correction that contradicts the initial decision made in block 182, the client device 110 can determine the ground truth output 105.

[0035] In some implementations, the gradient engine 126 can generate a gradient 106 based on a predicted output 103 relative to a ground truth output 105. For example, the gradient engine 126 can generate a gradient 106 based on comparing the predicted output 103 with the ground truth output 105. In some versions of those implementations, the client device 110 stores the predicted output 103 and the corresponding ground truth output 105 in local on-device storage 111, and the gradient engine 126 retrieves the predicted output 103 and the corresponding ground truth output 105 to generate a gradient 106 when one or more conditions are met. One or more conditions may include, for example, that the client device is charging, that the client device is in at least a threshold state of charging, that the temperature of the client device (based on one or more on-device temperature sensors) is below a threshold, and / or that the client device is not held by a user. In other versions of these implementations, the client device 110 provides the predicted output 103 and ground truth output 105 to the gradient engine 126 in real time, and the gradient engine 126 generates the gradient 106 in real time.

[0036] Furthermore, the gradient engine 126 can provide the generated gradient 106 to the on-device machine learning training engine 132A. When the on-device machine learning training engine 132A receives the gradient 106, it uses the gradient 106 to update the on-device machine learning model 152A. For example, the on-device machine learning training engine 132A may utilize backpropagation and / or other techniques to update the on-device machine learning model 152A. It should be noted that in some implementations, the on-device machine learning training engine 132A may utilize batch techniques to update the on-device machine learning model 152A based on the gradient 106 and additional gradients determined locally on the client device 110 based on additional corrections.

[0037] Furthermore, the client device 110 can transmit the generated gradient 106 to the remote system 160. When the remote system 160 receives the gradient 106, the remote training engine 162 of the remote system 160 uses the gradient 106 and additional gradients 107 from additional client devices 170 to update the global weights of the global speech recognition model 152A1. The additional gradients 107 from the additional client devices 170 may be generated based on the same or similar techniques described above in relation to the gradient 106 (but based on locally identified corrections specific to those client devices).

[0038] The update distribution engine 164 may, depending on the satisfaction of one or more conditions, provide the updated global weights and / or the updated global speech recognition model itself to the client device 110 and / or other client devices, as indicated by 108. One or more conditions may include, for example, the duration of the threshold and / or the amount of training since the updated weights and / or updated speech recognition model were last provided. Additional or alternative conditions may include, for example, the elapsed time of measured improvement to the updated speech recognition model and / or the duration of the threshold since the updated weights and / or updated speech recognition model were last provided. When the updated weights are provided to the client device 110, the client device 110 may replace the weights of the on-device machine learning model 152A with the updated weights. When the updated global speech recognition model is provided to the client device 110, the client device 110 may replace the on-device machine learning model 152A with the updated global speech recognition model.

[0039] In some implementations, the on-device machine learning model 152A is transmitted (for example, by a remote system 160 or other component) for storage and use on the client device 110, based on the geographical area and / or other properties of the client device 110 and / or the user of the client device 110. For example, the on-device machine learning model 152A could be one of N available machine learning models for a given language, but trained on corrections specific to a particular geographical area and provided to the client device 110 based on the fact that the client device 110 is primarily located within that particular geographical area.

[0040] Now, turning to Figure 2, we see a client device 110 in an implementation in which the various on-device machine learning engines of Figures 1A and 1B are included as part of one or more automated assistant clients 240 (e.g., a first automated assistant, a second automated assistant, and a meta-assistant) (or communicate with an automated assistant client 240). The respective machine learning models that interface with the various on-device machine learning engines of Figures 1A and 1B are also shown. Other components of Figures 1A and 1B are not shown in Figure 2 for simplicity. Figure 2 illustrates an example of how the various on-device machine learning engines of Figures 1A and 1B and their respective machine learning models may be utilized by the automated assistant clients 240 when performing various actions.

[0041] The client device 110 in Figure 2 is shown with one or more microphones 211, one or more speakers 212, one or more cameras and / or other visual components 213, and a display 214 (e.g., a touch display). The client device 110 may further include pressure sensors, proximity sensors, accelerometers, magnetometers, and / or other sensors used to generate additional sensor data in addition to audio data captured by one or more microphones 211. The client device 110 selectively runs at least an automated assistant client 240. In the example in Figure 2, the automated assistant client 240 includes an on-device hotword detection engine 122B, an on-device speech recognition engine 142, an on-device natural language understanding (NLU) engine 144, and an on-device performance engine 146. The automated assistant client 240 further includes a speech capture engine 242 and a visual capture engine 244. The automated assistant client 240 may include additional and / or alternative engines, such as a voice activity detector (VAD) engine, an endpoint detector engine, and / or other engines. In some implementations, one or more instances of the automated assistant client 240 may omit one or more of the elements shown in Figure 2.

[0042] One or more cloud-based automated assistant components 280 may optionally be implemented in one or more computing systems (collectively referred to as “cloud” computing systems) that are coupled together to communicate with client devices 110 via one or more local area and / or wide area networks (e.g., the Internet) as shown 290 as a whole. One or more cloud-based automated assistant components 280 may be implemented, for example, by a cluster of high-performance servers.

[0043] In various implementations, an instance of the automated assistant client 240 may, through its interaction with one or more cloud-based automated assistant components 280, form what appears to be a logical instance of the automated assistant 295 from the user's perspective, in which the user may engage in person-to-computer interactions (e.g., verbal interactions, gesture-based interactions, and / or touch-based interactions).

[0044] The client device 110 can be, for example, a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device in the user's vehicle (e.g., an in-car communication system, an in-car entertainment system, an in-car navigation system), a standalone interactive speaker, a smart home appliance such as a smart TV (or a regular TV with a network-connected dongle that has automated assistant capabilities), and / or a user's wearable device that includes a computing device (e.g., a user's wristwatch with a computing device, a user's glasses with a computing device, a virtual or augmented reality computing device). Additional and / or alternative client devices may be provided.

[0045] One or more visual components 213 can take various forms, such as a monographic camera, a stereographic camera, a LiDAR component (or other laser-based component), or a radar component. One or more visual components 213 may be used by a vision capture engine 244 to capture a vision frame (e.g., an image frame, a laser-based vision frame) of the environment in which the client device 110 is deployed. In some implementations, such vision frames may be used to determine whether a user is near the client device 110 and / or the distance of the user (e.g., the user's face) to the client device. Such determinations may be used, for example, to determine whether to activate the various on-device machine learning engines and / or other engines shown in Figure 2.

[0046] The voice capture engine 242 may be configured to capture user speech and / or other audio data captured by the microphone 211. Furthermore, the client device 110 may include pressure sensors, proximity sensors, accelerometers, magnetometers, and / or other sensors used to generate other sensor data in addition to the audio data captured by the microphone 211. As described herein, such audio data and other sensor data may be utilized by the hotword detection engine 122B and / or other engines to determine whether to start the functions of one or more currently dormant automated assistants, whether to refrain from starting the functions of one or more currently dormant automated assistants, and / or whether to stop the functions of one or more currently active automated assistants. The functions of the automated assistants may include an on-device speech recognition engine 142, an on-device NLU engine 144, an on-device performance engine 146, and additional and / or alternative engines. For example, the on-device speech recognition unit 142 can use the on-device speech recognition model 142A to process audio data capturing oral utterances and generate recognized text 143A corresponding to the oral utterances. The on-device NLU engine 144 can optionally use the on-device NLU model 144A to perform on-device natural language understanding on the recognized text 143A and generate NLU data 145A. The NLU data 145A may include, for example, the intent corresponding to the oral utterances and optionally parameters related to the intent (e.g., slot values). Furthermore, the on-device performance engine 146 can optionally use the on-device performance model 146A to generate performance data 147A based on the NLU data 145A.This performance data 147A may define local and / or remote responses (e.g., answers) to a verbal utterance, interactions with locally installed applications to be performed based on the verbal utterance, commands to be sent (directly or via a corresponding remote system) to an Internet of Things (IoT) device based on the verbal utterance, and / or other resolution actions to be performed based on the verbal utterance. The performance data 147A is then provided for the local and / or remote execution of the determined actions to resolve the verbal utterance. Execution may include, for example, rendering local and / or remote responses (e.g., rendering them visually and / or audibly (optionally using a local text-to-speech module)), interacting with locally installed applications, sending commands to IoT devices, and / or other actions.

[0047] The display 214 may be used to display recognized text 143A and / or further recognized text 143B from the on-device speech recognition unit 122, and / or one or more results from the execution unit 150. The display 214 may also be one of the user interface output components on which the visual portion of the response from the automated assistant client 240 is rendered.

[0048] In some implementations, the cloud-based automated assistant component 280 may include a remote ASR engine 281 that performs speech recognition, a remote NLU engine 282 that performs natural language understanding, and / or a remote execution engine 283 that generates executions. A remote execution module that performs remote executions based on locally or remotely determined execution data may also be optionally included. Additional and / or alternative remote engines may be included. As described herein, in various implementations, on-device speech processing, on-device NLU, on-device execution, and / or on-device execution may be preferred, at least for the reasons of the reduced latency and / or network usage they provide when resolving spoken utterances (by eliminating the need for client-server round trips to resolve spoken utterances). However, one or more cloud-based automated assistant components 280 may be used at least selectively. For example, such components may be used in parallel with on-device components, and the output from such components may be used when local components fail. For example, the on-device performance engine 146 may fail in certain circumstances (for example, due to the relatively limited resources of the client device 110), and the remote performance engine 283 can leverage the more robust resources of the cloud to generate performance data in such circumstances. The remote performance engine 283 can run in parallel with the on-device performance engine 146, and the results can be used when the on-device performance fails, or the remote performance engine 283 can be invoked in response to a failure of the on-device performance engine 146.

[0049] In various implementations, an NLU engine (on-device and / or remote) can generate NLU data containing one or more annotations of the recognized text and one or more (e.g., all) of the words in the natural language input. In some implementations, the NLU engine is configured to identify and annotate various types of grammatical information in the natural language input. For example, the NLU engine may include a morphological module that divides individual words into morphemes and / or annotates the morphemes by, for example, the class of those morphemes. The NLU engine may also include part-of-speech taggers configured to annotate words by their grammatical roles. Additionally, in some implementations, the NLU engine may include a dependency parser configured to determine syntactic relationships between words in the natural language input, additionally and / or alternatively.

[0050] In some implementations, the NLU engine may additionally and / or alternatively include entity taggers configured to annotate references to entities within one or more segments, such as references to people (including, for example, literary characters, celebrities, and famous people), organizations, and places (real and fictional). In some implementations, the NLU engine may additionally and / or alternatively include coreference resolvers (not shown) configured to group or "cluster" references to the same entity based on contextual cues from one or more situations. In some implementations, one or more components of the NLU engine may rely on annotations from one or more other components of the NLU engine.

[0051] The NLU engine may further include an intent matcher configured to determine the intent of a user engaging in interaction with the automated assistant 295. The intent matcher may employ various techniques to determine the user's intent. In some implementations, the intent matcher may have access to one or more local and / or remote data structures containing multiple mappings between, for example, grammars and response intents. For example, grammars included in the mappings may be selected and / or learned over time and may represent common user intents. For example, one grammar "play <artist>"but, <artist>This may map to an intent to invoke a response action that plays music on client device 110. Another grammar, "[weather | forecast] today," may be matchable to user inquiries such as "what's the weather today" and "what's the forecast for today?". In addition to or instead of grammars, in some implementations, intent matchers may use one or more trained machine learning models alone or in combination with one or more grammars. These trained machine learning models may be trained to identify intents by embedding recognized text from oral utterances into a reduced-dimensional space, and then determining which other embeddings (and therefore intents) are closest using techniques such as Euclidean distance or cosine similarity. <artist>As seen in the example grammar of ", some grammars have slots that can be filled with slot values ​​(or "parameters") (for example, <artist>) has. Slot values ​​can be determined in various ways. Often, the user provides the slot value in advance. For example, the grammar "Order me a <topping>Regarding "pizza," there is a high probability that the user will say the phrase "order me a sausage pizza," in which case the slot <topping>These are filled in automatically. Other slot values ​​may be inferred based on, for example, the user's location, the content currently being rendered, the user's preferences, and / or other clues.

[0052] An execution engine (local and / or remote) may be configured to receive predicted / estimated intents and any associated slot values ​​output by the NLU engine and to perform (or "resolve") the intent. In various implementations, the performance (or "resolution") of a user's intent may involve causing various performance information (also called performance data) to be generated / obtained by the execution engine, for example. This may include determining local and / or remote responses (e.g., answers) to the verbal utterance, interactions with locally installed applications to be executed based on the verbal utterance, commands to be sent (directly or via a corresponding remote system) to an Internet of Things (IoT) device based on the verbal utterance, and / or other resolution actions to be performed based on the verbal utterance. On-device performance may then initiate the local and / or remote execution of the determined actions to resolve the verbal utterance.

[0053] Figure 3 shows a flowchart illustrating an exemplary method 300 for detecting and handling failures in other automated assistants. For convenience, the operation of method 300 is described in relation to the system that performs the operation. This system of method 300 includes one or more processors and / or other components of a client device. Furthermore, the operation of method 300 is shown in a specific order, but this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0054] In block 310, the system runs a first automated assistant in an inactive state, at least partially, on a computing device operated by a user (e.g., a client device).

[0055] In block 320, while in an inactive state, the first automated assistant determines whether the second automated assistant has failed to fulfill the user's request. In some implementations, the failure of the second automated assistant to fulfill the user's request, as detected in block 320, may include a response from the second automated assistant indicating that the user's request cannot be fulfilled (e.g., "Sorry," "That's not possible," etc.). In other implementations, the failure detected in block 320 may include a response from the first automated assistant determining that the result provided by the second automated assistant in response to the user's request is suboptimal, inaccurate, or incomplete. In some implementations, the second automated assistant may be running on the client device in parallel with the first automated assistant. In other implementations, the second automated assistant may be running on another client device located near the client device on which the first automated assistant is running (e.g., in the same room).

[0056] In the iteration of block 320, if the first automated assistant determines that the second automated assistant did not fail to fulfill the user's request, the system proceeds to block 330 and the flow ends. On the other hand, in the iteration of block 320, if the first automated assistant determines that the second automated assistant failed to fulfill the user's request, the system proceeds to block 340.

[0057] Continuing with block 320, in some implementations, when determining whether the second automated assistant failed to fulfill the user's request, the first automated assistant receives audio data capturing the initial response and then uses speaker identification on the audio data capturing the initial response to determine whether the initial response was provided by the second automated assistant (for example, spoken in a voice known to be associated with the second automated assistant). If the first automated assistant determines that the initial response was not provided by the second automated assistant, the system proceeds to block 330 and the flow ends. On the other hand, if the first automated assistant determines that the initial response was provided by the second automated assistant, the first automated assistant determines whether the initial response indicates that the second automated assistant failed to fulfill the user's request.

[0058] In some implementations, the first automated assistant processes the audio data capturing the initial response using a hotword detection model to determine whether the initial response satisfies the user's request (for example, by detecting failure hotwords such as "sorry" or "that's not possible"). In other implementations, the first automated assistant uses automated speech recognition to process the audio data capturing the initial response to generate text, and then uses natural language processing techniques to process the text to determine whether the initial response satisfies the user's request. In some implementations, natural language processing techniques are used to identify suboptimal, inaccurate, or incomplete results, and based on the identification of suboptimal, inaccurate, or incomplete results, the first automated assistant may determine that the initial response fails to satisfy the user's request.

[0059] In block 340, in response to a determination that the second automated assistant has failed to satisfy the user's request, the first automated assistant processes cached audio data capturing the user's spoken utterance, including the request that the second automated assistant failed to satisfy, and / or processes features derived from the audio data (e.g., ASR transcript) to determine a response that satisfies the user's request. In some implementations, if the first automated assistant is unable to determine a response that satisfies the user's request in block 340, the system proceeds to block 330 and the flow terminates. In some implementations, the cached audio data further captures the initial response provided to the user by the second automated assistant.

[0060] In some implementations, a second automated assistant runs on the same computing device as the first automated assistant, and cached audio data and / or features derived from the audio data (e.g., ASR transcription) are received by the first automated assistant via a meta-assistant running on the computing device. In other implementations, the second automated assistant runs on a different computing device, and cached audio data is received by the first automated assistant via one or more microphones on the computing device. In other implementations, the second automated assistant runs on a different computing device, and the first automated assistant receives cached audio data and / or features derived from the audio data (e.g., ASR transcription) via an application programming interface (API).

[0061] In block 350, the first automated assistant provides the user with a response that satisfies the user's request (determined in block 340). In some implementations, the first automated assistant causes the response that satisfies the user's request to be provided on the computing device on which the first automated assistant is running (for example, by speaking through a speaker or displaying the response on the computing device's display). In other implementations, the first automated assistant causes the response that satisfies the user's request to be provided on another computing device (for example, by speaking through a speaker or display).

[0062] Figure 4 shows a flowchart illustrating an exemplary method 400 for detecting and handling failures in other automated assistants. For convenience, the operation of method 400 is described in relation to the system that performs the operation. This system of method 400 includes one or more processors and / or other components of a client device. Furthermore, the operation of method 400 is shown in a specific order, but this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0063] In block 410, the system runs a first automated assistant in an inactive state, at least partially, on a computing device operated by a user (e.g., a client device).

[0064] In block 420, while in an inactive state, the first automated assistant determines whether the second automated assistant has failed to fulfill the user's request. In some implementations, the failure of the second automated assistant to fulfill the user's request, as detected in block 420, may include a response from the second automated assistant indicating that the user's request cannot be fulfilled (e.g., "Sorry," "That's not possible," etc.). In other implementations, the failure detected in block 420 may include a response from the first automated assistant determining that the result provided by the second automated assistant in response to the user's request is suboptimal, inaccurate, or incomplete. In some implementations, the second automated assistant may be running on the client device in parallel with the first automated assistant. In other implementations, the second automated assistant may be running on another client device located near the client device on which the first automated assistant is running (e.g., in the same room).

[0065] If, in the iteration of block 420, the first automated assistant determines that the second automated assistant did not fail to fulfill the user's request, the system proceeds to block 430 and the flow ends. On the other hand, if, in the iteration of block 420, the first automated assistant determines that the second automated assistant failed to fulfill the user's request, the system proceeds to block 440.

[0066] Continuing with block 420, in some implementations, when determining whether the second automated assistant has failed to fulfill the user's request, the first automated assistant receives audio data capturing the initial response and then uses speaker identification on the audio data capturing the initial response to determine whether the initial response was provided by the second automated assistant. If the first automated assistant determines that the initial response was not provided by the second automated assistant, the system proceeds to block 430 and the flow ends. On the other hand, if the first automated assistant determines that the initial response was provided by the second automated assistant, the first automated assistant determines whether the initial response indicates that the second automated assistant has failed to fulfill the user's request.

[0067] In some implementations, the first automated assistant processes the audio data capturing the initial response using a hotword detection model to determine whether the initial response satisfies the user's request (for example, by detecting failure hotwords such as "sorry" or "that's not possible"). In other implementations, the first automated assistant uses automatic speech recognition to process the audio data capturing the initial response and generate text, and then uses natural language processing techniques to process the text and determine whether the initial response satisfies the user's request.

[0068] In block 440, in response to a determination that the second automated assistant has failed to satisfy the user's request, the first automated assistant processes cached audio data capturing the user's spoken utterance, including the request that the second automated assistant failed to satisfy, and / or processes features derived from the audio data (e.g., ASR transcript) to determine a response that satisfies the user's request. In some implementations, if the first automated assistant is unable to determine a response that satisfies the user's request in block 440, the system proceeds to block 430 and the flow terminates. In some implementations, the cached audio data further captures the initial response provided to the user by the second automated assistant.

[0069] In some implementations, a second automated assistant runs on the same computing device as the first automated assistant, and cached audio data and / or features derived from the audio data (e.g., ASR transcription) are received by the first automated assistant via a meta-assistant running on the computing device. In other implementations, the second automated assistant runs on a different computing device, and cached audio data is received by the first automated assistant via one or more microphones on the computing device. In other implementations, the second automated assistant runs on a different computing device, and the first automated assistant receives cached audio data and / or features derived from the audio data (e.g., ASR transcription) via an application programming interface (API).

[0070] In block 450, the first automated assistant provides the user with an indication of the availability of a response that will satisfy the user's request. In some implementations, the availability indication is a visual indication (e.g., an indication on a display or light) and / or an audio indication (e.g., a chime) provided by the computing device on which the first automated assistant is running.

[0071] In block 460, the first automated assistant determines whether a response request that satisfies the user's request is received (for example, from the user). If, in the iteration of block 460, the first automated assistant determines that no response request that satisfies the user's request was received, the system proceeds to block 430 and the flow ends. On the other hand, if, in the iteration of block 460, the first automated assistant determines that a response request that satisfies the user's request was received, the system proceeds to block 470.

[0072] In block 470, upon receiving a request for a response that satisfies the user's request, the first automated assistant provides a response that satisfies the user's request (as determined in block 440). In some implementations, the first automated assistant causes the response that satisfies the user's request to be delivered on the computing device on which the first automated assistant is running (for example, by using a speaker or displaying the response on the computing device's display). In other implementations, the first automated assistant causes the response that satisfies the user's request to be delivered on another computing device (for example, via a speaker or display).

[0073] Figure 5 shows a flowchart illustrating an exemplary method 500 for detecting and handling failures in other automated assistants. For convenience, the operation of method 500 is described in relation to the system that performs the operation. This system of method 500 includes one or more processors and / or other components of a client device. Furthermore, the operation of method 500 is shown in a specific order, but this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0074] In block 510, the system runs a first automated assistant in an inactive state, at least partially, on a computing device operated by a user (e.g., a client device).

[0075] In block 520, while in an inactive state, the first automated assistant determines whether the second automated assistant has failed to fulfill the user's request. In some implementations, the failure of the second automated assistant to fulfill the user's request, as detected in block 520, may include a response from the second automated assistant indicating that the user's request cannot be fulfilled (e.g., "Sorry," "That's not possible," etc.). In other implementations, the failure detected in block 520 may include a response from the first automated assistant determining that the result provided by the second automated assistant in response to the user's request is suboptimal, inaccurate, or incomplete. In some implementations, the second automated assistant may be running on the client device in parallel with the first automated assistant. In other implementations, the second automated assistant may be running on another client device located near the client device on which the first automated assistant is running (e.g., in the same room).

[0076] If, in the iteration of block 520, the first automated assistant determines that the second automated assistant did not fail to fulfill the user's request, the system proceeds to block 530 and the flow ends. On the other hand, if, in the iteration of block 520, the first automated assistant determines that the second automated assistant failed to fulfill the user's request, the system proceeds to block 540.

[0077] Continuing with block 520, in some implementations, when determining whether the second automated assistant failed to fulfill the user's request, the first automated assistant receives audio data capturing the initial response and then uses speaker identification on the audio data capturing the initial response to determine whether the initial response was provided by the second automated assistant. If the first automated assistant determines that the initial response was not provided by the second automated assistant, the system proceeds to block 530 and the flow ends. On the other hand, if the first automated assistant determines that the initial response was provided by the second automated assistant, the first automated assistant determines whether the initial response indicates that the second automated assistant failed to fulfill the user's request.

[0078] In some implementations, the first automated assistant processes the audio data capturing the initial response using a hotword detection model to determine whether the initial response satisfies the user's request (for example, by detecting failure hotwords such as "sorry" or "that's not possible"). In other implementations, the first automated assistant uses automatic speech recognition to process the audio data capturing the initial response and generate text, and then uses natural language processing techniques to process the text and determine whether the initial response satisfies the user's request.

[0079] In block 540, in response to a determination that the second automated assistant has failed to fulfill the user's request, the first automated assistant provides the user with an instruction that the first automated assistant is available to fulfill the user's request. In some implementations, the instruction is a visual instruction (e.g., an instruction on a display or light) and / or an audio instruction (e.g., a chime) provided by the computing device on which the first automated assistant is running.

[0080] In block 550, the first automated assistant determines whether a command satisfying the request is received from the user. If, in an iteration of block 550, the first automated assistant determines that no command satisfying the request was received from the user, the system proceeds to block 530 and the flow ends. On the other hand, if, in an iteration of block 550, the first automated assistant determines that a command satisfying the request was received from the user, the system proceeds to block 560.

[0081] In block 560, upon receiving a request-satisfying instruction from the user, the first automated assistant processes cached audio data capturing the user's spoken utterance, including the request that the second automated assistant failed to satisfy, and / or processes features derived from the audio data (e.g., ASR transcript) to determine a response that satisfies the user's request. In some implementations, the processed cached audio data further captures the initial response provided to the user by the second automated assistant.

[0082] In some implementations, a second automated assistant runs on the same computing device as the first automated assistant, and cached audio data and / or features derived from the audio data (e.g., ASR transcription) are received by the first automated assistant via a meta-assistant running on the computing device. In other implementations, the second automated assistant runs on a different computing device, and cached audio data is received by the first automated assistant via one or more microphones on the computing device. In other implementations, the second automated assistant runs on a different computing device, and the first automated assistant receives cached audio data and / or features derived from the audio data (e.g., ASR transcription) via an application programming interface (API).

[0083] In block 570, the first automated assistant provides the user with a response that satisfies the user's request (determined in block 560). In some implementations, the first automated assistant causes the response that satisfies the user's request to be provided on the computing device on which the first automated assistant is running (for example, by speaking through a speaker or displaying the response on the computing device's display). In other implementations, the first automated assistant causes the response that satisfies the user's request to be provided on another computing device (for example, by speaking through a speaker or display).

[0084] Figure 6 is a block diagram of an exemplary computing device 610 that may be optionally used to perform one or more embodiments of the technology described herein. In some implementations, one or more of the client device, cloud-based automated assistant components, and / or other components may include one or more components of the exemplary computing device 610.

[0085] Generally, the computing device 610 includes at least one processor 614 that communicates with several peripheral devices via a bus subsystem 612. These peripheral devices may include, for example, a storage subsystem 624 including a memory subsystem 625 and a file storage subsystem 626, a user interface output device 620, a user interface input device 622, and a network interface subsystem 616. The input and output devices enable user interaction with the computing device 610. The network interface subsystem 616 provides an interface to an external network and is coupled to corresponding interface devices of other computing devices.

[0086] The user interface input device 622 may include pointing devices such as keyboards, mice, trackballs, touchpads, or graphics tablets, audio input devices such as scanners, touchscreens integrated into displays, speech recognition systems, microphones, and / or other types of input devices. In general, the use of the term “input device” is intended to include all possible types of devices and methods for inputting information into the computing device 610 or a communication network.

[0087] The user interface output device 620 may include non-visual displays such as a display subsystem, printer, fax machine, or audio output device. The display subsystem may include flat panel devices such as cathode ray tubes (CRTs) or liquid crystal displays (LCDs), projection devices, or any other mechanism for generating visible images. The display subsystem may also provide non-visual displays, such as through an audio output device. In general, the use of the term “output device” is intended to include all possible types of devices and methods for outputting information from the computing device 610 to a user or another machine or computing device.

[0088] The storage subsystem 624 stores programming and data structures that provide some or all of the functionality of the modules described herein. For example, the storage subsystem 624 may include logic for performing selected embodiments of the methods disclosed herein and for implementing the various components shown in Figures 1A and 1B.

[0089] These software modules are generally executed by processor 614 alone or in combination with other processors. The memory subsystem 625 included in the storage subsystem 624 may include several memories, including primary random access memory (RAM) 630 for storing program instructions and data, and read-only memory (ROM) 632 for storing fixed instructions. The file storage subsystem 626 can provide persistent storage for program and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules implementing the functionality of a particular implementation may be stored by the file storage subsystem 626 within the storage subsystem 624, or on other machines that can be accessed by processor 614.

[0090] The bus subsystem 612 provides a mechanism for various components and subsystems of the computing device 610 to communicate with each other as intended. Although the bus subsystem 612 is schematically shown as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0091] The computing device 610 can be of various types, including workstations, servers, computing clusters, blade servers, server farms, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of the computing device 610 shown in Figure 6 is intended only as a specific example to illustrate several implementations. Many other configurations of the computing device 610 are possible, having more or fewer components than the computing device shown in Figure 6.

[0092] Where the systems described herein may collect or otherwise monitor personal information about a user, or use personal and / or monitored information, the user may be given the opportunity to control whether the program or feature collects user information (for example, information about the user's social networks, social behavior or activities, occupation, user preferences, or the user's current geographical location), or whether and / or how content that may be more relevant to the user should be received from the content server. Furthermore, certain data may be processed in one or more ways before being stored or used, such that personally identifiable information is removed. For example, a user's identity may be processed such that personally identifiable information cannot be determined about the user, or if geographical location information is obtained, the user's geographical location may be generalized (to the level of city, zip code, or state), and therefore the user's specific geographical location cannot be determined. Thus, the user may be able to control how and / or how information is collected about them.

[0093] While several implementations are described and illustrated herein, various other means and / or structures may be utilized to perform the functions described herein and / or to obtain one or more of the results and / or benefits, and each of such changes and / or modifications shall be considered within the scope of the implementations described herein. More broadly, all parameters, dimensions, materials and configurations described herein are intended to be illustrative, and actual parameters, dimensions, materials and / or configurations shall depend on the particular one or more applications in which the teaching is used. A person skilled in the art can recognize or determine many equivalents of the particular implementations described herein by means of ordinary experimentation alone. Thus, it should be understood that the implementations described herein are presented merely as examples, and within the scope of the appended claims and their equivalents, implementations may be carried out in ways different from those specifically described and claimed. Implementations of this disclosure shall apply to each individual feature, system, item, material, kit and / or method described herein. Furthermore, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included in the scope of this disclosure, provided that such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent. [Explanation of Symbols]

[0094] 101 Audio Data 102 Other sensor data 103 Predicted output 105 Ground Truth Output 106 gradient 107 Additional gradient 110 Client Devices 111 On-device storage 122A Machine Learning Engine 122B Hotword Detection Engine 124 Assistant Engine 126 Gradient Engine 128 Speech Activity Detector 132A On-device machine learning training engine 142 On-device voice recognition 142A On-device speech recognition model 143A Recognized text 143B Further recognized text 144 On-device NLU engine 144A On-device NLU model 145A NLU data 146 On-device performance engine 146A On-device performance model 147A Performance Data 150 executions 152A Machine Learning Model 152A1 Global speech recognition model 152B Hotword Detection Model 160 Remote servers, remote systems 162 Remote Training Engine 164 Update distribution engine 170 additional client devices 211 Microphone 212 speakers 213 Visual Components 214 displays 240 Automated Assistant Clients 242 Voice Capture Engine 244 Visual Capture Engine 280 Cloud-based automated assistant components 281 Remote ASR engine 282 Remote NLU engine 283 Remote Performance Engine 295 Automated Assistants 300 ways 400 ways 500 ways 610 Computing Devices 612 Bus Subsystem 614 Processors 616 Network Interface Subsystem 620 User Interface Output Devices 622 User Interface Input Devices 624 Storage Subsystems 625 Memory subsystem 626 File Storage Subsystem 630 RAM 632 ROM< / topping> < / topping> < / artist> < / artist> < / artist> < / artist>

Claims

1. A method carried out by one or more processors, A first automated assistant, which is at least partially run on a computing device operated by a user, determines that a second automated assistant has provided an inaccurate result in response to a user's request directed to the second automated assistant; A step in which, upon determination that the second automated assistant has provided an inaccurate result in response to a user's request directed to the second automated assistant, the first automated assistant automatically determines a response that satisfies the user's request, the step of the first automated assistant automatically determining a response that satisfies the user's request includes a step in which the first automated assistant automatically processes cached audio data that captures the user's oral utterance including the user's request, or features of the cached audio data, the cached audio data being received by the first automated assistant via a meta-assistant running on the computing device, The first automated assistant provides the user with the response that satisfies the user's request, and A method that includes [a certain feature].

2. The method according to claim 1, wherein the cached audio data captures the first response of the second automated assistant to the request, and the first response is provided to the user by the second automated assistant.

3. The step of determining that the second automated assistant has provided the inaccurate result in response to the user's request is: The steps include receiving audio data that captures the first response of the second automated assistant to the request, A step of determining that the first response was provided by the second automated assistant, using speaker identification on the audio data capturing the first response. The method according to claim 1, including the method described in claim 1.

4. The method according to claim 3, wherein the step of determining that the second automated assistant has provided the inaccurate result in response to the user's request further includes the step of processing the audio data capturing the initial response using a hotword detection model to determine that the initial response is the inaccurate result.

5. The step of determining that the second automated assistant has provided the inaccurate result in response to the user's request is: The steps include: using automatic speech recognition to process the audio data capturing the initial response and generate text; The method according to claim 3, further comprising the step of processing the text using natural language processing techniques to determine that the initial response is the inaccurate result.

6. The method according to claim 1, wherein the first automated assistant causes the response that satisfies the user's request to be provided on another computing device.

7. The method according to claim 1, wherein the first automated assistant causes the response that satisfies the user's request to be displayed on the display of the computing device.

8. One or more computer-readable storage media having program instructions stored together in one or more computer-readable storage media, wherein the program instructions are A first automated assistant, at least partially running on a computing device operated by a user, determines that a second automated assistant has provided an inaccurate result in response to a user request directed to the second automated assistant; In response to a determination that the second automated assistant has provided an inaccurate result in response to a user's request directed to the second automated assistant, the first automated assistant automatically determines a response that satisfies the user's request, and the first automated assistant automatically determines a response that satisfies the user's request, which includes the first automated assistant automatically processing cached audio data capturing the user's oral utterance including the user's request, or features of the cached audio data, the cached audio data being received by the first automated assistant via a meta-assistant running on the computing device. The first automated assistant provides the user with availability instructions that meet the user's requirements. One or more computer-readable storage media capable of performing the following actions.

9. One or more computer-readable storage media according to claim 8, wherein the cached audio data captures the first response of the second automated assistant to the request, and the first response is provided to the user by the second automated assistant.

10. The determination that the second automated assistant provided the inaccurate result in response to the user's request is Receiving audio data that captures the first response of the second automated assistant to the aforementioned request, Using speaker identification on the audio data capturing the initial response, it is determined that the initial response was provided by the second automated assistant. Using a hotword detection model, process the audio data capturing the initial response to determine that the initial response is the inaccurate result. One or more computer-readable storage media according to claim 8, including the following:

11. One or more computer-readable storage media according to claim 8, wherein the indication of availability is a visual indication provided by the computing device.

12. The aforementioned program instruction, The first automated assistant receives a request for the response that satisfies the user's request, One or more computer-readable storage media according to claim 11, wherein, in response to receiving the request for the response that satisfies the user's request, the first automated assistant is further capable of providing the user with the response that satisfies the user's request.

13. A system comprising a processor, computer-readable memory, one or more computer-readable storage media, and program instructions stored together in the one or more computer-readable storage media, wherein the program instructions are A first automated assistant, at least partially running on a computing device operated by a user, determines that a second automated assistant has provided an inaccurate result in response to a user request directed to the second automated assistant; In response to a determination that the second automated assistant has provided an inaccurate result in response to a user's request directed to the second automated assistant, the first automated assistant is provided to the user to indicate that it is available to provide a response that satisfies the user's request. In response to receiving instructions from the user to satisfy the request, the first automated assistant automatically determines a response that satisfies the user's request, and the first automated assistant's automatic determination of a response that satisfies the user's request includes the first automated assistant automatically processing cached audio data that captures the user's oral utterance including the user's request, or features of the cached audio data, the cached audio data being received by the first automated assistant via a meta-assistant running on the computing device. The first automated assistant provides the user with a response that satisfies the user's request. A system that is executable to perform the following actions.

Images