Detect close matches of hot words or phrases

Abstract

This document describes techniques for identifying failed hotword attempts. A method includes receiving first audio data, processing the first audio data to generate a first prediction output, determining that the first prediction output satisfies a secondary threshold but does not satisfy a primary threshold, receiving second audio data, processing the second audio data to generate a second prediction output, determining that the second prediction output satisfies the secondary threshold but does not satisfy the primary threshold, identifying a failed hotword attempt in response to the first prediction output and the second prediction output satisfying the secondary threshold but not the primary threshold, and in response to the first spoken utterance and the second spoken utterance satisfying one or more timing criteria relative to each other, and providing a prompt responsive to the failed hotword attempt in response to identifying the failed hotword attempt.

Description

Background Art

[0001] Humans can participate in human-computer conversations, where interactive software applications are referred to herein as "automated assistants" (also referred to as "digital agents," "interactive personal assistants," "intelligent personal assistants," "assistant applications," "conversational agents," etc.). For example, humans (who may be referred to as "users" when interacting with automated assistants) can provide commands and / or requests to the automated assistant using spoken natural language input (i.e., utterances), which in some cases can be converted to text and then processed by providing textual (e.g., typed) natural language input, and / or through touch and / or non-verbal physical movement (e.g., gestures, eye gaze, facial movement, etc.). The automated assistant responds to the request by providing responsive user interface output (e.g., audible and / or visual user interface output), controlling one or more smart devices, and / or controlling one or more functions of the device implementing the automated assistant (e.g., controlling other applications of the device).

[0002] As mentioned above, many automated assistants are configured to interact via spoken utterances. To protect user privacy and / or conserve resources, the automated assistant avoids executing one or more automated assistant functions based on all spoken utterances present in audio data detected (at least in part) via a microphone of a client device implementing the automated assistant. Instead, some processing based on the spoken utterances occurs only in response to a determination that certain conditions exist.

[0003] For example, many client devices that include automated assistants and / or interface with automated assistants include a hotword detection model. When the microphone of such a client device is not disabled, the client device can use the hotword detection model to continuously process audio data detected via the microphone to generate a predictive output indicating whether one or more hotwords (including multi-word phrases), such as "Hey Assistant," "OK Assistant," and / or "Assistant." When the predictive output indicates the presence of a hotword, any audio data that follows within a threshold amount of time (and is optionally determined to include voice activity) can be processed by one or more on-device and / or remote automated assistant components, such as a speech recognition component, a voice activity detection component, etc. In addition, a natural language understanding engine can be used to process the recognized text (from the speech recognition component), and / or actions can be performed based on the natural language understanding engine output. These actions can include, for example, generating and providing responses and / or controlling one or more applications and / or smart devices. Other hot words (e.g., "No," "Stop," "Cancel," "Volume up," "Volume down," "Next track," "Previous track," etc.) can be mapped to various commands, and when the prediction output indicates the presence of one of these hot words, the client device can process the mapped command. However, when the prediction output indicates the absence of the hot word, the corresponding audio data will be discarded without any further processing, thereby saving resources and user privacy.

[0004] The other machine learning models mentioned above and / or whose predicted output indicates whether the automated assistant function is activated (e.g., the additional machine learning models described below) perform well in many cases. However, in some cases, the automated assistant may not recognize the request and / or the response to the request. For example, if the user does not know, does not remember, or misremembers the hotword, the user's utterance may constitute a failed hotword attempt, and the automated assistant may not be able to recognize or respond to the failed hotword attempt.

[0005] In some cases, the automated assistant may recognize different sets of hotwords in different contexts (e.g., time of day) or based on the application running (e.g., foreground application). For example, if a music application is currently playing music, the automated assistant may recognize additional hotwords such as "pause music", "volume up", and "volume down". However, different sets of hotwords available at different times (e.g., "good morning" may only be active in the morning) may be confusing to the user. In cases where the user does not know or remember which hotwords are available in a particular context or misremembers the hotword, the user's utterance may constitute a failed hotword attempt, and the automated assistant may fail to recognize or respond to the failed hotword attempt. For example, the user may say "volume increase" or "loudness up" instead of "volume up", and the automated assistant may not recognize and respond to the request.

[0006] In cases where the automated assistant does not recognize and respond to a request because the user does not know, remember, or misremember a hotword, the automated assistant may not provide feedback to the user. As a result, it may be difficult for the user to determine whether the automated assistant failed to recognize the user's utterance or whether the user's utterance included an unsupported word or phrase (e.g., a word or phrase that is not a hotword, potentially because the user does not know, remember, or misremember a supported hotword). In cases where the user's utterance includes an unsupported word or phrase, the user may mistakenly believe that the automated assistant failed to recognize the user's utterance, and therefore the user may repeat the same unsupported word or phrase in one or more subsequent utterances, typically in short succession. However, in such cases, the automated assistant may also fail to respond to the subsequent utterances because they also fail to include a supported hotword and merely include repetitions of the same unsupported word or phrase.

[0007] The occurrence of failed hotword attempts can prolong the human / automated assistant interaction, forcing the human to repeat the utterance (and / or perform other actions) originally intended to activate the automated assistant function. In addition, the occurrence of failed hotword attempts can waste network and / or computing resources. Summary of the Invention

[0008] Some embodiments disclosed herein are directed to improving the performance of machine learning models by automatically identifying failed hotword attempts. As described in more detail herein, such machine learning models may include, for example, hotword detection models and / or other machine learning models. Various embodiments detect situations where a user is attempting to trigger a hotword but fails (i.e., a failed hotword attempt). In response to detecting such a situation, the system may provide the user with a prompt about available hotwords. In other embodiments, the system may adapt by adding user-preferred words or phrases to the available hotwords.

[0009] In some embodiments, the system can detect close matches to hot words. For example, an automated assistant may support a specific set of media playback commands, including "volume up", "volume down", "next track", "previous track". If a user misremembers one of these commands, the user might say "previous song" instead of "previous track". Conventional systems may not provide feedback about what is wrong or what the correct phrase should be. Various embodiments can address this vocabulary problem by providing the user with hints about one or more supported hot words, in which it may not be clear which phrases the user can say at any given point in time. In addition, embodiments can provide users with a mechanism to train custom hot words.

[0010] In some embodiments, the system may detect a series of weak hotword model triggers. For example, in the case where the user says a supported hotword, the hotword detection model may generate a prediction output indicating a high confidence detection of the hotword. In the case where the user says a variant or close match of the hotword (e.g., a word or phrase that is acoustically similar to the hotword but is not a supported hotword), the hotword detection model may generate a prediction output indicating a medium confidence detection of the hotword (e.g., higher than the baseline level but lower than the high confidence prediction output when the user says a supported hotword). In this case, the medium confidence detection may not be high enough for the system to determine that the hotword was spoken (e.g., the confidence level may be below the primary threshold). In an embodiment, the system looks for a series of similar close matching triggers and, based on this series of close matching triggers, automatically identifies failed hotword attempts. Therefore, this can improve performance and flexibility by allowing the automated assistant to respond to triggers that are close matches, reducing the processing and power usage of the automated assistant, and reducing the number of attempts the user needs to make in order to be understood. For example, in an embodiment, the automated assistant can respond to commands that are slight acoustic variants of supported hotwords.

[0011] In some embodiments, when a failed hotword attempt is identified, a set of gradients is generated locally on the client device based on comparing the predicted output with the true value output (e.g., a true value output that meets a threshold). In some embodiments, one or more processors of the client device use the generated gradients to update one or more weights of the machine learning model based on the generated gradients. For example, backpropagation and / or other techniques can be used to update the weights based on the gradients. In this way, the system can adapt by adding user-preferred words or phrases to the available hotwords. This can improve the performance of the machine learning model stored locally on the client device and mitigate the occurrence of failed hotword attempts based on the predicted output generated using the machine learning model.

[0012] In some embodiments, the generated gradients are additionally or alternatively transmitted by the client device over the network to a remote system. In those embodiments, the remote system utilizes the generated gradients and the additional gradients from the additional client devices to update the global weights of the corresponding global machine learning model using a federated learning technique. The additional gradients from the additional client devices can similarly be generated locally on the corresponding additional client devices. In various embodiments, the client device transmits the generated gradients without transmitting any data used to generate the prediction output (e.g., audio data and / or other sensor data) and without transmitting any data used to identify failed hotword attempts (e.g., further user interface input). The remote system can utilize the generated gradients when updating the global model using the federated learning technique without referencing or using such data. Transmitting only the gradients uses fewer network resources than transmitting the larger data size used to generate the prediction output and identify failed hotword attempts. Furthermore, the transmission of the gradients preserves the privacy and security of personal data, as the data used to generate the prediction output and identify failed hotword attempts may not be derived from the gradients. In some embodiments, one or more differential privacy techniques (e.g., adding Gaussian noise) can be utilized to further ensure that such data cannot be derived from the gradients.

[0013] In embodiments where the remote system updates the global weights of the speech recognition model, the remote system may thereafter provide the updated global weights to the client devices, causing the client devices to replace the weights of the machine learning models on their devices with the updated global weights. In some embodiments, the remote system may additionally or alternatively provide the updated machine learning model to the client devices, causing the client devices to replace the machine learning models on their devices with the updated global machine learning model. Thus, by utilizing the updated global weights or the updated global machine learning model, performance on the device is improved.

[0014] In various embodiments, first audio data capturing a first spoken utterance of a user may be received via one or more microphones of a client device; the first audio data may be processed using one or more machine learning models to generate a first prediction output indicating a probability that one or more hot words are present in the first audio data; it may be determined that the first prediction output satisfies a secondary threshold but does not satisfy a primary threshold, the secondary threshold being less than the primary threshold indicating that the one or more hot words are present in the audio data; second audio data capturing a second spoken utterance of the user may be received via the one or more microphones of the client device; the second audio data may be processed using the one or more machine learning models to generate a second prediction output indicating a probability that the one or more hot words are present in the second audio data; it may be determined that the second prediction output satisfies the secondary threshold but does not satisfy the primary threshold; in response to the first prediction output and the second prediction output satisfying the secondary threshold but not the primary threshold, and in response to the first spoken utterance and the second spoken utterance satisfying one or more timing criteria relative to each other, a failed hot word attempt may be identified; and in response to identifying the failed hot word attempt, a prompt responsive to the failed hot word attempt may be provided.

[0015] In some implementations, the failed hotword attempt can be further responsive to determining that a similarity between the first spoken utterance and the second spoken utterance exceeds a similarity threshold. In some implementations, the failed hotword attempt can be further responsive to determining that a probability indicated by the first prediction output and a probability indicated by the second prediction output correspond to the same hotword of the one or more hotwords. In some implementations, a model conditioned on acoustic features can be used to determine that the first audio data and the second audio data include a command, and the failed hotword attempt can be further responsive to the first audio data and the second audio data including the command.

[0016] In some embodiments, an intent hotword corresponding to a failed hotword attempt can be determined. In some embodiments, the intent hotword can be determined based on acoustic similarity between at least a portion of the first audio data, at least a portion of the second audio data, and the intent hotword. This can be implicit acoustic similarity (e.g., obtained via a probability from a hotword model). In some embodiments, providing a prompt can include displaying the intent hotword on a display of the client device, or providing an audio response including the intent hotword by the client device. In some embodiments, an action corresponding to the intent hotword can be performed.

[0017] In some additional or alternative embodiments, first audio data capturing a first spoken utterance of a user may be received via one or more microphones of a client device; the first audio data may be processed using each of a plurality of categories in a machine learning model to generate a corresponding probability associated with the first audio data, each category being associated with a corresponding hot word in a plurality of hot words, and each corresponding probability being associated with a probability that the corresponding hot word exists in the first audio data; the probability that one of the plurality of hot words exists in the first audio data may be determined to satisfy a secondary threshold but not satisfy a primary threshold, the secondary threshold being less than the primary threshold indicative of the presence of the one of the plurality of hot words in the audio data; second audio data capturing a second spoken utterance of the user may be received via the one or more microphones of the client device; the first audio data may be processed using Each of the multiple classes in the machine learning model processes the second audio data to generate a corresponding probability associated with the second audio data, each corresponding probability being associated with a probability that a corresponding hot word exists in the second audio data; the probability that the one of the multiple hot words exists in the second audio data can be determined to meet a secondary threshold but not a primary threshold; in response to the probability that the one of the multiple hot words exists in the first audio data meeting the secondary threshold but not the primary threshold and the probability that the one of the multiple hot words exists in the second audio data meeting the secondary threshold but not the primary threshold, and in response to the first spoken utterance and the second spoken utterance meeting one or more time criteria relative to each other, a failed hot word attempt can be identified; and in response to identifying the failed hot word attempt, a prompt responsive to the failed hot word attempt can be provided.

[0018] In some implementations, identifying a failed hotword attempt can be further responsive to determining that a similarity between the first spoken utterance and the second spoken utterance exceeds a similarity threshold. In some implementations, a model conditioned on acoustic features (e.g., prosody, non-verbal vocalizations, or other types of inferred audio command attributes) can be used to determine that the first audio data and the second audio data include a command; and identifying a failed hotword attempt can be further responsive to the first audio data and the second audio data including the command.

[0019] In some implementations, an intent hotword corresponding to the failed hotword attempt may be determined. In some implementations, the intent hotword may be determined based on acoustic similarity between at least a portion of the first audio data, at least a portion of the second audio data, and the intent hotword.

[0020] In some implementations, providing a prompt may include displaying the intent hotword on a display of the client device, or providing an audio response including the intent hotword by the client device. In some implementations, an action corresponding to the intent hotword may be performed.

[0021] By utilizing one or more of the techniques described herein, the occurrence of a failed hotword attempt can be identified locally on a corresponding client device. In addition, a prompt can be provided in response to the failed hotword attempt, and an action corresponding to the failed hotword attempt can be performed. In an embodiment, the system looks for a series of similar near-matching triggers, and based on such a series of near-matching triggers, automatically identifies a failed hotword attempt. This improves performance and flexibility by allowing the automated assistant to provide prompts about hotwords and respond to near-matching triggers, reducing processing and power usage at the automated assistant, and reducing the number of attempts a user needs to make to be understood.

[0022] The above description is provided as an overview of some embodiments of the present disclosure. Those embodiments, as well as further descriptions of other embodiments, will be described in more detail below.

[0023] Various embodiments may include a non-transitory computer-readable storage medium storing instructions executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), and / or a tensor processing unit (TPU)) to perform a method, such as one or more of the methods described herein. Other embodiments may include an automated assistant client device (e.g., a client device that includes at least an automated assistant interface for interfacing with a cloud-based automated assistant component) that includes a processor operable to execute the stored instructions to perform a method, such as one or more of the methods described herein. Still other embodiments may include a system of one or more servers that include one or more processors operable to execute stored instructions to perform a method, such as one or more of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1A and Figure 1B Depicted are exemplary process flows demonstrating various aspects of the disclosure according to various implementations.

[0025] Figure 2 Depicts including Figure 1A and Figure 1B FIG. 1 is a block diagram of various components of an example environment in which embodiments disclosed herein may be implemented.

[0026] Figure 3 Depicted is a flow chart illustrating an example method of identifying failed hotword attempts in accordance with various implementations.

[0027] Figure 4 Depicted is a flow chart illustrating an example method of identifying failed hotword attempts in accordance with various implementations.

[0028] Figure 5 An example architecture for a computing device is depicted. DETAILED DESCRIPTION

[0029] Figure 1A and Figure 1B An example process flow illustrating various aspects of the present disclosure is depicted. The client device 110 Figure 1A , and included in the representation of the client device 110 Figure 1A Components encompassed within the box of . The machine learning engine 122A can receive audio data 101 corresponding to spoken words detected via one or more microphones of the client device 110 and / or other sensor data 102 corresponding to speechless physical movements (e.g., gestures and / or movements, body postures and / or body movements, eye gaze, facial movements, mouth movements, etc.) detected via one or more non-microphone sensor components of the client device 110. The one or more non-microphone sensors may include a camera or other visual sensor, a proximity sensor, a pressure sensor, an accelerometer, a magnetometer, and / or other sensors. The machine learning engine 122A processes the audio data 101 and / or other sensor data 102 using a machine learning model 152A to generate a prediction output 103. As described herein, the machine learning engine 122A can be a hot word detection engine 122B or an alternative engine, such as a voice activity detector (VAD) engine, an endpoint detector engine, a speech recognition (ASR) engine, and / or other engines.

[0030] In some embodiments, when the machine learning engine 122A generates the prediction output 103, it can be stored locally on the client device in the on-device storage 111 and optionally associated with the corresponding audio data 101 and / or other sensor data 102. In some versions of those embodiments, the prediction output can be retrieved by the gradient engine 126 for use in generating the gradient 106 at a later time, such as when one or more conditions described herein are met. The on-device storage 111 can include, for example, read-only memory (ROM) and / or random access memory (RAM). In other embodiments, the prediction output 103 can be provided to the gradient engine 126 in real time.

[0031] Client device 110 may make a decision based on determining at block 182 whether predicted output 103 satisfies a threshold whether to initiate a currently dormant automated assistant function (e.g., Figure 2Automated assistant 295), avoid launching a currently dormant automated assistant function, and / or shut down a currently active automated assistant function using assistant activation engine 124. Automated assistant functions can include: speech recognition that generates recognized text, natural language understanding (NLU) that generates NLU output, generating a response based on the recognized text and / or NLU output, transmitting audio data to a remote server, transmitting recognized text to a remote server, and / or directly triggering one or more actions in response to audio data 101 (e.g., a common task such as changing the device volume). For example, assume that predicted output 103 is a probability (e.g., 0.80 or 0.90) and the threshold at box 182 is a threshold probability (e.g., 0.85). If client device 110 determines at box 182 that predicted output 103 (e.g., 0.90) meets the threshold (e.g., 0.85), then assistant activation engine 124 can launch the currently dormant automated assistant function.

[0032] In some embodiments, as Figure 1B As depicted, machine learning engine 122A may be hotword detection engine 122B. Note that various automated assistant functions, such as on-device speech recognizer 142, on-device NLU engine 144, and / or on-device fulfillment engine 146, are currently dormant (i.e., as indicated by the dashed lines). Furthermore, assume that hotword detection model 152B is used and the predicted output 103 generated based on audio data 101 satisfies the threshold at block 182, and that voice activity detector 128 detects user speech directed to client device 110.

[0033] In some versions of these embodiments, assistant activation engine 124 activates on-device speech recognizer 142, on-device NLU engine 144, and / or on-device fulfillment engine 146 as currently dormant automated assistant functionality. For example, on-device speech recognizer 142 can use on-device speech recognition model 142A to process audio data 101 including a spoken utterance of the hotword "OK Assistant" and additional commands and / or phrases following the hotword "OK Assistant" to generate recognized text 143A, on-device NLU engine 144 can use on-device NLU model 144A to process recognized text 143A to generate NLU data 145A, on-device fulfillment engine 146 can use on-device fulfillment model 146A to process the NLU data to generate fulfillment data 147A, and client device 110 can use fulfillment data 147A in performing 150 one or more actions in response to audio data 101.

[0034] In other versions of these embodiments, assistant activation engine 124 activates only on-device fulfillment engine 146, without activating on-device speech recognizer 142 and on-device NLU engine 144, to process various commands such as “No,” “Stop,” “Cancel,” “Volume up,” “Volume down,” “Nexttrack,” “Previous track,” and / or other commands that can be processed without on-device speech recognizer 142 and on-device NLU engine 144. For example, on-device fulfillment engine 146 processes audio data 101 using on-device fulfillment model 146A to generate fulfillment data 147A, and client device 110 can use fulfillment data 147A in performing 150 one or more actions in response to audio data 101. Furthermore, in versions of these embodiments, assistant activation engine 124 may initially activate the currently dormant automation functionality by initially activating only the on-device speech recognizer 142 to determine that audio data 101 includes the hotword “OK Assistant” to verify that the decision made at block 182 is correct (e.g., that audio data 101 does in fact include the hotword “OK Assistant”), and / or assistant activation engine 124 may transmit audio data 101 to one or more servers (e.g., remote server 160) to verify that the decision made at block 182 is correct (e.g., that audio data 101 does in fact include the hotword “OK Assistant”).

[0035] Back to Figure 1A If client device 110 determines at block 182 that predicted output 103 (e.g., 0.80) fails to satisfy a threshold value (e.g., 0.85), assistant activation engine 124 can refrain from initiating currently dormant automated assistant functionality and / or shut down any currently active automated assistant functionality. Furthermore, if client device 110 determines at block 182 that predicted output 103 (e.g., 0.80) fails to satisfy a threshold value (e.g., 0.85), client device 110 can determine at block 184 whether further user interface input has been received. For example, the further user interface input can be an additional spoken utterance including a hotword, an additional speechless physical movement serving as a proxy for the hotword, an actuation of an explicit automated assistant invocation button (e.g., a hardware button or a software button), a sensed “squeeze” of client device 110 (e.g., when squeezing client device 110 with at least a threshold amount of force to invoke an automated assistant), and / or other explicit automated assistant invocations. If the client device 110 determines at block 184 that no further user interface input has been received, the client device 110 may end at block 190 .

[0036] However, if the client device 110 determines that further user interface input was received at block 184, the system may determine at block 186 whether the further user interface input received at block 184, together with the previously received user interface input that failed to meet the threshold at block 182, indicates a failed hotword attempt. A failed hotword attempt may be the result of a user attempting to invoke the assistant (or a specific action of the assistant) using a word or phrase that is not a hotword but closely matches the hotword (e.g., is acoustically similar to the hotword). A failed hotword attempt may also be the result of a user attempting to invoke the assistant (or a specific action of the assistant) using a word or phrase that is not a hotword but is semantically similar to the hotword.

[0037] If client device 110 determines at block 186 that the further user interface input received at block 184 does not indicate a failed hotword attempt, client device 110 may end at block 190. However, if client device 110 determines at block 186 that the further user interface input received at block 184 indicates a failed hotword attempt, client device 110 may determine intent hotword 104 and provide prompt 105. In some implementations, client device 110 may also determine a true value output 109, which may be a probability (e.g., 1.00) indicating that client device 110 should have initiated the currently dormant automated assistant function and / or avoided closing the currently active automated assistant function.

[0038] In some implementations, if the client device 110 determines at box 186 that further user interface input received at box 184 indicates a failed hotword attempt, the assistant activation engine 124 of the client device 110 may initiate one or more currently dormant automated assistant functions as intended by the user (e.g., based on the intent hotword 104).

[0039] In some embodiments, gradient engine 126 can generate gradient 106 based on predicted output 103 of true value output 109. For example, gradient engine 126 can generate gradient 106 based on a comparison of predicted output 103 with true value output 109. In some versions of those embodiments, client device 110 locally stores predicted output 103 and corresponding true value output 109 in device storage 111, and when one or more conditions are met, gradient engine 126 retrieves predicted output 103 and corresponding true value output 109 to generate gradient 106. The one or more conditions can include, for example, that the client device is charging, that the client device has at least a threshold charge state, that the temperature of the client device (based on one or more on-device temperature sensors) is below a threshold, and / or that the client device is not being held by a user. In other versions of those embodiments, client device 110 provides predicted output 103 and true value output 109 to gradient engine 126 in real time, and gradient engine 126 generates gradient 106 in real time.

[0040] In addition, the gradient engine 126 can provide the generated gradients 106 to the on-device machine learning training engine 132. Upon receiving the gradients 106, the on-device machine learning training engine 132 uses the gradients 106 to update the on-device machine learning model 152A. For example, the on-device machine learning training engine 132 can utilize backpropagation and / or other techniques to update the on-device machine learning model 152A. Note that in some embodiments, the on-device machine learning training engine 132 can utilize batch processing techniques to update the on-device machine learning model 152A based on the gradients 106 and additional gradients determined locally at the client device 110 based on additional failed hotword attempts. In some embodiments, the on-device machine learning training engine 132 can require the user to train the new command on the device (by providing multiple examples) and update the on-device machine learning model 152A based on the training.

[0041] Additionally, client device 110 may transmit generated gradients 106 to remote system 160. When remote system 160 receives gradients 106, remote training engine 162 of remote system 160 updates the global weights of global hotword model 152A1 using gradients 106 and additional gradients 107 from additional client devices 170. Additional gradients 107 from additional client devices 170 may each be generated based on the same or similar techniques as described above with respect to gradients 106 (but based on locally identified failed hotword attempts specific to those client devices).

[0042] The update distribution engine 164 may provide the updated global weights and / or the updated global hotword model itself to the client device 110 and / or other client devices in response to one or more conditions being met, as indicated by 108. The one or more conditions may include, for example, a threshold duration and / or amount of training since the last time the updated weights and / or updated speech recognition model were provided. The one or more conditions may additionally or alternatively include, for example, a measured improvement in the updated speech recognition model and / or a threshold duration of time since the last time the updated weights and / or updated speech recognition model were provided. When the updated weights are provided to the client device 110, the client device 110 may replace the weights of the on-device machine learning model 152A with the updated weights. When the updated global hotword model is provided to the client device 110, the client device 110 may replace the on-device machine learning model 152A with the updated global hotword model. In other embodiments, the client device 110 may download one or more more appropriate hotword models from the server based on the type of command the user expects to speak, and replace the on-device machine learning model 152A with the downloaded hotword models.

[0043] In some implementations, the on-device machine learning model 152A is transmitted (e.g., by a remote system 160 or other component) for storage and use at the client device 110 based on the geographic region and / or other attributes of the client device 110 and / or the user of the client device 110. For example, the on-device machine learning model 152A can be one of N available machine learning models for a given language, but can be trained based on corrections specific to a particular geographic region, device type, context (e.g., music playback), etc., and provided to the client device 110 based on the client device 110 being primarily located in the particular geographic region.

[0044] Now turn Figure 2 , in an embodiment, a client device 110 is shown, wherein Figure 1A and Figure 1B The machine learning engines on various devices are included as part of (or in communication with) the automated assistant client 240. The corresponding machine learning models are also shown with Figure 1A and Figure 1B For simplicity, Figure 2 Not shown in Figure 1A and Figure 1B other components. Figure 2 shows how the automated assistant client 240 can utilize Figure 1A and Figure 1B An example of various on-device machine learning engines and their corresponding machine learning models to perform various actions.

[0045] Figure 2 The client device 110 in FIG. 1 is shown as having one or more microphones 211, one or more speakers 212, one or more cameras and / or other visual components 213, and a display 214 (e.g., a touch-sensitive display). The client device 110 may further include pressure sensors, proximity sensors, accelerometers, magnetometers, and / or other sensors for generating other sensor data in addition to the audio data captured by the one or more microphones 211. The client device 110 selectively executes at least an automated assistant client 240. Figure 2 In the example shown in FIG, automated assistant client 240 includes on-device hotword detection engine 122B, on-device speech recognizer 142, on-device natural language understanding (NLU) engine 144, and on-device fulfillment engine 146. Automated assistant client 240 further includes voice capture engine 242 and visual capture engine 244. Automated assistant client 140 may include additional and / or alternative engines, such as a voice activity detector (VAD) engine, an endpoint detector engine, and / or other engines.

[0046] One or more cloud-based automated assistant components 280 may optionally be implemented on one or more computing systems (collectively, “cloud” computing systems) that are communicatively coupled to client device 110 via one or more local area networks and / or wide area networks (e.g., the Internet), indicated generally at 290. Cloud-based automated assistant components 280 may be implemented, for example, via a high-performance server cluster.

[0047] In various embodiments, an instance of automated assistant client 240, through its interaction with one or more cloud-based automated assistant components 280, can form what appears to be a logical instance of automated assistant 295 from the user's perspective, with which the user can engage in human-computer interaction (e.g., verbal interaction, gesture-based interaction, and / or touch-based interaction).

[0048] Client device 110 may be, for example, a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device in a user's vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a standalone interactive speaker, a smart appliance such as a smart TV (or a standard TV equipped with a network-connected dongle with automated assistant capabilities), and / or a wearable device of a user including a computing device (e.g., a watch of the user including the computing device, glasses of the user including the computing device, a virtual or augmented reality computing device). Additional and / or alternative client devices may be provided.

[0049] The one or more visual components 213 can take various forms, such as a monographic camera, a stereo camera, a LIDAR component (or other laser-based component), a radar component, etc. The one or more visual components 213 can be used, for example, by the visual capture engine 242 to capture visual frames (e.g., image frames, laser-based visual frames) of the environment in which the client device 110 is deployed. In some embodiments, such visual frames can be used to determine whether a user is present in the vicinity of the client device 110 and / or a user (e.g., a user's face) is at a certain distance relative to the client device 110. For example, such a determination can be used to determine whether to activate a user. Figure 2 Various on-device machine learning engines and / or other engines are depicted.

[0050] The voice capture engine 242 can be configured to capture the user's voice and / or other audio data captured via the microphone 211. Furthermore, the client device 110 can include a pressure sensor, a proximity sensor, an accelerometer, a magnetometer, and / or other sensors for generating additional sensor data in addition to the audio data captured via the microphone 211. As described herein, such audio data and other sensor data can be used by the hotword detection engine 122B and / or other engines to determine whether to initiate one or more currently dormant automated assistant functions, to avoid initiating one or more currently dormant automated assistant functions, and / or to shut down one or more currently active automated assistant functions. The automated assistant functions can include an on-device voice recognizer 142, an on-device NLU engine 144, an on-device fulfillment engine 146, and additional and / or alternative engines. For example, the on-device voice recognizer 142 can utilize an on-device voice recognition model 142A to process audio data capturing a spoken utterance to generate recognized text 143A corresponding to the spoken utterance. The on-device NLU engine 144 can optionally utilize an on-device NLU model 144A to perform on-device natural language understanding on the recognized text 143A to generate NLU data 145A. NLU data 145A may include, for example, an intent corresponding to the spoken utterance and, optionally, parameters for the intent (e.g., slot values). Furthermore, an on-device fulfillment engine 146 generates fulfillment data 147A based on the NLU data 145A, optionally utilizing an on-device fulfillment model 146A. This fulfillment data 147A may define local and / or remote responses (e.g., answers) to the spoken utterance, interactions with locally installed applications based on the spoken utterance, commands transmitted to an Internet of Things (IoT) device (directly or via a corresponding remote system) based on the spoken utterance, and / or other parsed actions performed based on the spoken utterance. Fulfillment data 147A is then provided for local and / or remote execution / implementation of the actions determined to parse the spoken utterance. Execution may include, for example, presenting a local and / or remote response (e.g., visual and / or auditory presentation (optionally utilizing a local text-to-speech module), interacting with locally installed applications, transmitting commands to IoT devices, and / or other actions.

[0051] Display 214 may be used to display recognized text 143A and / or further recognized text 143B from on-device speech recognizer 122, and / or prompt 105 (which may include intent hotword 104), and / or one or more results from execution 150. Display 214 may further be one of the user interface output components through which a visual portion of a response from automated assistant client 240 is presented.

[0052] In some implementations, the one or more speakers 212 can be used to provide a prompt 105 , which can be an audio response including the intent hotword 104 .

[0053] In some embodiments, the cloud-based automated assistant component 280 may include a remote ASR engine 281 that performs speech recognition, a remote NLU engine 282 that performs natural language understanding, and / or a remote fulfillment engine 283 that generates fulfillment. A remote execution module may also optionally be included that performs remote execution based on fulfillment data determined locally or remotely. Additional and / or alternative remote engines may be included. As described herein, in various embodiments, on-device speech processing, on-device NLU, on-device fulfillment, and / or on-device execution may be prioritized, at least due to the latency and / or network usage reduction they provide when parsing spoken utterances (due to not requiring a client-server round trip to parse spoken utterances). However, one or more cloud-based automated assistant components 280 may be at least selectively utilized. For example, such components may be used in parallel with on-device components, and output from such components may be used when local components fail. For example, in certain circumstances (e.g., due to relatively limited resources of the client device 110), the on-device fulfillment engine 146 may fail, and in such circumstances, the remote fulfillment engine 283 may utilize the more robust resources of the cloud to generate fulfillment data. Remote fulfillment engine 283 may operate in parallel with on-device fulfillment engine 146 and utilize its results when on-device fulfillment fails, or may be invoked in response to determining a failure of on-device fulfillment engine 146 .

[0054] In various embodiments, an NLU engine (on-device and / or remote) can generate NLU data that includes one or more annotations of the recognized text and one or more (e.g., all) terms of the natural language input. In some embodiments, the NLU engine is configured to recognize and annotate various types of grammatical information in the natural language input. For example, the NLU engine can include a morphological module that can separate individual words into morphemes and / or annotate morphemes, for example, with categories of morphemes. The NLU engine can also include a part-of-speech tagger that is configured to annotate terms with their grammatical roles. Furthermore, for example, in some embodiments, the NLU engine can additionally and / or alternatively include a dependency parser that is configured to determine syntactic relationships between terms in the natural language input.

[0055] In some embodiments, the NLU engine may additionally and / or alternatively include an entity tagger configured to annotate entity references in one or more segments, such as references to people (including, for example, literary characters, celebrities, public figures, etc.), organizations, locations (real and imagined), etc. In some embodiments, the NLU engine may additionally and / or alternatively include a coreference resolver (not depicted) configured to group or "cluster" references to the same entity based on one or more contextual clues. In some embodiments, one or more components of the NLU engine may rely on annotations from one or more other components of the NLU engine.

[0056] The NLU engine may also include an intent matcher that is configured to determine the intent of a user participating in an interaction with the automated assistant 295. The intent matcher may use various techniques to determine the intent of the user. In some embodiments, the intent matcher may have access to one or more local and / or remote data structures that include, for example, a plurality of mappings between grammars and response intents. For example, the grammars included in the mappings may be selected and / or learned over time and may represent common intents of users. For example, a grammar "play <artist>(play <artist>)" may be mapped to an intent that invokes a response action that causes the music to be played on the client device 110. <artist>(<artist>)'s music. Another grammar "[weather|forecast]today" can match user queries such as "what's the weather today" and "what's the forecast for today?". In addition to or in place of grammars, in some embodiments, the intent matcher can also employ one or more trained machine learning models, alone or in combination with one or more grammars. These trained machine learning models can be trained to recognize intents, for example, by embedding recognized text from spoken utterances into a reduced dimensionality space, and then determining which other embeddings (and therefore intents) are closest, for example, using techniques such as Euclidean distance, cosine similarity, etc. As shown above, the "play <artist>As shown in the example grammars, some grammars have slots (e.g. <artist>). Slot values ​​can be determined in various ways. Users usually provide slot values ​​proactively. For example, for the syntax "Order me a <topping>pizza", the user may say the phrase "order me a sausage pizza", in which case the slot <topping>(<Toppings>) is automatically filled in. Other slot values ​​may be inferred based on, for example, user location, currently presented content, user preferences, and / or other cues.

[0057] A fulfillment engine (local and / or remote) can be configured to receive the predicted / estimated intent and any associated slot values ​​output by the NLU engine and fulfill (or "resolve") the intent. In various embodiments, the fulfillment (or "resolve") of the user intent can result in the generation / obtaining of various fulfillment information (also referred to as fulfillment data), for example, by the fulfillment engine. This can include determining local and / or remote responses (e.g., answers) to the spoken utterance, interactions with locally installed applications performed based on the spoken utterance, commands transmitted to Internet of Things (IoT) devices (directly or via corresponding remote systems) based on the spoken utterance, and / or other resolved actions performed based on the spoken utterance. The on-device fulfillment can then initiate local and / or remote execution / implementation of the determined actions to resolve the spoken utterance.

[0058] Figure 3 A flow chart illustrating an example method 300 for identifying failed hotword attempts is depicted. For convenience, the operations of method 300 are described with reference to a system performing the operations. The system of method 300 includes one or more processors and / or other components of a client device. Furthermore, while the operations of method 300 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0059] At block 305 , the system receives first audio data capturing a first spoken utterance of a user via one or more microphones of a client device.

[0060] At block 310, the system processes the first audio data received at block 305 using one or more machine learning models to generate a first prediction output indicating a probability of one or more hot words being present in the first audio data. The one or more machine learning models may be, for example, an on-device hot word detection model and / or other machine learning models. Each machine learning model may be a deep neural network or any other type of model and may be trained to recognize one or more hot words. Furthermore, the generated output may be, for example, a probability and / or other likelihood measure.

[0061] Still referring to box 310, in an example, the machine learning model can be a hot word model trained to identify hot words including multiple words as a single category. If the first audio data includes the word "next song" and the machine learning model is trained to identify "next track", then because there is a certain level of acoustic similarity between "next song" and "next track", the machine learning model can generate a first prediction output of 0.5. In another example, a single machine learning model with multiple categories can be used, and each category can detect a separate word from a specific set of hot words. Alternatively, multiple machine learning models can be used, and each model can detect a separate word from a specific set of hot words. In this example, the system can generate a high confidence match for the "next" category, but no other matches.

[0062] At block 315, the system determines whether the first prediction output generated at block 310 satisfies a secondary threshold that is less indicative of the presence of the one or more hot words in the audio data than the primary threshold (e.g., the secondary threshold may be a medium confidence threshold and the primary threshold may be a high confidence threshold). If, in an iteration of block 315, the system determines that the first prediction output generated at block 310 does not satisfy the secondary threshold, the system proceeds to block 320 and the process ends. On the other hand, if, in an iteration of block 315, the system determines that the first prediction output generated at block 310 does satisfy the secondary threshold, the system proceeds to block 325. In embodiments, the primary and secondary thresholds may vary based on the hot word, command, device, and / or context.

[0063] Still referring to block 315, in the example, assume that the first predicted output generated at block 310 is a probability, and that the probability must be greater than 0.65 to satisfy the secondary threshold at block 315, and that the predicted probability is 0.88. Based on the predicted probability 0.88 satisfying the secondary threshold 0.65, the system proceeds to block 325.

[0064] At block 325, the system determines whether the first predicted output generated at block 310 satisfies a preliminary threshold. If, in an iteration of block 325, the system determines that the first predicted output generated at block 310 does not satisfy the preliminary threshold, the system proceeds to block 335. On the other hand, if, in an iteration of block 325, the system determines that the first predicted output generated at block 310 satisfies the preliminary threshold, the system proceeds to block 330 and initiates one or more currently dormant automated assistant functions. In some implementations, the one or more automated assistant functions include speech recognition to generate recognized text, natural language understanding (NLU) to generate NLU output, generating a response based on the recognized text and / or NLU output, transmitting audio data to a remote server, transmitting recognized text to a remote server, and / or directly triggering one or more actions in response to the first audio data (e.g., a common task such as changing the volume of a device).

[0065] Still referring to block 330, in the example, assume that the first predicted output generated at block 310 is a probability, and that the probability must be greater than 0.85 to satisfy the preliminary threshold at block 325 and activate one or more currently dormant automated assistant functions at block 330, and that the predicted probability is 0.88. Based on the predicted probability of 0.88 satisfying the threshold of 0.85, the system proceeds to block 330 and initiates the one or more currently dormant automated assistant functions as intended by the user.

[0066] At block 335 , the system receives second audio data capturing a second spoken utterance of the user via one or more microphones of the client device.

[0067] At block 340, the system processes the second audio data received at block 335 using the one or more machine learning models to generate a second prediction output indicating a probability of the one or more hot words being present in the second audio data. The one or more machine learning models may be, for example, an on-device hot word detection model and / or other machine learning models. Furthermore, the generated output may be, for example, a probability and / or other likelihood measure.

[0068] At block 345, the system determines whether the second predicted output generated at block 340 satisfies the secondary threshold. If, at an iteration of block 345, the system determines that the second predicted output generated at block 340 does not satisfy the secondary threshold, the system proceeds to block 320, and the process ends. On the other hand, if, at an iteration of block 345, the system determines that the second predicted output generated at block 340 does satisfy the secondary threshold, the system proceeds to block 350.

[0069] Still referring to block 345, in the example, assume that the second prediction output generated at block 340 is a probability, and that the probability must be greater than 0.65 to satisfy the secondary threshold at block 345, and that the predicted probability is 0.88. Based on the predicted probability 0.88 satisfying the secondary threshold 0.65, the system proceeds to block 350.

[0070] At block 350, the system determines whether the second predicted output generated at block 340 satisfies a primary threshold. If, in an iteration of block 350, the system determines that the second predicted output generated at block 340 does not satisfy the primary threshold, the system proceeds to block 360. On the other hand, if, in an iteration of block 350, the system determines that the second predicted output generated at block 340 satisfies the primary threshold, the system proceeds to block 355 and initiates one or more currently dormant automated assistant functions. In some implementations, the one or more automated assistant functions include speech recognition to generate recognized text, natural language understanding (NLU) to generate NLU output, generating a response based on the recognized text and / or NLU output, transmitting audio data to a remote server, transmitting recognized text to a remote server, and / or directly triggering one or more actions in response to the second audio data (e.g., a common task such as changing the volume of a device).

[0071] Referring to block 350, in other embodiments, if, in an iteration of block 350, the system determines that the predicted output generated at block 340 does not satisfy the primary threshold, the system may return to block 335 one or more times and may repeat blocks 335 to 350 using third audio data capturing a third spoken utterance of the user, and so on.

[0072] Referring to block 355, in this example, assume that the second predicted output generated at block 340 is a probability, and that the probability must be greater than 0.85 to satisfy the preliminary threshold at block 350 and activate one or more currently dormant automated assistant functions at block 355, and that the predicted probability is 0.88. Based on the predicted probability of 0.88 satisfying the threshold of 0.85, the system proceeds to block 355 and initiates the one or more currently dormant automated assistant functions as intended by the user.

[0073] At block 360, the system determines whether the first spoken utterance captured by the first audio data received at block 305 and the second spoken utterance captured by the second audio data received at block 335 meet one or more timing criteria relative to each other (e.g., the utterances are separated by less than 10 seconds). If, in an iteration of block 360, the system determines that it is not the case that the first spoken utterance captured by the first audio data received at block 305 and the second spoken utterance captured by the second audio data received at block 335 meet one or more timing criteria relative to each other, the system proceeds to block 320 and the process ends. On the other hand, if, in an iteration of block 360, the system determines that the first spoken utterance captured by the first audio data received at block 305 and the second spoken utterance captured by the second audio data received at block 335 meet one or more timing criteria relative to each other, the system proceeds to block 365.

[0074] At box 365, in response to the first predicted output and the second predicted output satisfying the secondary threshold (determined at boxes 315 and box 345, respectively) but not satisfying the primary threshold (determined at boxes 325 and box 350, respectively), and in response to the first spoken utterance captured by the first audio data received at box 305 and the second spoken utterance captured by the second audio data received at box 335 satisfying one or more timing criteria relative to each other (as determined at box 360), the system identifies a failed hotword attempt (e.g., the system determines that the user made a failed attempt to invoke the assistant using a word or phrase that is not a hotword).

[0075] Still referring to block 365, in some embodiments, identifying a failed hotword attempt is further responsive to determining that the similarity between the first spoken utterance and the second spoken utterance exceeds a similarity threshold. The system can allow for some variability, as the user can try different variations of a word or phrase after multiple failed hotword attempts. Additionally, in some embodiments, identifying a failed hotword attempt is further responsive to determining that the probability indicated by the first prediction output and the probability indicated by the second prediction output correspond to the same hotword in the one or more hotwords (e.g., they are both close to the same hotword category).

[0076] Still referring to block 365, in some embodiments, the system uses a model conditioned on acoustic features (e.g., prosody, non-verbal vocalizations, or other types of inferred audio command attributes) to determine that the first audio data and the second audio data include a command. The model conditioned on acoustic features can be run as a second stage or in conjunction with the hotword model. In some embodiments, identifying a failed hotword attempt is further responsive to determining that the first audio data and the second audio data include a command.

[0077] In box 370, the system determines the intent hotword corresponding to the failed hotword attempt identified in box 365. In some embodiments, the intent hotword can be determined based on acoustic similarity between at least a portion of the first audio data, at least a portion of the second audio data, and the intent hotword. In other embodiments, the system can enter a higher power state (e.g., wake up) and perform a second stage of speech processing on the first audio data and the second audio data using an on-device speech recognition engine. In this case, the system can use natural language understanding to determine the intent hotword, for example, based on text embedding similarity between the transcription of the first audio data, the transcription of the second audio data, and the supported hotwords.

[0078] At block 375, in response to identifying a failed hotword attempt at block 365, the system provides a prompt responsive to the failed hotword attempt. In some implementations, providing the prompt includes displaying the intended hotword (determined at block 370) on a display of the client device, or providing an audio response by the client device including the intended hotword. The prompt may clarify the available vocabulary (i.e., supported hotwords). The prompt may indicate that the system has determined that the user may have attempted to say a particular supported hotword (e.g., the intended hotword determined at block 370). In cases where a failed hotword attempt (e.g., "volume up") is similar to multiple supported hotwords (e.g., "volume up" and "volume down"), the prompt may include multiple supported hotwords. In other implementations, the system may not provide a prompt at block 375 and may instead proceed directly from block 370 to block 380.

[0079] In box 380, the system performs an action corresponding to the intent hotword determined in box 370. For example, the system can initiate one or more currently dormant automated assistant functions. In some embodiments, the one or more automated assistant functions include speech recognition that generates recognized text, natural language understanding (NLU) that generates NLU output, generating a response based on the recognized text and / or NLU output, transmitting audio data to a remote server, transmitting the recognized text to a remote server, and / or directly triggering one or more actions based on the intent hotword (e.g., common tasks such as changing the device volume, advancing to the next track, pausing, etc.). In some embodiments, different intent hotwords can correspond to different actions (e.g., a "volume up" hotword can correspond to an action to increase the volume, and a "volume down" hotword can correspond to an action to decrease the volume).

[0080] Still referring to block 380, in some implementations, the system can determine a confidence level associated with the determination of the intent hotword at block 370, and can perform the actions at block 380 only if the confidence level exceeds a threshold level and / or only if a single intent hotword is determined at block 370. Additionally, in some implementations, the system can perform the actions at block 380 only if the intent hotword is within a particular subset of supported hotwords.

[0081] After block 380 , the system then returns to block 305 .

[0082] Figure 4 A flow chart illustrating an example method 400 for identifying failed hotword attempts is depicted. For convenience, the operations of method 400 are described with reference to a system performing the operations. The system of method 400 includes one or more processors and / or other components of a client device. Furthermore, while the operations of method 400 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0083] At block 405 , the system receives first audio data capturing a first spoken utterance of a user via one or more microphones of a client device.

[0084] At block 410, the system processes the first audio data received at block 405 using each of the multiple categories in the machine learning model to generate corresponding probabilities associated with the first audio data. In an embodiment, each category is associated with a corresponding hot word from the multiple hot words, and each corresponding probability is associated with a probability that the corresponding hot word is present in the first audio data. The machine learning model can be, for example, an on-device hot word detection model and / or other machine learning models.

[0085] Still referring to box 410, in an example, a first category in the machine learning model may correspond to the hot word "next track", a second category in the machine learning model may correspond to the hot word "previous track", and a third category in the machine learning model may correspond to the hot word "pause". In an embodiment, the system may selectively process audio data in an active subset of multiple categories in the machine learning model. The active subset may include categories corresponding to hot words that are active in a particular context (for example, categories in the machine learning model corresponding to "next track", "previous track", and "pause" may be included in the active subset only when audio and / or video are being actively presented).

[0086] In other implementations, in method 400 , instead of using a machine learning model with multiple categories, the system may use multiple machine learning models, where each machine learning model is associated with a corresponding hot word from the multiple hot words.

[0087] At block 415, the system determines whether the probability of one (or more) of the multiple hot words being present in the first audio data (generated at block 410) satisfies a secondary threshold, which is less than the primary threshold indicating that one of the multiple hot words is present in the audio data. If, in an iteration of block 415, the system determines that the probability of one of the multiple hot words being present in the first audio data does not satisfy the secondary threshold, the system proceeds to block 420 and the process ends. On the other hand, if, in an iteration of block 415, the system determines that the probability of one of the multiple hot words being present in the first audio data generated at block 410 does satisfy the secondary threshold, the system proceeds to block 425.

[0088] Still referring to block 415, in the example, assume that the probability of one of the multiple hot words being present in the first audio data (generated at block 410) is 0.88, and this probability must be greater than 0.65 to satisfy the secondary threshold at block 415. Based on the probability 0.88 satisfying the secondary threshold 0.65, the system proceeds to block 425.

[0089] In box 425, the system determines whether the probability of one of the multiple hot words being present in the first audio data (generated in box 410) meets the primary threshold. If, in an iteration of box 425, the system determines that it is not the case that the probability of one of the multiple hot words being present in the first audio data meets the primary threshold, the system proceeds to box 435. On the other hand, if, in an iteration of box 425, the system determines that the probability of one of the multiple hot words being present in the first audio data generated in box 410 meets the primary threshold, the system proceeds to box 430 and initiates one or more currently dormant automated assistant functions. In some embodiments, the one or more automated assistant functions include speech recognition to generate recognized text, natural language understanding (NLU) to generate NLU output, generating a response based on the recognized text and / or NLU output, transmitting audio data to a remote server, transmitting recognized text to a remote server, and / or directly triggering one or more actions in response to the first audio data (e.g., a common task such as changing the volume of a device).

[0090] Still referring to block 430, in the example, assume that the probability that one of the multiple hot words is present in the first audio data (generated at block 410) is 0.88, and this probability must be greater than 0.85 to satisfy the primary threshold at block 425 and activate one or more currently dormant automated assistant functions at block 430. Based on the probability 0.88 satisfying the threshold 0.85, the system proceeds to block 430 and initiates the one or more currently dormant automated assistant functions as intended by the user.

[0091] At block 435 , the system receives second audio data capturing a second spoken utterance of the user via one or more microphones of the client device.

[0092] At block 440, the system processes the second audio data received at block 435 using each of the multiple categories in the machine learning model to generate corresponding probabilities associated with the second audio data. In embodiments, each corresponding probability is associated with a probability of the corresponding hot word being present in the second audio data. In embodiments, the generated output may be, for example, a probability and / or other likelihood measure.

[0093] At block 445, the system determines whether the probability of one (or more) of the multiple hot words being present in the second audio data (generated at block 440) satisfies a secondary threshold. If, in an iteration of block 445, the system determines that the probability of one of the multiple hot words being present in the second audio data does not satisfy the secondary threshold, the system proceeds to block 420 and the process ends. On the other hand, if, in an iteration of block 445, the system determines that the probability of one of the multiple hot words being present in the second audio data does satisfy the secondary threshold, the system proceeds to block 450.

[0094] Still referring to block 445, in the example, assume that the probability of one of the multiple hot words being present in the second audio data (generated at block 440) is 0.88, and this probability must be greater than 0.65 to satisfy the secondary threshold at block 445. Based on the probability 0.88 satisfying the secondary threshold 0.65, the system proceeds to block 450.

[0095] At box 450, the system determines whether the probability of one of the multiple hot words being present in the second audio data (generated at box 440) satisfies a primary threshold. If, in an iteration of box 450, the system determines that it is not the case that the probability of one of the multiple hot words being present in the second audio data satisfies the primary threshold, the system proceeds to box 460. On the other hand, if, in an iteration of box 450, the system determines that the probability of one of the multiple hot words being present in the second audio data satisfies the primary threshold, the system proceeds to box 455 and initiates one or more currently dormant automated assistant functions. In some embodiments, the one or more automated assistant functions include speech recognition to generate recognized text, natural language understanding (NLU) to generate NLU output, generating a response based on the recognized text and / or NLU output, transmitting audio data to a remote server, transmitting recognized text to a remote server, and / or directly triggering one or more actions in response to the second audio data (e.g., a common task such as changing the volume of a device).

[0096] Still referring to block 450, in one example, assume that the probability that one of the plurality of hot words is present in the second audio data (generated at block 440) is 0.88, and that the probability must be greater than 0.85 to satisfy the primary threshold at block 450 and activate one or more currently dormant automated assistant functions at block 455. Based on the probability 0.88 satisfying the threshold 0.85, the system proceeds to block 455 and initiates the one or more currently dormant automated assistant functions as intended by the user.

[0097] At block 460, the system determines whether the first spoken utterance captured by the first audio data received at block 405 and the second spoken utterance captured by the second audio data received at block 435 meet one or more timing criteria relative to each other. If, in an iteration of block 460, the system determines that it is not the case that the first spoken utterance captured by the first audio data received at block 405 and the second spoken utterance captured by the second audio data received at block 435 meet one or more timing criteria relative to each other, the system proceeds to block 420 and the process ends. On the other hand, if, in an iteration of block 360, the system determines that the first spoken utterance captured by the first audio data received at block 405 and the second spoken utterance captured by the second audio data received at block 435 meet one or more timing criteria relative to each other, the system proceeds to block 465.

[0098] In block 465, in response to the probability that the one of the multiple hot words present in the first audio data satisfies the secondary threshold (as determined in block 415) but does not satisfy the primary threshold (as determined in block 425) and the probability that the one of the multiple hot words present in the second audio data satisfies the secondary threshold (as determined in block 445) but does not satisfy the primary threshold (as determined in block 450), and in response to the first spoken utterance and the second spoken utterance satisfying one or more timing criteria relative to each other (as determined in block 460), the system identifies a failed hot word attempt (e.g., the system determines that the user made a failed attempt to invoke the assistant using a word or phrase that is not a hot word).

[0099] Still referring to block 465, in some implementations, the failed hotword attempt is further responsive to determining that the similarity between the first spoken utterance and the second spoken utterance exceeds a similarity threshold. In some implementations, the system uses a model conditioned on the acoustic features to determine that the first audio data and the second audio data include a command. In some implementations, the failed hotword attempt is further responsive to determining that the first audio data and the second audio data include a command.

[0100] At block 470, the system determines an intent hotword corresponding to the failed hotword attempt identified at block 465. In some implementations, the intent hotword can be determined based on acoustic similarity between at least a portion of the first audio data, at least a portion of the second audio data, and the intent hotword.

[0101] In response to identifying a failed hotword attempt in block 465, the system provides a prompt in response to the failed hotword attempt at block 475. In some implementations, providing the prompt includes displaying the intent hotword on a display of the client device (determined at block 470), or providing an audio response including the intent hotword by the client device. In some implementations, the system may not provide the prompt at block 475 and may instead proceed directly from block 470 to block 480.

[0102] At block 480, the system performs an action corresponding to the intent hotword (determined at block 470). For example, the system can initiate one or more currently dormant automated assistant functions. In some embodiments, the one or more automated assistant functions include speech recognition to generate recognized text, natural language understanding (NLU) to generate NLU output, generating a response based on the recognized text and / or NLU output, transmitting audio data to a remote server, transmitting recognized text to a remote server, and / or directly triggering one or more actions based on the intent hotword (e.g., a common task such as changing the volume of a device). After block 480, the system then returns to block 405.

[0103] In embodiments, the system may identify new words to be added to the model as hot words based on audio data of failed hot word attempts. For example, the system may locally train / fine-tune a hot word model based on utterances captured in audio data associated with failed hot word attempts. In other embodiments, the system may call a server-side system to obtain an updated hot word model that includes words or phrases in utterances that the user attempted to use as hot words in failed hot word attempts (e.g., the system may download one or more more appropriate hot word models from the server based on the type of command the user expects to say). In some embodiments, the system may lower one or more hot word thresholds based on failed hot word attempts (e.g., to take into account the user's accent / pronunciation of the hot word).

[0104] In an embodiment, when a hot word is active in a context, the system can use the failed hot word attempts to make the hot word active in this new context. As a prompt, the system can provide feedback to indicate whether a specific hot word will work in the current context (e.g., time of day, running applications, etc.) in the future.

[0105] In other embodiments, in response to receiving audio data including multiple acoustically similar utterances in close temporal proximity, the system may process the audio data using an on-device speech recognition engine to generate text. In this case, the system may use natural language understanding to determine intent hotwords, for example, based on semantic similarity between the text generated by the speech recognition engine and supported hotwords.

[0106] Note that in various implementations of methods 300 and 400, the first audio data, the second audio data, the first predicted output, the second predicted output, the probability, the primary threshold, the secondary threshold, and / or the intent hotword can be stored locally on the client device. In these and other ways, the occurrence of failed hotword attempts can be mitigated. Furthermore, this enables improved performance of on-device machine learning models processing audio data that captures spoken utterances in the event that a user misremembers a hotword.

[0107] Figure 5 is a block diagram of an example computing device 510 that can optionally be used to perform one or more aspects of the techniques described herein. In some implementations, one or more of the client device, cloud-based automated assistant components, and / or other components can include one or more components of the example computing device 510.

[0108] The computing device 510 typically includes at least one processor 514 that communicates with a number of peripheral devices via a bus subsystem 512. These peripheral devices may include a storage subsystem 524, including, for example, a memory subsystem 525 and a file storage subsystem 526, a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input and output devices allow a user to interact with the computing device 510. The network interface subsystem 516 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0109] The user interface input devices 522 may include a keyboard, a pointing device such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touch screen incorporated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways of entering information into the computing device 510 or a communication network.

[0110] The user interface output device 520 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visual image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, the use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computing device 510 to a user or another machine or computing device.

[0111] The storage subsystem 524 stores programs and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 524 may include programs that execute selected aspects of the methods disclosed herein and implement Figure 1A and Figure 1B The logic of the various components is depicted in .

[0112] These software modules are typically executed by the processor 514 alone or in combination with other processors. The memory subsystem 525 included in the storage subsystem 524 may include multiple memories, including a main random access memory (RAM) 530 for storing instructions and data during program execution and a read-only memory (ROM) 532 for storing fixed instructions. The file storage subsystem 526 may provide persistent storage for program and data files and may include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules implementing the functionality of certain embodiments may be stored by the file storage subsystem 526 in the storage subsystem 524 or in other machines accessible to the processor 514.

[0113] The bus subsystem 512 provides a mechanism for the various components and subsystems of the computing device 510 to communicate with each other as intended. Although the bus subsystem 512 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0114] The computing device 510 can be of various types, including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Figure 5 The description of the computing device 510 depicted in FIG is intended only as a specific example for purposes of illustrating some embodiments. Many other configurations of the computing device 510 are possible with more Figure 5 The computing devices depicted in the drawings may have more or fewer components.

[0115] Where the systems described herein collect or otherwise monitor personal information about a user, or where personal and / or monitored information may be utilized, the user may be provided with an opportunity to control whether a program or feature collects user information (e.g., information about the user's social network, social actions or activities, occupation, user preferences, or the user's current geographic location), or to control whether and / or how content that may be more relevant to the user is received from a content server. In addition, certain data may be processed in one or more ways before it is stored or used to remove personally identifiable information. For example, the user's identity may be processed so that no personally identifiable information about the user can be determined, or where geographic location information is obtained, the user's geographic location may be summarized (such as at the city, zip code, or state level) so that the user's specific geographic location cannot be determined. Thus, the user may have control over how information about the user is collected and / or used.

[0116] Although several embodiments have been described and illustrated herein, various other means and / or structures for performing the functions and / or obtaining the results and / or one or more advantages described herein may be utilized, and each such variation and / or modification is considered to be within the scope of the embodiments described herein. More generally, all parameters, dimensions, materials, and configurations described herein are meant to be exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on the specific application or applications in which the teachings are used. Those skilled in the art will recognize or be able to determine many equivalents to the specific embodiments described herein using no more than routine experimentation. Therefore, it should be understood that the foregoing embodiments are presented only as examples, and within the scope of the appended claims and their equivalents, the embodiments may be practiced in a manner different from that specifically described and claimed. Embodiments of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods, if not mutually inconsistent, is included within the scope of the present disclosure.< / topping> < / topping> < / artist> < / artist> < / artist> < / artist>

Claims

1. A method implemented by one or more processors, the method comprising: receiving, via one or more microphones of the client device, first audio data capturing a first spoken utterance of a user; Processing the first audio data using one or more machine learning models to generate a first prediction output indicating a probability of one or more hot words being present in the first audio data; determining that the first prediction output satisfies a secondary threshold but does not satisfy a primary threshold, the secondary threshold being less indicative than the primary threshold of the presence of the one or more hot words in the audio data; receiving, via the one or more microphones of the client device, second audio data capturing a second spoken utterance of a user; processing the second audio data using the one or more machine learning models to generate a second prediction output indicating a probability of the one or more hot words being present in the second audio data; determining that the second predicted output satisfies the secondary threshold but does not satisfy the primary threshold; responsive to the first predicted output and the second predicted output satisfying the secondary threshold but not satisfying the primary threshold, and responsive to the first spoken utterance and the second spoken utterance satisfying one or more timing criteria relative to each other, identifying a failed hotword attempt; as well as In response to identifying the failed hotword attempt, providing a prompt responsive to the failed hotword attempt.

2. The method according to claim 1, wherein Identifying the failed hotword attempt is further responsive to determining that a similarity between the first spoken utterance and the second spoken utterance exceeds a similarity threshold.

3. The method according to claim 1, wherein Identifying the failed hotword attempt is further responsive to determining that the probability indicated by the first prediction output and the probability indicated by the second prediction output correspond to a same hotword among the one or more hotwords.

4. The method of any one of claims 1-3, further comprising determining that the first audio data and the second audio data comprise a command using a model conditioned on acoustic features, Wherein identifying the failed hotword attempt is further responsive to the first audio data and the second audio data including the command. The method of claim 1 , further comprising determining an intent hotword corresponding to the failed hotword attempt.

6. The method according to claim 5, wherein: The intent hotword is determined based on acoustic similarity between at least a portion of the first audio data, at least a portion of the second audio data, and the intent hotword.

7. The method according to claim 5, wherein: Providing the prompt includes displaying the intent hotword on a display of the client device, or providing an audio response including the intent hotword by the client device. 8 . The method according to claim 5 , further comprising executing an action corresponding to the intent hotword.

9. A method implemented by one or more processors, the method comprising: receiving, via one or more microphones of the client device, first audio data capturing a first spoken utterance of a user; processing the first audio data using each of a plurality of categories in a machine learning model to generate a corresponding probability associated with the first audio data, each of the categories being associated with a corresponding hot word in a plurality of hot words, and each of the corresponding probabilities being associated with a probability that the corresponding hot word is present in the first audio data; determining that a probability of one of the plurality of hot words being present in the first audio data satisfies a secondary threshold but does not satisfy a primary threshold, the secondary threshold being less than the primary threshold and indicating that the one of the plurality of hot words is present in the audio data; receiving, via the one or more microphones of the client device, second audio data capturing a second spoken utterance of a user; processing the second audio data using each of the plurality of categories in the machine learning model to generate a corresponding probability associated with the second audio data, each of the corresponding probabilities being associated with a probability that the corresponding hot word is present in the second audio data; determining that a probability of the one of the plurality of hot words existing in the second audio data satisfies the secondary threshold but does not satisfy the primary threshold; identifying a failed hotword attempt in response to a probability of the one of the plurality of hotwords being present in the first audio data satisfying the secondary threshold but not the primary threshold and a probability of the one of the plurality of hotwords being present in the second audio data satisfying the secondary threshold but not the primary threshold, and in response to the first spoken utterance and the second spoken utterance satisfying one or more timing criteria relative to each other; as well as In response to identifying the failed hotword attempt, providing a prompt responsive to the failed hotword attempt.

10. The method according to claim 9, wherein: Identifying the failed hotword attempt is further responsive to determining that a similarity between the first spoken utterance and the second spoken utterance exceeds a similarity threshold.

11. The method of claim 9 or 10, further comprising determining that the first audio data and the second audio data comprise a command using a model conditioned on acoustic features, Wherein identifying the failed hotword attempt is further responsive to the first audio data and the second audio data including the command.

12. The method of claim 9, further comprising determining an intent hotword corresponding to the failed hotword attempt.

13. The method according to claim 12, wherein: The intent hotword is determined based on acoustic similarity between at least a portion of the first audio data, at least a portion of the second audio data, and the intent hotword.

14. The method according to claim 12, wherein: Providing the prompt includes displaying the intent hotword on a display of the client device, or providing an audio response including the intent hotword by the client device.

15. The method according to any one of claims 12 to 14, further comprising performing an action corresponding to the intent hotword.

16. A computer program product comprising instructions which, when executed by one or more processors, cause the one or more processors to carry out the method according to any one of claims 1 to 15.

17. A computer-readable storage medium comprising instructions that, when executed by one or more processors, cause the one or more processors to carry out the method according to any one of claims 1 to 15.

18. A system comprising a processor, a computer-readable memory, one or more computer-readable storage media, and program instructions collectively stored on the one or more computer-readable storage media, the program instructions being executable to perform the method according to any one of claims 1 to 15.

Images