Degree-based hotword detection

Abstract

A method (300) for soft acceptance of hotwords (24) receives audio data (14) characterizing soft hotword events (202) detected by a hotword detector (200) in streaming audio (12) captured by a user device (110). The method also processes the audio data to determine that the audio data corresponds to a query (22) that specifies an action (148) to perform at the user device. Without triggering the performance of the action at the user device, the method provides a notification (204) of the output from the user device, the notification prompts a user (10) associated with the user device to provide a positive input indication (16) to trigger the performance of the action at the user device, and instructs the user device not to perform the action specified by the query when the user does not provide the positive input indication.

Description

[Technical Field]

[0001] TECHNICAL FIELD This disclosure relates to hotword detection by degree. [Background technology]

[0002] In a speech-enabled environment (e.g., a home, a workplace, a school, an automobile, etc.), a user may speak queries or commands aloud to a computer-based system that processes and responds to the query and / or performs a function based on the command. The speech-enabled environment may be implemented using a network of connected microphone devices distributed throughout various rooms or areas of the environment. These devices may use hot words to help identify when a given utterance is directed to the system, as opposed to utterance directed to another individual present in the environment. Thus, the device may operate in a sleep or hibernation state and only wake up when a detected utterance contains the hot word. Once woken up, the device may proceed to perform more expensive processing, such as full-on-device automatic speech recognition (ASR) or server-based ASR. Summary of the Invention [Means for solving the problem]

[0003] One aspect of the present disclosure provides a computer-implemented method for soft hotword acceptance. The computer-implemented method, when executed by data processing hardware, causes the data processing hardware to perform operations. The operations include receiving audio data characterizing soft hotword events detected by a hotword detector in streaming audio captured by a user device. The operations also include processing the audio data to determine that the audio data corresponds to a query specifying an action to perform at the user device. The operations include providing a notification of output from the user device without triggering performance of the action at the user device or another device, the notification prompting a user associated with the user device to provide a positive input indication to trigger performance of the action at the user device or another device, and instructing the user device or another device not to perform the action specified by the query when the user does not provide the positive input indication.

[0004] Another aspect of the present disclosure provides a system for soft acceptance of hotwords. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving audio data characterizing a soft hotword event detected by a hotword detector in streaming audio captured by a user device. The operations also include processing the audio data to determine that the audio data corresponds to a query specifying an action to perform at the user device. The operations include providing a notification of output from the user device without triggering performance of the action at the user device or another device, the notification prompting a user associated with the user device to provide a positive input indication to trigger performance of the action at the user device or another device, and instructing the user device or another device not to perform the action specified by the query when the user does not provide the positive input indication.

[0005] Implementations of the method or system may include one or more of the following features: In some implementations, a hotword detector resides on the user device and is configured to detect a soft hotword event when a confidence score indicating a likelihood that an audio stream captured by the user device contains the presence of a hotword meets a first threshold score but does not meet a second threshold score. In some examples, the operations further include determining whether a positive input indication is received within a predetermined duration after providing a notification about the output from the user device, and determining that the user has not provided a positive input indication when the positive input indication is not received within the predetermined duration. In some configurations, the operations include receiving a negative input indication indicating that the user does not want the user device or other device to perform the action specified by the query, and determining that the user has not provided a positive input indication in response to receiving the negative input indication. Here, the notification provided about the output from the user device also prompts the user to provide a negative input indication indicating that the user does not want the user device or other device to perform the action specified by the query. In some implementations, providing a notification about the output from the user device includes instructing a graphical user interface executing on the user device to render the notification as a visual notification for display on a screen of the user device. In these implementations, the operations further include instructing the graphical user interface to remove the visual notification displayed on the screen of the user device when the user does not provide an affirmative input indication. Providing a notification about the output from the user device may include instructing the user device to output the notification as an audible notification from a speaker of the user device.

[0006] The method or system may also include receiving an affirmative input indication from the user and, in response to receiving the affirmative input indication, triggering performance of the action specified by the query at the user device or other device. Receiving the affirmative input indication from the user may include receiving speech input from the user and determining that the speech input includes one or more terms that characterize an intent of the user device or other device to perform the action specified by the query. In some examples, receiving the affirmative input indication from the user includes receiving a user input indication indicating a selection of an affirmative graphical element displayed on a screen of the user device.

[0007] Some configurations of the method or system include an operation of, after providing notification of the output from the user device, activating a warm word model configured to detect the presence of at least one of one or more spoken terms characterizing a positive input indication or one or more other spoken terms characterizing a negative input indication, where the negative input indication indicates that the user does not want the user device or another device to perform the action specified by the query, without performing speech recognition. In some examples, the operation may include, after providing notification of the output from the user device, instructing an automatic speech recognizer in communication with the data processing hardware to operate at a reduced capability sufficient to recognize at least one of one or more spoken terms characterizing a positive input indication or one or more other spoken terms characterizing a negative input indication, where the negative input indication indicates that the user does not want the user device to perform the action specified by the query. Additionally or alternatively, the operation of the method or system may further include deleting any storage of the received audio segment when the user does not provide a positive input indication.

[0008] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0009] [Figure 1A] FIG. 1 is a schematic diagram of an exemplary speech environment. [Figure 1B] FIG. 1 is a perspective view of an exemplary speech environment. [Figure 1C] FIG. 1 is a perspective view of an exemplary speech environment. [Figure 1D] FIG. 1 is a perspective view of an exemplary speech environment. [Figure 2] FIG. 1B is a schematic diagram of the exemplary soft acceptor of FIG. 1A. [Figure 3] 1 is a flowchart of an exemplary configuration of operations of a method for soft acceptance of hot words. [Figure 4] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE INVENTION

[0010] Like reference symbols in the various drawings indicate like elements.

[0011] A voice-enabled device (e.g., a user device running a voice assistant) allows a user to vocalize queries or commands and process answers to the queries and / or execute functions based on the commands. The use of “hot words” (also known as “keywords,” “attention words,” “activation phrases / words,” “trigger phrases,” “invocation phrases,” or “voice action initiation commands”) allows predetermined terms / phrases to be spoken to attract the attention of a voice-enabled device, allowing the voice-enabled device to distinguish between speech directed to the system (i.e., to initiate a wake-up process to process one or more terms following the spoken hot word) and speech directed to an individual in the environment. Typically, a voice-enabled device operates in a sleep state to conserve battery power and does not process input audio data unless the input audio data follows a spoken hot word. For example, while in the sleep state, the voice-enabled device captures input audio via a microphone and uses a hot word detector trained to detect the presence of a hot word in the input audio. When a hot word is detected in the input audio, the voice-enabled device initiates a wake-up process to process the hot word and / or any other terms in the input audio that follow the hot word.

[0012] Hot word detection is like searching for a needle in a haystack because it requires continuously listening to streaming audio and triggering accurately and immediately when the presence of a hot word is detected in the streaming audio. In other words, the hot word detector is responsible for ignoring the streaming audio unless the presence of a hot word is detected. To address the complexity of detecting the presence of hot words in a continuous audio stream, hot word detectors typically employ neural networks. Typically, hot word detectors include a component that converts raw audio signals captured by a microphone on a user device into one or more audio features formatted for processing by a neural network. The neural network is trained to recognize audio sequences containing hot words.

[0013] With the proliferation of voice-enabled devices, it is generally necessary for these devices to be effective at speech processing (e.g., speech recognition or text-to-speech). Because speech processing depends on the ability of a hotword detector to reliably initiate speech processing when a hotword is detected, determining whether a hotword is detected in speech received by a hotword detector can affect a user's experience using the voice-enabled device. In other words, if a user speaks a hotword followed by an action requested of the voice-enabled device to perform (e.g., "Hey Google, play the Summer 2021 playlist"), but the hotword detector fails to detect the presence of the hotword, no speech processing occurs to understand and perform the requested action. In this situation, the user may repeat what they spoke, which can create a frustrating experience for the user. This error can cause users to lose trust in the capabilities of the voice-enabled device. Unfortunately, without user trust, voice-enabled devices may be underutilized.

[0014] One current problem with hotword detection is that the process is typically binary. That is, a hotword detector listens to an audio stream and either commits the audio stream to speech processing or does not commit the audio stream to speech processing. By committing or not, the hotword detector is essentially signaling that it recognizes either the presence of a hotword in the audio stream or the absence of a hotword in the audio stream. However, in practice, the hotword detector assigns a score to each sound byte or audio segment in the audio stream that corresponds to spoken speech, where the score represents the probability or likelihood that the hotword is present in the audio data. This results in a confidence gradient, with one end of the gradient representing the highest confidence that the hotword is present in the audio data and the other end representing the lowest confidence that the hotword is present in the audio data. Currently, in a binary approach, the confidence gradient is split into two outcome categories based on the confidence score: a first outcome of hotword acceptance or a second outcome of hotword rejection.

[0015] Unfortunately, having only two possible hotword outcomes (e.g., an accept outcome and a reject outcome) inherently induces responses that do not necessarily align with the confidence level regarding whether the audio data contains the hotword. That is, if a hotword detector assigns a 55% hotword detection score to an audio segment of audio data, representing a 55% confidence level that the audio segment contains the hotword, the hotword system does not take any subsequent action to reflect the 55% confidence level. Rather, because a 55% confidence level indicates that the hotword detector is more certain that the hotword is present than not, an audio segment with a 55% confidence level in a binary approach is simply categorized or classified as a hotword accept. In a binary approach, confidence scores closer to the middle of the confidence gradient (e.g., 40-60% confidence) fall into response categories that may not accurately classify the actual confidence level. In other words, if a hotword system is 51% confident that a hotword is present in a particular audio segment, it can simply classify this confidence as an acceptance result in the same way as another audio segment in which it is 99% confident that the hotword is present. Thus, the inherent inconsistency with a binary approach increases the likelihood that a hotword detector will mistakenly accept or reject an audio segment. In other words, a binary hotword detector will lead to false acceptance (i.e., categorizing an audio segment as an acceptance result despite the hotword not being present) or false rejection (i.e., categorizing an audio segment as a rejection result despite the hotword being present) errors.

[0016] In a binary approach, the hot word detector is configured with a threshold confidence that distinguishes between an acceptance result and a rejection result. For example, the threshold confidence can be set to 50% confidence so that any audio segment above 50% confidence is accepted and any audio segment below 50% confidence is rejected. In some examples, the threshold includes acceptance, meaning that audio segments with exactly 50% confidence are accepted. In other examples, the threshold instead includes rejection, meaning that audio segments with exactly 50% confidence are rejected. In many cases, the threshold confidence of a binary hot word detector attempts to balance errors on both sides. In other words, the threshold confidence is set to a confidence that minimizes the number of false acceptances and false rejections. Because the threshold is adjusted to minimize errors (i.e., false acceptances and false rejections), moving the threshold confidence in either direction does not reduce the rate of errors. That is, moving the threshold from 50% confidence to 55% confidence may reduce false accept errors but increase false reject errors, while moving the threshold from 50% confidence to 45% confidence may reduce false reject errors but increase false accept errors.

[0017] Because moving the threshold confidence simply trades off errors rather than reducing the overall number of errors, an approach that considers borderline confidence cases is needed. To address some of these issues, the approach described herein is a non-binary approach that introduces an outcome category called soft accept. This outcome category may be added to the accept and reject outcome categories. Soft accept refers to an outcome in which a hot word detector accepts that an audio segment contains the presence of a hot word, provided that a response (e.g., a responsive action) to a request or query contained in the audio stream is at least initially limited. For example, soft accept requires that an action to satisfy a query (i.e., a request or command) cannot be performed until the action is affirmatively accepted. Once an action is affirmatively accepted, the action is subsequently performed.

[0018] Generally, when a hotword detector in a speech processing system accepts the presence of a hotword in an audio stream, the hotword detector commits the audio stream for full speech processing. Full speech processing determines whether the audio stream requests a specific response (e.g., a responsive action) from the voice-enabled device. When the audio stream requests a specific response, the system in the voice-enabled device is typically configured to automatically process and / or execute the response. For example, if a user speaks the hotword "Hey, Google" followed by the requested action "Play the Summer 2021 Playlist" (i.e., utterance "Hey, Google, play the Summer 2021 Playlist" as shown in FIG. 1A), the voice-enabled device recognizes the hotword (e.g., via the hotword detector), opens a music playback application containing the "Summer 2021 Playlist," and plays the "Summer 2021 Playlist" using the music playback application (e.g., Google Play Music). In contrast, with soft acceptance, the user may not have spoken the hotword "Hey, Google" very clearly. Due to this lack of clarity, the hot word detector may indicate that the audio segment corresponding to "Hey Google" has a hot word confidence corresponding to a soft-accept result category. In the soft-accept result category, the hot word detector also commits to full speech processing of the audio "Play the Summer 2021 Playlist." In this soft-accept example, instead of generating (e.g., automatically generating) a response to play the "Summer 2021 Playlist," the system responds with a qualified response (or soft answer). For example, a qualified response is a response that notifies the user via a voice-enabled device of the system's interpreted response to the utterance. In some examples, this soft answer is a notification requesting user permission to proceed with the system's interpreted response.For example, the system generates a visual response (e.g., a prompt) on a voice-enabled device that says, "Play the Summer 2021 playlist on Google Play Music?" Here, the system does not generate the interpreted response (i.e., play the Summer 2021 playlist on Google Play Music) until the user grants permission for the interpreted response (e.g., performs an interaction that affirmatively approves the visual response).

[0019] By constraining the system's initial response to a limited response (e.g., requiring affirmative approval of an action from a query before executing the action), soft-accept results hedge against low hotword confidence, resulting in a limited response that may be less noticeable to the user than an inaccurate, complete response. In this sense, the system generates responses that reflect the hotword's confidence. In the "Hey Google, play the Summer 2021 playlist" example, the user may have actually uttered "You should add 'hey beautiful' to the Summer 2021 playlist" while talking to someone else within audible range of the voice-enabled device (i.e., the utterance was actually side speech, not follow-on speech). A misinterpretation by the system of "Hey Google, play the Summer 2021 playlist" could result in a highly annoying response for the user. In other words, the voice-enabled device may suddenly start playing music from a playlist over the user's ongoing conversation (e.g., as shown in Figure 1B). However, if the actual utterance, "You should add 'hey beautiful' to your summer 2021 playlist," had a hotword confidence that resulted in a soft-accept outcome, the result might simply be that the voice-enabled device generates an unobtrusive prompt on the voice-enabled device's display that says, "Do you want to play the summer 2021 playlist?" and the user either ignores the prompt or performs an interaction that negatively accepts the prompt (e.g., as shown in Figure 1C ).

[0020] In some instances, the interpreted full response (i.e., the action requested by the query) is an auditory response, while the limited response is a visual response. This follows the principle that there is often an asymmetry between visual and auditory responses. Generally, visual responses are disruptive only when the user is looking at the display displaying the visual response, whereas auditory responses are disruptive when the user can hear the auditory response. That is, human sensory perception may naturally make auditory responses more disruptive than visual responses. In other words, many users have a larger sensory area for hearing something (i.e., hearing it but not in their field of view) than for seeing something. Therefore, a visual response such as a notification or a small pop-up window may be a less intrusive response than changing the entire window of a voice-enabled device or audibly asking the user if they want to "play the Summer 2021 playlist."

[0021] The soft-accept result also serves as feedback to the speech processing system associated with the voice-enabled device. For example, the limited response may prompt the user for permission as to whether to proceed with generating a response to a request or query, so the user's granting or denial of permission may provide meaningful feedback to the hotword detector. That is, the user's interaction with the limited response indicates whether a hotword is present or absent. In other words, a user of a voice-enabled device is unlikely to grant permission to generate a response that was not originally requested. In this regard, if the user grants permission to generate a response, the interaction to grant permission may be interpreted by the system as a positive hotword example. The system may then store the positive hotword example to serve as a future training example for a model (e.g., a machine learning model) used by the hotword detector to predict the probability that a hotword is present in audio data input to the hotword detector. In this sense, the user's interaction may label the hotword example as a positive hotword example, generating supervised hotword training examples for the hotword detector (e.g., to improve or update the hotword detector's functionality). In some cases, these positive hot word examples do not serve as hot word training examples for the hot word detector in the broad sense, but rather serve to train the hot word detector to perform more accurate soft acceptance (i.e., serve as soft acceptance training examples).

[0022] Additionally or alternatively, the soft-accept output may be feedback to help a user device fine-tune the hotword sensitivity associated with a hotword detector. For example, the hotword detector may evaluate the frequency of soft-accept results compared to how many of the soft-accept results were identified as legitimate queries and, further, how many of the legitimate queries resulted in a positive user acknowledgement. That is, if the speech processing system fails to identify queries associated with a soft-accept result at a certain percentage (e.g., a threshold percentage), the hotword detector may be configured to modify the soft-accept confidence threshold to minimize or reduce that percentage. In other words, a soft-accept result likely occurs when a query is not present in audio data containing a soft hotword. In such a situation, it is likely that the hotword was not present in the audio data because the user initiated the query using the hotword as the query's invocation phrase. Without the query, the hotword is unlikely to be present, and therefore this situation may serve as feedback to teach the hotword detector that its soft-accept results are overly inclusive (i.e., contain false acceptances). In a similar regard, if a user does not affirmatively approve (or negatively approve) a query corresponding to a soft-accept result, this situation may represent that the user device picked up side speech rather than speech actually directed at the user device. Thus, the hot word detector may adjust one or more of its confidence thresholds corresponding to soft-accept results based on the frequency of occurrence of a lack of affirmative approval for an action resulting from the soft-accept result.

[0023] 1A-1D , in some examples, a speech environment (i.e., system) 100 includes a user 10 speaking an utterance 20 within audible range of a voice-enabled device 110 (also referred to as device 110 or user device 110) executing a digital assistant interface 120. Here, the utterance 20 spoken by the user 10 may be captured by the device 110 in streaming audio 12 and may correspond to a query 22 to perform an action, more specifically, a query 22 to the digital assistant interface 120 to perform an action. The user 10 can prefix the query 22 with a hotword 24 (e.g., an invocation phrase) to trigger the device 110 from a sleep or hibernation state when the hotword 24 is detected in the streaming audio 12 by a hotword detector (e.g., soft acceptor 200) running on the device 110 during the sleep or hibernation state. The action may also be referred to as an operation or a task. In this sense, the user 10 can conversationally interact with the digital assistant interface 120 running on the voice-enabled device 110 to perform computing activities or find answers to questions.

[0024] Device 110 may correspond to any computing device associated with user 10 and capable of capturing audio from environment 100. In some examples, user device 110 includes, but is not limited to, mobile devices (e.g., mobile phones, tablets, laptops, e-readers, etc.), computers, wearable devices (e.g., smart watches), music players, casting devices, smart appliances (e.g., smart TVs) and Internet of Things (IoT) devices, remote controls, smart speakers, etc. Device 110 includes data processing hardware 112 and memory hardware 114 that communicates with data processing hardware 112 and stores instructions that, when executed by data processing hardware 112, cause data processing hardware 112 to perform one or more operations related to speech processing.

[0025] Device 110 further includes an audio subsystem 116 having an audio capture device (e.g., an array of one or more microphones) for capturing and converting audio in speech environment 100 into electronic signals (e.g., audio data 14). In the illustrated example, device 110 implements audio capture device 116 (also commonly referred to as microphone 116), although audio capture device 116 may not be physically present on device 110 but may communicate with the audio subsystem (e.g., peripherals of device 110). For example, device 110 may correspond to a vehicle infotainment system that utilizes an array of microphones positioned throughout the vehicle. In another example, audio capture device 116a may reside on a separate device that communicates with user device 110 to perform actions. Additionally, audio subsystem 116 may include a playback device (e.g., one or more speakers, etc.) for playing audio (e.g., synthesized audio or synthesized speech) generated by user device 110.

[0026] Device 110 may also include display 118 for displaying graphical user interface (GUI) elements (e.g., windows, screens, icons, menus, etc.) and / or graphical content. For example, device 110 may load or launch an application (local or remote) that generates GUI elements or other graphical content for display 118. Additionally, the elements generated within display 118 may be selectable by user 10 and may also serve to provide some form of visual feedback for processing activities and / or operations occurring on device 110. Furthermore, because device 110 is voice-enabled, user 10 can interact with elements generated on display 118 using various voice commands. For example, display 118 may display an options menu for a particular application, and user 10 can select options via speech using interface 120.

[0027] Speech-enabled interface (e.g., digital assistant interface) 120 may process queries 22 or commands conveyed in spoken utterances 20 captured by device 110. Speech-enabled interface 120 (also referred to as interface 120 or assistant interface 120) generally facilitates receiving audio data 14 corresponding to the utterances 20 and coordinating speech processing or other activity resulting from the utterances 20 on the audio data 14. Interface 120 may execute on data processing hardware 112 of device 110. Interface 120 may stream audio data 14, including the utterances 20, to various systems related to speech processing or query fulfillment.

[0028] Additionally, device 110 is configured to communicate with remote system 140 via network 130. Remote system 140 may include scalable remote resources 142, such as remote data processing hardware 144 (e.g., a remote server or CPU) and / or remote memory hardware 146 (e.g., a remote database or other storage hardware). Device 110 can utilize remote resources 142 to perform various functions related to speech processing (e.g., by speech processing system 150) and / or soft acceptance (e.g., by soft acceptor 200). For example, device 110 is configured to perform speech recognition using speech recognition system 152 and / or speech interpretation using speech interpreter 154. In some examples, not shown, device 110 can further convert speech to text-to-speech (TTS) during speech processing using a TTS system.

[0029] The systems 150, 200 may reside on the device 110 (referred to as on-device systems) or may reside remotely (e.g., reside on a remote system 140) but communicate with the device 110. In some examples, some of these systems 150, 200 reside locally or on-device, while other systems reside remotely. In other words, any of these systems 150, 200 can be local or remote in any combination. For example, when the size or processing requirements of the systems 150, 200 are significant, the systems 150, 200 may reside on the remote system 140. Furthermore, when the device 110 can support the size or processing requirements of one or more systems 150, 200, one or more systems 150, 200 may reside on the device 110 using the data processing hardware 112 and / or memory hardware 114. Optionally, one or more systems 150, 200 may reside both locally / on-device and remotely. For example, one or more of systems 150, 200 may be defaulted to run on remote system 140 when a connection to network 130 between device 110 and remote system 140 is available, but when the connection is lost or network 130 is unavailable, systems 150, 200 instead run locally on device 110.

[0030] The device 110 is also configured to communicate with a speech processing system 150. The speech processing system 150 can generally perform various functions related to speech processing, such as speech recognition and speech interpretation (also referred to as query interpretation). For example, the speech processing system 150 of FIG. 1A is shown to include a speech recognizer 152 that performs automatic speech recognition (ASR), a speech interpreter 154 that determines the meaning of the recognized speech (i.e., understands the speech), and a search engine 156 that retrieves any search results in response to a query identified in the recognized speech. When the soft acceptor 200 detects a soft hotword event 202, the soft acceptor 200 passes the audio data 14 to the speech processing system 150. The soft hotword event 202 indicates that the soft acceptor 200 accepts a portion of the audio data 14 (e.g., a first audio segment) as a hotword 24. When a portion of the audio data 14 is identified as a hot word 24, the soft acceptor 200 communicates the audio data 14 as a soft hot word event 202 so that the speech processing system 150 can perform speech processing on the audio data 14. By performing speech processing on the audio data 14, the speech recognizer 152, in combination with the speech interpreter 154, can determine whether a second audio segment of the audio data 14 (e.g., denoted as query 22) is indicative of a spoken query-type utterance.

[0031] The speech recognizer 152 receives as input audio data 14 corresponding to the soft hotword event 202 and transcribes the audio data 14 into a transcript as output, referred to as a speech recognition result R. Generally speaking, by converting the audio data 14 into a transcript, the speech recognizer 152 enables the device 110 to recognize when a spoken utterance 20 from the user 10 corresponds to a query 22 (or command) or some other form of audio communication. A transcript refers to a sequence of text that the device 110 (e.g., the assistant interface 120 or the speech processing system 150) can use to generate a response to the query or command. The speech recognizer 152 and / or the interface 120 can provide the speech recognition result R to a speech interpreter 154 (e.g., a natural language understanding (NLU) module) to perform semantic interpretation on the result R to determine whether the audio data 14 includes a query 22 requesting the performance of a particular action 148. In other words, the speech interpreter 154 identifies queries 22 or commands within the audio data 14 and generates an interpretation I of the results R to enable the speech processing system 150 to respond to the queries 22 with corresponding actions 158 invoked by the queries 22. For example, if the query 22 is a command to play music, the corresponding action 158 invoked by the query 22 is to play the music (e.g., by executing an application capable of playing music). In some examples, the speech processing system 150 employs a search engine 156 to retrieve search results that enable the speech processing system 150 to respond to the queries 22 (i.e., satisfy the queries 22).

[0032] The soft acceptor 200 generally functions as a type of hotword detector. That is, the soft acceptor 200 is configured to detect the presence of hotwords 24 in the streaming audio 12. The soft acceptor 200 may be configured to be “always on” to detect the presence of hotwords. The soft acceptor 200 may include a hotword detection model, such as a neural network-based model, configured to detect acoustic features indicative of hotwords 24 without performing speech recognition or semantic analysis. By using the soft acceptor 200, the detection of hotwords 24 may be performed in low-power hardware, such as a digital signal processor (DSP) chip, avoiding consumption of the device's computing processing unit (CPU) (e.g., associated with the data processing hardware 112). In some examples, the functionality of the soft acceptor 200 may be divided into one or more stages of hotword detection. For example, a first stage that initially detects hotword candidates and a second stage that confirms the hotword candidates as hotwords. Even with multiple stages, when the detector detects a hot word 24, the hot word 24 may trigger the device to wake up and begin speech recognition that requires more expensive processing (such as ASR and natural language understanding (NLU)). Here, the device can perform on-device ASR by running a speech recognizer 152 on data processing hardware 112 (e.g., a CPU). Optionally, device 110 can establish a network connection with a server (e.g., remote system 140 of FIG. 1A ) and provide audio data 14 to the server to perform server-side ASR and / or NLU on the audio data 14.

[0033] Additionally, the soft acceptor 200 differs from conventional hotword detectors in that it can categorize hotword detection results as soft-acceptance results, more specifically, as soft hotword events 202. Here, a soft hotword event 202 refers to a situation in which the soft acceptor 200 determines that audio data 14 contains a hotword 24, but the soft acceptor 200 designates the hotword detection as a soft acceptance rather than a full acceptance due to the confidence level of the hotword detection. With either a full acceptance or a soft acceptance, the soft acceptor 200 passes the audio data 14 from the streaming audio 12 to the speech processing system 150 for processing by the system 150 to determine whether the audio data 14 corresponds to a query 22 specifying an action 148 to perform on the device 110. The difference between full acceptance and soft acceptance is that the device 110 is typically permitted to perform (e.g., automatically perform) the action 148 specified by the query 22 associated with the fully accepted hotword, whereas with soft acceptance, the device 110 cannot perform the action 148 specified by the query 22 associated with the soft acceptance until the soft acceptor 200 generates a notification 204 prompting the user 10 associated with the device 110 to provide a positive input indication 16 (i.e., affirmatively acknowledge the notification 204) to trigger the execution of the action 148 on the device 110. Thus, in a soft hotword event 202, if the user 10 does not provide a positive input indication 16, the action 148 does not occur.

[0034] 1B-1D illustrate some differences between traditional hot word detection (e.g., FIG. 1B) and hot word detection using a soft acceptor 200 (e.g., FIGS. 1C and 1D). In FIG. 1B, a first user 10, 10a, is having a conversation with a second user 10, 10b, both of whom are sitting at a table. A user device 110 is currently residing on a sideboard within audible range of the conversation between the users 10. During the conversation, the first user 10a utters a first utterance 20, 20a, saying, "I've started rewatching 'How I Met Your Mother.' All the actors look so young." In response to the first user 10a's utterance, the second user 10b utters a second utterance 20, 20b, saying, "Wow, you should definitely add the song 'Hey Beautiful' to your summer 2021 playlist." By mentioning the show "How I Met Your Mother," the first user 10a reminds the second user 10b that they should add the theme song from "How I Met Your Mother" to their summer 2021 playlist. Unfortunately, in this scenario, the hotword detector associated with the user device 110 mistakes the term "Hey Beautiful" for the hotword "Hey Google." Because the hotword detector of the device 110 in FIG. 1B is a purely binary hotword detector without soft acceptance, when a hotword is detected, the speech processing system of the device 110 automatically executes the action 148 associated with the detected hotword. Here, the device 110 interprets the action 148 as meaning that the device 110 should play the "Summer 2021 Playlist." The speaker paired with the device 110 then begins playing music from the "Summer 2021 Playlist." Unfortunately, neither the first user 10a nor the second user 10b wanted to listen to any music, so their conversation now competes with the unwanted music.

[0035] 1C and 1D, users 10a-b are having the same conversation, but instead of device 110 automatically performing action 148 (i.e., playing music), device 110 displays notification 204, "Want to play the Summer 2021 playlist?", asking whether action 148 should be performed. Because user 10 did not request a playlist, user 10 can either ignore notification 204 on device 110 or provide input 16 (i.e., negative input) to user device 110 indicating that action 148 is not desired. In some examples, if notification 204 is ignored, notification 204 is automatically removed from display 118 of device 110 after a certain time. That is, device 110 provides user 10 with a finite time window to affirmatively approve notification 204 (i.e., accept or grant permission) to perform action 148. Otherwise, action 148 may be forgotten (e.g., deleted from memory) without any approval. In this sense, if users 10a-b are engrossed in their conversation and do not intend to cause action 148 to occur, device 110 generates an unobtrusive notification 204 that is not intrusive and disappears when the time window expires. In some examples, the time window may be dynamic, such that the time window is extended or shortened depending on the user's current state. For example, device 110 may extend the time window when it senses that user 10 is approaching device 110 to affirmatively acknowledge notification 204.

[0036] FIG. 1D illustrates that the graphical size of the notification 204 can vary. For example, the size of the notification 204 can be proportional to the confidence of the hotword determined by the soft acceptor 200. For example, the number of pixels occupied by the notification 204 can be proportional to the confidence of the hotword. In FIG. 1D, the first notification 204, 204a for the action 148 is shown to be fairly small and merely serve as a way to visually identify that a potential action 148 is waiting. In some implementations, the user 10 can select the notification 204, which causes the notification 204 to change (e.g., enlarge or expand an interactable icon / button). For example, in FIG. 1D, the first notification 204, 204a is a small-corner graphical user interface (GUI) element. In this example, the user 10 selects a small GUI element in the corner (e.g., represented by a hand icon) to cause the device 110 to display a second notification 204, 204b, which is a pop-up window with a prompt asking whether the user 10 wants the user device 110 to “Play the Summer 2021 Playlist.” Now, when a second user input 16, 16A2 selects the “YES” icon in the second notification 204, 204b, the device 110 performs an action 148 in response to the user selection input 16.

[0037] Referring to FIG. 2 , in some implementations, the soft acceptor 200 includes a scorer 210 and an activator 220. When the soft acceptor 200 receives audio data 14 corresponding to an utterance 20, the scorer 210 is configured to generate a confidence score 212 for a portion of the audio data 14 (e.g., a first audio segment of the audio data 14). The confidence score 212 indicates the likelihood that the audio data 14 contains the presence of a hotword 24. In some examples, the scorer 210 corresponds to a hotword detection model (e.g., a neural network) trained to receive the audio data 14 as input and generate as output a probability of predicting whether the portion of the audio data 14 contains the hotword 24. This means that, as shown in FIG. 2 , the individual confidence scores 212 lie along a confidence gradient ranging from 100% confidence that the audio data 14 contains the hotword 24 to 0% confidence that the audio data 14 contains the hotword 24. The scorer 210 can then divide this gradient into different result categories. 2 depicts the confidence gradient of the scorer 210 with three result categories: accept, soft accept, and reject, where each of these result categories is separated by confidence thresholds 214, 216. The confidence thresholds 214, 216 refer to the confidence score 212 that distinguishes one result category from another. The first confidence threshold 214 refers to the confidence score 212 that distinguishes between when the scorer 210 accepts the presence of the hot word 24 in the audio data 14 and when the scorer 210 softly accepts the presence of the hot word 24 in the audio data 14. For example, if the confidence score 212 for an audio segment of the audio data 14 is greater than the first confidence threshold 214, the scorer 210 passes the audio data 14 to the speech processing system 150 to determine whether the audio data 14 corresponds to a query 22 that specifies an action 148 to be performed on the user device 110.If the audio data 14 corresponds to a query 22 that specifies an action 148, the soft acceptor 200 enables the action 148 to be automatically executed because the confidence score 212 was greater than the first confidence threshold 214 and was within the range of the confidence score 212 corresponding to the accepted result category.

[0038] Meanwhile, the second confidence threshold 216 refers to the confidence score 212 that distinguishes between when the scorer 210 rejects the presence of the hot word 24 in the audio data 14 and when the scorer 210 softly accepts the presence of the hot word 24 in the audio data 14. For example, if the confidence score 212 for an audio segment of the audio data 14 is less than the second confidence threshold 216, the scorer 210 interprets this confidence score 212 as a rejection result category, and therefore the scorer 210 does not pass the audio data 14 to the speech processing system 150, which means that the audio data 14 is unlikely to contain the presence of the hot word 24. Thus, the first confidence threshold 214 and the second confidence threshold 216 serve as upper and lower limits of a confidence score range that defines a soft-accept result category. In other words, the scorer 210 determines that the audio data 14 characterizes a soft hotword event 202 when it determines that the confidence score 212 is between a first confidence threshold 214 and a second confidence threshold 216. The scorer 210 determines that the confidence score 212 corresponds to a soft hotword event 202 when the confidence score 212 meets (e.g., exceeds) the second confidence threshold 216 but does not meet (e.g., fails to exceed) the first confidence threshold 214. For a soft hotword event 202, similar to the accept result category, the scorer 210 then passes the audio data 14 to the speech processing system 150 to determine whether the audio data 14 corresponds to a query 22 specifying an action 148 to perform at the user device 10.

[0039] Optionally, the first confidence threshold 214 and the second confidence threshold 216 may be dynamic in nature. For example, the scorer 210 adjusts these confidence thresholds 214, 216 based on the current acoustics in the vicinity of the device 110. For example, the user 10 may be hosting a large cocktail party. Typically, the first and second confidence thresholds 214, 216, which determine the boundaries of a soft-accept result, are user-defined, administrator-defined, or feedback-defined. In a large cocktail party situation with high background noise, there may be an increased amount of ambient noise that distorts the intended hotword 24 and query 22, or an increased amount of conversation that may result in a soft hotword event 202 without a corresponding query 22. For such situations, the soft acceptor 200 may be configured to dynamically adjust the confidence thresholds 214, 216 to account for the background acoustics. For example, the first confidence threshold 214 and the second confidence threshold 216 may be increased to result in a larger rejection result range.

[0040] In some examples, even if the soft acceptor 200 passes audio data 14 to the speech processing system 150 indicating that the audio data 14 includes the presence of a hotword 24, the speech processing system 150 (e.g., the voice interpreter 154) may nevertheless determine that the query 22 is not present in the audio data 14. For example, contrary to what is shown in FIGS. 1B-1D , the speech processing system 150 may determine that the utterance 20, “You should definitely add the song ‘Hey Beautiful’ to your summer 2021 playlist,” does not include the action 148. In this situation, because the speech processing system 150 does not identify the action 148, the soft acceptor 200 does not generate a notification 204 in response. Generally speaking, the soft acceptor 200 may prefer to have a soft hotword event 202 when the audio data 14 corresponds to the action 148. With this in mind, the soft acceptor 200 can collect occurrences when a soft hotword event 202 is designated but the speech processing system 150 cannot identify an action 148 corresponding to the soft hotword event 202. The collected occurrences can then be provided as feedback to the soft acceptor 200. For example, each occurrence serves as a training example for updating the hotword detection model. In some configurations, training the hotword detection model using the training examples adjusts one or more of the first confidence threshold 214 or the second confidence threshold 216. By adjusting these thresholds 214, 216, the soft acceptor 200 can be configured to limit or narrow the range of confidence scores 212 corresponding to soft-acceptance result categories.

[0041] When the speech processing system 150 determines that the audio data 14 corresponds to a query 22 that specifies an action 148 to be performed on the user device 110, the speech processing system 150 communicates the action 148 to the activator 220 of the soft acceptor 200. Upon receiving the action 148, the activator 220 generates a notification 204 for the user 10 of the user device 110 rather than triggering the execution of the action 148. The notification 204 prompts the user 10 to provide some kind of user input indication 16 to trigger the execution of the action 148 or to prevent the execution of the action 148. The notification 204 can take various forms. In some implementations, the notification 204 is a visual notification displayed on the display 118 of the user device 110. In some examples, the notification 204 is a user prompt with a GUI element that allows both negative and positive input indications 16N and 16A. 1C and 1D show a notification that includes a "YES" button that, when selected by the user 10, indicates a positive input indication 16A, and a "NO" button that, when selected by the user 10, indicates a negative input indication 16N. In other implementations, the notification 204 is an audible notification, such as a tone or chime (e.g., a discontinuous tone or chime) from a speaker associated with the device 110. For example, the user 10 or a device administrator sets a unique audible sound for the soft acceptor 200. The unique sound allows the user 10 to recognize what the sound means when they hear it and also allows them to ignore the audible notification if desired.

[0042] In response to the notification 204, the activator 220 is configured to determine whether the indication 16 by the user 10 corresponds to an affirmative input indication 16, 16A to affirmatively acknowledge the notification 204 (i.e., trigger the execution of the action 148) or a negative input indication 16, 16N indicating that the user 10 does not want the user device 110 to execute the action 148 specified by the query 22. In some configurations, when the activator 220 receives the affirmative input indication 16A, the activator 220 is further configured to determine whether the activator 220 received the affirmative input indication 16A within a predetermined duration, where the predetermined duration corresponds to a time window that begins when the activator 220 communicates the notification 204 to the user device 110 and ends at a specified end time (e.g., 20 seconds). Once the duration expires, the activator 220 may remove the notification 204 and / or remove the action 148 (along with any other data associated with the action 148, such as, for example, the spoken utterance 20 that resulted in the action 148). In this regard, when the user 10 does not provide an affirmative input indication 16A within the predetermined duration, the activator 220 does not perform the action 148. In contrast, when the activator 220 receives an affirmative input indication 16A within the predetermined duration, the activator 220 allows the action 148 to occur.

[0043] In some implementations, the positive input indication 16A or the negative input indication 16N is an audible command from the user 10 (e.g., rather than a tactile input). In these implementations, when the activator 220 receives speech input from the user 10, the activator 220 determines whether the speech input includes one or more terms that characterize the user device 110's intent to perform or not perform the action 148 specified by the query 22. To assist in receiving the audible command in response to the notification 204, the activator 220 may activate a warm word model without performing speech recognition. Here, the warm word model is configured to detect the presence of either one or more spoken terms that characterize the positive input indication 16A or one or more other spoken terms that characterize the negative input indication 16N. Alternatively, the activator 220 may use the speech recognizer 152 to process the receipt of the audible command. When using the speech recognizer 152, the speech recognizer 152 may operate at a reduced capacity. In this regard, reduced functionality refers to the speech recognizer 152 being at least sufficient to recognize one or more spoken terms that characterize a positive input instruction 16A or one or more other spoken terms that characterize a negative input instruction 16N.

[0044] 3 is a flowchart illustrating an exemplary configuration of operations of a method 300 for softly accepting a hotword 24. At operation 302, the method 300 receives audio data 14 characterizing a soft hotword event 202 detected by a hotword detector 200 in streaming audio 12 captured by the user device 110. At operation 304, the method 300 processes the audio data 14 to determine that the audio data 14 corresponds to a query 22 specifying an action 148 to perform at the user device 110. Without triggering the execution of the action 148 at the user device 110, the method 300 performs two sub-operations 306a, 306b at operation 306a. At operation 306a, the method 300 provides a notification 204 of the output from the user device 110, which prompts a user 10 associated with the user device 110 to provide an affirmative input indication 16 to trigger the execution of the action 148 at the user device 110. At operation 306 b , when the user 10 does not provide an affirmative input indication 16 , the method 300 instructs the user device 110 not to perform the action 148 specified by the query 22 .

[0045] 4 is a schematic diagram of an exemplary computing device 400 that can be used to implement the systems (e.g., assistant interface 120, speech processing system 150, remote system 140, and soft acceptor 200) and methods (e.g., method 300) described herein. Computing device 400 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components, their connections and relationships, and their functions shown herein are merely exemplary and do not limit the implementation of the invention described and / or claimed herein.

[0046] Computing device 400 includes a processor 410 (e.g., data processing hardware 112, 134), a memory 420 (e.g., memory hardware 114, 136), a storage device 430, a high-speed interface / controller 440 connecting to memory 420 and a high-speed expansion port 450, and a low-speed interface / controller 460 connecting to a low-speed bus 470 and storage device 430. Each of components 410, 420, 430, 440, 450, and 460 are interconnected using various buses and may be implemented on a common motherboard or in other manners, as appropriate. Processor 410 can process instructions for execution within computing device 400, including instructions stored in memory 420 or storage device 430, to display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 480 coupled to high-speed interface 440. In other implementations, multiple processors and / or multiple buses may be used, as appropriate, along with multiple memories and multiple types of memory. Also, multiple computing devices 400 may be connected, each providing a portion of the required operations (eg, as a server bank, a group of blade servers, or a multi-processor system).

[0047] The memory 420 stores information non-temporarily within the computing device 400. The memory 420 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-temporary memory 420 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) used by the computing device 400. Examples of non-volatile memory include, but are not limited to, flash memory, read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), disk, or tape, etc.

[0048] The storage device 430 can provide mass storage for the computing device 400. In some implementations, the storage device 430 is a computer-readable medium. In various different implementations, the storage device 430 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices including devices in a storage area network or other configuration. In additional implementations, the computer program product is tangibly embodied on an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as the memory 420, the storage device 430, or memory on the processor 410.

[0049] The high-speed controller 440 manages bandwidth-intensive operations for the computing device 400, and the low-speed controller 460 manages less bandwidth-intensive operations. Such an allocation of duties is merely exemplary. In some implementations, the high-speed controller 440 is coupled to memory 420 (e.g., via a graphics processor or accelerator), a display 480, and to a high-speed expansion port 450 that can accept various expansion cards (not shown). In this implementation, the low-speed controller 460 is coupled to a storage device 430 and a low-speed expansion port 490. The low-speed expansion port 490, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet, etc.), may be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, for example, via a network adapter.

[0050] As shown, computing device 400 may be implemented in several different forms. For example, it may be implemented as a standard server 400a, or multiple times within a group of such servers 400a, as a laptop computer 400b, or as part of a rack server system 400c.

[0051] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be special-purpose or general-purpose, coupled to transmit and receive data and instructions from a storage system, at least one input device, and at least one output device.

[0052] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language and / or in an assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device used to provide machine instructions and / or data to a programmable processor (such as a magnetic disk, optical disk, memory, programmable logic device (PLD), etc.), including, for example, a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0053] The processes and logic flows described herein can be implemented by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be implemented by special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). Processors suitable for executing computer programs include, by way of example, both general-purpose and special purpose microprocessors, and any one or more processors of any type of digital computer. Generally, a processor receives instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic, magneto-optical, or optical disks, or is operatively coupled to receive data from, transfer data to, or both of, the one or more mass storage devices. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0054] To provide for user interaction, one or more aspects of the present disclosure can be implemented on a computer having a display device, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touchscreen, for displaying information to the user, and optionally a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide for user interaction; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from devices used by the user, such as, for example, by sending web pages to a web browser on the user's client device in response to a request received from the web browser.

[0055] Although several implementations have been described, it will be understood that various modifications can be made without departing from the scope and spirit of the present disclosure. Accordingly, other implementations are within the scope of the following claims. [Explanation of symbols]

[0056] 12. Streaming Audio 14 Audio Data 16 Positive input instruction 16A Positive input indication 16N Negative input instruction 20 utterances 22 queries 24 Hot Words 100 System, Speech Environment 110 User Devices, Voice-Enabled Devices 112 Data Processing Hardware 114 Memory Hardware 116 Audio subsystem, audio capture devices, microphones 118 Display 120 Interface 130 Network 134 Data Processing Hardware 136 Memory Hardware 140 Remote Systems 142 Remote Resources 144 Remote Data Processing Hardware 146 Remote Memory Hardware 148 Actions 150 Speech Processing System 152 Speech Recognition System 154 Speech Interpreter 156 search engines 200 Hot Word Detector, Soft Acceptor, System 202 Soft Hot Word Event 204 Notification 210 Scorer 212 Trust Score 214 First Threshold Score 216 Second Threshold Score 220 Activator 300 ways 400 computing devices 400a Standard Server 400b laptop computer 400c Rack Server System 410 Data Processing Hardware, Processors 420 Memory Hardware 430 Storage Devices 440 High-Speed Interface / Controller 450 High-Speed Expansion Port 460 Low-Speed Interface / Controller 470 Slow Bus 480 display 490 Low-Speed Expansion Port

Claims

1. A computer-implemented method (300), when executed by data processing hardware (410), causing the data processing hardware (410) to: receiving audio data (14) corresponding to speech captured in the streaming audio (12) by a user device (110); generating a confidence score indicative of the likelihood of hot words being present in the audio data by a hot word detector (200); determining, based on the confidence score, that the received audio data (14) characterizes a soft hotword event (202); processing the audio data (14) to determine that the audio data (14) corresponds to a query (22) specifying an action (148) to perform at the user device (110); without triggering the execution of the action (148) on the user device (110) or another device; providing a notification (204) of the output from the user device (110), the notification (204) prompting a user (10) associated with the user device (110) to provide a positive input indication (16) to trigger execution of the action (148) on the user device (110) or the other device, the notification being a visual notification having a size proportional to the confidence score; instructing the user device (110) or the other device not to perform the action (148) specified by the query (22) when the user (10) does not provide the affirmative input indication (16); A computer-implemented method (300) for performing operations including:

2. 2. The computer-implemented method of claim 1, wherein the hotword detector resides on the user device and is configured to detect the soft hotword event when a confidence score indicating a likelihood that the streaming audio captured by the user device contains the presence of a hotword satisfies a first threshold score but does not satisfy a second threshold score.

3. The operation is determining whether the positive input indication (16) is received within a predetermined duration of time after providing the notification (204) of the output from the user device (110); determining that the user (10) has not provided the affirmative input indication (16) when the affirmative input indication (16) is not received within the predetermined duration; and 3. The computer-implemented method (300) of claim 1 or 2, further comprising:

4. The operation is receiving a negative input indication (16N) indicating that the user (10) does not want the user device (110) or the other device to perform the action (148) specified by the query (22); determining, in response to receiving the negative input indication (16N), that the user (10) did not provide the positive input indication (16); 4. The computer-implemented method (300) of claim 1, further comprising:

5. 5. The computer-implemented method of claim 4, wherein the notification provided about the output from the user device also prompts the user to provide the negative input indication indicating that the user does not want the user device or another device to perform the action specified by the query.

6. 6. The computer-implemented method of claim 1, wherein providing the notification of the output from the user device comprises instructing a graphical user interface executing on the user device to render the notification as a visual notification for display on a screen of the user device.

7. 7. The computer-implemented method of claim 6, wherein the actions further include instructing the graphical user interface to remove the visual notification displayed on the screen of the user device when the user does not provide the affirmative input indication.

8. 8. The computer-implemented method of claim 1, wherein providing the notification of output from the user device comprises instructing the user device to output the notification as an audible notification from a speaker of the user device.

9. The operation is receiving the affirmative input indication (16) from the user (10); triggering execution of the action (148) specified by the query (22) at the user device (110) or the other device in response to receiving the positive input indication (16); 9. The computer-implemented method (300) of any one of claims 1 to 8, further comprising:

10. receiving the affirmative input indication (16) from the user (10); receiving speech input from the user (10); determining that the speech input includes one or more terms that characterize an intent of the user device (110) to perform the action (148) specified by the query (22); and 10. The computer-implemented method (300) of claim 9, comprising:

11. 11. The computer-implemented method (300) of claim 9 or 10, wherein receiving the affirmative input indication (16) from the user (10) comprises receiving a user input indication indicating a selection of an affirmative graphical element displayed on a screen (118) of the user device (110).

12. After the operation provides the notification (204) about the output from the user device (110), without performing speech recognition, one or more spoken terms characterizing said positive input indication (16); or one or more other spoken terms that characterize a negative input instruction (16N), the negative input instruction (16N) indicating that the user (10) does not want the user device (110) or the other device to perform the action (148) specified by the query (22).

12. The computer-implemented method (300) of claim 1, further comprising: invoking a warm word model configured to detect the presence of at least one of:

13. After the operation provides the notification (204) of the output from the user device (110), one or more spoken terms characterizing said positive input indication (16); or one or more other spoken terms that characterize a negative input instruction (16N), the negative input instruction (16N) indicating that the user (10) does not want the user device (110) or the other device to perform the action (148) specified by the query (22).

13. The computer-implemented method (300) of claim 1, further comprising instructing an automatic speech recognizer (152) in communication with the data processing hardware (410) to operate at a reduced level sufficient to recognize at least one of:

14. 14. The computer-implemented method (300) of any one of claims 1 to 13, wherein the operations further include deleting any storage of the received audio segment when the user (10) does not provide the affirmative input indication (16).

15. data processing hardware (410); memory hardware (420) in communication with the data processing hardware (410), the memory hardware (420) storing instructions that, when executed on the data processing hardware (410), cause the data processing hardware (410) to: receiving audio data (14) corresponding to speech captured in the streaming audio (12) by a user device (110); generating a confidence score indicative of the likelihood of hot words being present in the audio data by a hot word detector (200); determining, based on the confidence score, that the received audio data (14) characterizes a soft hotword event (202); processing the audio data (14) to determine that the audio data (14) corresponds to a query (22) specifying an action (148) to perform at the user device (110); without triggering the execution of the action (148) on the user device (110) or another device; providing a notification (204) of the output from the user device (110), the notification (204) prompting a user (10) associated with the user device (110) to provide a positive input indication (16) to trigger execution of the action (148) on the user device (110) or the other device, the notification being a visual notification having a size proportional to the confidence score; instructing the user device (110) or the other device not to perform the action (148) specified by the query (22) when the user (10) does not provide the affirmative input indication (16); and memory hardware (420) that performs operations including: A system (100) comprising:

16. 16. The system of claim 15, wherein the hotword detector resides on the user device and is configured to detect the soft hotword event when a confidence score indicating a likelihood that the streaming audio captured by the user device contains the presence of a hotword satisfies a first threshold score but does not satisfy a second threshold score.

17. The operation is determining whether the positive input indication (16) is received within a predetermined duration of time after providing the notification (204) of the output from the user device (110); determining that the user (10) has not provided the affirmative input indication (16) when the affirmative input indication (16) is not received within the predetermined duration; and 17. The system (100) of claim 15 or 16, further comprising:

18. The operation is receiving a negative input indication (16N) indicating that the user (10) does not want the user device (110) or the other device to perform the action (148) specified by the query (22); determining, in response to receiving the negative input indication (16N), that the user (10) did not provide the positive input indication (16); 18. The system (100) of any one of claims 15 to 17, further comprising:

19. 20. The system of claim 18, wherein the notification provided regarding the output from the user device also prompts the user to provide a negative input indication indicating that the user does not want the user device or another device to perform the action specified by the query.

20. 20. The system (100) of any one of claims 15 to 19, wherein providing the notification (204) of the output from the user device (110) comprises instructing a graphical user interface executing on the user device (110) to render the notification (204) as a visual notification for display on a screen (118) of the user device (110).

21. 21. The system (100) of claim 20, wherein the actions further include instructing the graphical user interface to remove the visual notification displayed on the screen (118) of the user device (110) when the user (10) does not provide the affirmative input indication (16).

22. 22. The system (100) of claim 15, wherein providing the notification (204) about the output from the user device (110) comprises instructing the user device (110) to output the notification (204) as an audible notification from a speaker (116) of the user device (110).

23. The operation is receiving the affirmative input indication (16) from the user (10); triggering execution of the action (148) specified by the query (22) at the user device (110) or the other device in response to receiving the positive input indication (16); 23. The system (100) of any one of claims 15 to 22, further comprising:

24. receiving the affirmative input indication (16) from the user (10); receiving speech input from the user (10); determining that the speech input includes one or more terms that characterize an intent of the user device (110) to perform the action (148) specified by the query (22); and 24. The system (100) of claim 23, comprising:

25. 25. The system (100) of claim 23 or 24, wherein receiving the affirmative input indication (16) from the user (10) comprises receiving a user input indication indicating a selection of an affirmative graphical element displayed on a screen (118) of the user device (110).

26. After the operation provides the notification (204) about the output from the user device (110), without performing speech recognition, one or more spoken terms characterizing said positive input indication (16); or one or more other spoken terms that characterize a negative input instruction (16N), the negative input instruction (16N) indicating that the user (10) does not want the user device (110) or the other device to perform the action (148) specified by the query (22).

26. The system (100) of any one of claims 15 to 25, further comprising: activating a warm word model configured to detect the presence of at least one of:

27. After the operation provides the notification (204) of the output from the user device (110), one or more spoken terms characterizing said positive input indication (16); or one or more other spoken terms that characterize a negative input instruction (16N), the negative input instruction (16N) indicating that the user (10) does not want the user device (110) or the other device to perform the action (148) specified by the query (22).

27. The system (100) of any one of claims 15 to 26, further comprising instructing an automatic speech recognizer (152) in communication with the data processing hardware (410) to operate at a reduced level sufficient to recognize at least one of:

28. 28. The system (100) of any one of claims 15 to 27, wherein the operations further include deleting any storage of the received audio segment when the user (10) does not provide the affirmative input indication (16).

Images