Freeze Word

Abstract

A method (300) for detecting a freeze word includes receiving voice data (118) corresponding to an utterance (119) spoken by a user (10) and captured by a user device (102) associated with the user. The method also includes processing the voice data using a voice recognizer (200) to determine that the utterance includes a query (122) for the digital assistant (109) to take an action. The voice recognizer is configured to trigger termination of the utterance after a predetermined duration of non-speech in the voice data. Before the predetermined duration of non-speech, the method includes detecting a freeze word (123) in the voice data. In response to detecting the freeze word in the voice data, the method also includes triggering a hard microphone block event (125) at the user device. The hard microphone block event prevents the user device from capturing any voice after the freeze word.

Description

[Technical field]

[0001] The present disclosure relates to freeze words. [Background technology]

[0002] A voice-enabled environment (e.g., at home, at work, at school, in an automobile, etc.) allows a user to speak queries or commands out loud to a computer-based system that responds to the query with an answer and / or performs a function based on the command. A voice-enabled environment can be implemented using a network of connected microphone devices distributed throughout various rooms or areas of the environment. These devices may use hot words to help distinguish when a given utterance is directed to the system, as opposed to an utterance directed to another individual present in the environment. Thus, the device may operate in a sleep or hibernation state and wake up only when a detected utterance contains the hot word. Once awake, the device can continue to perform more expensive processing, such as full on-device automatic speech recognition (ASR) or server-based ASR. Summary of the Invention [Means for solving the problem]

[0003] One aspect of the present disclosure provides a method for detecting a freeze word. The method includes receiving, in the data processing hardware, voice data corresponding to an utterance spoken by a user and captured by a user device associated with the user. The method also includes processing the voice data using a voice recognizer to determine, by the data processing hardware, that the utterance includes a query for the digital assistant to perform an action. The voice recognizer is configured to trigger a termination process of the utterance after a predetermined duration of non-speech in the voice data. Before the predetermined duration of non-speech in the voice data, the method includes detecting, by the data processing hardware, a freeze word in the voice data. The freeze word follows the query in the utterance spoken by the user and captured by the user device. In response to detecting the freeze word in the voice data, the method includes triggering, by the data processing hardware, a hard microphone blocking event in the user device to prevent the user device from capturing any voice following the freeze word.

[0004] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, the freeze word includes a default freeze word including one or more fixed words across all users in a given language, a user-selected freeze word including one or more words specified by a user of a user device, or one of the action-specific freeze words associated with an action to be taken by the digital assistant. In some examples, detecting the freeze word in the audio data includes extracting audio features from the audio data, generating a freeze word confidence score by processing the extracted audio features using a freeze word detection model, the freeze word detection model executing on data processing hardware, and determining that the audio data corresponding to the utterance includes a freeze word if the freeze word confidence score satisfies a freeze word confidence threshold.

[0005] Detecting the freeze word in the voice data may include recognizing the freeze word in the voice data using a voice recognizer executing on the data processing hardware. Optionally, the method may further include, in response to detecting the freeze word in the voice data, instructing the voice recognizer by the data processing hardware to cease any active processing on the voice data, and instructing the digital assistant by the data processing hardware to perform the action.

[0006] In some implementations, processing the voice data to determine that the utterance includes a query for the digital assistant to perform an action includes processing the voice data using a speech recognizer to generate a voice recognition result for the voice data, and performing a semantic interpretation on the voice recognition result for the voice data to determine that the voice data includes a query to perform an action. In these implementations, in response to detecting the freeze word in the voice data, the method also includes modifying, by the data processing hardware, the voice recognition result for the voice data by removing the freeze word from the voice recognition result, and instructing, by the data processing hardware, the digital assistant to perform the action requested by the query using the modified voice recognition result.

[0007] In some examples, prior to processing the audio data using the speech recognizer, the method further includes detecting, by the data processing hardware, a hot word in the audio data preceding the query using a hot word detection model, and triggering, by the data processing hardware, in response to detecting the hot word, the audio recognizer to process the audio data by performing speech recognition on the hot word and / or one or more words following the hot word in the audio data. In these examples, the method may also include verifying, by the data processing hardware, the presence of a hot word detected by the hot word detection model based on detecting the freeze word in the audio data. Optionally, detecting the freeze word in the audio data may include executing a freeze word detection model on the data processing hardware configured to detect the freeze word in the audio data without performing speech recognition on the audio data. Here, the freeze word detection model and the hot word detection model may each include the same or different neural network-based models.

[0008] Another aspect of the present disclosure provides a method for detecting a freeze word. The method includes receiving, in the data processing hardware, a first instance of speech data corresponding to a dictation-based query for a digital assistant to render a representation of audible content spoken by a user. The dictation-based query is spoken by the user and captured by an assistant-enabled device associated with the user. The method also includes receiving, in the data processing hardware, a second instance of speech data corresponding to an utterance of the audible content spoken by the user and captured by the assistant-enabled device. The method also includes processing, by the data processing hardware, the second instance of speech data using a speech recognizer to generate a representation of the audible content. During processing of the second instance of speech data, the method includes detecting, by the data processing hardware, a freeze word in the second instance of speech data. The freeze word follows the audible content in the utterance spoken by the user and captured by the assistant-enabled device. In response to detecting the freeze word in the second instance of speech data, the method includes providing, by the data processing hardware, a representation of the audible content spoken by the user for output from the assistant-enabled device.

[0009] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, in response to detecting the freeze word in the second instance of the audio data, the method further includes: initiating, by the data processing hardware, a hard microphone blocking event in the assistant-enabled device to prevent the assistant-enabled device from capturing any audio following the freeze word; ceasing, by the data processing hardware, any active processing of the second instance of the audio data; and removing, by the data processing hardware, the freeze word from the end of the representation prior to providing the representation of the audible content for output from the assistant-enabled device.

[0010] Optionally, the method may further include processing, by the data processing hardware, the first instance of the audio data using a speech recognizer to generate a speech recognition result, and performing, by the data processing hardware, a semantic interpretation on the speech recognition result for the first instance of the audio data to determine that the first instance of the audio data includes a dictation-based query that causes a transcription of audible content spoken by a user. In some examples, prior to commencing processing on the second instance of the audio data to generate a transcription, the method also includes determining, by the data processing hardware, that the dictation-based query identifies a freeze word based on a semantic interpretation performed on the speech recognition result for the first instance of the audio data, and instructing, by the data processing hardware, a terminator to increase a termination timeout period for terminating the speech of the audible content.

[0011] Another aspect of the present disclosure provides a system for detecting a freeze word. The system includes data processing hardware and memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform an operation. The operation includes receiving voice data corresponding to an utterance spoken by a user and captured by a user device associated with the user. The operation also includes processing the voice data using a voice recognizer to determine that the utterance includes a query for the digital assistant to perform an operation. The voice recognizer is configured to trigger a termination of the utterance after a predetermined duration of non-speech in the voice data. Before the predetermined duration of non-speech in the voice data, the operation includes detecting a freeze word in the voice data. The freeze word follows the query in the utterance spoken by the user and captured by the user device. In response to detecting the freeze word in the voice data, the operation includes triggering a hard microphone blocking event in the user device to prevent the user device from capturing any sound after the freeze word.

[0012] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, the freeze word includes a default freeze word including one or more fixed words across all users in a given language, a user-selected freeze word including one or more words specified by a user of a user device, or one of the action-specific freeze words associated with an action to be taken by the digital assistant. In some examples, detecting the freeze word in the audio data includes extracting audio features from the audio data, generating a freeze word confidence score by processing the extracted audio features using a freeze word detection model, and determining that the audio data corresponding to the utterance includes the freeze word if the freeze word confidence score meets a freeze word confidence threshold. In these examples, the freeze word detection model executes on data processing hardware.

[0013] Detecting the freeze word in the audio data may include recognizing the freeze word in the audio data using a speech recognizer executing on the data processing hardware. Optionally, the operations may further include, in response to detecting the freeze word in the audio data, instructing the speech recognizer to cease any active processing on the audio data and instructing the digital assistant to perform the action.

[0014] In some implementations, processing the voice data to determine that the utterance includes a query for the digital assistant to perform an action includes processing the voice data using a speech recognizer to generate a voice recognition result for the voice data, and performing a semantic interpretation on the voice recognition result for the voice data to determine that the voice data includes a query to perform an action. In these implementations, in response to detecting the freeze word in the voice data, the operations also include modifying the voice recognition result for the voice data by removing the freeze word from the voice recognition result, and using the modified voice recognition result to instruct the digital assistant to perform the action requested by the query.

[0015] In some examples, prior to processing the audio data using the speech recognizer, the operations further include detecting hot words in the audio data preceding the query using the hot word detection model, and triggering the speech recognizer to process the audio data by performing speech recognition on the hot words and / or one or more words following the hot words in the audio data in response to detecting the hot words. In these examples, the operations also include verifying the presence of hot words detected by the hot word detection model based on detecting the freeze word in the audio data. Optionally, detecting the freeze word in the audio data may include executing a freeze word detection model on data processing hardware configured to detect the freeze word in the audio data without performing speech recognition on the audio data. The freeze word detection model and the hot word detection model each include the same or different neural network-based models.

[0016] Another aspect of the present disclosure provides a system for detecting a freeze word. The system includes data processing hardware and memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform an operation. The operation includes receiving a first instance of voice data corresponding to a dictation-based query for a digital assistant to render a representation of audible content spoken by a user. The dictation-based query is spoken by the user and captured by an assistant-enabled device associated with the user. The operation also includes receiving a second instance of voice data corresponding to an utterance of the audible content spoken by the user and captured by the assistant-enabled device. The operation also includes processing the second instance of voice data using a speech recognizer to generate a representation of the audible content. During processing of the second instance of voice data, the operation includes detecting a freeze word in the second instance of voice data. The freeze word follows the audible content in the utterance spoken by the user and captured by the assistant-enabled device. In response to detecting the freeze word in the second instance of the voice data, the operations include providing a representation of the audible content spoken by the user for output from the assistant-enabled device.

[0017] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, in response to detecting the freeze word in the second instance of the audio data, the operations further include initiating a hard microphone blocking event in the assistant-enabled device to prevent the assistant-enabled device from capturing any audio following the freeze word, ceasing any active processing on the second instance of the audio data, and removing the freeze word from the end of the representation prior to providing the representation of the audible content for output from the assistant-enabled device.

[0018] Optionally, the operations may further include processing the first instance of the audio data using a speech recognizer to generate a speech recognition result, and performing a semantic interpretation on the speech recognition result for the first instance of the audio data to determine that the first instance of the audio data includes a dictation-based query that causes a transcription of audible content spoken by the user. In some examples, prior to commencing processing on the second instance of the audio data to generate a transcription, the operations also include determining that the dictation-based query identifies a freeze word based on the semantic interpretation performed on the speech recognition result for the first instance of the audio data, and instructing a terminator to increase a termination timeout period for terminating the speech of the audible content.

[0019] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0020] [Figure 1] 1 is an example system that includes an assistant-enabled device configured to detect freeze words. [Figure 2A] FIG. 1 is a schematic diagram of a first instance of a user utterance corresponding to a dictation-based query identifying a freeze word for terminating a second instance of the user utterance; [Figure 2B] 1 is a schematic diagram of an acoustic feature detector that instructs a terminator to terminate an utterance in response to detecting a freeze word in the utterance; [Diagram 3] 1 is a flowchart of an example arrangement of operations for a method for detecting freeze words in an utterance. [Figure 4] 1 is a flowchart of an example arrangement of operations for a method for detecting freeze words in an utterance. [Diagram 5]FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Like reference symbols in the various drawings indicate like elements.

[0022] Voice-based interfaces such as digital assistants are becoming increasingly common across a variety of devices, including, but not limited to, mobile phones and smart speakers / displays that include microphones for capturing voice. A common way to initiate a voice interaction with an assistant-enabled device is to speak a fixed phrase, e.g., a hotword, that when detected by the voice-enabled device in streaming voice, triggers the assistant-enabled device to begin a wake-up process that begins recording and processing subsequent voice to confirm the query spoken by the user. Thus, hotwords are a key component in the overall digital assistant interface stack, as they allow users to wake up their assistant-enabled devices from a low-power state so that the assistant-enabled devices can continue to perform more expensive processing, such as full automatic speech recognition ASR or server-based ASR.

[0023] Queries spoken by a user directed to an assistant-enabled device typically fall within two categories of queries: conversational queries and non-conversational queries. Conversational queries refer to standard digital assistant queries that ask the digital assistant to perform an action, such as "set a timer," "remind me to buy milk," "how tall is Barack Obama," etc. Non-conversational queries, on the other hand, refer to dictation-based queries, which are longer-form queries that a user speaks to have an email, message, document, social media post, or some other content rendered. For example, a user may speak the query "send Alex the next email" and then continue speaking the content of the email message that the digital assistant will have rendered and then send from the user's email client to the recipient's (e.g., Alex's) email client.

[0024] ASR systems typically use a terminator to determine when a user begins and ends speaking. Once defined, the portion of speech that represents the user's words can be processed to generate speech recognition results, and in some cases, semantic interpretation can be performed on the speech recognition results to confirm the query spoken by the user. The terminator typically evaluates the duration of pauses between words in determining when an utterance begins or ends. For example, if a user says "What is <long pause> dinner", the terminator may split the speech input at the long pause and have the ASR system only process the incomplete phrase "What is" instead of the complete phrase "What is dinner". If the terminator specifies an incorrect termination for an utterance, the results of processing the utterance may be inaccurate and undesirable. At the same time, while allowing for a longer duration of pauses between words in determining when an utterance begins or ends prevents premature termination, the microphone of the user's assistant-enabled device that detects the utterance remains open and may detect sounds that are not intended for the user device. Additionally, delaying the blocking of the microphone results in a delay in the execution of the action specified by the utterance. For example, if the utterance spoken by the user was a query for the digital assistant to perform the action "call my mother", there would necessarily be a delay in the digital assistant initiating the call while the terminator waits for the termination timeout to elapse to confirm that the user has likely stopped speaking. In this scenario, the assistant-enabled device may also detect additional unintended speech that may lead to the performance of an action different from that intended by the user, resulting in the waste of computational resources in interpreting and acting on the detected additional speech due to the inability to timely determine when the user has likely stopped speaking.

[0025] To mitigate the drawbacks associated with too short termination timeout times, such as the possibility of the user interrupting before finishing speaking, and the drawbacks associated with too long termination timeout times, such as increasing the possibility of capturing unintended words and increasing the latency to perform an action specified by an utterance, implementations herein aim at a freeze word that, when spoken at the end of an utterance, identifies when the user has finished speaking to the assistant-enabled device. To some extent, a "freeze word" is the opposite of a hot word by allowing a user to manually terminate an utterance to end a voice-based conversation or a long-form utterance and initiate a hard microphone shutoff event. That is, while a hot word will trigger the assistant-enabled device to wake up from a sleep or hibernation state and start processing the voice, a freeze word will do the opposite by ceasing all active processing on the voice and shutting off the microphone on the assistant-enabled device, thereby transitioning the assistant-enabled device back to a sleep or hibernation state.

[0026] In addition to turning off some or all of the ongoing voice processing, once a freeze word is detected, the assistant-enabled device may additionally disable or adapt future processing for some amount of time to effectively make the assistant-enabled device less responsive. For example, the hotword detection threshold may be temporarily raised to make it harder / less difficult for the user to issue subsequent queries for some time window after speaking the freeze word. In this scenario, the raised hotword detection threshold may then be gradually lowered to a default value over time. Additionally or alternatively, after detecting a freeze word spoken by a particular user, voice input is disabled for that particular user.

[0027] The assistant-enabled device executes an acoustic feature detection model configured to detect the presence of freeze words in the voice data corresponding to the utterance without performing speech recognition or semantic interpretation on the voice data. Here, the acoustic feature detection model may be a neural network-based model trained to detect one or more freeze words. The assistant-enabled device may utilize the same or different acoustic feature detection model for detecting the presence of hot words in the voice data. When the same acoustic feature detection model is used for both hot word detection and freeze word detection, functionality for only one of hot word detection and freeze word detection may be active at a time. In particular, the acoustic feature detection models are capable of running on the user device due to their relatively compact size and low processing requirements compared to ASR models.

[0028] In some configurations, in addition to triggering a hard microphone blocking event, detection of the freeze word in the voice data verifies the presence of a hot word recently detected in the voice data while the assistant-enabled device was in a sleep or hibernation state. Here, the detected hot word may be associated with a low hot word detection confidence score, but subsequent detection of the freeze word may serve as a proxy to verify that the hot word is present in the voice data. In these configurations, the voice data may be buffered on the assistant-enabled device while hot word detection and freeze word detection are performed, and once detection of the freeze word is detected in the buffered voice data, the assistant-enabled device may initiate a wake-up process to perform voice recognition on the buffered voice data.

[0029] In some additional implementations, freeze word detection leverages an automatic speech recognizer currently running on the device or server side to recognize the presence of freeze words. The speech recognizer may be biased to recognize one or more particular freeze words.

[0030] In some examples, a language model may be utilized to determine whether a freeze word is detected in the audio. In these examples, the language model may enable the assistant-enabled device to identify scenarios in which the freeze word is actually part of the user's utterance / query and is not spoken by the user to terminate the utterance / query. Furthermore, the language model may also allow for near matches of the freeze word, where the phrase is similar to the freeze word and the freeze word is unlikely to be part of the user's query / utterance per the language model score.

[0031] An Assistant-enabled device may be capable of recognizing one or more different types / categories of freeze words, such as, without limitation, predefined freeze words, custom freeze words, user-selected freeze words, action-specific freeze words, and query-specific freeze words. Predefined freeze words may include one or more fixed phrases across all users in a given language. For example, for the conversational queries "call my mom right now" and "tell me what the temperature is outside, thank you Google," the phrases "right now" and "thank you Google" correspond to freeze words to allow the user to manually terminate the respective queries.

[0032] The user-selected freeze word may correspond to a freeze word specified by a particular user in advance, for example, during setup of the digital assistant. For example, the user may select a freeze word from a list of suggested freeze words. Optionally, the user may specify one or more custom words to use as freeze words by typing or speaking the words. In some scenarios, the user specifies a user-selected freeze word to be active for a particular type of query. Here, the user may assign a different user-selected freeze word for a dictation-based query than for a conversational query. For example, in the dictation-based query "Hey Google, send Alex the next message: 'I'm going to be late for the meeting' done." In this example, "Hey Google" corresponds to a hot word, the phrase "send Alex the next message" corresponds to a query for the digital assistant to have the message written and sent to the recipient, the phrase "I'm going to be late for the meeting" corresponds to the content of the message, and the phrase "Done" includes a user-selected freeze word for manually terminating the query. Thus, upon detecting the freeze word "Done", the assistant-enabled device will immediately terminate the speech and have the speech recognizer remove the freeze word from the transcribed message before sending it to the recipient. Alternatively, the phrase "send Alex the next message" may instead represent a query for the digital assistant to facilitate voice-based communication between the user and the recipient, and the content of the message "I'm going to be late for the meeting" or simply be sent as a voice message to the recipient for audible playback on the recipient's device. In particular, the freeze word "Done", when detected by the assistant-enabled device, will immediately terminate the speech and remove the voice of the freeze word from the voice message before sending it to the recipient.

[0033] The action-specific freeze word is associated with a specific behavior / action specified by the query for the digital assistant to perform. For example, a user who speaks the query "Hey Google, broadcast I'm back, broadcast done" includes the specific freeze word "broadcast done" in the broadcast action performed by the digital assistant. In this example, the words "broadcast I'm back" specify the action of broadcasting an audible notification through one or more speakers to indicate to other individuals that the user has returned home. The audible notification may include a specific melody or chime that allows individuals who hear the audible notification to confirm that the user has returned home. In some implementations, the action-specific freeze word is enabled in parallel with the user-specified freeze word and / or the default freeze word.

[0034] A query-specific freeze word may be specified as part of a query spoken by a user. For example, the following utterance "Hey Google, let me render the following journal entry until I say it's finished <content of journal entry> END" includes a dictation-based query to cause the digital assistant to render the content spoken by the user for the journal entry. Additionally, the dictation-based query further specifies the freeze word "END" before the user begins speaking the content of the journal entry. Here, the freeze word "END" specified as part of the dictation-based query instructs the terminator to wait to trigger a termination process until the freeze word "END" is detected, or at least to extend the termination timeout period. Extending the termination duration allows for longer pauses while the user is speaking the content of the journal entry that would otherwise trigger a termination process. In some examples, the query-specific freeze word is enabled in parallel with the user-specified freeze word and / or the default freeze word.

[0035] 1, in some implementations, an example system 100 includes an assistant-enabled device (AED) 102 associated with one or more users 10 and in communication with a remote system 111 via a network 104. The AED 102 may represent a computing device, such as a mobile phone, a computer (laptop or desktop), a tablet, a smart speaker / display, a smart appliance, a smart headphone, a wearable, a vehicle infotainment system, etc., and is equipped with data processing hardware 103 and memory hardware 105. The AED 102 includes or is in communication with one or more microphones 106 for capturing speech from the respective user 10. The remote system 111 may be a single computer, multiple computers, or a distributed system (e.g., a cloud environment) with scalable / elastic computational resources 113 (e.g., data processing hardware) and / or storage resources 115 (e.g., memory hardware).

[0036] The AED 102 includes an acoustic feature detector 110 configured to detect the presence of hot words 121 and / or freeze words 123 in the streaming audio 118 without performing semantic analysis or speech recognition processing on the streaming audio 118. The AED 102 also includes an acoustic feature extractor 112, which may be implemented as part of the acoustic feature detector 110 or as a separate component from the acoustic feature detector 110. The acoustic feature extractor 112 is configured to extract acoustic features from the utterance 119. For example, the acoustic feature extractor 112 may receive streaming audio 118 captured by one or more microphones 106 of the AED 102, corresponding to an utterance 119 spoken by the user 10, and extract acoustic features from the audio data 120 corresponding to the utterance 119. The acoustic features may include Mel-frequency cepstral coefficients (MFCCs) or filter bank energies calculated over a window of the audio data 120 corresponding to the utterance 119.

[0037] The acoustic feature detector 110 may receive the speech data 120 including the acoustic features extracted by the acoustic feature extractor 112, and based on the extracted features, the hot word classifier 150 is configured to classify whether the utterance 119 includes a particular hot word 121 spoken by the user 10. The AED 102 may store the extracted acoustic features in a buffer of the memory hardware 105, and the hot word classifier 150 may use the acoustic features in the buffer to detect whether the speech data 120 includes the hot word 121. The hot word classifier 150 may also be referred to as a hot word detection model 150. The AED 102 may include multiple hot word classifiers 150, each trained to detect different hot words associated with a particular word / phrase. These hot words may be predefined hot words and / or custom hot words assigned by the user 10. In some implementations, the hot word classifier 150 includes a trained neural network-based model received from the remote system 111 via the network 104.

[0038] The acoustic feature detector 110 also includes a freeze word classifier 160 configured to classify whether the utterance 119 includes a freeze word 123 spoken by the user 10. The freeze word classifier 160 may also be referred to as a freeze word detection model 160. The AED 102 may include multiple freeze word classifiers 160, each trained to detect different freeze words associated with a particular word / phrase. As described in the above notes, the freeze words may include pre-defined freeze words, user-selected freeze words, action-specific freeze words, and / or query-specific freeze words. Like the hot word classifier 150, the freeze word classifier 160 may include a trained neural network-based model received from the remote system 111. In some examples, the freeze word classifier 160 and the hot word classifier 150 are incorporated into the same neural network-based model. In these examples, the respective portions of the neural network model corresponding to the hot word classifier 150 and the freeze word classifier 160 are never active at the same time. For example, while the AED 102 is in a sleep state, the hot word classifier 150 may be active to listen for hot words 121 in the streaming audio 118, and the freeze word classifier 160 may be inactive. Once a hot word 121 is detected, triggering the AED 102 to wake up and process subsequent audio, the hot word classifier 150 may be currently inactive, and the freeze word classifier 160 may be active to listen for freeze words 123 in the streaming audio 118. The classifiers 150, 160 of the acoustic feature detector 110 may run on a first processor of the AED 102, such as a digital signal processor (DSP), and / or a second processor of the AED 102, such as an application processor (AP) or CPU, that consumes more power while operating than the first processor.

[0039] In some implementations, the hotword classifier 150 is configured to identify hotwords in an initial portion of the utterance 119. In the illustrated example, the hotword classifier 150 may determine that the utterance 119, "Ok Google, broadcast I'm back, done broadcasting," includes the hotword 121, "Ok Google," if the hotword classifier 150 detects acoustic features in the speech data 120 that are specific to the hotword 121. For example, the hotword classifier 150 may detect that the utterance 119, "Ok Google, broadcast I'm back, done broadcasting," includes the hotword 121, "Ok Google," based on generating MFCCs from the speech data and classifying the MFCCs as including MFCCs similar to MFCCs specific to the hotword "Ok Google" stored in a model of the hotword classifier 150. As another example, the hot word classifier 150 may detect that the utterance 119 “Ok Google, broadcast I’m back, broadcast finished” contains the hot word 121 “Ok Google” based on generating mel-scale filter bank energies from the speech data and classifying the mel-scale filter bank energies as containing mel-scale filter bank energies similar to mel-scale filter bank energies that are specific to the hot word “Ok Google” stored in a model of the hot word classifier 150.

[0040] When the hot word classifier 150 determines that the voice data 120 corresponding to the utterance 119 includes the hot word 121 at stage A of the acoustic feature detector 110, the AED 102 may trigger a wake-up process to begin speech recognition on the voice data 120 corresponding to the utterance 119. For example, an automatic speech recognition (ASR) engine 200 (interchangeably referred to as a "speech recognizer" 200) executing on the AED 102 may perform speech recognition or semantic interpretation on the voice data corresponding to the utterance 119. The speech recognizer 200 may include an ASR model 210, a natural language understanding (NLU) module 220, and a terminator 230. The ASR model 210 may process the voice data 120 to generate a speech recognition result 215, and the NLU module 220 may perform semantic interpretation on the speech recognition result 215 to determine that the voice data 120 includes a query 122 for the digital assistant 109 to perform an action. In this example, the ASR model 210 may process the speech data 120 to generate speech recognition results 215 for "broadcast I'm back, broadcast finished," and the NLU module 220 may identify "broadcast I'm back" as a query 122 for the digital assistant 109 to take the action of broadcasting an audible notification for audible output from one or more speakers indicating to other individuals that the user has returned home. Alternatively, the query 122 may be for the digital assistant 109 to broadcast a voice message in which the user speaks "I'm back" for audible output from one or more speakers. The NLU module 220 may be utilized to determine whether the presence of a freeze word detected in the speech data 120 was actually part of the query 122 and thus was not spoken by the user to terminate the utterance. As such, the NLU 220 may override detection of the freeze word in scenarios where the freeze word is actually part of the utterance. The NLU 220 may utilize the language model score in these scenarios.

[0041] In some implementations, the speech recognizer 200 is provided in the remote system 111 in addition to or instead of the AED 102. In response to the hot word classifier 150 detecting the hot word 121 in the utterance 119, the AED 102 may be triggered to wake up and the AED 102 may transmit the speech data 120 corresponding to the utterance 119 to the remote system 111 over the network 104. The AED 102 may transmit the portion of the speech data including the hot word 121 for the remote system 111 to perform speech recognition via the ASR model 210 to confirm the presence of the hot word 121. Alternatively, the AED 102 may transmit only the portion of the speech data 120 corresponding to the portion of the utterance 119 after the hot word 121 to the remote system 111. The remote system 111 executes the ASR model 210 to generate a speech recognition result 215 for the speech data 120. The remote system 111 may also execute the NLU module 220 to perform semantic interpretation on the speech recognition results 215 to identify a query 122 for the digital assistant 109 to perform an action. Alternatively, the remote system 111 may send the speech recognition results 215 to the AED 102, and the AED 102 may execute the NLU module 220 to identify the query 122.

[0042] Continuing with reference to FIG. 1, the terminator 230 is configured to trigger speech termination after a predetermined duration of non-speech in the audio data 120. Here, the predetermined duration of non-speech may correspond to a termination timeout period in which the terminator 230 will terminate the speech upon detecting a period of non-speech for at least the predetermined duration. That is, the terminator 230 terminates the speech by making a hard microphone shutoff decision to instruct one or more microphones 106 in the AED 102 to shut off and no longer capture the streaming audio 118. The termination timeout period is typically set to a default value that is long enough to prevent premature termination of the speech so that the content of the speech is not interrupted before the user has finished speaking. At the same time, setting a longer termination timeout period allows for longer pauses between words in the speech to prevent processing of incomplete phrases, whereas the microphone of the assistant-enabled device may remain open and detect sounds that are not directed toward the assistant-enabled device. Additionally, delaying the shutoff of the microphone results in a delay in the execution of the action / operation identified by the query.

[0043] While the speech recognizer 200 is processing the speech data 120 and before the terminator 230 detects a predetermined duration of non-speech in the speech data, the freeze word classifier 160 is simultaneously running on the AED 102 to detect the freeze word 123 "end broadcast" in the speech data 120. Here, the freeze word 123 "end broadcast" follows the query 122 at the end of the utterance 119 spoken by the user 10 and corresponds to an action-specific freeze word 123. That is, the freeze word 123 "end broadcast" is specific to the action / action of broadcasting a notice or message through an audio speaker. In some examples, the NLU 220 provides instructions 222 to the acoustic feature detector 110 to activate / enable the freeze word 123 "end broadcast" in response to determining that the speech recognition result 215 for the speech data 120 includes the query 122 for the digital assistant 109 to take an action to broadcast. In these examples, the acoustic features detector 110 may activate / enable a freeze word detection model that is configured to detect the freeze word 123 "end of broadcast."

[0044] In some implementations, the freeze word classifier 160 executing on the AED 102 is configured to identify the freeze word 123 at the end of the utterance 119 without performing speech recognition or semantic interpretation. For example, in this example, the freeze word classifier 160 may determine that the utterance 119 "Ok Google, broadcast I'm back, broadcast end" includes the freeze word 123 "broadcast end" if the freeze word classifier 160 detects acoustic features in the speech data 120 that are specific to the freeze word 123. For example, the freeze word classifier 160 may detect that the utterance 119 "Ok Google, broadcast I'm back, broadcast end" includes the freeze word 123 "broadcast end" based on generating MFCCs from the speech data and classifying the MFCCs as including MFCCs similar to MFCCs specific to the freeze word 123 "broadcast end" stored in the model of the freeze word classifier 160. As another example, the freeze word classifier 160 may detect that the utterance 119 “Ok Google, broadcast now, broadcast end” includes the freeze word 123 “broadcast end” based on generating mel-scale filter bank energy from the speech data and classifying the mel-scale filter bank energy as including mel-scale filter bank energy similar to the mel-scale filter bank energy that is characteristic of the freeze word “broadcast end” stored in the model of the freeze word classifier 160. The freeze word classifier 160 may generate a freeze word confidence score by processing extracted speech features in the speech data 120 and determine that the speech data 120 corresponding to the utterance 119 includes the freeze word 123 if the freeze word confidence score meets a freeze word confidence threshold.

[0045] In response to the freeze word classifier 160 detecting the freeze word 123 in the audio data 120 before the terminator 230 detects a predetermined duration of non-speech in the audio data at stage B of the acoustic feature detector 110, the AED 102 may trigger a hard microphone block event 125 in the AED 102 that prevents the AED 102 from capturing any streaming audio 118 after the freeze word 123. For example, triggering the hard microphone block event 125 may include the AED 102 shutting off one or more microphones 106. Thus, the freeze word 123 is spoken by the user 10 as a manual cue that indicates when the user 10 is done speaking the query 122, thereby triggering the hard microphone block event 125 without waiting for the terminator timeout period to elapse so that the terminator 230 can terminate the utterance. Instead, triggering a hard microphone blocking event 125 in response to detecting the freeze word 123 causes the AED 102 to instruct the terminator 230 and / or the ASR model 210 to immediately terminate the utterance. Triggering a hard microphone blocking event 125 also causes the AED 102 to instruct the ASR system 200 to cease any active processing on the voice data and to instruct the digital assistant 109 to perform the action. As a result, speech recognition accuracy is improved because no subsequent speech or background noise is captured by the microphone 106 after the user speaks the freeze word 123, while simultaneously improving latency because the utterance 119 is manually terminated, allowing the digital assistant 109 to begin performing the action specified by the query 122 without having to wait for the termination timeout period to elapse. In the illustrated example, the ASR system 200 provides an output 250 to the digital assistant 109 that causes the digital assistant 109 to perform the action specified by the query 122. The output 250 may include instructions to perform an action.

[0046] In some scenarios, the output 250 also includes speech recognition results 215 for the speech data 120 corresponding to the utterance 119. These scenarios may occur when the query 122 identified by the ASR system 200 corresponds to a search query, in which case the speech recognition results 215 for the search query 122 are provided as output 250 to a search engine (not shown) that retrieves search results. For example, the utterance 119 "Hey Google, what's the weather going to be like tomorrow, Google now" may include the hot word "Hey Google", the conversational search query 122 "What's the weather going to be like tomorrow", and finally the freeze word "Google now". The ASR system 200 may process the speech data 120 to generate speech recognition results 215 for the utterance 119 and perform semantic interpretation on the speech recognition results 215 to identify the search query 122. Continuing with this example, in response to the freeze word classifier 160 detecting the freeze word "Google now," the AED 102 may trigger a hard microphone block event 125, and the ASR system 200 may remove the phrase "Google now" from the end of the speech recognition result 215 (e.g., transliteration 225) and provide the speech recognition result 215 as a search query to a search engine for retrieving search results for tomorrow's weather forecast. In this example, the freeze word "Google now" may comprise a default freeze word 123 common to all users of a given language that manually triggers a hard microphone block event 125 when spoken while speech recognition is active.

[0047] In some implementations, the digital assistant 109 is enabled for continued conversation in which the microphone 106 may remain open to accept subsequent queries from the user after the digital assistant 109 outputs a response to a previous query. For example, using the example above, the digital assistant 109 may audibly output search results for tomorrow's weather forecast as synthesized speech and then instruct the microphone 106 to remain open so that the user 10 can speak the subsequent query without having to repeat speaking the hot word 121 as a prefix to the subsequent query. In this example, if the user 10 does not have a subsequent query, the user 10 may speak the phrase "Thank you Google" (or other phrases of one or more fixed words) to trigger a hard microphone blocking event as the freeze word 123. Leaving the microphone 106 open for a fixed time to accept subsequent queries that the user 10 may or may not speak would necessarily require further processing since audio processing is active while the microphone 106 is open, thereby increasing power consumption and / or bandwidth usage. Therefore, the user 10 may speak a freeze word to prevent the AED 102 from capturing unintended audio, triggering a hard microphone disconnect event, and the AED 102 may transition to a low-power sleep or hibernation state, thus providing power and bandwidth savings.

[0048] In some examples, if the user 10 speaks a freeze word to shut off the microphone 106 and end the ongoing conversation, the AED 102 temporarily raises the hot word detection threshold and / or ignores subsequent words spoken by the same user 10 for some period of time. The AED 102 may store a reference speaker embedding for the user 10 that indicates the user's voice characteristics that can be compared to the verification speaker embedding extracted from the utterance. For example, the verification speaker embedding can be text-dependent, where the embedding is extracted from a spoken hot word, and the reference speaker embedding can be extracted from the user 10 who speaks the same hot word one or more times during enrollment and / or from one or more previous interactions with the digital assistant 109. If the verification speaker embedding extracted from the subsequent utterance matches the reference speaker embedding for the user 10, the words may be ignored if the subsequent utterance was provided immediately after the user 10 spoke the freeze word to trigger a hard microphone shutoff.

[0049] 2A and 2B illustrate the ASR engine 200 receiving a first instance 120a (FIG. 2A) of speech data corresponding to a dictation-based query 122 for the digital assistant 109 to render audible content 124 and a second instance 120b (FIG. 2B) of speech data 120 corresponding to an utterance 119, 119b of the audible content 124. Referring to FIG. 2A, the AED 102 captures a first instance 119a of an utterance 119 spoken by a user 10, which includes "Hey Google, let me render a message for Alex until you say it's over." In this example, "Hey Google" corresponds to a hot word 121, the phrase "let me render a message for Alex" corresponds to a dictation-based query 122 for the digital assistant 109 to render a message for Alex, and the phrase "until you say it's over" identifies a freeze word for terminating the audible content 124 for the message, with the phrase "end" corresponding to the freeze word 123.

[0050] The acoustic feature detector 110 receives streaming audio 118 captured by one or more microphones 106 of the AED 102, which corresponds to a first instance 119a of an utterance 119. The hotword classifier 150 determines that the streaming audio 118 includes a hotword 121. For example, the hotword classifier 150 determines that the streaming audio 118 includes the hotword 121 "Hey Google." After the hotword classifier 150 determines that the streaming audio 118 includes the hotword 121, the AED 102 triggers a wake-up process to begin speech recognition on the first instance 120a of the audio data 120 corresponding to the first instance 119a of the utterance 119.

[0051] The ASR 200 receives the first instance 120a of the speech data 120 from the acoustic feature detector 110. The ASR model 210 may process the first instance 120a of the speech data 120 to generate a speech recognition result 215. For example, the ASR model 210 may receive the first instance 120a of the speech data 120 corresponding to the utterance 119a "Let me transcribe a message for Alex until you say it's over," and generate a corresponding speech recognition result 215. The NLU module 220 may receive the speech recognition result 215 from the ASR model 210 and perform semantic interpretation on the speech recognition result 215 to determine that the first instance 120a of the speech data 120 includes a dictation-based query 122 for the digital assistant 109 to transcribe an audible content 124 spoken by the user 10. In particular, semantic interpretation performed by NLU 220 on speech recognition results 215 identifies the phrase "Let me render a message for Alex" as a dictation-based query 122 for the digital assistant 109 to render audible content 124 for a message (e.g., an electronic message or email) to recipient Alex. In addition to messages, dictation-based query 122 may be associated with rendering other types of content, such as audible content corresponding to a journal entry or a note to be stored in a document.

[0052] In some implementations, the ASR 200 further determines that the dictation-based query 122 identifies the freeze word 123 based on a semantic interpretation performed on the speech recognition results 215 for the first instance of the audio data 120. For example, in the illustrated example, the NLU 220 identifies the phrase "until you say it's over" as an instruction to set the phrase "end" as the freeze word 123 for terminating the audible content 124 of the message. In some examples, the NLU 220 provides instructions 222 to the acoustic feature detector 110 to activate / enable the freeze word 123 "end" in response to determining that the speech recognition results 215 for the first instance of the audio data 120a identify the freeze word 123. In these examples, the acoustic feature detector 110 may activate / enable the freeze word classifier (e.g., freeze word detection model) 160 to detect the freeze word 123 "end" in the subsequent streaming audio 118 captured by the AED 102.

[0053] In some examples, the freeze word classifier 160 and the hot word classifier 150 are never active at the same time. The dotted line surrounding the freeze word classifier 160 in FIG. 2A indicates that the freeze word classifier 160 is currently inactive, while the solid line surrounding the hot word classifier 150 indicates that the hot word classifier 150 is active. For example, before the NLU 220 sends instructions 222 to the acoustic feature detector 110 to activate / enable the freeze word classifier 160 to detect the freeze word 123 "end", the hot word classifier 150 may be active (e.g., indicated by a solid line) to listen for the hot word 121 in the streaming audio 118, and the freeze word classifier 160 may be inactive (e.g., indicated by a dotted line). Once the NLU 220 determines that the speech recognition results 215 include a dictation-based query 122 and that the dictation-based query 122 identifies a freeze word 123, the NLU 220 sends instructions 222 to the acoustic features detector 110 to cause the acoustic features detector 110 to activate the freeze word classifier 160 to detect the freeze word “end” in the subsequent streaming audio 118 and to stop the hot word classifier 150.

[0054] In the illustrated example, the freeze word 123 "END" corresponds to a query-specific freeze word identified as part of the query 122 spoken by the user 10. In particular, the dictation-based query 122 identifies the freeze word 123 "END" before the user begins speaking the audible content 124 of the message. Here, the freeze word "END" identified as part of the dictation-based query instructs the terminator to wait to trigger a termination process until the freeze word "END" is detected, or at least to extend the termination timeout period. In some implementations, the NLU 220 sends instructions 224 to the terminator 230 to increase the termination timeout period. Extending the termination duration allows for long pauses once the user 10 begins speaking the audible content 124 of the message that would otherwise trigger a termination process. In some examples, query-specific freeze words are made available in parallel with action-specific freeze words (e.g., "message ended") and / or user-specified freeze words (e.g., "done") and / or default freeze words (e.g., "Thanks Google").

[0055] In a scenario in which the NLU 220 determines that the speech recognition results 215 include a dictation-based query 122, but that the dictation-based query 122 does not identify a query-specific freeze word 123, the NLU 220 will not send instructions 222 to the acoustic features detector 110 and will cause the acoustic features detector 110 to activate / enable the freeze word classifier 160 to detect any query-specific freeze words, since none are identified by the query 122. However, the NLU 220 may still send instructions 222 to the acoustic features detector 110 to activate / enable the freeze word classifier 160 to detect at least one of an action-specific freeze word, a user-defined freeze word, or a predefined freeze word. Optionally, the acoustic feature detector 110 may detect hot words 121 in the streaming audio 118 corresponding to the first instance 119a of the utterance 119 and then automatically activate / enable the freeze word classifier 160 to detect user-defined and / or pre-defined freeze words in the subsequent streaming audio 118.

[0056] 2B, after the user 10 speaks the first instance 119a of the utterance 119 conveying the dictation-based query 122 identifying the hot word 121 and the query-specific freeze word 123, the user 10 then speaks the second instance 119b of the utterance 119 to convey the audible content 124 of the message the user 10 would like the digital assistant 109 to render, followed by the query-specific freeze word 123 indicating that the user 10 has completed speaking the audible content 124 of the message. In particular, since the AED 102 has now woken in response to the hot word classifier 150 detecting the hot word 121 "Hey Google" in the first instance 119a of the utterance 119, and the ASR 200 remains active, the user 10 does not need to prefix the second instance 119b of the utterance 119 with the hot word 121. In the illustrated example, the second instance 119b of the utterance 119 includes "Alex, I'm running late, I'm done." In this example, the phrase "Alex, you're going to be late" corresponds to the message audible content 124, and the phrase "the end" corresponds to the query-specific freeze word 123 identified by the dictation-based query 122 in the first instance 119a of the utterance 119 spoken by the user 10 in Figure 2A. Instead of the query-specific freeze word "the end" following the audible content 124, other types of freeze words 123 may follow the audible content 124 to similarly trigger termination processing of the audible content 124.

[0057] The acoustic feature detector 110 executing on the AED 102 receives streaming audio 118 captured by one or more microphones 106 of the AED, corresponding to a second instance 119b of the utterance 119. In response to the acoustic feature detector 110 receiving instructions 222 from the NLU 220 in FIG. 2A to activate / enable the freeze word classifier 160 to listen to the streaming audio 118 for the presence of a query-specific freeze word 123, the hot word classifier 150 is now inactive (e.g., indicated by a dotted line) and the freeze word classifier 160 is now active (e.g., indicated by a solid line). The acoustic feature detector 110 utilizes the freeze word classifier 160 to determine whether the streaming audio 118 includes the freeze word 123. The acoustic feature detector 110 transmits the second instance 120b of the audio data 120 to the ASR 200. The ASR 200 receives the second instance 120b of the audio data 120 corresponding to the second instance 119b of the utterance 119 of the audible content 124 spoken by the user 10 and captured by the AED 102. Additionally, the terminator 230 has applied an extended termination timeout period in response to receiving an instruction 224 from the NLU 220 based on the dictation-based query 122 identifying the freeze word 123 “the end” to the first instance 119a of the utterance 119 of FIG.

[0058] The ASR 200 processes the second instance 120b of the speech data 120 to generate a representation 225 of the audible content 124. For example, the ASR 200 generates a representation 225 of the audible content 124, "Alex, I'm going to be late." During processing of the second instance 120b of the speech data 120 in the ASR 200, the acoustic feature detector 110 detects a freeze word 123 in the second instance 120b of the speech data 120. In particular, the freeze word classifier (e.g., freeze word detection model) 160 detects the presence of the freeze word 123 in the second instance 120b of the speech data 120. In the illustrated example, the freeze word 123 includes a query specific freeze word "end" to indicate the end of the audible content 124. The freeze word 123 follows the audible content 124 in the second instance 119a of the utterance 119 spoken by the user 10.

[0059] In response to detecting the freeze word 123 in the second instance 120b of the voice data 120, the ASR 200 provides a representation 225 of the audible content 124 spoken by the user 10 for output from the AED 102. The AED 102 may output the representation 225 by transmitting the representation 225 to a recipient device (not shown) associated with the recipient Alex. In a scenario where the representation 225 represents the audible content 124 related to a memo or journal entry, the AED 102 may provide the representation 225 for output by storing the representation 225 in the document or sending the representation 225 to an associated application. Additionally, the AED 102 may output the representation 225 by displaying the representation in a graphical user interface (if available) of the AED. Here, if the user 10 wishes to have the representation 225 re-represented, correct a misrepresented word, and / or change any of the content of the message, the user 10 can view the representation 225 before sending it to the recipient device. Additionally or alternatively, the AED 102 may utilize a text-to-speech (TTS) module to convert the representation 225 to synthetic speech for audible playback to the user 10 so that the user 10 can confirm that the representation 225 is desired to be sent to the recipient device. In a configuration in which the ASR 200 executes server side on the remote system 111 (FIG. 1), the ASR 200 may transmit the representation 225 to the AED 102 and / or transmit the representation 225 to the recipient device. That is, the ASR 200 provides the representation 225, “Alex, I’m going to be late,” which corresponds to the audible content 124 of the second instance 120b of the voice data 120.

[0060] In some examples, in response to detecting the freeze word 123 in the second instance 120b of the audio data 120, the acoustic feature detector 110 initiates / triggers a hard microphone block event 125 in the AED 102. The hard microphone block event 125 prevents the AED 102 from capturing any audio following the freeze word 123. That is, triggering the hard microphone block event 125 in the AED 102 may include the AED 102 shutting down one or more microphones 106. As such, the freeze word 123 is spoken by the user 10 as a manual cue that indicates when the user 10 has finished speaking the audible content 124 for the dictation-based query 122, thereby triggering the hard microphone block event 125 without waiting for the terminator timeout period to elapse so that the terminator 230 can immediately terminate the second instance 119b of the utterance 119. Instead, triggering a hard microphone block event 125 in response to detecting the freeze word 123 causes the AED 102 to instruct the terminator 230 and / or the ASR model 210 to immediately terminate the speech. Triggering a hard microphone block event 125 also causes the AED 102 to instruct the ASR system 200 to cease any active processing of the second instance 120b of the voice data 120 and to instruct the digital assistant 109 to perform the action.

[0061] In some additional implementations, the ASR system 200 detects the presence of the freeze word 123 in the second instance 120b of the speech data 120 in addition to or instead of the freeze word classifier 160 of the acoustic feature detector 110. That is, the ASR 200 is able to recognize the presence of the freeze word 123 in the second instance 120b of the speech data 120 because the ASR 200 is already actively processing the second instance 120b of the speech data 120 in response to the hot word classifier 150 detecting the hot word 121 in the first instance 120a of the speech data 120. Thus, the ASR 200 may be configured to initiate a hard microphone block event 125 at the AED 102, cease active processing of the second instance 120b of the speech data 120, and remove the recognized freeze word 123 from the end of the notation 225. Further extending this capability of the ASR system 200, the freeze word classifier 160 may run on the AED 102 as a first stage freeze word detector, and the ASR system 200 may be utilized as a second stage freeze word detector to confirm the presence of freeze words detected in the audio data by the freeze word classifier 160.

[0062] In some implementations, while processing the second instance 120b of the speech data 120 to generate the representation 225 of the audible content 124, the ASR 200 also renders the freeze word 123 for inclusion in the representation 225. For example, the representation 225 of the audible content 124 may include "Alex, I'm going to be late, the end." Here, the representation 225 of the audible content 124 inadvertently includes the freeze word 123 "the end" as part of the audible content 124 of the message. That is, the user 10 did not intend for the digital assistant 109 to render the freeze word 123 as part of the audible content 124 to be included in the representation 225, but rather it was spoken to signify the end of the audible content 124. Thus, a hard microphone block event 125 initiated in the AED 102 in response to detecting the freeze word 123 may cause the ASR 200 to remove the freeze word 123 from the end of the representation 225 prior to providing the representation 225 of the audible content 124 for output from the AED 102. Additionally or alternatively, the ASR 200 may recognize the presence of the freeze word 123 at the end of the representation 225 and remove the freeze word 123 from the end of the representation 225 accordingly. In the illustrated example, the ASR 200 removes the freeze word 123 “end” from the representation 225 “Alex, I’m going to be late, the end” prior to providing the representation 225 for output 250. Thus, after the ASR 200 removes the freeze word 123 from the representation 225, the ASR 200 provides the representation 225 “Alex, I’m going to be late” for output 250 from the AED 102.

[0063] 3 is a flowchart of an illustrative arrangement of operations for a method 300 for detecting freeze words. The method 300 includes, at operation 302, receiving, at the data processing hardware 113, voice data 120 corresponding to an utterance 119 spoken by the user 10 and captured by a user device 102 associated with the user 10. At operation 304, the method 300 includes processing, by the data processing hardware 113, the voice data 120 using a voice recognizer 200 to determine that the utterance 119 includes a query 122 for the digital assistant 109 to perform an action. The voice recognizer 200 is configured to trigger termination processing of the utterance 119 after a predetermined duration of non-speech in the voice data 120.

[0064] At operation 306, prior to a predetermined duration of non-speech in the audio data 120, the method 300 includes detecting, by the data processing hardware 113, a freeze word 123 in the audio data 120. The freeze word 123 follows a query 122 in an utterance 119 spoken by the user 10 and captured by the user device 102. At operation 308, in response to detecting the freeze word 123 in the audio data 120, the method 300 includes triggering, by the data processing hardware 113, a hard microphone blockage event 125 at the user device 102. The hard microphone blockage event 125 prevents the user device 102 from capturing any audio following the freeze word 123.

[0065] 4 is a flowchart of an illustrative arrangement of operations for a method 400 for detecting freeze words. At operation 402, the method 400 includes receiving, at the data processing hardware 113, a first instance 120a of speech data 120 corresponding to a dictation-based query 122 for the digital assistant 109 to render an audible content 124 spoken by the user 10. The dictation-based query 122 is spoken by the user 10 and captured by an assistant-enabled device (AED) 102 associated with the user 10. At operation 404, the method 400 includes receiving, at the data processing hardware 113, a second instance 120b of speech data 120 corresponding to an utterance 119 of the audible content 124 spoken by the user 10 and captured by the assistant-enabled device 102. At operation 406 , the method 400 includes processing, by the data processing hardware 113 , the second instance 120 b of the audio data 120 using the speech recognizer 200 to generate a representation 225 of the audible content 124 .

[0066] At operation 408, during processing of the second instance 120b of the audio data 120, the method 400 includes detecting, by the data processing hardware 113, a freeze word 123 in the second instance 120b of the audio data 120. The freeze word 123 follows the audible content 124 in the utterance 119 spoken by the user 10 and captured by the assistant-enabled device 102. At operation 410, in response to detecting the freeze word 123 in the second instance 120b of the audio data 120, the method 400 includes providing, by the data processing hardware 113, a representation 225 of the audible content 124 spoken by the user 10 for output 250 from the assistant-enabled device 102.

[0067] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform a task. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0068] Non-transient memory may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by a computing device. Non-transient memory may be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), as well as disk or tape.

[0069] 5 is a schematic diagram of an example computing device 500 that may be used to implement the systems and methods described herein. Computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components illustrated herein, their connections and relationships, and their functions are merely illustrative and do not limit the example implementations of the invention described and / or claimed herein.

[0070] The computing device 500 includes a processor 510, a memory 520, a storage device 530, a high-speed interface / controller 540 that connects to the memory 520 and a high-speed expansion port 550, and a low-speed interface / controller 560 that connects to a low-speed bus 570 and the storage device 530. Each of the components 510, 520, 530, 540, 550, and 560 may be interconnected using various buses and mounted on a common motherboard or in other manners as appropriate. The processor 510 may include data processing hardware 103, 113 of the user device 102 or the remote system 111. The data processing hardware 510 may process instructions for execution within the computing device 500, including instructions stored in the memory 520 or in the storage device 530, and display graphic information for a graphical user interface (GUI) on an external input / output device, such as a display 580 coupled to the high-speed interface 540. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and types of memories, as appropriate. Likewise, multiple computing devices 500 may be connected, with each device providing a portion of the required operations (eg, as a server bank, a collection of blade servers, or a multi-processor system).

[0071] The memory 520 stores information non-transiently within the computing device 500. The memory 520 may include memory hardware 105, 115 of the user device 102 or the remote system 111. The memory hardware 520 may be a computer-readable medium, a volatile memory unit or a non-volatile memory unit. The non-transient memory 520 may be a physical device used to store programs (e.g., sequences of instructions) or data (e.g., program state information) temporarily or permanently for use by the computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), as well as disks or tapes.

[0072] The storage device 530 can provide mass storage for the computing device 500. In some implementations, the storage device 530 is a computer-readable medium. In various different implementations, the storage device 530 can be an array of devices, including a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or a storage area network or other configuration of devices. In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as the memory 520, the storage device 530, or a memory on the processor 510.

[0073] While the high-speed controller 540 manages bandwidth-intensive operations for the computing device 500, the low-speed controller 560 manages low-bandwidth-intensive operations. Such role assignments are merely illustrative. In some implementations, the high-speed controller 540 is coupled to the memory 520, the display 580 (e.g., through a graphics processor or accelerator), and to a high-speed expansion port 550, which may accept various expansion cards (not shown). In some implementations, the low-speed controller 560 is coupled to the storage device 530 and to a low-speed expansion port 590. The low-speed expansion port 590 may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) and may be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a network device, such as a switch or a router, for example, through a network adapter.

[0074] The computing device 500 may be implemented in a number of different forms, as illustrated in the figure: For example, it may be implemented as a standard server 500a or a collection of multiple such servers 500a, as a laptop computer 500b, or as part of a rack server system 500c.

[0075] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations with one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0076] These computer programs (also known as programs, software, software applications or codes) contain machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives the machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0077] The processes and logic flows described herein can be performed by one or more programmable processors, also referred to as data processing hardware, which execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, such as FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits). Processors suitable for executing computer programs include, by way of example, both general purpose and special purpose microprocessors, and any one or more processors of any kind of digital computer. In general, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for performing instructions, and one or more memory devices for storing instructions and data. In general, a computer will also include, or be operatively coupled to, one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, to receive data therefrom or to transfer data thereto, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, by way of example, semiconductor memory devices, such as EPROM, EEPROM and flash memory devices, magnetic disks, such as internal hard disks or removable disks, magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0078] To provide for user interaction, one or more aspects of the present disclosure can be implemented on a computer having a display device, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, for displaying information to the user, and optionally a keyboard and pointing device, such as a mouse or trackball, for allowing the user to provide input to the computer. Other types of devices can also be used to provide user interaction, for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, the computer can interact with the user by sending documents to and receiving documents from a device used by the user, for example by sending a web page to the web browser in response to a request received from the web browser on the user's client device.

[0079] Although several implementations have been described, it will be understood that various modifications may be made without departing from the spirit and scope of the disclosure. Accordingly, other implementations are within the scope of the following claims. [Explanation of symbols]

[0080] 10 users 100 Systems 102 Assistant Enabled Device (AED) 103 Data Processing Hardware 104 Network 105 Memory Hardware 106 Microphone 109 Digital Assistant 110 Acoustic Feature Detector 111 Remote Systems 112 Acoustic Feature Extractor 113 Computational resources 115 Memory resources 118 Streaming Audio 119, 119a, 119b utterances 120, 120a, 120b Audio data 121 Hot Words 122 queries 123 Freeze Word 124 Audible Content 125 Hard Microphone Blocking Events 150 Hot Words Classifier 160 Freeze Word Classifier 200 Automatic Speech Recognition (ASR) Engines 210 ASR model 215 Voice Recognition Results 220 Natural Language Understanding (NLU) Module 222 Command 224 Command 225 Notation 230 Terminator 250 Output 500 computing devices 510 Processor 520 Memory 530 Storage Devices 540 High Speed ​​Interface / Controller 550 High Speed ​​Expansion Port 560 Low Speed ​​Interface / Controller 570 Slow Bus 580 Display 590 Low Speed ​​Expansion Port 500a Server 500b Laptop Computer 500c Rack Server System

Claims

1. receiving, at the data processing hardware (510), voice data (120) corresponding to speech (119) spoken by a user and captured by a user device (102) associated with said user; Processing the voice data (120) using a voice recognizer (200) to determine, by the data processing hardware (510), that the utterance (119) includes a query (122) for the digital assistant (109) to perform an action, the voice recognizer (200) being configured to trigger termination processing of the utterance (119) after a predetermined duration of non-speech in the voice data (120); before the predetermined duration of non-voice in the voice data (120), detecting, by the data processing hardware (510), a freeze word (123) in the audio data (120), the freeze word (123) following the query (122) in the utterance (119) spoken by the user and captured by the user device (102); in response to detecting the freeze word (123) in the audio data (120), triggering, by the data processing hardware (510), a hard microphone blocking event (125) at the user device (102) to prevent the user device (102) from capturing any audio following the freeze word (123); verifying the presence of hot words in the audio data (120) based on detecting the freeze words (123) in the audio data (120); A method (300) comprising:

2. The freeze word (123) is A predefined set of freeze words (123) containing one or more freeze words across all users in a given language; a user-selected freeze word (123) that includes one or more words specified by the user of the user device (102); or 2. The method of claim 1, further comprising: providing a freeze word associated with the action to be taken by the digital assistant;

3. The step of detecting the freeze word (123) in the audio data (120) comprises: extracting speech features from the speech data (120); generating a freeze word confidence score by processing the extracted speech features using a freeze word detection model (160), the freeze word detection model (160) executing on the data processing hardware (510); and determining that the audio data (120) corresponding to the utterance (119) contains the freeze word (123) if the freeze word confidence score satisfies a freeze word (123) confidence threshold.

4. 4. The method (300) of claim 1, wherein detecting the freeze word (123) in the speech data (120) comprises recognizing the freeze word (123) in the speech data (120) using the speech recognizer (200) executing on the data processing hardware (510).

5. in response to detecting the freeze word (123) in the audio data (120), instructing, by the data processing hardware (510), the speech recognizer (200) to cease any active processing on the speech data (120); The method (300) of any one of claims 1 to 4, further comprising the step of instructing the digital assistant (109) to perform the action by the data processing hardware (510).

6. Processing the voice data (120) to determine that the utterance (119) includes the query (122) for the digital assistant (109) to perform the action, processing the speech data (120) using the speech recognizer (200) to generate speech recognition results (215) for the speech data (120); and performing a semantic interpretation on the speech recognition results (215) for the speech data (120) to determine that the speech data (120) includes the query (122) for performing the action.

7. in response to detecting the freeze word (123) in the audio data (120), modifying, by said data processing hardware (510), said speech recognition results (215) for said audio data (120) by removing said freeze words (123) from said speech recognition results (215); and instructing, by the data processing hardware (510), the digital assistant (109) to perform the action requested by the query (122) using the modified speech recognition results (215).

8. Prior to processing the speech data (120) using the speech recognizer (200), detecting, by said data processing hardware (510), hot words (121) in said speech data (120) preceding said query (122) using a hot word detection model (150); and in response to detecting the hot word (121), triggering, by the data processing hardware (510), the speech recognizer (200) to process the audio data (120) by performing speech recognition on the hot word (121) and / or one or more words following the hot word (121) in the audio data (120). The method (300) of any one of claims 1 to 7, further comprising:

9. 10. The method (300) of claim 8, further comprising the step of verifying, by the data processing hardware (510), the presence of the hot word (121) detected by the hot word detection model (150) based on detecting the freeze word (123) in the audio data (120).

10. detecting the freeze word (123) in the audio data (120) comprises executing a freeze word detection model (160) on the data processing hardware (510) configured to detect the freeze word (123) in the audio data (120) without performing speech recognition on the audio data (120); the freeze word detection model (160) and the hot word detection model (150) each comprise the same or different neural network based models; The method (300) according to claim 8 when relying on any one of claims 1 to 3.

11. Data processing hardware (510); and memory hardware (520) in communication with the data processing hardware (510), the memory hardware (520) storing instructions that, when executed on the data processing hardware (510), cause the data processing hardware (510) to perform operations, the operations comprising: receiving voice data (120) corresponding to speech (119) spoken by a user and captured by a user device (102) associated with said user; Processing the voice data (120) using a voice recognizer (200) to determine that the utterance (119) includes a query (122) for the digital assistant (109) to perform an action, the voice recognizer (200) being configured to trigger termination processing of the utterance (119) after a predetermined duration of non-speech in the voice data (120); before the predetermined duration of non-voice in the voice data (120), detecting a freeze word (123) in the audio data (120), the freeze word (123) following the query (122) in the utterance (119) spoken by the user and captured by the user device (102); in response to detecting the freeze word (123) in the audio data (120), triggering a hard microphone blocking event (125) at the user device (102) to prevent the user device (102) from capturing any audio following the freeze word (123); and verifying the presence of a hot word in the audio data (120) based on detecting the freeze word (123) in the audio data (120). System (100).

12. The freeze word (123) is A predefined set of freeze words (123) containing one or more freeze words across all users in a given language; a user-selected freeze word (123) that includes one or more words specified by the user of the user device (102); or 12. The system (100) of claim 11, further comprising one of an action-specific freeze word (123) associated with the action to be taken by the digital assistant (109).

13. Detecting the freeze word (123) in the audio data (120), extracting speech features from the speech data (120); generating a freeze word confidence score by processing the extracted speech features using a freeze word detection model (160), the freeze word detection model (160) executing on the data processing hardware (510); and determining that the audio data (120) corresponding to the utterance (119) contains the freeze word (123) if the freeze word confidence score satisfies a freeze word (123) confidence threshold.

14. 14. The system (100) of claim 11, wherein detecting the freeze word (123) in the audio data (120) comprises recognizing the freeze word (123) in the audio data (120) using the speech recognizer (200) executing on the data processing hardware (510).

15. In response to detecting the freeze word (123) in the audio data (120), the operation instructing the speech recognizer (200) to cease any active processing on the speech data (120); 15. The system (100) of claim 11, further comprising: instructing the digital assistant (109) to perform the action.

16. Processing the voice data (120) to determine that the utterance (119) includes the query (122) for the digital assistant (109) to perform the action, processing the speech data (120) using the speech recognizer (200) to generate speech recognition results (215) for the speech data (120); and performing a semantic interpretation on the speech recognition results (215) for the speech data (120) to determine that the speech data (120) includes the query (122) to perform the action.

17. In response to detecting the freeze word (123) in the audio data (120), the operation modifying the speech recognition results (215) for the speech data (120) by removing the freeze words (123) from the speech recognition results (215); and instructing the digital assistant to perform the action requested by the query using the modified speech recognition result.

18. Prior to said operation processing said speech data (120) using said speech recognizer (200), detecting hot words (121) in the speech data (120) preceding the query (122) using a hot word detection model (150); 18. The system (100) of any one of claims 11 to 17, further comprising, in response to detecting the hot word (121), triggering the speech recognizer (200) to process the audio data (120) by performing speech recognition on the hot word (121) and / or one or more words following the hot word (121) in the audio data (120).

19. 20. The system (100) of claim 18, wherein the operations further comprise verifying the presence of the hotword (121) detected by the hotword detection model (150) based on detecting the freeze word (123) in the audio data (120).

20. detecting the freeze word (123) in the audio data (120) comprises executing a freeze word detection model (160) on the data processing hardware (510) configured to detect the freeze word (123) in the audio data (120) without performing speech recognition on the audio data (120); the freeze word detection model (160) and the hot word detection model (150) each comprise the same or different neural network based models; A system (100) according to claim 18 when relying on any one of claims 11 to 13.

Images