Adapting hot word identification based on personalized negation

By using a cascaded hot word detection architecture and negative hot word classification, the problem of false hot word detection in voice-enabled devices is solved, improving detection accuracy and user experience while saving battery power.

CN116648743BActive Publication Date: 2026-11-17GOOGLE LLC
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202180085989.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2020-11-20
Filing Date
2021-11-11
Publication Date
2026-11-17
Estimated Expiration
2041-11-11

AI Technical Summary

Technical Problem

In existing technologies, when enabling voice devices to detect hot words in streaming audio, false positives and false negatives are prone to occur, leading to unnecessary wake-up processes, affecting user experience and increasing power consumption.

Method used

It adopts a cascaded hot word detection architecture, combining first-level and second-level hot word detectors. By classifying negative hot words and updating the first-level hot word detector, it personalizes and adapts to users' pronunciation differences and environmental changes, thus preventing false detections.

Benefits of technology

It improves the accuracy of hot word detection, reduces false wake-up events, enhances user experience, and saves battery life.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116648743B_ABST
    Figure CN116648743B_ABST
Patent Text Reader

Abstract

A method (600) includes receiving audio data (120) characterizing hot word events detected by a first-level hot word detector (210) in audio (118) captured by a user equipment (102). The method further includes processing the audio data using a second-level hot word detector (220) to determine whether a hot word is detected in a first segment (121) of the audio data. When a hot word is not detected by the second-level hot word detector, the method includes classifying the first segment of the audio data as containing a negative hot word that causes a false detection of a hot word event in the streaming audio by the first-level hot word detector. Based on the first segment of audio data classified as containing a negative hot word, the method includes updating the first-level hot word detector to prevent the triggering of hot word events in subsequent audio data containing negative hot words.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to adapting hot word identification based on personalized negation. Background Technology

[0002] Voice-enabled environments (e.g., home, workplace, school, car, etc.) allow users to speak queries or commands aloud to a computer-based system, which then responds to and answers the queries and / or performs functions based on the commands. Voice-enabled environments can be implemented using a network of connected microphone devices distributed across various rooms or areas of the environment. Instead of utterances directed at another person in the environment, these devices can use hot words to help identify when a given utterance is directed at the system. Therefore, the device can operate in sleep or hibernation mode and wake up only when a detected utterance includes a hot word. Typically, systems used to detect hot words in streaming audio generate probability scores indicating the probability that a hot word is present in the streaming audio. When the probability score meets a predetermined threshold, the device initiates a wake-up process. Summary of the Invention

[0003] One aspect of this disclosure provides a method for adaptive hot word identification based on personalized negation. The method includes receiving, at data processing hardware, audio data characterizing hot word events detected by a first-level hot word detector in streaming audio captured by a user device. The method further includes processing the audio data by the data processing hardware using a second-level hot word detector to determine whether the second-level hot word detector detected a hot word in a first segment of the audio data. When the second-level hot word detector did not detect a hot word in the first segment of the audio data, the method includes classifying the first segment of the audio data by the data processing hardware as containing a negative hot word that causes a false detection of a hot word event in the streaming audio by the first-level hot word detector. Based on the first segment of audio data classified as containing a negative hot word, the method includes updating the first-level hot word detector by the data processing hardware to prevent the triggering of hot word events in subsequent audio data containing negative hot words.

[0004] Implementations of this disclosure may include one or more of the following optional features. In some implementations, the method further includes: when a second-level hot word detector does not detect a hot word in a first segment of audio data, suppressing a wake-up process on a user device by data processing hardware for processing the hot word and one or more other terms following the hot word in the streaming audio; and determining by the data processing hardware whether the user of the user device provides an immediate follow-up query after suppressing the wake-up process on the user device. In these implementations, classifying the first segment of audio data as containing a negative hot word is further based on determining that the user of the user device did not provide a follow-up query after suppressing the wake-up process.

[0005] In some examples, when a second-level hot word detector detects a hot word in the first segment of the audio data, the method further includes processing a second segment of the audio data following the first segment of the audio data by data processing hardware to determine whether the second segment of the audio data indicates a spoken query type utterance. In these examples, when the second audio segment of the audio data does not indicate a spoken query type utterance, the method further includes: classifying the first segment of the audio data as containing a negative word by the data processing hardware; and updating the first-level hot word detector by the data processing hardware based on the first segment of the audio data classified as containing a negative hot word to prevent triggering a hot word event in subsequent audio data containing a negative hot word. In some implementations, the method further includes: when the second audio segment of the audio data does not indicate a spoken query type utterance, determining by the data processing hardware whether the user of the user device provided an immediate follow-up query. Here, classifying the first segment of the audio data as containing a negative hot word is further based on determining that the user of the user device did not provide a follow-up query. When the second audio segment of the audio data indicates a spoken query type utterance, the method may further include: receiving a negative interaction result at the data processing hardware, the negative interaction result indicating that the user of the user device has negatively interacted with the result of the spoken query type utterance provided to the user device; classifying the first segment of the audio data as containing negative hot words by the data processing hardware based on the received negative interaction result; and updating the first-level hot word detector by the data processing hardware based on the first segment of the audio data classified as containing negative hot words to prevent the detection of hot word events in subsequent audio data containing negative hot words.

[0006] In some examples, after receiving audio data representing hot word events detected by the first-level hot word detector, the method further includes receiving, at the data processing hardware, a negative user interaction indicating user suppression of the wake-up process on the user device. Here, the first segment of the audio data is classified as a negative user interaction containing negative hot words, further based on the indication of user suppression of the wake-up process.

[0007] Optionally, updating the first-level hot word detector to prevent hot word events from being triggered in subsequent audio data may include providing a first segment of audio data classified as containing negative hot words to the user device. The user device is configured to retrain the first-level hot word detector using the first segment of audio data classified as containing negative hot words. In some embodiments, the user device is configured to retrain the first-level hot word detector by storing each instance of the first segment of audio data classified as containing negative hot words in the user device's memory hardware in the user device's memory hardware and retraining the first-level hot word detector based on an aggregation of the number of instances of the first segment of audio data classified as containing negative hot words stored in the memory hardware. In these embodiments, the user device is further configured to: determine, before retraining the first-level hot word detector, that the corresponding confidence score associated with each instance of the first segment of audio data classified as containing negative hot words fails to meet a negative hot word threshold score, and determine that the number of instances exceeds a threshold number of instances.

[0008] Updating the first-level hot word detector to prevent the triggering of hot word events in subsequent audio data may include providing a first segment of audio data, classified as containing a negative hot word, to a user device. The user device is configured to obtain an embedded representation of the first segment of the audio data and store the embedded representation of the first segment of the audio data in the user device's memory hardware. Additionally, the user device is configured to determine when subsequent audio data characterizing a hot word event detected by the first-level hot word detector includes a negative hot word by computing an evaluated embedded representation of the audio data; to determine a similarity score between the embedded representation of the first segment of the audio data classified as a negative hot word and the evaluated embedded representation for subsequent audio data; and to determine that subsequent audio data includes a negative hot word when the similarity score meets a similarity score threshold.

[0009] In some implementations, the data processing hardware resides on a server that communicates with the data processing hardware, and the first-level hot word detector executes on the processor of the user device. Processing the audio data to determine whether the second-level hot word detector has detected a hot word in a first segment of the audio data may include performing automatic speech recognition to determine whether the hot word is recognized in the first segment of the audio data.

[0010] In some examples, data processing resides on the user equipment. In these examples, the first-level hot word detector can be executed on the digital signal processor (DSP) of the data processing hardware, and the second-level hot word detector can be executed on the application processor of the data processing hardware. The first-level hot word detector can be configured to generate probability scores indicating the presence of hot words in audio features of streaming audio captured by the user equipment, and to detect hot word events in the streaming audio when the probability scores meet the hot word detection threshold of the first-level hot word detector.

[0011] Another aspect of this disclosure provides a system for adaptive hot word recognition based on personalized negation. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving audio data characterizing hot word events detected by a first-level hot word detector in streaming audio captured by a user equipment. The operations also include processing the audio data using a second-level hot word detector to determine whether the second-level hot word detector detected a hot word in a first segment of the audio data. When the second-level hot word detector did not detect a hot word in the first segment of the audio data, the operations include classifying the first segment of the audio data as containing a negative hot word that causes a false detection of a hot word event in the streaming audio by the first-level hot word detector. Based on the first segment of audio data classified as containing a negative hot word, the operations include updating the first-level hot word detector to prevent triggering hot word events in subsequent audio data containing negative hot words.

[0012] Implementations of this disclosure may include one or more of the following optional features. In some implementations, operation further includes: suppressing a wake-up process on a user device when a second-level hot word detector does not detect a hot word in a first segment of audio data, for processing the hot word and one or more other terms following the hot word in the streaming audio; and after suppressing the wake-up process on the user device, determining whether an immediate follow-up query was provided by a user of the user device. In these implementations, classifying the first segment of audio data as containing a negative hot word is further based on determining that a follow-up query was not provided by a user of the user device after the wake-up process was suppressed.

[0013] In some examples, when a second-level hot word detector detects a hot word in the first segment of the audio data, the operation further includes processing a second segment of the audio data following the first segment to determine whether the second segment of the audio data indicates a spoken query type utterance. In these examples, when the second audio segment of the audio data does not indicate a spoken query type utterance, the operation further includes classifying the first segment of the audio data as containing a negative word; and updating the first-level hot word detector based on the first segment of the audio data classified as containing a negative hot word to prevent triggering a hot word event in subsequent audio data containing a negative hot word. In some implementations, the operation further includes determining whether the user of the user device provided an immediate follow-up query when the second audio segment of the audio data does not indicate a spoken query type utterance. Here, classifying the first segment of the audio data as containing a negative hot word is further based on determining that the user of the user device did not provide a follow-up query. When the second audio segment of the audio data indicates a spoken query type utterance, the operation may further include: receiving a negative interaction result indicating that the user of the user device has negatively interacted with the result of the spoken query type utterance provided to the user device; classifying the first segment of the audio data as containing negative hot words based on the received negative interaction result; and updating the first-level hot word detector based on the first segment of the audio data classified as containing negative hot words to prevent the detection of hot word events in subsequent audio data containing negative hot words.

[0014] In some examples, after receiving audio data characterizing hot word events detected by the first-level hot word detector, the operation further includes receiving a negative user interaction indicating user suppression of the wake-up process on the user's device. Here, the first segment of the audio data is classified as a negative user interaction containing a negative hot word, further based on the indication of user suppression of the wake-up process.

[0015] Optionally, updating the first-level hot word detector to prevent hot word events from being triggered in subsequent audio data may include providing a first segment of audio data classified as containing negative hot words to the user device. The user device is configured to retrain the first-level hot word detector using the first segment of audio data classified as containing negative hot words. In some embodiments, the user device is configured to retrain the first-level hot word detector by storing each instance of the first segment of audio data classified as containing negative hot words in the user device's memory hardware in the user device's memory hardware, and retraining the first-level hot word detector based on an aggregation of the number of instances of the first segment of audio data classified as containing negative hot words stored in the memory hardware. In these embodiments, the user device is further configured to: determine, before retraining the first-level hot word detector, that the corresponding confidence score associated with each instance of the first segment of audio data classified as containing negative hot words fails to meet a negative hot word threshold score, and determine that the number of instances exceeds a threshold number of instances.

[0016] Updating the first-level hot word detector to prevent the triggering of hot word events in subsequent audio data may include providing a first segment of audio data, which will be classified as containing a negative hot word, to a user device. The user device is configured to obtain an embedded representation of the first segment of the audio data and store the embedded representation of the first segment of the audio data in the user device's memory hardware. Additionally, the user device is configured to determine when subsequent audio data characterizing a hot word event detected by the first-level hot word detector includes a negative hot word by: calculating an evaluated embedded representation of the audio data; determining a similarity score between the embedded representation of the first segment of the audio data classified as a negative hot word and the evaluated embedded representation for subsequent audio data; and determining that the subsequent audio data includes a negative hot word when the similarity score meets a similarity score threshold.

[0017] In some implementations, the data processing hardware resides on a server that communicates with the data processing hardware, and the first-level hot word detector executes on the processor of the user device. Processing the audio data to determine whether the second-level hot word detector has detected a hot word in a first segment of the audio data may include performing automatic speech recognition to determine whether the hot word is recognized in the first segment of the audio data.

[0018] In some examples, data processing resides on the user equipment. In these examples, the first-level hot word detector can be executed on the digital signal processor (DSP) of the data processing hardware, and the second-level hot word detector can be executed on the application processor of the data processing hardware. The first-level hot word detector can be configured to generate probability scores indicating the presence of hot words in audio features of streaming audio captured by the user equipment, and to detect hot word events in the streaming audio when the probability scores meet the hot word detection threshold of the first-level hot word detector.

[0019] Details of one or more embodiments of this disclosure are set forth in the accompanying drawings and the following description. Other aspects, features, and advantages will be apparent from the specification, drawings, and claims. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of an example system used to classify negative hot words and update the first-level hot word detector to prevent the detection of hot word events in audio containing negative hot words.

[0021] Figure 2 This is a schematic diagram of the hot word detection architecture.

[0022] Figure 3 yes Figure 1 A schematic diagram of an example negative hot word classifier for a system.

[0023] Figure 4This is a diagram of a user device that stores the classification results of audio data that contains negative hot words.

[0024] Figure 5 This is a schematic diagram of an example user device that identifies the presence of personal negative hot words in captured audio data to prevent triggering hot word events in the audio data.

[0025] Figure 6 This is a flowchart of an example setup for classifying personalized negative hot words and updating the first-level hot word detector to prevent hot word events from being triggered in audio data containing personalized negative hot words.

[0026] Figure 7 This is a schematic diagram of an example computing device that can be used to implement the systems and methods described in this paper.

[0027] Similar reference numerals in the various figures indicate similar elements. Detailed Implementation

[0028] Voice-based interfaces, such as digital assistants, are becoming increasingly prevalent across multiple devices, including but not limited to smart speakers / displays and mobile phones that include microphones for capturing speech. The general way to initiate voice interaction with a voice-enabled device is to speak a fixed phrase, such as a hotword. When the voice-enabled device detects this fixed phrase in the streaming audio, it triggers a wake-up process to begin recording and processing subsequent speech to determine the user's spoken query. Therefore, as the primary entry point for voice-based interfaces, it is crucial that hotword detection / recognition works reliably in terms of recall and accuracy, minimizing the number of false wake-up events.

[0029] False denial (also known as 'false rejection') occurs when a hot word spoken by a user is not detected in the streaming audio when the user intends to interact with a voice-based interface (e.g., a digital assistant). Here, the voice-enabled device fails to respond to the user and prompts the user to try again by speaking the hot word aloud—usually louder and with a different pronunciation—to ensure the hot word is detected. False affirmation (also known as 'false acceptance'), on the other hand, occurs when a hot word is detected in the streaming audio when it does not actually contain it, often due to the presence of a word / phrase in the streaming audio that sounds similar to the hot word when spoken. False affirmation causes the voice-enabled device to initiate a wake-up process, surprising and / or confusing the user by responding when the voice-enabled device should be asleep.

[0030] The cascaded hotword detection architecture comprises a first-level hotword detector running on the device to detect the presence of hotwords in streaming audio, and a second-level hotword detector that confirms the presence of hotwords detected by the first-level hotword detector. The second-level hotword detector is associated with higher accuracy in detecting the presence of hotwords in streaming audio and therefore includes higher power requirements than the first-level hotword detector. Typically, the second-level hotword detector is implemented on a server that communicates with the first-level hotword detector implemented on the voice-enabled device. Even during partial false positives, where the first-level hotword detector detects the presence of a hotword locally, the second-level hotword detector implemented on the server rejects the presence of the hotword, which has a negative impact on the user experience, even if the server eventually suppresses the wake-up process. That is, the detection of hotwords by the first-level hotword detector still causes the device to wake up and connect to the server, which is obvious to the user (e.g., a visible notification or flashing light), and this is further undesirable from a privacy and power conservation perspective. Therefore, eliminating the occurrence of partial false positives is desirable to improve the user experience.

[0031] Typically, voice-enabled devices use the same, fixed hotword model for all users in a given language (or location), which is periodically updated with new versions pushed from the server to the device. That is, despite significant variations across user speech, accents, vocabulary, and / or the acoustic environment in which the voice-enabled device is operating, the same hotword model is used to detect hotwords in the streaming audio of all users. As a result, it becomes nearly impossible to achieve the stringent precision / recall requirements for hotword detection when a single hotword model is shared across all users in a given language / location.

[0032] For a given user and / or environment, hot word false positives are likely to cluster together. In a non-restrictive example, a specific user uttering the term "pood le" might cause a hot word detection model on a speech-enabled device to falsely detect the specified hot word "Hey Google," while a different user might cause the same hot word detection model implemented on another speech-enabled device to detect the specified hot word when that user utters "doodle." These variations in false positives across different users can be attributed to differences in user pronunciation and / or the frequency of those terms in those users' respective vocabularies. Since the same false positives are likely to recur based on similar acoustic patterns of the same user in the same environment, the hot word detector should ideally learn to adapt to avoid repeatedly omitting the same false positives when a user pronounces a word similar to the specified hot word.

[0033] The implementation described herein relates to personalizing a user's voice-enabled device's hot word detector based on specific terms that lead to false positives in hot word detection, as identified as negative hot words. These specific terms can be user-specific, such that any hot word detection is suppressed when the hot word detector detects an audio segment derived from the specific user who uttered the term. Additionally or alternatively, these specific terms can be device-specific, such that hot word detection is suppressed when an audio segment derived from the user who uttered the term is detected by a hot word detector implemented on that specific device but not on other voice-enabled devices associated with the same user. This follows the concept that devices in some environments may be more prone to false positives in hot word detection than devices in others, due to acoustic variations, vocabulary variations, and variations across users who typically speak in those environments.

[0034] refer to Figure 1 In some implementations, example system 100 includes user equipment 102 associated with one or more users 10 and communicating with remote system 110 via network 104. User equipment 102 may correspond to computing devices such as mobile phones, computers (laptops or desktops), tablets, smart speakers / displays, smart appliances, smart headphones, wearable devices, vehicle infotainment systems, etc. User equipment 102 includes one or more microphones 106 for capturing or communicating with one or more microphones 106 from the corresponding user 10. Remote system 110 may be a single computer, multiple computers, or a distributed system (e.g., a cloud environment) with scalable / elastic computing resources 112 (e.g., data processing hardware) and / or storage resources 114 (e.g., memory hardware).

[0035] User equipment 102 includes a first-level hot word detector 210 (also referred to as a hot word detection model) configured to detect the presence of hot words in the streaming audio 118 without performing semantic analysis or speech recognition processing on the streaming audio 118. In some embodiments, user equipment 102 also includes an initial coarse hot word detector 205 ( Figure 2The initial coarse hot word detector 205 is trained to initially detect the presence of hot words before invoking the first-level (fine) hot word detector 210 to receive the streaming audio 118, and to confirm whether hot words are detected in the streaming audio 118. In some embodiments, the first-level hot word detector 210 includes a trained neural network (e.g., a memory neural network) received from a remote system 110 via network 104. The remote system 110 may push updates or new versions of the first-level hot word detector 210 to a given group of user devices 102 associated with local and / or language-specific users. That is, all user devices 102 associated with American English-speaking users receive the same first-level hot word detector 210, which may include a neural network trained to detect the presence of hot words in the streaming audio 118 when spoken by an American English-speaking user.

[0036] In some examples, a first-level hot word detector 210 executed on user device 102 is configured to detect the presence of the hot word "Hey Google" in streaming audio 118 to initiate a wake-up process on user device 102 for processing the hot word and / or one or more other terms (e.g., queries or commands) following the hot word in streaming audio 118. The first-level hot word detector 210 may be configured to generate a probability score indicating the presence of a hot word in the audio features of streaming audio 118 captured by user device 102, and to detect a hot word in streaming audio 118 when the probability score meets a hot word detection threshold of the first-level hot word detector 210. Therefore, the first-level hot word detector 210 can detect a hot word event in streaming audio 118 captured by user device 102 when the probability score generated for the audio features of streaming audio 118 meets the hot word detection threshold.

[0037] In the example shown, user 10 utters 119, which includes a word / phrase (e.g., “Hey Poodle”) captured as streaming audio 118 by user device 102. This word / phrase has a similar pronunciation to a fixed word / phrase (e.g., “Hey Google”) designated as a hot word to be detected by a first-level hot word detector 210. Notably, user 10 can pronounce the word as “Hey Poodle,” making it more difficult for the first-level hot word detector to distinguish it from the designated hot word “Hey Google” compared to when the word is spoken by another user using a slightly different pronunciation. As a result, the first-level hot word detector 210, executed on user device 102, incorrectly detects the presence of the hot word by generating probability scores of audio features associated with “Hey Poodle” that meet the hot word detection threshold, thereby triggering an unintended wake-up process initiated by user 10.

[0038] When the probability score of the audio features associated with “Hey Poodle” meets the hot word detection threshold, the first-level hot word detector 210 further sends audio data 120 characterizing the hot word event to a second-level hot word detector 220 executed on the remote system 110. In some examples, the audio data 120 is a direct representation of the streaming audio 118, while in other examples, the audio data 136 represents the streaming audio 118 after processing by the first-level hot word detector 210 (e.g., to identify and / or isolate specific audio characteristics of the streaming audio 118 or to convert the streaming audio 118 into a format suitable for transmission and / or processing by the second-level hot word detector 220). For example, the audio data 120 includes a first segment 121 from the chomp processing of the streaming audio 120, comprising relevant audio features associated with the presence of the hot word detected by the first-level hot word detector 210. Audio data 120 also includes a second segment 122, which includes audio features captured by user device 102 in streaming audio 118 after the first segment 121. Typically, the first segment 121 usually has a fixed duration sufficient to include audio features associated with a specified hot word. However, the second segment 122 may have a variable duration, which includes audio captured by user device 102 when microphone 106 is turned on. The second segment 122 may capture query-type utterances that require further processing of one or more terms (e.g., automatic speech recognition and / or semantic interpretation) to identify queries or commands in the audio data. In an example scenario, since the first segment 121 includes audio features captured by the user device that are not associated with the user uttering the hot word, but rather with another term / phrase similar to the hot word pronounced by user 10, user 10 does not intend to invoke device 102 by voice, and therefore, the second segment 122 may not include query-type utterances, but instead includes background noise captured from the environment of user device 102 in streaming audio 118.

[0039] The first-level hot word detector 210 and the second-level hot word detector 220 collaborate to form a cascaded hot word detection architecture 200, whereby the second-level hot word detector 220 is configured to confirm whether a hot word detected by the first-level hot word detector 210 exists in the audio data 120. Specifically, the second-level hot word detector 220, executed on the remote system 110, processes the audio data 120 to determine whether the second-level hot word detector has detected a hot word in the first segment 121 of the audio data 120. In some examples, the second-level hot word detector 210 is implemented as an automatic speech recognition (ASR) engine, which performs speech recognition on the first segment 121 of the audio data 120 to determine whether a hot word exists. When the probability of identifying a hot word meets a hot word detection threshold, the second-level hot word detector 210 can detect the presence of the hot word in the first segment 121.

[0040] In other examples, the second-level hot word detector 220 is similar to the first-level hot word detector 210 because it is implemented as a trained neural network model configured to detect the presence of hot words in the first audio segment 121 without performing semantic analysis or speech recognition processing. In these examples, the second-level hot word detector 220 may be associated with a larger version of the hot word detection model used by the first-level hot word detector 210, and may include a different neural network that is potentially more computationally intensive than the neural network of the first-level hot word detector 210, thereby providing increased hot word detection accuracy on top of the first-level hot word detector 210, which is limited by the resources of the user device 102. The second-level hot word detector 220 may generate probability scores indicating the presence of hot words in the first segment of the audio data 120, and detect the presence of hot words when the probability scores meet a hot word detection threshold for the second-level hot word detector 220. Here, the value of the hot word detection threshold at the second-level hot word detector 220 may be the same as or different from the value of the hot word detection threshold at the first-level hot word detector 210. In some examples, when it is determined whether a hot word exists in the first audio segment 121, the hot word detection threshold at the second-level hot word detector 220 is set to be higher, so that the second-level hot word detector 220 is more certain.

[0041] In some implementations, the second-level hot word detector 220 is executed on user equipment 102 (e.g., data processing hardware 103) to implement the entire cascaded hot word detection architecture 200 on the device without using remote system 110. When executed on user equipment 102, the second-level hot word detector 220 can be implemented as an on-device ASR engine to detect the presence of hot words by performing speech recognition on the first audio segment 121 or as a larger version of the hot word detection model implemented by the first-level hot word detector 210 to detect the presence of hot words in the first audio segment 121 without performing speech recognition.

[0042] Figure 2 Provided Figure 1An example of a cascaded hot word detection architecture 200 includes a first-level hot word detector 210, a second-level hot word detector 220, and optionally an initial coarse hot word detector 205. In some examples, when the user equipment 102 is a battery-powered device, the data processing hardware 103 of the user equipment 102 collectively includes a first processor 60 (e.g., a digital signal processor (DSP)) and a second processor 70 (e.g., an application processor (AP)). The first processor 60 consumes less power during operation compared to the second processor 70. As used herein, the first processor 60 is interchangeably referred to as a DSP, and the second processor 70 is interchangeably referred to as an "AP" or "device SoC". The initial coarse hot word detector 205 may run on the first processor 60, and the first-level hot word detector 210 may run on the second processor 70. The second-level hot word detector 220 may be executed on a server (e.g., a remote system 110) communicating with the user equipment 102 to provide server-side hot word confirmation that fully utilizes the increased processing power at the server. Alternatively, the second-level hot word detector 220 can run on the second processor 70 of the user device 102 to implement the entire cascaded hot word detection architecture on the device.

[0043] Typically, the coarse hot word detector 205 resides on a dedicated DSP 60, includes a smaller model size than the model associated with the first-level hot word detector 210, and is computationally efficient for coarsely filtering the input streaming audio 118 for hot word detection. Therefore, the dedicated DSP 60 (e.g., the first processor) can be "always on," so that the coarse hot word detector 205 is always running to coarsely filter hot word candidates in the multi-channel audio 118, while all other components of the user equipment 102, including the main AP 70 (e.g., the second processor), are in sleep / mode to conserve battery life. On the other hand, the first-level hot word detector 210 resides on the main AP 70, includes a larger model size, and provides more computational output than the coarse-level hot word detector 205 for providing more accurate detection of hot words initially detected by the coarse hot word detector 205. Therefore, the first-level hot word detector 210 can be more stringent in determining the presence of hot words in the audio 118. When DSP60 is "always on", the more power-intensive main AP 70 operates in sleep mode to conserve battery life until the coarse hot word detector 205 at DSP 60 detects candidate hot words in the streaming audio 118. Therefore, DSP 60 only triggers the main AP 70 to switch from sleep mode to hot word detection mode for running the first-level hot word detector 210 once a candidate hot word is detected.

[0044] Upon receiving streaming audio 118, the always-on DSP 60 executes / runs a coarse hot word detector 205 to determine whether hot words are detected in the corresponding audio features of the streaming audio 118. It is worth noting that the AP 70 can operate in sleep mode when receiving multi-channel audio at the DSP 60, and the AP 70 can also operate in sleep mode when the DSP 60 processes the corresponding audio features of the streaming audio 118.

[0045] When the coarse hot word detector 205 detects a hot word in the streaming audio 118, the DSP 60 provides the AP 70 with segmented audio data 120. In some examples, the DSP 60 providing the segmented audio data 120 to the AP 70 triggers / invokes the AP 70 to transition from sleep mode to hot word detection mode. In some implementations, the segmented audio data 120 from the streaming audio 118 includes a first segment 121 characterizing the hot word detected by the coarse hot word detector 205 in the streaming audio 118. That is, the first segment includes a duration sufficient to safely contain the detected hot word. Additionally, the audio data 212 includes a second segment 122 following the first segment 121, which may include the duration of audio containing a spoken query. The coarse hot word detector 205 is optional, and the first-level hot word detector 210 can initially detect hot word events in the streaming audio 118, the segment processing includes audio data 120 of the first segment 121 and the second segment 122, and provides the segment-processed audio data 120 to the second-level hot word detector 220.

[0046] When the first-level hot word detector 210 detects a hot word in the first segment 121 of audio data 120, AP 70 initiates a wake-up process on user equipment 102 and provides audio data 120 to the second-level hot word detector 220 for processing to determine / confirm whether the second-level hot word detector 220 detected a hot word in the first segment 121 of audio data 120. In an example where the cascaded hot word detection architecture 200 is implemented entirely on the device, AP 70 simply passes audio data 120 to the second-level hot word detector 220, which also runs on AP 70. In other examples, where the second-level hot word detector 220 is implemented at server 110, AP 70 instructs user equipment 102 to transmit audio data 120 to the second-level hot word detector 220 via network 104.

[0047] Return to reference Figure 1In some examples, when the second-level hot word detector 220 does not detect a hot word in the first segment 121 of audio data 120 (i.e., the probability score meets the hot word detection threshold), the negative hot word classifier 300, executed on the remote system 110, classifies the first segment 121 of audio data 120 as containing a negative hot word (e.g., "Hey Poodle"), which leads to a false detection of hot word events in the streaming audio 118 by the first-level hot word detector 210. In a configuration where the second-level hot word detector 220 runs on user device 102, the negative hot word classifier 300 can also be executed on user device 102. The negative hot word classifier 300 can provide a classification result 170 to a negative hot word updater 400, which indicates that the first segment 121 of audio data 120 contains a negative hot word. The classification result 170 can provide a probability score 171 generated by the second-level hot word detector 220 for the first audio segment 121 and / or other relevant information 172 associated with the corresponding classification result 170, which can be useful for the negative hot word updater 400 to update the first-level hot word detector 210. Other relevant information 172 may include, but is not limited to, transcription of utterance 119, speaker identification scores (e.g., speaker embeddings) identifying the speaker characteristics of the user 10 who uttered utterance 119, timestamps indicating hot word events of the time of day and / or day of week, and a negative hot word confidence score 304 indicating the confidence level used to classify the first segment 121 as a negative hot word. Figure 3 In the example shown, the negative hot word updater 400 is executed on user device 102 to personalize hot word detection on user device 102 by updating the first-level hot word detector 210 to prevent hot word events from being triggered in subsequent audio data containing negative hot words.

[0048] Sending the classification result 170 to the negative hot word updater 400 can cause the negative hot word updater 400 to update the first-level hot word detector 210 to prevent the triggering of a hot word event in subsequent audio data containing a negative hot word (e.g., Hey Poodle). In some embodiments, when the second-level hot word detector 220 does not detect a hot word in the first segment 121 of the audio data 120, the remote system 110 (or user equipment 102) suppresses the wake-up process on the user equipment 102 for processing the hot word and / or one or more other terms following the hot word in the streaming audio 118. In some embodiments, the remote system 110 suppresses the wake-up process by sending a suppression instruction 160 to the user equipment 102 that causes the user equipment 102 to suppress the wake-up process. In other embodiments, the provision of a classification result 170 indicating that the first segment 121 of the audio data 120 contains a negative hot word causes the user equipment 102 to suppress the wake-up process. In other embodiments, the remote system 110 suppresses the wake-up process by not responding to the user device 102 after receiving the audio data 120 (e.g., by closing the network connection). The lack of a response from the remote system 110 can cause the user device 102 to suppress the wake-up process. That is, in some examples, the user device 102 initiates the wake-up process only upon receiving confirmation from the second-level hot word detector 220 that a hot word exists in the streaming audio 118. The user device 102 can suppress the wake-up process independently. For example, the user device 102 can automatically suppress the wake-up process when the query or command following the hot word is empty (i.e., the streaming audio 118 following the hot word fails to include a command or query directed to the user device 102).

[0049] In some scenarios, after suppressing the wake-up process caused by the second-level hot word detector 220 failing to detect the presence of a hot word in the first segment 121 of audio data 120, the negative hot word classifier 300 determines whether the user 10 of user device 102 provides an immediate follow-up query. This determination can be made when the second-level hot word detector 220 does not receive a subsequent hot word event detected by the first-level hot word detector 210. Here, the determination that the user 10 did not provide an immediate follow-up query serves as additional confirmation that the user 10 did not previously intend to say the hot word in utterance 119 but instead said the word ("Hey Google") with a pronunciation similar to the specific word / phrase designated as the hot word ("Hey Google"). Therefore, classifying the first segment 121 of audio data 120 as containing a negative hot word can be further based on the determination that no follow-up query was received from user device 102 after suppressing the wake-up process.

[0050] In an additional example, when the second-level hot word detector 220 detects the presence of the hot word (“Hey Google”) in the first segment 121 of audio data 120 (i.e., the probability score meets the hot word detection threshold), even though user 10 actually utters another similar vocal phrase (“Hey Poodle”), the second segment 122 of audio data 120 (and optionally the first segment 121) is provided to the query processor 180. Here, the query processor 180 processes the second segment 122 of audio data 120 to determine whether the second segment 122 of audio data 120 indicates a spoken query type utterance. In an example where the second-level hot word detector 220 is implemented as an ASR engine, the query processor 180 processes the resulting speech recognition result by performing semantic analysis to determine whether the second segment 122 indicates a query type utterance. In other examples, when the second-level hot word detector 220 is implemented as a hot word detection model, the query processor 180 is implemented as an ASR engine that processes the second segment 122 of audio data 120 by performing speech recognition and then performing semantic analysis on the speech recognition result. As used herein, a query-type utterance corresponds to a utterance directed to user device 102, such as a utterance directed to the digital assistant interface used to query the digital assistant to perform an operation or action. Therefore, when the second segment 122 of audio data 120 indicates a query-type utterance, there is an accurate strong likelihood that the second-level hot word detector 220 in the first segment 121 of audio data 120 detected the presence of a hot word. Otherwise, when the query processor 180 determines that the second segment 122 does not indicate a query-type utterance, there is an incorrect strong likelihood that the second-level hot word detector 220 in the first segment 121 detected the presence of a hot word.

[0051] The query processor 180 can provide a score 182 indicating whether the second segment 122 indicates a query-type utterance. In some examples, the score 182 is binary, where a score of zero or one (1) indicates a query-type utterance, and a score of the other of zero or one (1) does not indicate a query-type utterance. In other examples, the score 182 provides a likelihood (e.g., probability) that the second segment 122 indicates a query-type utterance. Here, the second segment 122 may not indicate a query-type utterance if the score 182 fails to meet a query-type utterance threshold. In the example shown, the negative hot word classifier 300 can receive the score 182 as input, in addition to the determination made by the second-level hot word detector 220 to determine whether the first segment 121 of the audio data 120 should be classified as containing a negative hot word.

[0052] Therefore, when the negative hot word classifier 300 receives the second audio segment 122 of the audio data from the query processor 180, which does not indicate a spoken query type utterance, the negative hot word classifier 300 can classify the first segment 121 of the audio data 120 as containing a negative hot word, to indicate that the second-stage hot word detection 220 provides a false acceptance. The negative hot word classifier 300 may additionally receive a probability score 171 generated by the second-stage hot word detector 220 for the first segment 121, whereby only probability scores that satisfy the hot word detection threshold with narrow margin can further bias the negative hot word classifier 300 to classify the first segment 121 as containing a negative hot word. Furthermore, after the query processor 180 determines that the second segment 122 does not indicate a query type utterance, the negative hot word classifier 300 can also determine whether the user 10 of the user device provides an immediate follow-up query. As stated above, user 10 did not provide an immediate follow-up query as additional confirmation that user 10 did not previously intend to say the hot word in utterance 119 but instead pronounced the term ("Hey Google") in a similar manner to the specific term / phrase designated as the hot word ("Hey Google"). Therefore, classifying the first segment 121 of audio data 120 as containing a negative hot word can be further based on the determination that no follow-up query was received from user device 102.

[0053] In some examples, after receiving audio data 120 characterizing a hot word event detected by the first-level hot word detector 210, the remote system 110 receives a negative user interaction 162 indicating user suppression of the wake-up process on the user device 102. That is, when the user utters the negative hot word “Hey Poodle,” a false acceptance instance of the first-level hot word detector 210 in detecting the hot word event can trigger an initial wake-up of the user device 102, while awaiting confirmation or rejection of the hot word's presence by the second-level hot word detector 220. Here, the user device 102 can provide auditory and / or visual notifications to inform the user that the user device 102 is awake, and the user 10 can provide the negative user interaction 162 to return the device 102 to sleep mode because the user 10 does not intend to trigger the wake-up process. For example, the user 10 can press a physical button on the user device, provide a gesture, or, if the user device 102 includes a display, select a graphic rendered in a graphical user interface displayed on the display, which causes the user device 102 to return to sleep mode. In some implementations, the negative hot word classifier 300 uses a negative user interaction 162 that indicates user inhibition of the wake-up process on the user device as input for classifying the first segment 121 of the audio data 120 as containing negative hot words.

[0054] In some additional examples, when query processor 180 determines that the second segment 122 of audio data 120 indicates a spoken query-type utterance, query processor 180 provides a query 185 containing a transcription of the second segment 122 of audio data 120 to search engine 190 (or other downstream application). Here, search engine 190 provides a result 192 in response to query 185 back to user device 102. Here, query processor 180 may have already identified the second segment 122 as indicating a query-type utterance, even if the second segment 122 of audio data 120 corresponds to background speech or other background audio captured by user device 102 in streaming audio 118 after the first-level hot word detector 210 detects a false positive hot word event when the user says “Hey Poodle.” This background audio can be captured in streaming audio 118, and query processor 180 can identify the query-type utterance and provide the corresponding query 185 to search engine 190 to obtain result 192. Result 192 may be audibly and / or visually output to user 10 by user device 102, even if user 10 never intends to invoke user device 102. Consequently, user 10 may provide a negative user interaction 162 instructing user 10 to negatively interact with result 192. For example, user 10 may provide verbal input instructing user 10 to be confused with the result or a statement for which user 10 has not provided a query. Additionally or alternatively, user 10 may provide input instructions / commands instructing user 10 to deselect result 192.

[0055] In other scenarios, result 192 is a prompt from the digital assistant stating an audible output from device 102, requiring user 10 to provide confirmation of the action performed, such as, “You asked for the current weather, is that correct?” Thus, negating user interaction could be user 10 saying, “No, I did not ask about the weather.” Similarly, result 192 could be a prompt requesting the user to repeat the query because the query processor 180 is unsure about the query, such as, “I did not understand your question, please repeat?” Thus, negating user interaction could be user 10 expressing confusion by saying “Huh,” user 10 assertively eliminating the prompt by saying “I did not ask anything,” or simply by the user not responding within a predetermined time period. Therefore, in addition to one or more of the other inputs discussed above, negative user interactions 162 can be provided to the negative hot word classifier 300, such as an indication that the second-level hot word detector 220 did not detect a hot word in the first segment of audio data 120, an indication that the second segment 122 of audio data 120 is not associated with the query type utterance, or an indication that the user 10 did provide an immediate follow-up query after the wake-up process was suppressed.

[0056] Figure 3 It shows Figure 1 The negative hot word classifier 300 receives one or more input features 302 to determine whether the first segment 121 of the audio data 120 should be classified as a negative hot word. When the negative hot word classifier 300 determines based on one or more input features 302 that the first segment 121 of the audio should be classified as a negative hot word, the negative hot word classifier 300 will generate a classification result 170 indicating that the first segment 121 of the audio data 120 is classified as a negative hot word, as described above. Figure 1As discussed herein, one or more input features 302 received by the negative hot word classifier 300 may include, but are not limited to: whether the second hot word detector 220 detects the presence of a hot word and / or probability score 171 in the first segment 121 of audio data 120, whether an immediate follow-up query is received from user device 102, whether the second segment 122 of audio data 120 is provided with a score 182 indicating whether the second segment 122 includes a query-type utterance (e.g., by providing a score 182 indicating whether the second segment 122 indicates a query-type utterance), transcription of the first segment 121 and / or the second segment 122 of audio data 120, and whether a negative user interaction 162 indicating user inhibition of a wake-up process on user device 102 is received, and / or whether user 10 negatively interacts with result 192 in response to processing the second segment 122 (and / or optionally the first segment 121) of audio data 120 as a query-type utterance (e.g., providing query 185 to search engine 190 or other downstream applications).

[0057] When determining whether the first segment 121 of audio data 120 should be classified as a negative hot word, some input features 302 can be weighted more heavily. For example, a second-level hot word detector 220 that fails to detect the presence of a hot word in the first segment 121 is a strong indication that the first segment 121 includes a negative hot word that would lead to a false acceptance instance at the first-level hot word detector 210. The magnitude of the probability score 171 can bias the classification result 170. For example, at the second-level hot word detector 220, a probability score 171 failing to meet the hot word detection threshold with a large margin provides a greater likelihood of a negative hot word than a probability score 171 missing the hot word detection threshold with a small margin.

[0058] In some configurations, the negation hot word classifier 300 includes a trained classifier (which may include a neural network model trained via machine learning) configured to generate a negation hot word confidence score 304 indicating the likelihood that the first segment 121 of the audio data 120 contains a negation hot word. When the negation hot word confidence score 304 meets a confidence threshold, the classifier 300 can classify the first segment 121 as containing a negation hot word. The negation hot word confidence score 304 may be included in the list of negative hot words in the audio data 120. Figure 1 The negative hot word updater 400 receives the classification results 170 to update the first-level hot word detector 210 so as not to detect hot word events in subsequent audio containing negative hot words. In some examples, the negative hot word confidence score 304 is a binary score indicating whether the first segment 121 of the audio data 120 contains a negative hot word—and should therefore be classified as a negative hot word—or whether the first segment 121 does not contain a negative hot word.

[0059] refer to Figure 1 and Figure 4In some examples, the negative hot word updater 400 updates the first-level hot word detector 210 by providing the user device 102 with a classification result 170 that includes the first segment 121 of the audio data 120 to prevent the triggering of hot word events in subsequent audio data that includes negative hot words. Here, the user device 102 can be configured to retain the first-level hot word detector 210 using the first segment 121 of the audio data 120 that is classified as containing negative hot words. For example, the first segment 121 of the audio data 120 can be labeled as a negative hot word and provided as training input to the first-level hot word detector 210, so that the first-level hot word detector 210 learns not to detect the presence of hot words ("Hey Google") in subsequent audio data that contain negative hot words ("Hey Poodle"). As used herein, retraining the first-level hot word detector 210 may include retraining an existing hot word detector 210 running on user device 102 that initially detected hot word events incorrectly, or may include a new first-level hot word detector 210 that is later pushed to user device 102.

[0060] Furthermore, updating the first-level hot word detector 210 may also include updating the DSP 60 when the user device 102 includes a battery-powered device. Figure 2 An optional initial coarse hot word detector 205 runs on the system. Similar to the first-level hot word detector 210, updating the initial coarse hot word detector 205 may include retraining the initial coarse hot word detector 205 to prevent hot word events from being triggered in audio data that includes negative hot words. In some examples, only the initial coarse hot word detector 205 is updated to prevent hot word events from being triggered in subsequent audio data that includes negative hot words.

[0061] like Figure 4 As shown, the negation hot word updater 400 is executed on user device 102, storing each instance of the first segment 121 of audio data 120 classified by the negation hot word classifier 300 as containing the corresponding negation hot word in memory hardware 105. In the example shown, user device 102 stores each instance of the first segment 121 of audio data 120 classified as a negation hot word by storing a corresponding classification result 170 for each instance. Here, the classification result 170 includes the first segment 121 being classified as containing a negation hot word, a probability score 171 indicating the likelihood that the first segment 121 contains the actual hot word, and other relevant information 172 such as the transcription of utterance 119, a speaker identification score (e.g., speaker embedding) identifying the speaker characteristics of the user 10 who uttered utterance 119, a timestamp indicating the hot word event of the time of day and / or day of week, and a negation hot word confidence score 304 indicating the confidence used to classify the first segment 121 as a negation hot word. Figure 3In the example shown, user device 102 stores one or more classification results 170, each associated with a different corresponding negative hot word. For example, one or more classification results 170Aa-n, 170Ba-n, 170Ca-n may be stored for each of the negative hot words “Poodle,” “Doodle,” and “Noodle,” which sound similar to the specified hot word “Hey Google” when spoken by user 10.

[0062] In some implementations, user device 102 (via negation hot word updater 400) is configured to retrain the first-level hot word detector 210 based on an aggregation of the number of instances (e.g., the number of classification results 170) of the first segment 121 of audio data 120 classified as containing negation hot words stored in memory hardware 105. Here, the number of instances of audio data classified as containing the same hot word that meets a threshold number of instances can establish a pattern in which user 10 regularly utters negation hot words that are incorrectly detected as specified hot words by the first-level hot word detector 210. In some examples, when the negation hot word confidence score 304 associated with the score is relatively low—for example, the negation hot word confidence score 304 only meets the threshold with narrow margin—user device 102 requires a specified number of misacceptance instances generated by user 10 uttering the same word.

[0063] Continue to refer to Figure 4 In some examples, the negation hot word updater 400 may append the embedding representation 12 to each classification result 170 stored in the memory hardware 105. Here, the first-level hot word detector 210 may compute the embedding representation 12 of any audio data characterizing the hot word events detected by the first-level hot word detector 210, and when the audio data 120 is subsequently classified by the negation hot word classifier 300 as containing a negation hot word, the negation hot word updater 400 may append the embedding representation 12 to the corresponding instance of the classification result 170. In some implementations, the negation hot word updater 400 aggregates / averages the embedding representations 12 stored in the memory hardware 105 for each corresponding negation hot word to generate a reference embedding 15 for each corresponding negation hot word. For example, a corresponding reference embedding 15 may be generated for each negation hot word “Poodle”, “Noodle”, and “Doodle”.

[0064] Figure 5A schematic diagram 500 depicting an example is shown, in which user device 102 captures subsequent audio data 120 corresponding to another utterance 519 spoken by user 10, including the term "My Poodle," which causes the first-level hot word detector 210 to erroneously detect another hot word event. The first-level hot word detector 210, executed on user device 102, computes an evaluation embedding representation 18 of the subsequent audio data 120 (e.g., a portion of the subsequent audio data 120 characterizing the hot word event). User device 102 can simultaneously access classification results 170 stored in memory hardware 105, which may each include an embedding representation 12 of the corresponding first segment 121 of the audio data 120 classified as one of the negative hot words (e.g., "Poodle," "Noodle," and "Doodle"). Additionally or alternatively, user device 102 can access corresponding reference embeddings 15 generated for each negative hot word, as referenced above. Figure 4 As described.

[0065] In some implementations, the scorer 510 compares the evaluation embedding representation 18 computed for subsequent audio data 120 with all reference embeddings 15 generated and stored for each of the negative hot words. In these implementations, the reference embedding representations 12, 15 associated with each negative hot word “Poodle,” “Doodle,” and “Noodle” are all clustered together in an embedding representation space that is different from the clustering of reference embeddings associated with other negative hot words. Therefore, the scorer 510 can determine a similarity score 515 between the reference embedding representation 12 computed for each first segment 121 of audio data 120 classified as any of the negative hot words and the evaluation embedding representation 18 for subsequent audio data 120. Additionally or alternatively, the scorer 510 can determine a similarity score 515 between each corresponding reference embedding representation 15 representing the aggregated / averaged embedding representation of a corresponding one of the negative hot words (e.g., “Poodle,” “Doodle,” and “Noodle”). In some examples, each similarity score 515 is associated with the distance (e.g., cosine distance) between the reference embedding representations 12 and 15 in the embedding representation space and the evaluation embedding representation 18.

[0066] After the scorer 510 determines / generates a similarity score 515, the classifier 520 can compare each similarity score 515 with a similarity score threshold, and determine / classify subsequent audio data 120 as including negative hot words when the similarity score 515 meets the similarity score threshold. In some examples, the similarity score threshold represents the maximum permissible cosine distance between embedding representations associated with the same negative hot words. In some scenarios, multiple similarity scores 515 may meet the similarity score threshold when calculating the similarity score 515 between the evaluated embedding 18 for subsequent audio and the corresponding reference embedding representation 12 computed for each instance of the first segment 121 of the audio data 120 classified as a negative hot word. To illustrate, in the example shown, the similarity score 515 between the evaluation embedding 18 classified as the negative hot word "Poodle" and the corresponding reference embedding representation 12 will satisfy a similarity score threshold, indicating that the evaluation embedding 18 falls within the cluster of the embedding representation 12 classified as the negative hot word "Poodle" and outside the cluster of the embedding representations 12 classified as other negative hot words "Doodle" and "Noodle". Therefore, when classifier 520 determines that the similarity score 515 satisfies the similarity threshold, classifier 520 determines that subsequent audio data 120 includes negative hot words. As a result, if the first-level hot word detector 210 assertively and incorrectly detects a hot word event and triggers the initiation of a wake-up process on the user device, classifier 520 can instruct the first-level hot word detector 210 to suppress the detection of hot word events in subsequent audio data, or guide the user device 102 to return to a sleep state.

[0067] Figure 6 A flowchart of example operations is provided for a method 600 for personalizing a hot word detector on a user device based on negative hot words from previous instances that resulted in false positives in hot word detection caused by the hot word detector. At operation 602, method 600 includes receiving audio data 120 at data processing hardware 103, 112, the audio data 120 representing hot word events detected by a first-level hot word detector 210 in streaming audio 118 captured by user device 102. The first-level hot word detector 210 may be executed on a digital signal processor (DSP) of the data processing hardware 103 of user device 102, or on an application processor of the data processing hardware 103 of user device 102.

[0068] At operation 604, method 600 includes processing audio data 120 using a second-level hot word detector 220 by data processing hardware 103, 112 to determine whether the second-level hot word detector 220 detected a hot word in a first segment 121 of the audio data 120. The second-level hot word detector 220 may be implemented as an ASR engine that performs automatic speech recognition to determine whether a hot word is recognized in the first segment 121. In other configurations, the second-level hot word detector 220 may be implemented as a hot word detection model that determines whether a hot word is detected in the first segment 121 without performing speech recognition.

[0069] At operation 606, when the second-level hot word detector 220 does not detect a hot word in the first segment 121 of audio data 120, method 600 includes classifying the first segment 121 of audio data 120 as containing a negative hot word that causes a false detection of a hot word event in the streaming audio 118 by the first-level hot word detector 210. At operation 608, method 600 includes updating the first-level hot word detector 210 by the data processing hardware 103, 112 to prevent the triggering of a hot word event in subsequent audio data 120 containing a negative hot word.

[0070] A software application (i.e., a software resource) can refer to computer software that causes a computing device to perform a task. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0071] Figure 7 This is a schematic diagram of an example computing device 700 that can be used to implement the systems and methods described in this document. The computing device 700 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the inventions described and / or claimed in this document.

[0072] Computing device 700 includes a processor 710, a memory 720, a storage device 730, a high-speed interface / controller 740 connected to the memory 720 and a high-speed expansion port 750, and a low-speed interface / controller 760 connected to a low-speed bus 770 and the storage device 730. Each of components 710, 720, 730, 740, 750, and 760 is interconnected using various buses and may be mounted on a common motherboard or otherwise suitably mounted. The processor (e.g., data processing hardware) 710 can process instructions for execution within computing device 700, including instructions stored in the memory 720 (e.g., memory hardware) or on the storage device 730, to display graphical information of a graphical user interface (GUI) on an external input / output device, such as a display 780 coupled to the high-speed interface 740. In other embodiments, multiple processors and / or multiple buses, as well as multiple memories and memory types, may be suitably used. Furthermore, multiple computing devices 700 can be connected, each providing a portion of the necessary operation (e.g., as a server library, a set of blade servers, or a multiprocessor system). The processor 710 may include residing in... Figure 1 Data processing hardware 103 on user equipment 102 or Figure 1 Data processing hardware 112 on remote system 110.

[0073] The memory 720 non-transitory stores information within the computing device 700. The memory 720 may include memory residing in... Figure 1 Memory hardware 105 on user equipment 102 or Figure 1 The memory hardware 114 is located on the remote system 110. The memory 720 can be a computer-readable medium, a volatile memory cell, or a non-volatile memory cell. The non-volatile memory 720 can be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by the computing device 700. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used in firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.

[0074] Storage device 730 provides mass storage for computing device 700. In some embodiments, storage device 730 is a computer-readable medium. In various embodiments, storage device 730 may be a floppy disk device, hard disk device, optical disk device, magnetic tape device, flash memory or other similar solid-state storage device, or device array, including devices in a storage area network or other configuration. In additional embodiments, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as memory 720, storage device 730, or memory on processor 710.

[0075] High-speed controller 740 manages bandwidth-intensive operations for computing device 700, while low-speed controller 760 manages lower bandwidth-intensive operations. This assignment of responsibilities is merely exemplary. In some embodiments, high-speed controller 740 is coupled to memory 720, display 780 (e.g., via a graphics processor or accelerator), and to high-speed expansion port 750, which can accept various expansion cards (not shown). In some embodiments, low-speed controller 760 is coupled to storage device 730 and low-speed expansion port 790. Low-speed expansion port 790, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), may be coupled to one or more input / output devices, such as keyboards, pointing devices, scanners, or networking devices such as switches or routers, for example, via a network adapter.

[0076] The computing device 700 can be implemented in many different forms, as shown in the figure. For example, it can be implemented as a standard server 700a or multiple servers 700a in a group, as a laptop computer 700b, or as part of a rack server system 700c.

[0077] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuit systems, integrated circuit systems, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs executable and / or interpretable on a programmable system, the programmable system including at least one programmable processor, which may be dedicated or general-purpose, coupled to receive and send data and instructions from and to a storage system, at least one input device, and at least one output device.

[0078] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0079] The processes and logical flows described in this specification can be executed by one or more programmable processors—also known as data processing hardware—that execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logical flows can also be executed by special-purpose logic circuit systems (e.g., FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits)). Processors suitable for executing computer programs include, for example, general-purpose and special-purpose microprocessors and any one or more processors of any kind of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include, or be operatively coupled to, receiving data from or transferring data to one or more mass storage devices, or both, for storing data, such as magnetic, magneto-optical, or optical discs. However, a computer does not need to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry systems.

[0080] To provide interaction with a user, one or more aspects of this disclosure can be implemented on a computer having a display device for displaying information to the user—such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touchscreen—and optionally including a keyboard and pointing device through which the user can provide input to the computer—such as a mouse or trackball. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, the computer can interact with the user by sending and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's client device in response to a request received from a web browser.

[0081] Several embodiments have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other embodiments are within the scope of the appended claims.

Claims

1. A computer-implemented method (600) for adaptive hot word recognition based on personalized negation, said method, when executed on data processing hardware (710), causing said data processing hardware (710) to perform operations, said operations including: Receive audio data (120) representing hot word events detected by a first-level hot word detector (210) in streaming audio (118) captured by the user equipment (102). The audio data (120) is processed using a second-level hot word detector (220) to determine whether the second-level hot word detector (220) detects a hot word in the first segment (121) of the audio data (120); as well as When the second-level hot word detector (220) does not detect the hot word in the first segment (121) of the audio data (120): The first segment (121) of the audio data (120) is classified as containing negative hot words, which cause false detection of the hot word event in the streaming audio (118) by the first-level hot word detector (210); as well as Based on the first segment (121) of the audio data (120) classified as containing the negative hot word, the first-level hot word detector (210) is updated to prevent the hot word event from being triggered in subsequent audio data (120) containing the negative hot word. The first-level hot word detector is updated based on at least one of the following: The hot word detection probability score determined by the second-level hot word detector (220) for the first segment (121) of the audio data; or The negative hot word confidence score determined by the second-level hot word detector (220) classifies the first segment (121) of the audio data as the negative hot word.

2. The method (600) according to claim 1, wherein, The operation further includes, when the second-level hot word detector (220) does not detect the hot word in the first segment (121) of the audio data (120): Suppress the wake-up process on the user equipment (102) for processing the hot word and / or one or more other terms following the hot word in the streaming audio (118); as well as After suppressing the wake-up process on the user equipment (102), it is determined whether the user of the user equipment (102) has provided an immediate follow-up query. The first segment (121) of the audio data (120) is classified as containing the negative hot words, further based on the determination that the user of the user device (102) did not provide a follow-up query after the wake-up process was suppressed.

3. The method (600) according to claim 1, wherein, The operation further includes: when the second-level hot word detector (220) detects the hot word in the first segment (121) of the audio data (120): Processing the second segment (122) of the audio data (120) following the first segment (121) of the audio data (120) to determine whether the second segment (122) of the audio data (120) indicates a spoken query type utterance; and When the second segment of the audio data (120) does not indicate the spoken query type utterance: The first segment (121) of the audio data (120) is classified as containing the negative hot words; and Based on the first segment (121) of the audio data (120) classified as containing the negative hot word, the first-level hot word detector (210) is updated to prevent the hot word event from being triggered in subsequent audio data (120) containing the negative hot word.

4. The method (600) according to claim 3, wherein, The operation further includes: when the second segment of the audio data (120) does not indicate the spoken query type utterance: Determine whether the user of the user equipment (102) provides real-time follow-up queries. The first segment (121) of the audio data (120) is classified as containing the negative hot words, further based on the determination that the user of the user device (102) has not provided follow-up queries.

5. The method (600) according to claim 3, wherein, The operation further includes: when the second segment of the audio data (120) indicates the spoken query type utterance: Receive a negative interaction result, the negative interaction result instructing the user of the user equipment (102) to negatively interact with the result (192) of the verbal query type utterance provided to the user equipment (102); Based on the received negative interaction results, the first segment (121) of the audio data (120) is classified as containing the negative hot words; and Based on the first segment (121) of the audio data (120) classified as containing the negative hot word, the first-level hot word detector (210) is updated to prevent the detection of the hot word event in subsequent audio data (120) containing the negative hot word.

6. The method (600) according to claim 1, wherein, The operation further includes: after receiving the audio data (120) characterizing the hot word event detected by the first-level hot word detector (210): Receive a negative user interaction (162) that indicates user suppression of the wake-up process on the user equipment (102). The first segment (121) of the audio data (120) is classified as the negative user interaction (162) that includes the negative hot words and further indicates user inhibition of the wake-up process.

7. The method (600) according to claim 1, wherein, Updating the first-level hot word detector (210) to prevent the hot word event from being triggered in subsequent audio data (120) includes providing the user device (102) with the first segment (121) of the audio data (120) classified as containing the negative hot word, the user device (102) being configured to retrain the first-level hot word detector (210) using the first segment (121) of the audio data (120) classified as containing the negative hot word.

8. The method (600) according to claim 7, wherein, The user equipment (102) is configured to retrain the first-level hot word detector (210) by: Each instance of the first segment (121) of the audio data (120) classified as contained in the memory hardware (720) of the user equipment (102) is stored in the memory hardware (720) of the user equipment (102); and The first-level hot word detector (210) is retrained based on the aggregation of the number of instances of the first segment (121) of the audio data (120) of the negative hot words stored in the memory hardware (720).

9. The method (600) according to claim 8, wherein, The user equipment (102) is further configured to: before retraining the first-level hot word detector (210): The confidence score associated with each instance of the first segment (121) of the audio data (120) classified as containing the negative hot word failed to meet the negative hot word threshold score; and The number of instances is determined to exceed the threshold number of instances.

10. The method (600) according to claim 1, wherein: Updating the first-level hot word detector (210) to prevent the hot word event from being triggered in subsequent audio data (120) includes providing the user device (102) with the first segment (121) of the audio data (120) classified as containing the negative hot word, the user device (102) being configured to: Obtain the embedded representation (12) of the first segment (121) of the audio data (120); and The embedded representation (12) of the first segment (121) of the audio data (120) is stored in the memory hardware (720) of the user equipment (102); and The user equipment (102) is configured to determine when subsequent audio data (120) characterizing the hot word event detected by the first-level hot word detector (210) includes the negative hot word by: Calculate the evaluation embedding representation (18) of the subsequent audio data (120); Determine a similarity score (515) between the embedding representation (12) of the first segment (121) of the audio data (120) classified as the negative hot word and the evaluation embedding representation (18) of the subsequent audio data (120); and When the similarity score (515) meets the similarity score threshold, it is determined that the subsequent audio data (120) includes the negative hot words.

11. The method (600) according to claim 1, wherein: The data processing hardware (710) resides on a server (110) that communicates with the data processing hardware (710); and The first-level hot word detector (210) is executed on the processor of the user equipment (102).

12. The method (600) according to claim 11, wherein, Processing the audio data (120) to determine whether the second-level hot word detector (220) detects the hot word in the first segment (121) of the audio data (120) includes performing automatic speech recognition to determine whether the hot word is recognized in the first segment (121) of the audio data (120).

13. The method (600) according to claim 1, wherein, The data processing resides on the user equipment (102).

14. The method (600) according to claim 13, wherein: The first-level hot word detector (210) is executed on the digital signal processor (DSP) (60) of the data processing hardware (710); as well as The second-level hot word detector (220) is executed on the application processor (70) of the data processing hardware (710).

15. The method (600) according to any one of claims 1 to 14, wherein, The first-level hot word detector (210) is configured as follows: Generate a probability score (171) indicating the presence of the hot words in the audio features of the streaming audio (118) captured by the user equipment (102); and When the probability score (171) meets the hot word detection threshold of the first-level hot word detector (210), the hot word event in the streaming audio (118) is detected.

16. A system (100) for adaptive hot word recognition based on personalized negation, comprising: Data processing hardware (710); as well as A memory hardware (720) communicating with the data processing hardware (710), the memory hardware (720) storing instructions that, when executed on the data processing hardware (710), cause the data processing hardware (710) to perform operations, the operations including: Receive audio data representing hot word events detected by a first-level hot word detector in streaming audio captured by a user equipment; The audio data (120) is processed using a second-level hot word detector (220) to determine whether the second-level hot word detector (220) detects a hot word in the first segment (121) of the audio data (120); and When the second-level hot word detector (220) does not detect the hot word in the first segment (121) of the audio data (120): The data processing hardware (710) classifies the first segment (121) of the audio data (120) as containing a negative hot word, which causes a false detection of the hot word event in the streaming audio (118) by the first-level hot word detector (210); and Based on the first segment (121) of the audio data (120) classified as containing the negative hot word, the first-level hot word detector (210) is updated by the data processing hardware (710) to prevent the hot word event from being triggered in subsequent audio data (120) containing the negative hot word. The first-level hot word detector is updated based on at least one of the following: The hot word detection probability score determined by the second-level hot word detector (220) for the first segment (121) of the audio data; or The negative hot word confidence score determined by the second-level hot word detector (220) classifies the first segment (121) of the audio data as the negative hot word.

17. The system (100) according to claim 16, wherein, The operation further includes: when the second-level hot word detector (220) does not detect the hot word in the first segment (121) of the audio data (120): Suppress the wake-up process on the user equipment (102) for processing the hot word and / or one or more other terms following the hot word in the streaming audio (118); and After suppressing the wake-up process on the user equipment (102), it is determined whether the user of the user equipment (102) has provided an immediate follow-up query. The first segment (121) of the audio data (120) is classified as containing the negative hot words, further based on the determination that the user of the user device (102) did not provide a follow-up query after the wake-up process was suppressed.

18. The system (100) according to claim 16, wherein, The operation further includes: when the second-level hot word detector (220) detects the hot word in the first segment (121) of the audio data (120): Processing the second segment (122) of the audio data (120) following the first segment (121) of the audio data (120) to determine whether the second segment (122) of the audio data (120) indicates a spoken query type utterance; and When the second segment of the audio data (120) does not indicate the spoken query type utterance: The first segment (121) of the audio data (120) is classified as containing the negative hot words; and Based on the first segment (121) of the audio data (120) classified as containing the negative hot word, the first-level hot word detector (210) is updated to prevent the hot word event from being triggered in subsequent audio data (120) containing the negative hot word.

19. The system (100) according to claim 18, wherein, The operation further includes: when the second segment of the audio data (120) does not indicate the spoken query type utterance: Determine whether the user of the user equipment (102) provides real-time follow-up queries. The first segment (121) of the audio data (120) is classified as containing the negative hot words, further based on the determination that the user of the user device (102) has not provided follow-up queries.

20. The system (100) according to claim 18, wherein, The operation further includes: when the second segment of the audio data (120) indicates the spoken query type utterance: Receive a negative interaction result, the negative interaction result instructing the user of the user equipment (102) to negatively interact with the result (192) of the verbal query type utterance provided to the user equipment (102); Based on the received negative interaction results, the first segment (121) of the audio data (120) is classified as containing the negative hot words; and Based on the first segment (121) of the audio data (120) classified as containing the negative hot word, the first-level hot word detector (210) is updated to prevent the detection of the hot word event in subsequent audio data (120) containing the negative hot word.

21. The system (100) according to claim 16, wherein, The operation further includes: after receiving the audio data (120) characterizing the hot word event detected by the first-level hot word detector (210): Receive a negative user interaction (162) that indicates user suppression of the wake-up process on the user equipment (102). The first segment (121) of the audio data (120) is classified as the negative user interaction (162) that includes the negative hot words and further indicates user inhibition of the wake-up process.

22. The system (100) according to claim 16, wherein, Updating the first-level hot word detector (210) to prevent the hot word event from being triggered in subsequent audio data (120) includes providing the user device (102) with the first segment (121) of the audio data (120) classified as containing the negative hot word, the user device (102) being configured to retrain the first-level hot word detector (210) using the first segment (121) of the audio data (120) classified as containing the negative hot word.

23. The system (100) according to claim 22, wherein, The user equipment (102) is configured to retrain the first-level hot word detector (210) by: Each instance of the first segment (121) of the audio data (120) classified as contained in the memory hardware (720) of the user equipment (102) is stored in the memory hardware (720) of the user equipment (102); and The first-level hot word detector (210) is retrained based on the aggregation of instances of the first segment (121) of the audio data (120) that are classified as negative hot words stored in the memory hardware (720).

24. The system (100) according to claim 23, wherein, The user equipment (102) is further configured to: before retraining the first-level hot word detector (210): The confidence score associated with each instance of the first segment (121) of the audio data (120) classified as containing the negative hot word failed to meet the negative hot word threshold score; and It is determined that the number of instances exceeds a threshold number of instances.

25. The system (100) according to claim 16, wherein: Updating the first-level hot word detector (210) to prevent the hot word event from being triggered in subsequent audio data (120) includes providing the user device (102) with the first segment (121) of the audio data (120) classified as containing the negative hot word, the user device (102) being configured to: Obtain the embedded representation (12) of the first segment (121) of the audio data (120); and The embedded representation (12) of the first segment (121) of the audio data (120) is stored in the memory hardware (720) of the user equipment (102); and The user equipment (102) is configured to determine when subsequent audio data (120) characterizing the hot word event detected by the first-level hot word detector (210) includes the negative hot word by: Calculate the evaluation embedding representation (18) of the subsequent audio data (120); Determine a similarity score (515) between the embedding representation (12) of the first segment (121) of the audio data (120) classified as the negative hot word and the evaluation embedding representation (18) of the subsequent audio data (120); and When the similarity score (515) meets the similarity score threshold, it is determined that the subsequent audio data (120) includes the negative hot words.

26. The system (100) according to claim 16, wherein: The data processing hardware (710) resides on a server (110) that communicates with the data processing hardware (710); and The first-level hot word detector (210) is executed on the processor of the user equipment (102).

27. The system (100) according to claim 26, wherein, Processing the audio data (120) to determine whether the second-level hot word detector (220) detects the hot word in the first segment (121) of the audio data (120) includes performing automatic speech recognition to determine whether the hot word is recognized in the first segment (121) of the audio data (120).

28. The system (100) according to claim 16, wherein, The data processing hardware resides on the user equipment (102).

29. The system (100) according to claim 28, wherein: The first-level hot word detector (210) is executed on the digital signal processor (DSP) (60) of the data processing hardware (710); as well as The second-level hot word detector (220) is executed on the application processor (70) of the data processing hardware (710).

30. The system (100) according to any one of claims 16 to 29, wherein, The first-level hot word detector (210) is configured as follows: Generate a probability score (171) indicating the presence of the hot words in the audio features of the streaming audio (118) captured by the user equipment (102); and When the probability score (171) meets the hot word detection threshold of the first-level hot word detector, the hot word event in the streaming audio (118) is detected.

Citation Information

Patent Citations

  • Method and device for outputting information

    CN111640426A