Automatic hot word threshold tuning
By using cascaded hot word detection technology, the threshold of the first-level hot word detector is dynamically adjusted, which solves the problem of incorrect acceptance and rejection in hot word detection of voice-enabled devices, and achieves higher detection accuracy and user experience.
Patent Information
- Application Number
- CN202180040933.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-06-10
- Filing Date
- 2021-03-15
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2041-03-15
AI Technical Summary
Existing voice-enabled devices are prone to incorrect acceptance and rejection during hot word detection, leading to inaccuracies in the wake-up process and an inability to adapt to different environments and individual user differences.
A cascaded hot word detection technology is adopted, using first-level and second-level hot word detectors. The results of the first-level detector are verified by a remote computing device, and the threshold of the first-level hot word detector is dynamically adjusted to reduce false acceptances and false rejections.
It improves the accuracy of hot word detection, reduces unnecessary wake-up processes, enhances user experience, and adapts to different environments and individual user differences.
Smart Images

Figure CN115699165B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to automatic hot word threshold tuning. Background Technology
[0002] Voice-enabled environments (e.g., home, workplace, school, car, etc.) allow users to speak queries or commands aloud to a computer-based system, which responds and answers the queries and / or performs command-based functions. Voice-enabled environments can be achieved using a network of connected microphone devices distributed across various rooms or areas of the environment. These devices can use hot words to help identify when a given utterance is directed at the system, rather than at another person present in the environment. Therefore, the device can operate in sleep or hibernation mode and only wake up when the detected utterance includes a hot word. Typically, the system used to detect hot words in streaming audio generates a probability score indicating the probability that the hot word is present in the streaming audio. When the probability score meets a predetermined threshold, the device initiates a wake-up process. Summary of the Invention
[0003] One aspect of this disclosure provides a method for automatic hot word threshold tuning. The method includes: at data processing hardware, receiving audio data from a user equipment performing a first-level hot word detector, representing hot words detected by the first-level hot word detector in streaming audio captured by the user equipment. The first-level hot word detector is configured to generate probability scores indicating the presence of hot words in audio features of the streaming audio captured by the user equipment, and to detect hot words in the streaming audio when the probability scores satisfy a hot word detection threshold of the first-level hot word detector.
[0004] The method also includes processing the audio data using a second-level hot word detector by data processing hardware to determine whether the second-level hot word detector detected a hot word in the audio data. When the second-level hot word detector does not detect a hot word in the audio data, the method includes identifying an erroneous acceptance instance at the first-level hot word detector, indicating that the first-level hot word detector incorrectly detected a hot word in the streaming audio, by the data processing hardware.
[0005] The method also includes determining, by data processing hardware, whether the false acceptance rate associated with the first-level hot word detector of the user device meets a false acceptance rate threshold. The false acceptance rate is based on the number of false acceptance instances identified at the first-level hot word detector during the false acceptance period. When the false acceptance rate associated with the first-level hot word detector meets the false acceptance rate threshold, the method includes adjusting the hot word detection threshold of the first-level hot word detector by the data processing hardware.
[0006] Implementations of this disclosure may include one or more of the following optional features. In some implementations, the method further includes, when the second-level hot word detector does not detect a hot word in the audio data, the data processing hardware suppresses a wake-up process on the user device for processing the hot word and / or one or more other terms following the hot word in the streaming audio. In some examples, the method further includes, when the second-level hot word detector detects a hot word in the audio data, the data processing hardware determines whether subsequent audio data representing a verbal query following the hot word in the streaming audio is received from the user device. When no subsequent audio data representing a verbal query is received from the user device, the method may include: the data processing hardware identifying an erroneous acceptance instance at the first-level hot word detector indicating that the first-level hot word detector incorrectly detected the hot word in the streaming audio.
[0007] Optionally, the method further includes processing the verbal query by data processing hardware when subsequent audio representing the verbal query is received from the user equipment. The user equipment can be configured to initiate a wake-up procedure to process the hot word and / or one or more other terms following the hot word in the streaming audio when the first-level hot word detector detects a hot word in the streaming audio. In some examples, adjusting the hot word detection threshold of the first-level hot word detector includes increasing the value of the hot word detection threshold.
[0008] The method may further include, upon receiving audio data characterizing hot words detected in the streaming audio by the first-level hot word detector, receiving at the data processing hardware a near-miss indication from the user equipment indicating that the first-level hot word detector detected hot words in the streaming audio within a threshold time period after generating a previous probability score that failed to meet the hot word detection threshold with a threshold margin. The previous probability score indicates the presence of hot words in the previous audio features of the streaming audio captured by the user equipment.
[0009] When the second-level hot word detector detects a hot word in the audio data, the method may include: having data processing hardware identify erroneous rejection instances at the first-level hot word detector based on near-miss indications, indicating that the first-level hot word detector failed to initially detect the hot word in previous audio features of the streaming audio; and having the data processing hardware determine whether the erroneous rejection rate associated with the first-level hot word detector of the user device meets an erroneous rejection rate threshold. The erroneous rejection rate is based on the number of erroneous rejection instances identified at the first-level hot word detector during the erroneous rejection period. When the erroneous rejection rate associated with the first-level hot word detector meets the erroneous rejection rate threshold, the method may include adjusting the hot word detection threshold of the first-level hot word detector by the data processing hardware. In some examples, adjusting the hot word detection threshold includes lowering the hot word detection threshold of the first-level hot word detector.
[0010] Another aspect of this disclosure provides a different method for providing automatic hot word threshold tuning. The method includes receiving streaming audio captured by one or more microphones in communication with the data processing hardware at a user device. The method further includes generating a probability score by the data processing hardware using a first-level hot word detector, indicating the presence of hot words in audio features of the streaming audio. The method includes determining by the data processing hardware whether the probability score satisfies a hot word detection threshold.
[0011] When the probability score meets the hot word detection threshold, the method includes detecting hot words in the streaming audio by data processing hardware, and transmitting audio data representing the hot words detected in the streaming audio using the first-level hot word detector to a remote computing device performing a second-level hot word detector. The remote computing device is configured to determine whether the second-level hot word detector detected a hot word in the audio data, and when the second-level hot word detector did not detect a hot word in the audio data, to identify an erroneous acceptance instance at the first-level hot word detector indicating that the first-level hot word detector incorrectly detected a hot word in the streaming audio. When the erroneous acceptance rate based on the number of erroneous acceptance instances identified at the first-level hot word detector within the erroneous acceptance period meets an erroneous acceptance rate threshold, the method includes adjusting the hot word detection threshold of the first-level hot word detector by the data processing hardware.
[0012] This aspect may include one or more of the following optional features. Adjusting the hot word detection threshold of the first-level hot word detector may include increasing the value of the hot word detection threshold. In some examples, when the probability score meets the hot word detection threshold, the method includes initiating a wake-up process on the user device by the data processing hardware to process the hot word and / or one or more other terms following the hot word in the streaming audio. When the second-level hot word detector does not detect a hot word in the audio data, the method may include suppressing the wake-up process on the user device by the data processing hardware.
[0013] In some examples, the method further includes, when the probability score meets the hot word detection threshold, the data processing hardware determines a near miss indication generated by the first-level hot word detector within a threshold time period prior to the hot word detection in the detected streaming audio, for previous probability scores that failed to meet the hot word detection threshold with a threshold margin. The method may also include transmitting the near miss indication from the data processing hardware to a remote computing device.
[0014] A remote computing device can be configured to identify erroneous rejection instances at a first-level hot word detector based on a near-miss indication when a second-level hot word detector detects a hot word in the audio data. An erroneous rejection instance indicates that the first-level hot word detector failed to initially detect a hot word in previous audio features of the streaming audio. In some embodiments, the method includes adjusting a hot word detection threshold of the first-level hot word detector by data processing hardware when the erroneous rejection rate based on the number of erroneous rejection instances identified at the first-level hot word detector within an erroneous rejection period meets an erroneous rejection threshold. Optionally, adjusting the hot word detection threshold includes decreasing the value of the hot word detection threshold.
[0015] Another aspect of this disclosure provides a system for providing automatic hot word threshold tuning. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving audio data from a user equipment performing a first-level hot word detector, representing hot words detected by the first-level hot word detector in streaming audio captured by the user equipment. The first-level hot word detector is configured to generate probability scores indicating the presence of hot words in audio features of the streaming audio captured by the user equipment, and to detect hot words in the streaming audio when the probability scores meet a hot word detection threshold of the first-level hot word detector.
[0016] The operation also includes processing the audio data using a second-level hot word detector to determine whether the second-level hot word detector detected hot words in the audio data. When the second-level hot word detector does not detect hot words in the audio data, the operation includes identifying an erroneous acceptance instance at the first-level hot word detector that indicates the first-level hot word detector incorrectly detected hot words in the streaming audio.
[0017] The operation also includes determining whether the false acceptance rate associated with the first-level hot word detector of the user device meets a false acceptance rate threshold. The false acceptance rate is based on the number of false acceptance instances identified at the first-level hot word detector during the false acceptance period. When the false acceptance rate associated with the first-level hot word detector meets the false acceptance rate threshold, the operation includes adjusting the hot word detection threshold of the first-level hot word detector.
[0018] Implementations of this disclosure may include one or more of the following optional features. In some implementations, operation further includes suppressing a wake-up process on the user device for processing the hot word and / or one or more other terms following the hot word in the streaming audio when the second-level hot word detector does not detect the hot word in the audio data. In some examples, operation further includes determining whether subsequent audio data representing a verbal query following the hot word in the streaming audio is received from the user device when the second-level hot word detector detects the hot word in the audio data. When no subsequent audio data representing a verbal query is received from the user device, this operation may include identifying an erroneous acceptance instance at the first-level hot word detector indicating that the first-level hot word detector incorrectly detected the hot word in the streaming audio.
[0019] Optionally, the operation further includes processing the verbal query when subsequent audio representing the verbal query is received from the user equipment. The user equipment can be configured to initiate a wake-up procedure to process the hot word and / or one or more other terms following the hot word in the streaming audio when the first-level hot word detector detects a hot word in the streaming audio. In some examples, adjusting the hot word detection threshold of the first-level hot word detector includes increasing the value of the hot word detection threshold.
[0020] The operation may further include, when receiving audio data characterizing hot words detected in the streaming audio by the first-level hot word detector, receiving from the user equipment a near-miss indication indicating that the first-level hot word detector detected hot words in the streaming audio within a threshold time period after generating a previous probability score that failed to meet the hot word detection threshold with a threshold margin. The previous probability score indicates the presence of hot words in the previous audio features of the streaming audio captured by the user equipment.
[0021] When the second-level hot word detector detects a hot word in the audio data, the operation may include identifying erroneous rejection instances at the first-level hot word detector based on a near-miss indication, indicating that the first-level hot word detector failed to initially detect the hot word in previous audio features of the streaming audio, and determining whether the erroneous rejection rate associated with the first-level hot word detector of the user device meets an erroneous rejection rate threshold. The erroneous rejection rate is based on the number of erroneous rejection instances identified at the first-level hot word detector during the erroneous rejection period. When the erroneous rejection rate associated with the first-level hot word detector meets the erroneous rejection rate threshold, the operation may include adjusting the hot word detection threshold of the first-level hot word detector. In some examples, adjusting the hot word detection threshold includes lowering the hot word detection threshold of the first-level hot word detector.
[0022] Another aspect of this disclosure provides a system for providing automatic hot word threshold tuning. The system includes data processing hardware of a user device and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. These operations include receiving streaming audio captured by one or more microphones in communication with the data processing hardware. The operations also include generating probability scores using a first-level hot word detector to indicate the presence of hot words in audio features of the streaming audio. The operations further include determining whether the probability scores meet a hot word detection threshold.
[0023] When the probability score meets the hot word detection threshold, the operation includes detecting a hot word in the streaming audio and transmitting audio data representing the hot word detected in the streaming audio using the first-level hot word detector to a remote computing device performing the second-level hot word detector. The remote computing device is configured to determine whether the second-level hot word detector detected a hot word in the audio data, and when the second-level hot word detector did not detect a hot word in the audio data, to identify an erroneous acceptance instance at the first-level hot word detector indicating that the first-level hot word detector incorrectly detected a hot word in the streaming audio.
[0024] When the error acceptance rate based on the number of error acceptance instances identified at the first-level hot word detector during the error acceptance period meets the error acceptance rate threshold, the operation includes adjusting the hot word detection threshold of the first-level hot word detector.
[0025] This aspect may include one or more of the following optional features. Adjusting the hot word detection threshold of the first-level hot word detector may include increasing the value of the hot word detection threshold. In some examples, when the probability score meets the hot word detection threshold, the operation includes initiating a wake-up process on the user device to process the hot word and / or one or more other words following the hot word in the streaming audio. When the second-level hot word detector does not detect a hot word in the audio data, the operation may include suppressing the wake-up process on the user device.
[0026] In some examples, the operation further includes, when the probability score meets the hot word detection threshold, determining a near miss indication generated by the first-level hot word detector within a threshold time period prior to the hot word detection in the detected audio stream for a previous probability score that failed to meet the hot word detection threshold with a threshold margin. The operation may also include transmitting the near miss indication to a remote computing device.
[0027] A remote computing device can be configured to identify erroneous rejection instances at a first-level hot word detector based on a near-miss indication when a second-level hot word detector detects a hot word in the audio data. An erroneous rejection instance indicates that the first-level hot word detector failed to initially detect a hot word in previous audio features of the streaming audio. In some implementations, operation includes adjusting a hot word detection threshold for the first-level hot word detector when the erroneous rejection rate based on the number of erroneous rejection instances identified at the first-level hot word detector within an erroneous rejection period meets an erroneous rejection threshold. Optionally, adjusting the hot word detection threshold includes decreasing the value of the hot word detection threshold.
[0028] Details of one or more embodiments of this disclosure are set forth in the following drawings and description. Further aspects, features, and advantages will become apparent from the specification, drawings, and claims. Attached Figure Description
[0029] Figure 1 This is a schematic diagram of an example system that provides automatic hot word threshold tuning.
[0030] Figure 2 This is a schematic diagram of an exemplary component of a hot word detection threshold adjuster.
[0031] Figure 3 This is a schematic diagram of the hot word detection threshold adjuster incrementing the error acceptance count.
[0032] Figure 4 This is a diagram illustrating an example of an error accepting an instance.
[0033] Figure 5A and 5B This is a diagram illustrating an example of an error rejecting an instance.
[0034] Figure 6 This is a flowchart of an example layout for providing a method for automatic hot word threshold tuning.
[0035] Figure 7 This is a flowchart of another example arrangement of operations used to provide an automatic threshold tuning method.
[0036] Figure 8 This is a schematic diagram of an example computing device that can be used to implement the systems and methods described in this paper.
[0037] In the various figures, the same reference numerals denote the same elements. Detailed Implementation
[0038] Voice-enabled devices (e.g., user devices running a voice assistant) allow users to speak queries or commands aloud and respond to and answer queries and / or perform functions based on commands. By using “hot words” (also known as “keywords,” “attention words,” “wake-up phrases / words,” “trigger phrases,” or “voice action initiation commands”), where predetermined words / phrases spoken to attract the attention of a voice-enabled device are preserved via a protocol, a voice-enabled device is able to distinguish between utterances directed at the system (i.e., those used to initiate a wake-up process for processing one or more words following the hot word in the utterance) and utterances directed at individuals in the environment. Typically, voice-enabled devices operate in sleep mode to conserve power and do not process input audio data unless the input audio data follows the spoken hot word. For example, when in sleep mode, a voice-enabled device captures input audio via one or more microphones and uses a trained hot word detector to detect the presence of hot words in the input audio. When a hot word is detected in the input audio, the voice-enabled device initiates a wake-up process for processing the hot word and / or any other words following the hot word in the input audio.
[0039] Hotword detection is akin to finding a needle in a haystack because the hotword detector must continuously listen to streaming audio and correctly and immediately trigger when a hotword is detected within it, while ignoring most of the audio stream. Hotword detectors typically employ neural networks to address the complexity of detecting hotwords in a continuous audio stream. Generally, the neural network generates probability scores indicating the presence of hotwords in the stream based on the received audio. The hotword detector determines whether the probability scores meet a detection threshold. When the probability scores meet the threshold, the hotword detector determines that a hotword exists in the streaming audio. The hotword detector can then initiate the device's wake-up process.
[0040] Hot word detection thresholds are traditionally set at predetermined values to balance false acceptance and false rejection rates. False acceptance occurs when the hot word detector detects a hot word (i.e., the probability score meets the hot word detection threshold), even though the streaming audio may not actually contain the hot word. Despite a false acceptance, the hot word detector will initiate a wake-up process on a voice-enabled device, even if the user does not intend to invoke that device. False rejection, on the other hand, occurs when the streaming audio includes a hot word, but the hot word detector determines that the hot word is not present in the streaming audio (i.e., the probability score fails to meet the hot word detection threshold). False rejections by the hot word detector are frustrating for the user, as they must subsequently try to invoke the voice-enabled device by saying the hot word again, often loudly, and / or by moving closer to the device, to ensure that the spoken hot word is not falsely rejected again. Therefore, choosing a hot word detection threshold is very difficult due to the wide variety of devices, environments, and users. Traditionally, the detection threshold is not customized for each individual device. However, each device may experience significantly different acoustic environments. For example, a device near a television often experiences significantly more false acceptances compared to the same device with the same hot word detection threshold in a quiet office. Furthermore, each user can have significantly different tolerances for false rejections and false acceptances. That is, while one user might tolerate a moderate number of false acceptances, another user might not tolerate the same number.
[0041] The embodiments described herein pertain to a hot word detection threshold adjuster system that dynamically adjusts the hot word detection threshold of a user device performing a first-level hot word detector to individually customize the hot word detector to the environment. As used herein, the term "hot word detection threshold" refers to a value or probability score that streaming audio must satisfy for the hot word detector to determine / detect the presence of a predetermined hot word in the audio features of the streaming audio, and thus trigger a wake-up process on the user device. The first-level hot word detector of the user device detects hot words in the streaming audio based on a first probability score indicating the presence of hot words in the audio features of the streaming audio. Here, the first probability score satisfies the hot word detection threshold associated with the first-level hot word detector, such that the user device transmits audio data characterizing the hot words detected by the first-level hot word detector to a remote second-level hot word detector for verification. For example, the user device transmits the audio data via the Internet to a server performing the second-level hot word detector. The second-level hot word detector can utilize a hot word detection model that is more accurate in detecting the presence of hot words in the audio compared to the hot word detection model used by the first-level hot word detector running on the user device. The second-level hot word detector processes the audio data to determine whether the second-level hot word detector has detected a hot word. When the second-level hot word detector fails to detect a hot word, the system identifies an erroneous acceptance instance at the first-level hot word indicator, indicating that the first-level hot word detector incorrectly detected the hot word. The system determines whether the erroneous acceptance rate associated with the first-level hot word detector meets an erroneous acceptance rate threshold and adjusts the hot word detection threshold of the first-level hot word detector accordingly. For example, the system can increase the value of the hot word detection threshold to reduce the sensitivity of the first-level hot word detector, thereby reducing / eliminating the occurrence of future erroneous acceptance instances.
[0042] Therefore, the system described in this paper includes a cascaded hot word detection technique that uses multiple models to improve accuracy and validate and / or confirm hot word detection. The system individually determines the ratio of false acceptance instances to false rejection instances for each device and customizes the hot word detection threshold accordingly based on the false acceptance rate and false rejection rate of the respective device.
[0043] refer to Figure 1In some embodiments, example system 100 includes one or more user devices 102, each associated with a corresponding user 10 and communicating with a remote system 110 via network 104. Each user device 102 may correspond to a computing device, such as a mobile phone, computer, smart speaker, smart appliance, smart headset, wearable device, etc., and is equipped with data processing hardware 103 and memory hardware 105. User device 102 includes or communicates with one or more microphones 106 for capturing or communicating with speech from the corresponding user 10. Remote system 110 may be a single computer, multiple computers, or a distributed system (e.g., a cloud environment) with scalable / elastic computing resources 112 (e.g., data processing hardware) and / or storage resources 114 (e.g., memory hardware). In some embodiments, user device 102 receives a trained neural network 130 (e.g., a memory-based neural network) from remote system 110 via network 104 and executes the trained neural network 130 to detect hot words in streaming audio 118. The trained neural network 130 resides in the first-level hot word detector 120 (also called a hot word detector) of the user device 102, which is configured to detect the presence of hot words in the streaming audio 118 without performing semantic analysis or speech recognition processing on the streaming audio 118.
[0044] In the example shown, when user 10 utters 119, which includes a hot word (e.g., “Hey Google”), captured as streaming audio 118 by user device 102, a first-level hot word detector 120 executed on user device 102 is configured to detect the presence of the hot word in 119 to initiate a wake-up process on user device 102 for processing the hot word and / or one or more other terms (e.g., queries or commands) that follow the hot word in 119. That is, user device 102 can be configured to initiate a wake-up process to process the hot word and / or one or more other terms that follow the hot word in streaming audio 118 when the first-level hot word detector 120 detects a hot word in streaming audio 118.
[0045] A first-level hot word detector 120 (e.g., from a neural network 130) generates a probability score 132 indicating the presence of hot words in the audio features of the streaming audio 118 captured by user device 102. When the probability score 132 satisfies a hot word detection threshold 134 for the first-level hot word detector 120, the first-level hot word detector 120 detects hot words in the streaming audio 118. When the probability score 132 satisfies the hot word detection threshold 134, the first-level hot word detector 120 transmits audio data 136 representing the streaming audio 118 to a second-level hot word detector 140 executed on a remote system 110. In some examples, the audio data 136 is a direct representation of the streaming audio 118, while in other examples, the audio data 136 represents the streaming audio 118 after processing by the first-level hot word detector 120 (e.g., for identifying and / or isolating specific audio characteristics of the streaming audio 118 or converting the streaming audio 118 into a format suitable for transmission and / or processing by the second-level hot word detector 140). For example, audio data 136 can be chomped from streaming audio 118 to include relevant segments that include audio features associated with hot words such as those detected by the first-level hot word detector 120.
[0046] Similar to the first-level hot word detector 120, the second-level hot word detector 140 is configured to detect the presence of hot words in the audio data 136. The second-level hot word detector 140 differs from the first-level hot word detector 120. For example, the second-level hot word detector 140 includes a different neural network, which is potentially more computationally dense than the neural network 130 of the first-level hot word detector 120. The second-level hot word detector 140 can provide higher accuracy than the first-level hot word detector 120, which is limited by the resources of the user device 102.
[0047] The second-level hot word detector 140 processes the audio data 136 to determine whether hot words exist in the audio data 136. The second-level hot word detector 140 may generate probability scores similar to the first-level hot word detector 120 for comparison with a hot word detection threshold, or the second-level hot word detector 140 may use entirely different methods to determine the presence of hot words. When the second-level hot word detector 140 does not detect a hot word in the audio data 136, the hot word detection threshold adjusters 200, 200a-b (…) Figure 2 The first-level hot word detector 120 is marked to indicate that the first-level hot word detector 120 incorrectly detected the hot word erroneous acceptance instance 210 in the streaming audio 118.
[0048] Now for reference Figure 2The hot word detection threshold adjuster 200 maintains the false acceptance count 220. The hot word detection threshold adjuster 200 increments the false acceptance count 220 in response to identifying a false acceptance instance 210. Based on the false acceptance count 220, the hot word detection threshold adjuster determines the current false acceptance rate 230. The false acceptance rate 230 represents the number of false acceptance instances 210 identified by the hot word detection threshold adjuster 200 within a false acceptance period. For example, the false acceptance period could be 1 hour, 4 hours, or 24 hours, etc. The false acceptance count 220 may only include the number of false acceptance instances 210 within the most recent false acceptance period. Therefore, the false acceptance rate 230 indicates that the first-level hot word detector 120 incorrectly determines the frequency at which hot words exist in the streaming audio 118.
[0049] The hot word detection threshold adjuster 200 can determine whether the false acceptance rate 230 meets the false acceptance rate threshold 240. For example, when the false acceptance period is 1 hour and the false acceptance rate threshold 240 is three per hour, the false acceptance rate 230 meets the false acceptance rate threshold 240 when the hot word detection threshold adjuster 200 identifies three or more false acceptance instances 210 in the most recent 1 hour.
[0050] Back Figure 1 When the false acceptance rate 230 associated with the first-level hot word detector 120 meets the false acceptance rate threshold 240, the hot word detection threshold adjuster 200 adjusts the hot word detection threshold 134 of the first-level hot word detector 120. In some embodiments, the hot word detection threshold adjuster 200 is executed on a remote system 110 (i.e., hot word detection threshold adjuster 200a) and transmits a hot word detection threshold tuning instruction 150 to the first-level hot word detector 120. When the tuning instruction 150 is received by the user equipment 102, the tuning instruction 150 causes the user equipment 102 to adjust the hot word detection threshold 134 of the first-level hot word detector 120. In other embodiments, the hot word detection threshold adjuster 200 is executed on the user equipment 102 (i.e., hot word detection threshold adjuster 200b) and receives an indication 142 of a false acceptance instance 210 from a second-level hot word detector 140 executed on the remote system 110. Here, user equipment 102 maintains the false acceptance count 220 and determines the current false acceptance rate 230. Hot word detection threshold adjuster 200b provides hot word detection threshold tuning instruction 150 to first-level hot word detector 120 to adjust hot word detection threshold 134 based on false acceptance rate threshold 240 and the current false acceptance rate 230.
[0051] In some implementations, when the false acceptance rate 230 exceeds the false acceptance rate threshold 240, the hot word detection threshold adjuster 200 increases the value of the hot word detection threshold 134. That is, the probability score 132 required to detect the presence of hot words in the streaming audio 118 increases, thereby making the false acceptance instance 210 less likely. In some examples, the hot word detection threshold adjuster 200 adjusts or changes the false acceptance rate threshold 240 based on the adjusted hot word detection threshold 134. In some configurations, the user 10 of the user device 102 can set and / or adjust the false acceptance rate threshold 240.
[0052] In some examples, when the second-level hot word detector 140 does not detect a hot word in the audio data 136, the remote server 110 suppresses the wake-up process on the user device 102. The wake-up process allows the user device 102 to process the hot word and / or one or more other terms (e.g., queries or commands) that follow the hot word in the streaming audio 118. In some embodiments, the remote system 110 suppresses the wake-up process by sending a suppression instruction 162 to the user device 102, causing the user device 102 to suppress the wake-up process. In other embodiments, the remote system 110 suppresses the wake-up process by sending an indication 164 to the user device 102 that the second-level hot word detector 140 failed to confirm the presence of a hot word in the audio data 136, causing the user device 102 to suppress the wake-up process (i.e., remain in or return to a sleep state). In other embodiments, the remote system 110 suppresses the wake-up process by not responding to the user device 102 after receiving the audio data 136 (e.g., by closing the network connection). The lack of a response from the remote system 110 can cause the user device 102 to suppress the wake-up process. That is, in some examples, user device 102 initiates a wake-up process only upon receiving confirmation from the second-level hot word detector 140 that a hot word exists in the streaming audio 118. User device 102 can independently suppress the wake-up process. For example, user device 102 can automatically suppress the wake-up process when the query or command following the hot word is empty (i.e., the streaming audio 118 following the hot word fails to include a command or query directed to user device 102). In this instance, user device 102 can detect the erroneous acceptance instance 210 and notify the hot word detection threshold adjuster to increment the erroneous acceptance count 220.
[0053] Now for reference Figure 3In some examples, when the second-level hot word detector 140 detects a hot word in audio data 136, the remote system 110 determines whether subsequent audio data 136 representing a verbal query following the hot word in the streaming audio 118 has been received from the user equipment 102. When no subsequent audio data 136 representing a verbal query has been received from the user equipment 102, the hot word detection threshold adjuster 200 identifies an erroneous acceptance instance 210 at the first-level hot word detector 120 indicating that the first-level hot word detector 120 incorrectly detected the hot word in the streaming audio 118. That is, in some embodiments, the hot word detection threshold adjuster 200 identifies an erroneous acceptance instance 210 based on the absence of a query or command in the subsequent audio data 136 following the detected hot word. For example, when audio that is not intended to trigger a wake-up process (e.g., ambient noise in the environment such as from a television) unintentionally or undesirably triggers hot word detection, the absence of any subsequent query or command (which would occur during an intentional wake-up command) allows the hot word detection threshold adjuster 200 to identify an erroneous acceptance instance 210.
[0054] In some examples, the remote system processes the spoken query when it receives subsequent audio data 136 representing the spoken query from user device 102 (i.e., after both the first-level hot word detector 120 and the second-level hot word detector 140 have detected the presence of hot words in the streaming audio 118). In these examples, processing the query may include passing the audio data 136 to a speech recognition system for transcribing the spoken query. The remote system 110 may use the transcription to perform natural language understanding and / or provide the transcription to a search engine and / or other applications for processing the query.
[0055] In some implementations, the remote system 110 does not include a second-level hot word detector 140, but instead executes a query / command processor 430. Figure 4 The query / command processor 430 is configured to perform speech recognition on the audio data 136 to confirm whether the first-level hot word detector 120 has correctly detected the presence of hot words in the streaming audio 118. That is, in some embodiments, after detecting the presence of hot words in the streaming audio 118, the first-level hot word detector 120 transmits the audio data 136 to a remote server to process subsequent queries from the user 10. In this case, the hot word detection threshold adjuster 200 can identify an erroneous acceptance instance 210 when the processor 430 fails to identify hot words in the received audio data 136 and when the processor 430 determines that the subsequent audio data 136 received from the user equipment 102 is empty (i.e., when the subsequent audio data 136 fails to include a query or command). Here, either case causes the hot word detection threshold adjuster 200 to increment the erroneous acceptance count 220.
[0056] Now for reference Figure 4Schematic diagram 400 depicts a television 410 transmitting playback audio 420 including the utterance “Hey you all!”. Due to the speech similarity between the utterance “Hey you all” and the hot word “Hey Google”, a first-level hot word detector 120 determines a probability score 132 that meets the hot word detection threshold, and thereby detects the presence of the hot word in the streaming audio 118 representing the playback audio 420a from the television 410. As described above, a second-level hot word detector 140 can confirm / verify whether the first-level hot word detector 120 has correctly detected the hot word. When the second-level hot word detector 140 fails to detect the hot word, it can notify the hot word detection threshold adjuster 200, causing the adjuster 200 to identify an incorrect acceptance instance 210 and increase the incorrect acceptance rate 230. On the other hand, the second-level hot word detector 140 may also incorrectly detect the hot word in the utterance “Hey you all” and pass the corresponding audio to the query processor 430. Here, the query processor 430 can perform speech recognition on the audio data and determine each of the hot word detectors 120, 140 that has erroneously detected a hot word. Alternatively, the processor 430 can determine that no subsequent audio data 136 containing a query or command was received after the erroneous detection of a hot word. In any of these scenarios, the query processor 430 can notify the hot word detection threshold adjuster 200 to identify the erroneous acceptance instance 210.
[0057] Return to reference Figure 1 and 2 In some implementations, the hot word detection threshold adjuster 200 identifies an erroneous rejection instance 250 indicating an instance where the first-level hot word detector 120 fails to detect the presence of a hot word in the streaming audio 118 when the hot word is present. In response, the hot word detection threshold adjuster 200 increments the erroneous rejection count 260 and determines the current erroneous rejection rate 270. When the erroneous rejection rate 270 meets the erroneous rejection threshold 280, the hot word detection threshold adjuster 200 adjusts the hot word detection threshold 134. Here, the adjuster 200 provides the first-level hot word detector 120 with a hot word detection threshold tuning instruction 150 to lower the hot word detection threshold, thereby increasing the sensitivity of the first-level hot word detector 120 for detecting hot words in the streaming audio 118.
[0058] The hot word detection threshold adjuster 200 can identify an erroneous rejection instance 250 in response to receiving a near miss indication 510 indicating that the first-level hot word detector 120 detected a hot word in the streaming audio within a threshold time period after generating a previous probability score, which failed to meet the hot word detection threshold with a threshold margin. For example, the first-level hot word detector 120 running on user device 102 may fail to detect a hot word in the first utterance spoken by the user. Here, the first-level hot word detector 120 may determine that a probability score equal to 0.7 fails to meet the hot word detection threshold set to 0.75. The near miss threshold can be set to a value less than the hot word detection threshold (0.65) such that the range of values between the near miss threshold (0.65) and the hot word detection threshold (0.75) corresponds to the “threshold margin”. For example, the near miss threshold can be set to 0.65, such that any streaming audio 118 associated with a probability score greater than or equal to the near miss threshold but less than the hot word detection threshold of 0.75 fails to meet the hot word detection threshold through the threshold margin. Continuing this example, in a subsequent attempt by user 10 to invoke user device 102, the first-level hot word detector 120 correctly detects a hot word in a second utterance spoken by user 10 within a threshold time period (e.g., 5 seconds). Conditioned that the probability score associated with the first utterance determined by the first-level hot word detector 120 fails to meet the hot word detection threshold through the threshold margin and a hot word is subsequently detected in the second utterance within the threshold time period, the hot word detection threshold adjuster 200 can receive a near miss indication 510 and identify an error rejection instance 250 when it is confirmed that the second-level hot word detector 140 also detected the presence of a hot word in the second utterance. It is worth noting that although the more accurate second-level hot word detector 140 may have detected the presence of hot words in the first utterance, the first-level hot word detector 120 never transmits the corresponding audio data 136 to the second-level hot word detector 140 because the first-level hot word detector 120 generates a probability score associated with the first utterance that fails to meet the hot word detection threshold. The hot word detection threshold adjuster 200 can determine whether the false rejection rate 270 (based on the false rejection count 260) meets the false rejection threshold 280, and when it does, adjust the hot word detection threshold 134 of the first-level hot word detector 120. In some examples, the first-level hot word detector 120 provides a near miss indication 510 to the hot word detection threshold adjuster 200, and the hot word detection threshold adjuster 200 only identifies the false rejection instance 550 upon receiving confirmation from the second-level hot word detector 140 that it has detected a hot word in the audio data 136.
[0059] Now for reference Figure 5AAs an illustration of a near miss as a proxy used for erroneous rejection instance 250, schematic diagram 500a depicts user 10 uttering the first utterance 119a (“Hey Google”) received at a first-level hot word detector 120 on user device 102 (not shown). Here, the first-level hot word detector 120 generates a probability score 132 that fails to meet the hot word detection threshold 134 but does meet the near miss threshold 520. For example, when the hot word detection threshold 134 is 0.75 and the near miss threshold 520 is 0.65 (i.e., less than but generally close to the hot word detection threshold 134), a probability score of 0.70 (or some other value between 0.65 and 0.75) might fail to meet the hot word detection threshold 134 by the threshold margin because the probability score meets the near miss threshold 520.
[0060] Within a threshold time period (e.g., 5 seconds, 10 seconds, 30 seconds, etc.) after receiving the first utterance 119a, Figure 5B Schematic diagram 500b depicts user 10 uttering / saying a second utterance 119b (“HEYGOOGLE”) in another attempt to invoke wake-up of user device 102. This utterance 119b may be said more forcefully and / or with more notification (because user device 102 failed to initiate a wake-up process in response to the previous utterance 119a). In this case, both the first-level hot word detector 120 and the second-level hot word detector 140 determine the presence of a hot word in the streaming audio 118 associated with the second utterance 119b. Although the first-level hot word detector 120 calculates a probability score 132 for the first utterance 119a that fails to meet the hot word detection threshold 134, the hot word detection threshold adjuster 200 receives a near miss indicator 510 because the probability score 132 of the first utterance meets the near miss threshold 520, and the second utterance 119b is within the threshold time period, thus indicating to the first-level hot word detector 120 that it erroneously missed a hot word in the streaming audio 118. Once the second-level hot word detector 140 confirms that a hot word exists in the second discourse 119b, the hot word detection threshold adjuster 200 can identify the false rejection instance 250 and increment the false rejection count 260. The hot word detection threshold adjuster 200 can reduce the hot word detection threshold 134 of the first-level hot word detector 120 in response to the false rejection rate 270 meeting the false rejection threshold 280.
[0061] In some examples, the hot word detection threshold adjuster 200 can adjust the hot word detection threshold 134 of the first-level hot word detector 120 by applying a predetermined threshold to a combined value representing hot word usage, false acceptance count 220, and false rejection count 260. For example, the combined value is the ratio of false acceptances to false rejections (i.e., because false acceptance count 220 and false rejection count 260 are typically inversely related). In other examples, the hot word detection threshold adjuster 200 adjusts the hot word detection threshold 134 of the first-level hot word detector 120 based on information collected from other user devices 102 performing the first-level hot word detector 120. In these examples, the remote system 110 estimates a multivariate distribution of hot word usage, false acceptance count 220 (or false acceptance rate 230), and false rejection count 260 (or false acceptance rate 270) from a large number of user devices 102 and identifies outliers in this distribution to trigger threshold tuning by the hot word detection threshold adjuster 200 for outliers. That is, a device with a false acceptance count 220 or a false rejection count 260 that deviates sufficiently from the general population can be a candidate for threshold tuning.
[0062] Figure 6 This is a flowchart illustrating an exemplary arrangement of the operation of a method 600 for automatic hot word threshold tuning. Method 600 includes, at operation 602, at data processing hardware 112, receiving from a user device 102 executing a first-level hot word detector 120 audio data representing hot words detected by the first-level hot word detector 120 in streaming audio 118 captured by the user device 102. The first-level hot word detector 120 is configured to generate a probability score 132 indicating the presence of hot words in audio features of the streaming audio 118 captured by the user device 102, and to detect hot words in the streaming audio 118 when the probability score 132 satisfies a hot word detection threshold 134 of the first-level hot word detector 120.
[0063] In operation 604, method 600 includes processing audio data 136 by data processing hardware 112 using a second-level hot word detector 140 to determine whether the second-level hot word detector 140 detected a hot word in the audio data 136. When the second-level hot word detector 140 does not detect a hot word in the audio data 136, method 600 includes, in operation 606, identifying an erroneous acceptance instance 210 at the first-level hot word detector 120 indicating that the first-level hot word detector 120 incorrectly detected a hot word in the streaming audio 118.
[0064] In operation 608, method 600 includes determining, by data processing hardware 112, whether the false acceptance rate 230 associated with the first-level hot word detector 120 of user equipment 102 meets an false acceptance rate threshold 240. The false acceptance rate 230 is based on the number of false acceptance instances 210 identified at the first-level hot word detector 120 during the false acceptance period. When the false acceptance rate 230 associated with the first-level hot word detector 120 meets the false acceptance rate threshold 240, in operation 610, method 600 adjusts the hot word detection threshold 134 of the first-level hot word detector 120 via data processing hardware 112.
[0065] Figure 7 This is a flowchart of another exemplary arrangement of the operation of a method 700 for automatic hot word threshold tuning. Method 700 includes, at operation 702, receiving streaming audio 118 captured by one or more microphones 106 communicating with the data processing hardware 103 of the user equipment 102. At operation 704, method 700 includes generating a probability score 132 by the data processing hardware 103 using a first-level hot word detector 120, indicating the presence of hot words in the audio features of the streaming audio 118.
[0066] In operation 706, method 700 includes determining, by data processing hardware 103, whether probability score 132 meets hot word detection threshold 134. When probability score 132 meets hot word detection threshold 134, method 700 includes: in operation 708, detecting hot words in streaming audio 118 by data processing hardware 103, and in operation 710, transmitting, by data processing hardware 103, audio data 136 representing hot words detected in streaming audio 118 using first-level hot word detector 120 to a remote computing device 110 executing second-level hot word detector 140.
[0067] Remote computing device 110 is configured to determine, in operation 712, whether a second-level hot word detector 140 detected a hot word in audio data 136. Remote computing device is configured, in operation 714, when the second-level hot word detector 140 did not detect a hot word in audio data 136, to identify an erroneous acceptance instance 210 at the first-level hot word detector 120 indicating that the first-level hot word detector 120 incorrectly detected a hot word in streaming audio 118. When an error acceptance rate 230 based on the number of erroneous acceptance instances 210 identified at the first-level hot word detector 120 within the error acceptance period meets an error acceptance rate threshold 240, method 700 includes, in operation 716, adjusting the hot word detection threshold 134 of the first-level hot word detector 120 by data processing hardware 103.
[0068] Figure 8This is a schematic diagram of an example computing device 800 that can be used to implement the systems and methods described in this document. The computing device 800 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit the implementation of the inventions described and / or claimed in this document.
[0069] Computing device 800 includes a processor 810, a memory 820, a storage device 830, a high-speed interface / controller 840 connected to the memory 820 and a high-speed expansion port 850, and a low-speed interface / controller 860 connected to a low-speed bus 870 and the storage device 830. Each of components 810, 820, 830, 840, 850, and 860 is interconnected using various buses and may be mounted on a common motherboard or otherwise, as appropriate. Processor 810 is capable of processing instructions for execution within computing device 800, including instructions stored in memory 820 or on storage device 830 for displaying graphical information for a graphical user interface (GUI) on an external input / output device such as a display 880 coupled to the high-speed interface 840. In other embodiments, multiple processors and / or multiple buses, as well as multiple memories and various types of memory, may be used as appropriate. Additionally, multiple computing devices 800 may be connected, with each device providing a portion of the necessary operation (e.g., as a server library, blade server group, or multiprocessor system).
[0070] Memory 820 stores information non-transitorily within computing device 800. Memory 820 may be a computer-readable medium, one or more volatile memory cells, or one or more non-volatile memory cells. Non-transitory memory 820 may be a physical device for storing programs (e.g., instruction sequences) or data (e.g., program state information) on a temporary or permanent basis for use by computing device 800. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used in firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.
[0071] Storage device 830 provides mass storage for computing device 800. In some embodiments, storage device 830 is a computer-readable medium. In various embodiments, storage device 830 may be a floppy disk device, hard disk device, optical disk device, magnetic tape device, flash memory, or other similar solid-state storage device or array of devices, including devices in a storage area network or other configuration. In additional embodiments, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as memory 820, storage device 830, or memory on processor 810.
[0072] High-speed controller 840 manages bandwidth-intensive operations of computing device 800, while low-speed controller 860 manages lower bandwidth-intensive operations. This allocation of responsibilities is merely exemplary. In some embodiments, high-speed controller 840 is coupled to memory 820, display 880 (e.g., via a graphics processor or accelerator), and is coupled to high-speed expansion port 850 which can accept various expansion cards (not shown). In some embodiments, low-speed controller 860 is coupled to storage device 830 and low-speed expansion port 890. Low-speed expansion port 890, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), may be coupled to one or more input / output devices, such as keyboards, pointing devices, scanners, or networking devices such as switches or routers, for example, via a network adapter.
[0073] As shown in the figure, the computing device 800 can be implemented in many different forms. For example, it can be implemented as a standard server 800a or multiple times in a group of such servers 800a, as a laptop computer 800b, or as part of a rack server system 800c.
[0074] Various implementations of the systems and techniques described herein can be carried out in digital electronic and / or optical circuit systems, integrated circuit systems, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations within one or more computer programs operable and / or interpretable on a programmable system comprising at least one programmable processor, which may be dedicated or general-purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transmit data and instructions to the storage system, at least one input device, and at least one output device.
[0075] A software application (i.e., a software resource) can refer to computer software that enables a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.
[0076] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.
[0077] The processes and logical flows described in this specification can be executed by one or more programmable processors, also known as data processing hardware, running one or more computer programs to perform functions by manipulating input data and generating output. The processes and logical flows can also be executed by special-purpose logic circuit systems, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). As an example, processors suitable for running computer programs include both general-purpose and special-purpose microprocessors, as well as any one or more processors of any kind of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. Essential components of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include, or be operatively coupled to, receiving data from or transferring data to one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Processors and memory can be supplemented by dedicated logic circuitry systems or incorporated into dedicated logic circuitry systems.
[0078] To provide interaction with a user, one or more aspects of this disclosure can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, and optionally including a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on the user's client device in response to a request received from a web browser.
[0079] Many embodiments have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other embodiments are within the scope of the following claims.
Claims
1. A computer-implemented method, the method comprising: Audio data representing hot words detected by the first-level hot word detector in streaming audio captured by the user equipment is received from the user equipment at the data processing hardware. The first-level hot word detector is configured to: Generate probability scores indicating the presence of hot words in the audio features of the streaming audio captured by the user equipment; as well as When the probability score meets the hot word detection threshold of the first-level hot word detector, the hot word is detected in the streaming audio. The audio data is processed by the data processing hardware using a second-level hot word detector to determine whether the second-level hot word detector detects the hot word in the audio data. The data processing hardware receives from the user equipment a near miss indication that the first-level hot word detector detected the hot word in the streaming audio within a threshold time period after generating a previous probability score that failed to meet the hot word detection threshold within a threshold margin, the previous probability score indicating the presence of the hot word in the previous audio features of the streaming audio captured by the user equipment. When the second-level hot word detector detects the hot word in the audio data, the data processing hardware identifies an erroneous rejection instance at the first-level hot word detector based on the near miss indication, indicating that the first-level hot word detector did not detect the hot word in the previous audio features of the streaming audio. The data processing hardware determines whether the false rejection rate associated with the first-level hot word detector of the user device meets the false rejection rate threshold, the false rejection rate being based on the number of false acceptance instances identified at the first-level hot word detector during the false rejection time period; as well as When the false rejection rate associated with the first-level hot word detector meets the false rejection rate threshold, the hot word detection threshold of the first-level hot word detector is adjusted by the data processing hardware.
2. The method according to claim 1, further comprising: When the second-level hot word detector does not detect the hot word in the audio data, the data processing hardware suppresses the wake-up process on the user device for processing the hot word and / or one or more other words following the hot word in the streaming audio.
3. The method according to claim 1, further comprising: when the second-level hot word detector detects the hot word in the audio data: The data processing hardware determines whether subsequent audio data representing a verbal query following the hot words in the streaming audio has been received from the user equipment; and When no subsequent audio data characterizing the verbal query is received from the user equipment, the data processing hardware identifies an erroneous acceptance instance at the first-level hot word detector that the first-level hot word detector incorrectly detected the hot word in the streaming audio.
4. The method of claim 3 further includes processing the verbal query via the data processing hardware when subsequent audio representing the verbal query is received from the user equipment.
5. The method according to claim 1, wherein, The user equipment is configured to initiate a wake-up process to process the hot word and / or one or more other words that follow the hot word in the streaming audio when the first-level hot word detector detects the hot word in the streaming audio.
6. The method according to claim 1, wherein, Adjusting the hot word detection threshold includes lowering the hot word detection threshold of the first-level hot word detector.
7. A computer-implemented method, the method comprising: Receive streaming audio captured by one or more microphones in communication with the data processing hardware at the user equipment; The data processing hardware uses a first-level hot word detector to generate probability scores indicating the presence of hot words in the audio features of the streaming audio. The data processing hardware determines whether the probability score meets the hot word detection threshold. When the probability score satisfies the hot word detection threshold within a threshold time period after the generation of a previous probability score that failed to satisfy the hot word detection threshold within the threshold margin, the previous probability score indicates the presence of hot words in the previous audio features of the streaming audio captured by the user device: The hot words are detected in the streaming audio by the data processing hardware. as well as The data processing hardware transmits audio data representing the hot words detected in the streaming audio using the first-level hot word detector and a near-miss indication to a remote computing device performing the second-level hot word detector. The remote computing device is configured to: Determine whether the second-level hot word detector detects the hot word in the audio data; as well as When the second-level hot word detector detects the hot word in the audio data, the near miss indication is identified as an erroneous rejection instance at the first-level hot word detector indicating that the first-level hot word detector did not detect the hot word in the previous audio features of the streaming audio; as well as When the error rejection rate based on the number of error rejection instances identified at the first-level hot word detector within the error rejection time period meets the error rejection rate threshold, the hot word detection threshold of the first-level hot word detector is adjusted by the data processing hardware.
8. The method of claim 7, further comprising: When the probability score meets the hot word detection threshold, the data processing hardware initiates a wake-up process on the user device to process the hot word and / or one or more other words that follow the hot word in the streaming audio. as well as When the second-level hot word detector does not detect the hot word in the audio data, the wake-up process on the user device is suppressed by the data processing hardware.
9. The method according to claim 8, wherein, Adjusting the hot word detection threshold includes lowering the value of the hot word detection threshold.
10. A system comprising: User equipment data processing hardware; as well as Memory hardware that communicates with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations including the following: Audio data representing hot words detected by the first-level hot word detector in streaming audio captured by the user equipment is received from the user equipment performing the first-level hot word detector, wherein the first-level hot word detector is configured to: Generate probability scores indicating the presence of hot words in the audio features of the streaming audio captured by the user equipment; as well as When the probability score meets the hot word detection threshold of the first-level hot word detector, the hot word is detected in the streaming audio. The audio data is processed using a second-level hot word detector to determine whether the second-level hot word detector detects the hot word in the audio data; The user equipment receives a near miss indication indicating that the first-level hot word detector detected the hot word in the streaming audio within a threshold time period after generating a previous probability score that failed to meet the hot word detection threshold within a threshold margin, the previous probability score indicating the presence of the hot word in the previous audio features of the streaming audio captured by the user equipment. When the second-level hot word detector detects the hot word in the audio data, it identifies an erroneous rejection instance at the first-level hot word detector based on the near miss indication, indicating that the first-level hot word detector did not detect the hot word in the previous audio features of the streaming audio. Determine whether the false rejection rate associated with the first-level hot word detector of the user device meets the false rejection rate threshold, the false rejection rate being based on the number of false rejection instances identified at the first-level hot word detector during the false acceptance time period; as well as When the false rejection rate associated with the first-level hot word detector meets the false rejection rate threshold, the hot word detection threshold of the first-level hot word detector is adjusted.
11. The system according to claim 10, wherein, The operation further includes: when the second-level hot word detector does not detect the hot word in the audio data, suppressing the wake-up process on the user device for processing the hot word and / or one or more other words following the hot word in the streaming audio.
12. The system according to claim 10, wherein, The operation further includes, when the second-level hot word detector detects the hot word in the audio data: Determine whether subsequent audio data representing a verbal query following the hot words in the streaming audio has been received from the user equipment; as well as When no subsequent audio data representing the verbal query is received from the user equipment, an erroneous acceptance instance indicating that the first-level hot word detector incorrectly detected the hot word in the streaming audio is identified at the first-level hot word detector.
13. The system according to claim 12, wherein, The operation further includes processing the verbal query when subsequent audio representing the verbal query is received from the user equipment.
14. The system according to claim 10, wherein, The user equipment is configured to initiate a wake-up process to process the hot word and / or one or more other words that follow the hot word in the streaming audio when the first-level hot word detector detects the hot word in the streaming audio.
15. The system according to claim 10, wherein, Adjusting the hot word detection threshold includes lowering the hot word detection threshold of the first-level hot word detector.
16. A system comprising: User equipment data processing hardware; as well as Memory hardware that communicates with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations including the following: Receive streaming audio captured by one or more microphones in communication with the data processing hardware at the user equipment; The data processing hardware uses a first-level hot word detector to generate probability scores indicating the presence of hot words in the audio features of the streaming audio. The data processing hardware determines whether the probability score meets the hot word detection threshold. When the probability score satisfies the hot word detection threshold within a threshold time period after the generation of a previous probability score that failed to satisfy the hot word detection threshold within the threshold margin, the previous probability score indicates the presence of hot words in the previous audio features of the streaming audio captured by the user device: The hot words in the streaming audio are detected by the data processing hardware. as well as The data processing hardware transmits audio data representing the hot words detected in the streaming audio using the first-level hot word detector and a near-miss indication to a remote computing device performing the second-level hot word detector. The remote computing device is configured to: Determine whether the second-level hot word detector detects the hot word in the audio data; as well as When the second-level hot word detector detects the hot word in the audio data, the near miss indication is identified as an erroneous rejection instance at the first-level hot word detector indicating that the first-level hot word detector did not detect the hot word in the previous audio features of the streaming audio; as well as When the error rejection rate based on the number of error rejection instances identified at the first-level hot word detector within the error rejection time period meets the error rejection rate threshold, the hot word detection threshold of the first-level hot word detector is adjusted by the data processing hardware.
17. The system according to claim 16, wherein, The operation further includes: When the probability score meets the hot word detection threshold, a wake-up process is initiated on the user device to process the hot word and / or one or more other words following the hot word in the streaming audio; and When the second-level hot word detector does not detect the hot word in the audio data, the wake-up process on the user device is suppressed.
18. The system according to claim 16, wherein, Adjusting the hot word detection threshold includes lowering the value of the hot word detection threshold.
Citation Information
Patent Citations
Server side hotwording
US20180233150A1
Method in the recognition of speech and a wireless communication device to be controlled by speech
US6697782B1
