Detection sequence-based popular phrase trigger

Abstract

A method (300) includes receiving audio data (202) corresponding to an utterance (110) captured by a user device (104). The utterance includes a command for a digital assistant (109) to perform an operation. The method also includes determining whether any trigger words in a set of trigger words associated with a hotphrase (210) are detected in the audio data during a corresponding fixed duration time window (220) using a hotphrase detector (200) configured to detect each trigger word in the set of trigger words associated with the hotphrase. The method also includes identifying the hotphrase in the audio corresponding to the utterance when each other trigger word in the set of trigger words is also detected in the audio data. The method also includes triggering an automatic speech recognizer (116) to perform speech recognition on the audio data when the hotphrase is identified in the audio data corresponding to the utterance.

Description

Technical Field

[0001] This disclosure relates to popular phrase triggering based on detection sequences. Background Technology

[0002] Voice-enabled environments allow users to simply speak their queries or commands aloud, and the digital assistant will respond and answer the queries and / or execute the commands. Voice-enabled environments (e.g., homes, workplaces, schools, etc.) can be achieved using a network of connected microphone devices distributed throughout various rooms and / or areas of the environment. With such a microphone network, users have the ability to verbally query the digital assistant from virtually anywhere in the environment, without needing a computer or other device in front of or even nearby them. These devices can use hotwords to help distinguish when a given utterance is directed at the system, rather than at another person present in the environment. Therefore, the device can operate in sleep or hibernation mode and only wake up when the detected utterance includes the hotword. Once awakened, the device is able to proceed with more expensive processing, such as fully on-device automatic speech recognition (ASR) or server-based ASR. For example, while cooking in the kitchen, a user might say a specific hot word, “Hey Computer,” to trigger the wake-up of a voice-enabled device, and then ask the digital assistant running on the voice-enabled device, “please set timer for 20 minutes.” In response, the digital assistant will acknowledge that the timer has been set (e.g., in the form of synthesized speech output), and then alert the user once 20 minutes have elapsed (e.g., in the form of an alarm from an acoustic speaker or other audible reminder). Summary of the Invention

[0003] One aspect of this disclosure provides a method for detecting popular phrases. The method includes receiving audio data at data processing hardware of a user device associated with a user, corresponding to a utterance spoken by the user and captured by the user device. The utterance includes commands for a digital assistant to perform operations. During each of a plurality of fixed-duration time windows of the audio data: the method includes: the data processing hardware using a popular phrase detector configured to detect each trigger word in a set of trigger words associated with a popular phrase to determine whether any trigger word in the set of trigger words is detected in the audio data during the corresponding fixed-duration time window; when one of the trigger words in the set of trigger words associated with a popular phrase is detected in the audio data during the corresponding fixed-duration time window, the data processing hardware determines whether each other trigger word in the set of trigger words associated with a popular phrase is also detected in the audio data; and when each other trigger word in the set of trigger words is also detected in the audio data, the data processing hardware identifies the popular phrase in the audio data corresponding to the utterance. The method further includes: when a popular phrase is identified in the audio data corresponding to the utterance, the data processing hardware triggers an automatic speech recognition (ASR) to perform speech recognition on the audio data.

[0004] Another aspect of this disclosure provides a system for detecting popular phrases in audio data. The system includes data processing hardware and memory hardware, the memory hardware communicating with the data processing hardware and storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform an operation. The operation includes receiving audio data corresponding to a utterance spoken by a user and captured by a user device associated with the user. The utterance includes commands for a digital assistant to perform an operation. During each of a plurality of fixed-duration time windows of the audio data: the operation further includes: using a popular phrase detector configured to detect each trigger word in a set of trigger words associated with a popular phrase to determine whether any trigger word in the set of trigger words is detected in the audio data during the corresponding fixed-duration time window; when one of the trigger words in the set of trigger words associated with a popular phrase is detected in the audio data during the corresponding fixed-duration time window, determining whether each other trigger word in the set of trigger words associated with a popular phrase is also detected in the audio data; and when each other trigger word in the set of trigger words is also detected in the audio data, identifying the popular phrase in the audio data corresponding to the utterance. The operation also includes triggering an automatic speech recognizer (ASR) to perform speech recognition on the audio data when popular phrases are identified in the audio data corresponding to the utterance.

[0005] Details of one or more embodiments of this disclosure are set forth in the accompanying drawings and the following description. Other aspects, features, and advantages will be apparent from the description and drawings, as well as from the claims. Attached Figure Description

[0006] Figure 1 This is an exemplary system that includes a popular phrase detector for detecting popular phrases in audio.

[0007] Figure 2 yes Figure 1 Examples of popular phrase detectors.

[0008] Figure 3 This is a flowchart illustrating an exemplary layout of operations for a method used to detect popular phrases in audio.

[0009] Figure 4 This is a schematic diagram of an exemplary computing device that can be used to implement the systems and methods described herein.

[0010] The same reference numerals in the various figures indicate the same elements. Detailed Implementation

[0011] Through such a network of one or more assistant-enabled devices, users have the ability to speak queries or commands aloud and have a digital assistant respond and answer the queries and / or execute the commands. Ideally, users should be able to communicate with the digital assistant by speaking queries / commands directed to the assistant-enabled device as if they were talking to another person. However, making the digital assistant always responsive to the user is challenging because continuously running full voice recognition on assistant-enabled devices with limited resources—such as smartphones or smartwatches—is extremely expensive.

[0012] Therefore, these assistant-enabled devices typically operate in sleep or hibernation mode, where a low-power hotword model detects predefined hotwords in the audio without performing speech recognition. Upon detecting a predefined hotword in spoken utterance, the assistant-enabled device can wake up and continue with more expensive processing, such as fully on-device automatic speech recognition (ASR) or server-based ASR. To alleviate the need for users to speak predefined hotwords and thus create experiences that support always-on speech, much of the current effort focuses on activating the digital assistant directly for a narrow set of common phrases (e.g., "set a timer," "lower volume," etc.). When in a low-power state, the assistant-enabled device can run a low-power model capable of detecting / recognizing fixed hotwords in the audio, such as a compact hotword (or hot phrase) model or a low-power speech recognizer. When a fixed hotword is detected / recognized by the low-power model, the speech-enabled device triggers a higher-power, more accurate model to wake up and verify the presence of the fixed phrase in the audio.

[0013] One challenge with popular phrase detection models is their lack of flexibility, as they require users to utter the exact command the model is trained to recognize. That is, users must utter the exact popular phrase the model expects, without the ability to accept variations / flexibility with different phrases. In many scenarios, the word sequence for a given command will not always be spoken consecutively in the utterance, making it difficult to represent the given command with popular phrases. For example, when executing the command to send a text message, a user might say, "send a message to John saying I'm running late." Here, the command includes a fixed part as well as some variable parts that are difficult to detect / recognize using conventional low-power popular phrase detection models. Therefore, conventional low-power popular phrase detection models lack flexibility and only support a limited number of different popular phrases.

[0014] The implementation described herein relates to achieving a more flexible hot word detection model that can operate at low power while allowing users to communicate more naturally with a always-on Assistant-Enabled Device (AED) with increased flexibility. More specifically, the AED can execute a first-level hot phrase detector, which either runs a single hot word detection model configured to detect multiple different hot words in a utterance, or runs a set of hot word detection models in parallel, each configured to detect a corresponding hot word in the utterance. When the set of hot word detection models detects multiple hot words in a given utterance, the first-level hot phrase detector can aggregate the multiple hot words to detect a complete hot phrase. That is, when multiple hot words are detected in a desired order and within a predefined time window, a complete hot phrase can be detected, enabling the AED to wake from a low-power state and execute a second-level hot phrase detector to verify the detected hot phrase. The second-level hot phrase detector can be used to verify the hot words detected by the first level and / or allow the identification of parameters not detected / recognized by the first-level hot phrase detector within a predefined time window. These parameters can include, for example, intermediate words / terms that the hot word model was not trained to detect but are otherwise scattered throughout spoken utterance as part of an issued query / command.

[0015] A hot phrase detector can be activated / initialized to detect multiple hot words / trigger words based on the context related to the currently used application and / or the content displayed on the AED's screen. For example, if the user views "end message" and "answer the call" displayed on the screen, the hot phrase detector can activate the words send / answer / call / message.

[0016] Figure 1 An exemplary system 100 is shown that includes an Assistant-Enabled Device (AED) 104 that performs a digital assistant 109 that a user 102 can interact with via voice. In the example shown, AED 104 corresponds to a smart speaker. However, AED 104 can include other computing devices, such as, but not limited to, smartphones, tablets, smart displays, desktop / laptop computers, smartwatches, smart appliances, headphones, or vehicle infotainment devices. AED 104 includes data processing hardware 10 and memory hardware 12 that stores instructions that, when executed on the data processing hardware 10, cause the data processing hardware 10 to perform operations. AED 104 includes an array of one or more microphones 16 configured to capture acoustic sounds, such as voice directed at AED 104. AED 104 may also include or communicate with an audio output device (e.g., a speaker) that can output audio for playback to user 102.

[0017] In the example shown, user 102 speaks 110, “Send a message to John saying 'I'm running late',” near AED 104. AED 104's microphone 16 receives 110 and processes corresponding audio data 202. Initial processing of the audio data may involve filtering the audio data and converting it from an analog signal to a digital signal. While AED 104 processes the audio data, it can store the audio data in a buffer in memory hardware 12 for further processing. Using the audio data in the buffer, AED... 104 can use a popular phrase detector 200 to detect whether the audio data 202 includes popular phrases. More specifically, the popular phrase detector is configured to detect each trigger word in the audio data associated with a popular phrase within a set of trigger words during a fixed-duration time window 220 of the audio data 202. Therefore, the popular phrase detector 200 is configured to identify trigger words included in the audio data without performing speech recognition on the audio data. In the example shown, if the popular phrase detector 210 detects acoustic features in the audio data as characteristics of each of the trigger words “send,” “message,” and “saying” during the fixed-duration time window 220 of the audio data 202, then the popular phrase detector 200 can determine the utterance 110 “Send a message to John saying ‘I’m running’”. "late" (sending a message to John saying "I'm late") includes the popular phrase 210 "send<..>message<..>saying<..>". Acoustic features can be Mel-frequency cepstral coefficients (MFCCs) representing the short-term power spectrum of utterance 110, or Mel-scale filter bank energy used for utterance 110. While the example depicts each trigger word as a complete word, trigger words can also include subwords or word fragments.

[0018] As used herein, the hot phrase 210 refers to a narrow set of trigger words (e.g., hot words) that the AED 104 is configured to recognize / detect in audio without performing speech recognition to directly trigger the corresponding action. That is, the hot phrase 210 serves a dual purpose: a calling phrase to wake the AED 104 from a low-power state (e.g., sleep or hibernation) and a command specifying the action to be performed by the digital assistant 109. In this example, the hot phrase 210 “send<..>message<..>saying” allows the user to call the AED 104 to trigger the execution of the corresponding action (e.g., sending message content to a recipient) without requiring the user to prefix the utterance 110 with a predefined calling phrase (e.g., hot word, wake word) to first wake the AED 104 to process the subsequent audio corresponding to the command / query.

[0019] It is worth noting that the popular phrase detector 200 is configured to detect the popular phrase 210 as long as each trigger word in the set of trigger words associated with the popular phrase 210 is detected in the audio data in a predefined sequence of audio data that matches the popular phrase 210 and within a time window 220 of fixed duration. That is, in addition to the fixed portion corresponding to the set of trigger words that the popular phrase detector 200 is configured to detect, the utterance 110 may also include some variable portions not associated with the popular phrase, such as words / terms spoken by the user 102 between the first trigger word (e.g., “send”) and the last trigger word (e.g., “saying”). Therefore, the popular phrase detector 200 does not require the user 102 to utter the exact command that the popular phrase detector 200 is trained to detect. In other words, the popular phrase detector 200 has the ability to accept variations / flexibility regarding different phrases associated with the popular phrase, and therefore does not require the user to continuously utter the set of trigger words and allows the user to embed open-ended parameters within the popular phrase. While some popular phrases are typically spoken consecutively in a discourse (e.g., “volume up,” “volume down,” “next track,” “set timer,” “stop alarm,” etc.), the popular phrase detector 200 disclosed herein is also capable of detecting trigger word sequences for popular phrases that are not always spoken consecutively, thus allowing the AED 104 to detect a wider range of popular phrases 210. For example, in Figure 1In the example, user 102 could convey the same command for digital assistant 109 to perform an operation by saying a slightly different utterance: “Please send a nice message to my colleague John saying ‘I’m running late’”. Here, the utterance still includes the set of trigger words “send<..>message<..>saying” associated with popular phrase 110, but with different kinds of words / terms spoken by user 102 between the first trigger word (e.g., “send”) and the last trigger word (e.g., “saying”). Therefore, popular phrase detector 200 can still detect popular phrase 210 to invoke AED 104 to wake up and trigger the execution of the corresponding operation (e.g., sending message content to the recipient).

[0020] When the AED 104 is in a low-power state, the hot phrase detector 200 can run / execute continuously on the AED 104 to listen for each trigger word in the trigger word set in the streaming audio. When the AED 104 includes a battery-powered device such as a smartphone, the hot phrase detector 200 can execute on low-power hardware such as a digital signal processor (DSP) chip. The hot phrase detector 200 can run / execute on the application process (AP) / CPU of other types of AEDs, but consumes less power and requires less processing than performing speech recognition.

[0021] When the popular phrase detector 200 identifies the popular phrase 210 in the audio data 202 by detecting each trigger word in the set of trigger words during a fixed-duration time window 220 of the audio data 202, the AED 204 can trigger a wake-up process to initiate speech recognition of the audio data 202 corresponding to the utterance 110. For example, an automatic speech recognizer (ASR) 116 running on the AED 104 can perform speech recognition on the audio data 202 as a verification phase to confirm the presence of the popular phrase 210 in the audio data 202. The popular phrase detector 200 can rewind the audio data cached in the memory hardware 12 to a time before or after the detection of the first trigger word and provide the audio data 202 that begins at or before the detection of the first trigger word to the ASR 116 for processing. Therefore, the cached audio data 202 provided to the ASR 116 may include some preamble audio that begins before the first trigger word. The duration of the preamble audio may depend on the specific popular phrase 210 based on the expected position of the first trigger word relative to other terms in a given utterance. The audio data 202 provided to ASR116 includes a portion that characterizes the set of detected trigger words corresponding to the preceding audio and a time window 220 of fixed duration, as well as a subsequent portion 222 including the message content “I'm running late”.

[0022] Here, ASR 116 generates a transcription 120 of utterance 110 by processing audio data 202 and determines whether each trigger word in the set of trigger words associated with popular phrase 210 is identified in the transcription 120. ASR 116 may also process portion 222 of audio data 202 corresponding to the message content “I’m running late” following the last trigger word (e.g., saying) for inclusion in transcription 120. When ASR 116 determines that each trigger word in the set of trigger words is identified in transcription 120, ASR 116 may provide transcription 120 to query processing 180 to perform query interpretation on transcription 120 to identify commands for digital assistant 109 to perform operations. Query processing 180 may execute a dedicated model configured to receive transcription 120 of utterance 110 and classify the likelihood that utterance 110 corresponds to query / command class utterances directed to digital assistant 109. Query processing 180 may additionally or alternatively perform query interpretation via a natural language processing (NLP) layer for performing intent classification. In this example, the query interpretation performed by query processing 180 on transcription 120 may identify a command to send a message to the receiver device associated with John, and provide the portion of transcription 120 including the message content “I’m running late” to a messaging application for transmission to the receiver device associated with John.

[0023] On the other hand, when ASR 116 determines that one or more trigger words from the trigger word set are not identified in transcription 120, ASR 116 determines that a false triggering event has occurred at popular phrase detector 200, and therefore, popular phrase 210 is not spoken in the user's utterance 110. In the example shown, ASR 116 instructs AED 104 to suppress the wake-up process and return to a low-power state upon determining a false triggering event. In some examples, when one or more trigger words detected by popular phrase detector 200 are falsely identified by ASR, AED 104 performs a refinement process to fine-tune the popular phrase detector based on each trigger word falsely identified by ASR.

[0024] Optionally, ASR 116 can be performed on a remote server (not shown) that communicates with AED 104 via a network. In some examples, in addition to or instead of the verification performed by ASR 116, a computationally denser second-level hot phrase detector confirms the presence of hot phrase 210 in the audio data 202.

[0025] refer to Figure 2In some implementations, the popular phrase detector 200 includes a trigger word detection model 205 trained to detect each trigger word in a set of trigger words associated with the popular phrase 210. Audio data 202 converted from streaming audio captured by the microphone 16 of the AED 104 is cached at a buffer and fed to the trigger word detection model 205. The buffer may reside on memory hardware 12. The model 205 is configured to output a confidence score 207 that includes a range of supported trigger words from the set of trigger words associated with the popular phrase 210. The range of supported trigger words may include other trigger words from different sets of trigger words associated with one or more additional popular phrases. Some trigger words may belong to multiple sets of trigger words. For example, the trigger word "message" could also belong to a different set of trigger words associated with another popular phrase, "(dictate<..>message)". In some examples, model 205 includes a fixed-window audio model with several neural network layers configured to process audio frames to generate audio classifications (e.g., confidence scores 207) every N ms. Here, the neural network layers may include convolutional blocks. At each of multiple time steps, the model's output layer may output a confidence score 207 for each supported trigger word. Thus, each trigger word supported by the model can be referred to as a target class. When model 205 outputs the corresponding confidence score 207 for a trigger word that meets the trigger word confidence threshold, popular phrase detector 200 detects a corresponding trigger event 260 indicating the presence of a trigger word in the audio data 202 and caches the corresponding trigger event 260 in a buffer.

[0026] In the example shown, each corresponding trigger event 260 in the buffer indicates a corresponding confidence score 207 for the corresponding trigger word and a corresponding timestamp 209 indicating when the corresponding trigger word was detected in the audio data 202. For example, assuming the trigger word confidence threshold is equal to 0.7: the corresponding trigger event 260 can be detected when model 205 outputs a corresponding confidence score of 0.95 for the trigger word “send” at 0 (0) milliseconds (ms) at the start time of the time window 220 indicating the current fixed duration; the corresponding trigger event 260 can be detected when model 205 outputs a corresponding confidence score of 0.8 for the trigger word “message” at 300 (300) ms; and the corresponding trigger event 260 can be detected when model 205 outputs a corresponding confidence score of 0.85 for the trigger word “saying” at 1,000 ms. It is worth noting that the popular phrase detector 200 does not initiate a wake-up process in response to the detection of a trigger event 260 for each individual trigger word.

[0027] The popular phrase detector 200 is further configured to execute a trigger word aggregation routine 280 whenever the trigger word detection model 205 detects a corresponding trigger event 260. Here, routine 280 is configured to: determine whether the corresponding trigger event 260 for each other corresponding trigger word in the trigger word set is in the buffer; and when the corresponding trigger event 260 for each other corresponding trigger word in the trigger word set is also in the buffer, determine a popular phrase confidence score 282 indicating the likelihood that the utterance spoken by the user includes the popular phrase 210. In some examples, when the popular phrase confidence score 282 meets a popular phrase confidence threshold, the popular phrase detector 200 identifies the popular phrase in the audio data 202.

[0028] Routine 280 can be configured to determine a popular phrase confidence score 282 for each corresponding trigger word in the trigger word set, based on the corresponding trigger word confidence score 207 indicated by the corresponding trigger event 260 and the corresponding timestamp 209 in a buffer. In practice, the corresponding trigger event 260 may include multiple corresponding timestamps 209 indicating the position where the trigger word confidence score 207 exceeds the trigger word confidence threshold, thereby allowing the use of multiple techniques to combine successive detections. For example, the timestamp 209 associated with the highest trigger word confidence score 207 may be indicated by the trigger event 260 stored in the buffer. Performing the trigger word aggregation routine 280 may include performing a neural network-based model. The neural network-based model may include a sequence-based machine learning model, such as a model with a recurrent neural network (RNN) architecture. In other examples, performing the trigger word aggregation routine 280 may include performing a grammatical or heuristic-based model. Routine 280 also considers detecting a sequence of trigger words during a time window 220 of fixed duration. In other words, the sequence of trigger words detected in the audio data 202 must match a predefined order associated with the popular phrase 210 in order to identify the popular phrase 210. For example, in the example shown, upon receiving a trigger event 260 indicating that the trigger word "saying" has been detected, routine 280 can use the corresponding timestamp 209 in the buffer to determine that the trigger word "message" was detected after the trigger word "send" and before the trigger word "saying".

[0029] In some examples, the popular phrase confidence score 282 generated by routine 280 is further based on the corresponding time interval between each pair of adjacent trigger words in the set of trigger words detected in the audio data. For example, routine 280 may compare each corresponding time interval with a corresponding reference time interval expected between adjacent trigger word pairs for a particular phrase. That is, for the popular phrase 210 “send<..>message<..>saying”, the expected reference time interval between the trigger words “send” and “message” is shorter than the expected reference time interval between the trigger words “message” and “saying”. Routine 280 may also constrain the popular phrase based on the maximum time interval between a particular trigger word pair.

[0030] The grammar (e.g., target class / trigger word) for trigger word detection model 205 can be manually constructed or learned / trained. When learned, AED queries can be used for specific verticals or intents. For example, to represent a command to dictate and send a message to a recipient hands-free, the query transcripts of the user who dictated and sent the message command can be used to learn the minimum set of trigger words that covers the largest portion of the query transcript for the message sending command and are used by trigger word detection model 205. That is, the minimum set of trigger words covering the largest portion of the transcript is associated with the trigger words that appear in the transcripts with the highest frequency. In another example, when building trigger word detection model 205 to support a low-power command for playing music, the query transcripts for the music playing command can be obtained to identify the minimum set of trigger words covering the largest portion of the transcripts in the obtained query transcripts. Notably, trigger word detection model 205 can be built on a device and / or on a per-user basis, such that trigger word detection model 205 is designed to detect personalized popular phrases spoken by the user and / or the user of a specific AED. Trigger word detection models for common / general popular phrases can also be detected on the server side and pushed to the AED community.

[0031] Figure 3 This is a flowchart illustrating an exemplary arrangement of the operation of a method 300 for detecting popular phrases 210 in audio data 202 during a fixed-duration time window 220 of audio data 202. At operation 302, method 300 includes receiving audio data 202 corresponding to utterances 110 spoken by user 102 and captured by user device 104 at a data processing hardware 10 of user device 104 associated with user 102. The utterances 110 include commands for digital assistant 109 to perform operations. User device 104 may include an assistant-enabled device (AED) that performs the digital assistant 109.

[0032] Operations 304, 306, and 308 of method 300 are performed during each of multiple fixed-duration time windows 220 of audio data 202. At operation 304, method 300 includes the data processing hardware 10 using a popular phrase detector 200 configured to detect each trigger word in a set of trigger words associated with popular phrase 210 to determine whether any trigger word in the set of trigger words is detected in the audio data 202 during the corresponding fixed-duration time window 220. At operation 306, when one of the trigger words in the set of trigger words associated with popular phrase 210 is detected in the audio data 202 during the corresponding fixed-duration time window 220, method 300 further includes the data processing hardware 10 determining whether each other trigger word in the set of trigger words associated with popular phrase 210 is also detected in the audio data. At operation 308, when each other trigger word in the set of trigger words is also detected in the audio data, method 300 further includes the data processing hardware identifying the popular phrase in the audio data corresponding to the utterance.

[0033] At operation 310, method 300 includes triggering an automatic speech recognizer (ASR) to perform speech recognition on the audio data when a popular phrase is identified in the audio data corresponding to the utterance. Here, the ASR can process the audio that begins at or before the detection of the first trigger word to generate a transcription 120 for the utterance to determine whether each trigger word in the set of trigger words is identified in the transcription 120.

[0034] A software application (i.e., a software resource) can refer to computer software that enables a computing device to perform tasks. In some examples, a software application can mean "application," "app," or "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and game applications.

[0035] Non-transitory memory can be a physical device used to store programs (e.g., instruction sequences) or data (e.g., program state information) on a temporary or permanent basis for use by a computing device. Non-transitory memory can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.

[0036] Figure 4 This is a schematic diagram of an exemplary computing device 400 that can be used to implement the systems and methods described in this document. The computing device 400 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit the implementation of the inventions described and / or claimed in this document.

[0037] Computing device 400 includes a processor 410, a memory 420, a storage device 430, a high-speed interface / controller 440 connected to the memory 420 and a high-speed expansion port 450, and a low-speed interface / controller 460 connected to a low-speed bus 470 and the storage device 430. Each of the components 410, 420, 430, 440, 450, and 460 is interconnected using various buses and may be mounted on a common motherboard or otherwise suitably mounted. The processor 410 is capable of processing instructions for execution within the computing device 400, including instructions stored in the memory 420 or on the storage device 430, to display graphical information for a graphical user interface (GUI) on an external input / output device such as a display 480 coupled to the high-speed interface 440. In other embodiments, multiple processors and / or multiple buses, as well as multiple memories and memory types, may be suitably used. Moreover, multiple computing devices 400 may be connected, with each device providing a portion of the necessary operation (e.g., as a server library, blade server group, or multiprocessor system).

[0038] Memory 420 stores information non-transitory within computing device 400. Memory 420 may be a computer-readable medium, a volatile memory cell, or a non-volatile memory cell. Non-transitory memory 420 may be a physical device for storing programs (e.g., instruction sequences) or data (e.g., program state information) on a temporary or permanent basis for use by computing device 400. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.

[0039] Storage device 430 provides mass storage for computing device 400. In some embodiments, storage device 430 is a computer-readable medium. In various embodiments, storage device 430 may be a floppy disk device, hard disk device, optical disk device, magnetic tape device, flash memory or other similar solid-state storage device, or device array, including devices in a storage area network or other configuration. In additional embodiments, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods such as those described above. The information carrier is a computer or machine-readable medium, such as memory 420, storage device 430, or memory on processor 410.

[0040] High-speed controller 440 manages bandwidth-intensive operations of computing device 400, while low-speed controller 460 manages lower bandwidth-intensive operations. This allocation of responsibilities is merely exemplary. In some embodiments, high-speed controller 440 is coupled to memory 420, display 480 (e.g., via a graphics processor or accelerator), and high-speed expansion port 450 which can accept various expansion cards (not shown). In some embodiments, low-speed controller 460 is coupled to storage device 430 and low-speed expansion port 490. Low-speed expansion port 490, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), may be coupled to one or more input / output devices, such as keyboards, pointing devices, scanners, or network devices such as switches or routers (e.g., via network adapters).

[0041] As shown in the figure, the computing device 400 can be implemented in a variety of different forms. For example, the computing device 400 can be implemented as a standard server 400a or multiple times in a set of such servers 400a, as a laptop computer 400b, or as part of a rack server system 400c.

[0042] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in the form of one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be dedicated or general-purpose, coupled to receive and transmit data and instructions from and to a storage system, at least one input device, and at least one output device.

[0043] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine language. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0044] The processes and logical flows described in this specification can be executed by one or more programmable processors—also known as data processing hardware—that perform functions by executing one or more computer programs to manipulate input data and generate output. The processes and logical flows can also be executed by special-purpose logic circuitry—such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). For example, processors suitable for executing computer programs include both general-purpose and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include, or be operatively coupled to, one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, for receiving data from or transferring data to, or both. However, a computer does not need to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. Processors and memory can be supplemented by or incorporated into dedicated logic circuitry.

[0045] To provide interaction with a user, one or more aspects of this disclosure can be implemented on a computer having a display device for displaying information to the user—such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touchscreen—and optionally a keyboard and pointing device, such as a mouse and trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending web pages to a web browser on the user's client device in response to a request received from a web browser.

[0046] Several embodiments have been described. However, it will be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other embodiments are also within the scope of the appended claims.

Claims

1. A method for detecting popular phrases, comprising: Audio data corresponding to a speech uttered by the user and captured by the user device is received at the data processing hardware of the user device associated with the user, the speech including: Commands used by digital assistants to perform operations; Popular phrases associated with the set of trigger words; and One or more other words that are not associated with the popular phrase spoken between the first trigger word in the set of trigger words and the last trigger word in the set of trigger words; During each of the multiple fixed-duration time windows of the audio data: The data processing hardware uses a popular phrase detector configured to detect each trigger word in the set of trigger words associated with the popular phrase to determine whether the first trigger word in the set of trigger words is detected in the audio data during a time window of a corresponding fixed duration; After determining that the first trigger word in the set of trigger words was detected in the audio data during a corresponding fixed-duration time window, the data processing hardware determines that each other trigger word in the set of trigger words associated with the popular phrase was also detected in the audio data during the corresponding fixed-duration time window; Determine a predefined order in which sequence matches of the set of trigger words detected in the audio data during a corresponding fixed-duration time window are associated with the popular phrase; and Based on the predefined order in which the sequence matching of the set of trigger words detected in the audio data during a corresponding fixed-duration time window is associated with the popular phrase, the data processing hardware identifies the popular phrase in the audio data corresponding to the utterance; and When the popular phrase is identified in the audio data corresponding to the utterance, the data processing hardware triggers an automatic speech recognizer (ASR) to perform speech recognition on the audio data.

2. The method according to claim 1, wherein: When the user utters the words, the user equipment is in a low-power state; and The utterance spoken by the user does not include a predetermined hot word configured to trigger the user device to wake up from the low-power state.

3. The method according to claim 1, wherein, Determining that the first trigger word in the set of trigger words is detected in the audio data includes: The popular phrase detector is used to generate a trigger word confidence score that indicates the probability that the first trigger word is present in the audio data during a time window of corresponding fixed duration; When the trigger word confidence score meets the trigger word confidence threshold, the first trigger word in the audio data is detected during a time window of the corresponding fixed duration; and The audio data and a trigger event for the first trigger word detected in the audio data are cached in memory hardware that communicates with the data processing hardware. The trigger event indicates the confidence score of the trigger word and a timestamp indicating when the first trigger word was detected in the audio data.

4. The method of claim 3, further comprising: Based on determining that a first trigger word in the set of trigger words associated with the popular phrase is detected in the audio data during a corresponding fixed-duration time window, a trigger word aggregation routine is executed, the trigger word aggregation routine being configured to: Determine whether the corresponding trigger events for each other corresponding trigger word in the trigger word set are also cached in the memory hardware; as well as While the corresponding trigger events for each other corresponding trigger word in the trigger word set are also cached in the memory hardware, a popular phrase confidence score is determined, indicating the likelihood that the utterance spoken by the user includes the popular phrase. The identification of popular phrases includes identifying popular phrases when the confidence score of the popular phrase meets the confidence threshold of popular phrases.

5. The method according to claim 4, wherein, The trigger word aggregation routine is configured to determine the popular phrase confidence score based on the corresponding trigger word confidence score and the corresponding timestamp for each corresponding trigger word in the trigger word set, indicated by the corresponding trigger event cached in the memory hardware.

6. The method according to claim 4, wherein, Executing the trigger word aggregation routine includes executing a neural network-based model.

7. The method according to claim 4, wherein, Executing the trigger word aggregation routine includes executing a heuristic-based model.

8. The method of claim 1, further comprising: The data processing hardware determines the corresponding time interval between each pair of adjacent trigger words in the set of trigger words detected in the audio data. The popular phrases are identified in the audio data corresponding to the utterance based on the corresponding time interval between each pair of adjacent trigger words.

9. The method according to claim 1, wherein, Triggering the ASR to perform speech recognition on the audio data includes: The transcript of the speech is generated by processing the audio data; Determine whether each trigger word in the set of trigger words associated with the popular phrase is identified in the transcription of the utterance; and When each trigger word in the set of trigger words associated with the popular phrase is identified in the transcription, a query interpretation is performed on the transcription to identify that the transcription includes the command for the digital assistant to perform the operation.

10. The method according to claim 9, wherein, The generation of the transcript includes: The audio data cached in memory hardware communicating with the data processing hardware is rewound to a time prior to or when the first trigger word in the trigger word set was detected in the audio data; and The audio data, starting at or before the first trigger word in the trigger word sequence, is processed to generate the transcription of the utterance.

11. The method according to claim 9, wherein, The transcription includes one or more other words that are not associated with the popular phrase between the first trigger word in the set of trigger words identified in the transcription and the last trigger word in the set of trigger words identified in the transcription.

12. The method according to any one of claims 1 to 11, wherein, The popular phrase detector includes a trigger word detection model, which is trained to detect each trigger word in the set of trigger words associated with the popular phrase.

13. A system for detecting popular phrases, comprising: Data processing hardware; as well as Memory hardware communicating with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations, the operations including: Receive audio data corresponding to a utterance spoken by a user and captured by a user device associated with the user, the utterance including: Commands used by digital assistants to perform operations; Popular phrases associated with the set of trigger words; and One or more other words that are not associated with the popular phrase spoken between the first trigger word in the set of trigger words and the last trigger word in the set of trigger words; During each of the multiple fixed-duration time windows of the audio data: A popular phrase detector configured to detect each trigger word in the set of trigger words associated with the popular phrase is used to determine the first trigger word in the set of trigger words detected in the audio data during a time window of a corresponding fixed duration; After determining that the first trigger word in the trigger word set was detected in the audio data during a corresponding fixed-duration time window, it is determined that each other trigger word in the trigger word set associated with the popular phrase was also detected in the audio data during the corresponding fixed-duration time window; Determine a predefined order in which sequence matches of the set of trigger words detected in the audio data during a corresponding fixed-duration time window are associated with the popular phrase; and Based on the predefined order in which the sequence matching of the set of trigger words detected in the audio data during a corresponding fixed-duration time window is associated with the popular phrase, the popular phrase is identified in the audio data corresponding to the utterance; and When the popular phrase is identified in the audio data corresponding to the utterance, an automatic speech recognizer (ASR) is triggered to perform speech recognition on the audio data.

14. The system according to claim 13, wherein: When the user utters the words, the user equipment is in a low-power state; and The utterance spoken by the user does not include a predetermined hot word configured to trigger the user device to wake up from the low-power state.

15. The system according to claim 13, wherein, Determining that the first trigger word in the set of trigger words is detected in the audio data includes: The popular phrase detector is used to generate a trigger word confidence score that indicates the probability that the first trigger word is present in the audio data during a time window of corresponding fixed duration; When the trigger word confidence score meets the trigger word confidence threshold, the trigger word in the audio data is detected during a time window of the corresponding fixed duration; and The audio data and a trigger event for the first trigger word detected in the audio data are cached in memory hardware that communicates with the data processing hardware. The trigger event indicates a confidence score for the first trigger word and a timestamp indicating when the first trigger word was detected in the audio data.

16. The system according to claim 15, wherein, The operation further includes: based on determining that a first trigger word in the set of trigger words associated with the popular phrase is detected in the audio data during a corresponding fixed-duration time window, executing a trigger word aggregation routine, the trigger word aggregation routine being configured to: Determine whether the corresponding trigger events for each other corresponding trigger word in the trigger word set are also cached in the memory hardware; and While the corresponding trigger events for each other corresponding trigger word in the trigger word set are also cached in the memory hardware, a popular phrase confidence score is determined, indicating the likelihood that the utterance spoken by the user includes the popular phrase. The identification of popular phrases includes identifying popular phrases when the confidence score of the popular phrase meets the confidence threshold of popular phrases.

17. The system according to claim 16, wherein, The trigger word aggregation routine is configured to determine the popular phrase confidence score based on the corresponding trigger word confidence score and the corresponding timestamp for each corresponding trigger word in the trigger word set, indicated by the corresponding trigger event cached in the memory hardware.

18. The system according to claim 16, wherein, Executing the trigger word aggregation routine includes executing a neural network-based model.

19. The system according to claim 16, wherein, Executing the trigger word aggregation routine includes executing a heuristic-based model.

20. The system according to claim 13, wherein, The operation further includes: Determine the corresponding time interval between each pair of adjacent trigger words in the set of trigger words detected in the audio data. The popular phrases are identified in the audio data corresponding to the utterance based on the corresponding time interval between each pair of adjacent trigger words.

21. The system according to claim 13, wherein, Triggering the ASR to perform speech recognition on the audio data includes: The transcript of the speech is generated by processing the audio data; Determine whether each trigger word in the set of trigger words associated with the popular phrase is identified in the transcription of the utterance; and When each trigger word in the set of trigger words associated with the popular phrase is identified in the transcription, a query interpretation is performed on the transcription to identify that the transcription includes the command for the digital assistant to perform the operation.

22. The system according to claim 21, wherein, The generation of the transcript includes: The audio data cached in memory hardware communicating with the data processing hardware is rewound to a time prior to or when the first trigger word in the trigger word set was detected in the audio data; and The audio data, starting at or before the first trigger word in the trigger word sequence, is processed to generate the transcription of the utterance.

23. The system according to claim 21, wherein, The transcription includes one or more other words that are not associated with the popular phrase, between the first trigger word in the set of trigger words identified in the transcription and the last trigger word in the set of trigger words identified in the transcription.

24. The system according to any one of claims 13 to 23, wherein, The popular phrase detector includes a trigger word detection model, which is trained to detect each trigger word in the set of trigger words associated with the popular phrase.

Images