Two-stage user-customizable wake-word detection

Abstract

Described herein are devices, methods, and systems for detecting a phrase from an uttered speech. A processing device can determine a first model for using a set of training utterances for phrase recognition based on a likelihood ratio. The set of utterances can be analyzed by the first model to determine a second model that includes a training state sequence for each of the set of training utterances, and wherein each training state sequence indicates a possible state for each time interval of a corresponding training utterance. A determination as to whether a detected utterance corresponds to a phrase can be based on a concatenation of the first model and the second model.

Description

[0001] Related Applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 020,984, filed May 6, 2020, the entire disclosure of which is hereby incorporated by reference. TECHNICAL FIELD

[0003] The present disclosure relates generally to speech recognition systems, and more particularly to wake-word detection. BACKGROUND

[0004] Increasingly, modern computing devices feature speech recognition capabilities, allowing users to perform a wide variety of computing tasks via voice commands and natural speech. Devices such as mobile phones or smart speakers provide integrated virtual assistants that can respond to user commands or natural language requests by communicating over a local area network and / or a wide area network to retrieve requested information or control other devices, e.g., lights, heating and air conditioning controls, audio or video equipment, etc. Devices with speech recognition capabilities typically remain in a low-power mode until a particular word or phrase (i.e., a wake-word or wake-phrase) is spoken, allowing the user to use voice commands to control the device after it has thus been activated.

[0005] To initiate a voice-based user interface, wake-word detection (WWD) is typically deployed. Here, a key word or key phrase is continuously monitored, and when detected, further voice-based interaction is enabled. Early WWD systems used Gaussian Mixture Model-Hidden Markov Model (GMM-HMM) for acoustic modeling. Recently, deep learning or deep neural networks (NNs) have become an attractive option due to their higher accuracy than traditional methods. BRIEF DESCRIPTION OF DRAWINGS

[0006] The present embodiments are illustrated by way of example, and not limitation, in the figures of the accompanying drawings in which:

[0007] Figure 1 FIG. 1 is a block diagram illustrating a system in accordance with some embodiments of the present disclosure.

[0008] Figure 2 FIG. 2 is a block diagram illustrating an audio processing device in accordance with some embodiments of the present disclosure.

[0009] Figures 3A-3C FIG. 3 illustrates a wake-word recognition model derivation process in accordance with some embodiments of the present disclosure.

[0010] Figure 3D FIG. 4 illustrates a conventional wake-word recognition process in accordance with some embodiments of the present disclosure.

[0011] Figure 4AA 2-stage model training and 2-stage wake word recognition process is shown in accordance with some embodiments of the disclosure.

[0012] Figure 4B A wake word recognition model is shown in accordance with some embodiments of the disclosure.

[0013] Figure 5 A graph of state sequences for various utterances is shown in accordance with some embodiments of the disclosure.

[0014] Figure 6 A flowchart of a method for recognizing a wake word is shown in accordance with some embodiments of the disclosure.

[0015] Figure 7 A flowchart of a method for recognizing a wake word is shown in accordance with some embodiments of the disclosure.

[0016] Figure 8 A flowchart of a method for recognizing a wake word is shown in accordance with some embodiments of the disclosure.

[0017] Figure 9 A programmable system-on-a-chip Embodiments of a core architecture of a processing device. DETAILED DESCRIPTION

[0018] In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments. It will be apparent, however, to one skilled in the art that the embodiments can be practiced without some or all of these specific details. In other instances, well known circuits, structures, and techniques have not been shown to avoid obscuring the understanding of this description.

[0019] Reference in the specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. The appearances of the phrase “in one embodiment” in various places in the specification are not necessarily referring to the same embodiment.

[0020] As discussed above, to use a wake-word to initiate a device, typically wake-word detection (WWD) is utilized. Most approaches employ a pre-selected wake-word (e.g., "hello") that cannot be modified by the user, and require tens of thousands of backend training utterances. These pre-selected wake-words are trained offline and work for all speakers, and thus are speaker independent. Detection of the wake-word can be performed locally on the device, and then can be verified in the cloud by more complex algorithms. Speaker recognition is an additional feature that can provide some level of security or customization (e.g., user-specific playlists). However, speaker recognition is a complex task that is typically performed in the cloud and requires a cumbersome enrollment phase (reading text).

[0021] Many personal devices (e.g., earpieces, hearables, wearables, cameras, etc.) now feature a voice interface. These devices are typically used by few or even a single user. Since they are battery powered, connectivity to the cloud is limited to conserve power. Thus, it is desirable for the functionality to remain local. Nonetheless, features such as wake-word detection and speaker recognition are desirable since hands-free is an important advantage for many of these products. One approach to achieve these goals is to enable user personalized wake-words. By having the user train their own (or alternatively, a few speakers share training of the same) wake-word, it becomes speaker dependent and thus optimized for use by a particular speaker or a small number of speakers. Since these devices are used by a small number or even a single user, speaker independence is not necessarily a requirement. Additionally, customization of the wake-word inherently identifies the user, and the privacy of the user customizable wake-word provides a level of security without the need for explicit and costly speaker recognition. However, it is challenging to implement such a system with as few training utterances as possible.

[0022] Deep learning or deep neural networks (NNs) have become an attractive option compared to traditional methods due to their improved accuracy. However, these systems are trained offline for a fixed or given wake-word (e.g., "hello"). They require thousands of utterances from thousands of speakers. Some solutions do provide an option to adapt to the user's voice later (enrollment phase, or adaptation based on usage), but typically do not demonstrate the ability to train arbitrary wake-words (user customizable) with only a small number of training utterances. Other available solutions (e.g., isolated word training and detection) suffer from the problem of sensitivity to "spoof" phrases that share important phonemes (or building block sounds) with the wake-word. Indeed, such systems are susceptible to a relatively high degree of false detection in the case of similarly pronounced words.

[0023] Embodiments described herein are directed to devices, methods, and systems for detecting a wake word from an emitted speech. A processing device can determine a first model for using a set of training utterances to perform wake word recognition based on a likelihood ratio. The set of utterances can be analyzed by the first model to determine a second model that includes a training state sequence for each of the set of training utterances, and wherein each training state sequence indicates a possible state for each time interval of the corresponding training utterance. Whether a detected utterance corresponds to a wake word can be determined based on a concatenation of the first model and the second model. More specifically, the processing device can measure a distance between each training state sequence and a state sequence of the detected utterance to generate a set of distances, and can determine a minimum distance among the set of distances. The processing device can determine whether the detected utterance corresponds to a wake word based at least in part on a likelihood ratio of the detected utterance and the minimum distance among the set of distances.

[0024] Figure 1 is a block diagram of a system 100 in accordance with various embodiments, showing an audio processing device 102 communicatively coupled to other devices over one or more networks 114. The audio processing device 102 is used to facilitate audio pattern recognition, and can control devices or applications such as device 103 based on the recognized audio pattern. The audio processing device 102 is shown receiving sound waves 105 from an audio pattern source 104 and sound waves 107 from an audio interference source 106. The audio processing device 102 can itself emit audio interference (not shown) (e.g., through a speaker).

[0025] The audio processing device 102 is also shown interacting with the network 114 over a communication link. To facilitate pattern recognition, the audio processing device 102 uses corresponding audio data received from the audio interference source 106 over the network 114 or generated internally to provide noise cancellation to remove some or all of the audio interference. In embodiments, the noise cancellation can be implemented using independent component analysis (ICA), in which incoming signals (e.g., from a microphone) are separated by source (e.g., signals from the audio pattern source and the audio interference source), and then the audio data is compared to the separated signals to determine which signals should be removed to leave the estimated audio pattern. Noise cancellation in other embodiments can utilize adaptive filters, neural networks, or any technique known in the art that can be used to attenuate non-target components of a signal. In some embodiments, the audio processing device 102 can be integrated with the controlled device 103, which can be controlled based on the recognized audio pattern.

[0026] Audio pattern source 104 is configured to provide sound waves 105 corresponding to an identifiable audio pattern (e.g., a wake word). In some embodiments, audio pattern source 104 can interact with network 114 over a communication link. In some embodiments, the audio pattern is a predetermined audio pattern and / or an audio pattern that can be recognized by pattern recognition software or firmware associated with audio processing device 102. Audio pattern source 104 can be a living object (e.g., a person) or a non-living object (e.g., a machine).

[0027] Audio interference source 106 can be a source of sound waves 107 that interfere with the recognition of the audio pattern corresponding to sound waves 105. Audio interference source 106 is shown interacting with network 114 over a communication link. Audio interference source 106 can provide audio data corresponding to audio interference to audio processing device 102 over network 114. Audio interference source can include a speaker, a television, a video game, an industrial noise source, or any other noise source whose sound output is digitized or can be digitized and provided to audio processing device 102 via network 114.

[0028] Controlled second device 108 is shown coupled to network 114 via a link. Controlled devices 108 and 103 can include any device that can be initiated in response to audio pattern recognition facilitated by audio processing device 102. Exemplary controlled devices include white goods, home automation controllers, thermostat controllers, lighting devices, automated blinds, automated door locks, car controls, windows, industrial controls, and brakes. As used herein, a controlled device can include any logic, firmware, or software application run by the controlled device 110.

[0029] Network 114 can include one or more types of wired and / or wireless networks to Figure 1 communicate network nodes to each other. For example, and without limitation, the network can include a wireless local area network (WLAN) (e.g., Wi-Fi, compliant with 802.11), a PAN (e.g., Bluetooth SIG standard or Zigbee, compliant with IEEE 802.15.4), and the Internet. In embodiments, audio processing device 102 is communicatively coupled to pattern recognition application 112 over Wi-Fi and the Internet. Audio processing device 102 can be communicatively coupled to audio interference source 106 and controlled device 108 over Bluetooth and / or Wi-Fi.

[0030] Figure 2This is a block diagram illustrating the components of an audio processing device 202 according to an embodiment. The audio processing device 202 may be a microcontroller from the Cypress PSoC family, developed by Cypress Semiconductor, Inc., San Jose, California. The audio processing device 202 is shown as including functional blocks such as a microphone array 220, an audio interface 221, a threshold calculation module 222, a speech start detector (SOD) 223, an audio interface control 224, a buffer 225, a combiner 226, and a central processing unit (CPU) 228. Each functional block may be coupled to a bus system 227 (e.g., I2C, I2S) and may be implemented using hardware (e.g., circuitry), instructions (e.g., software and / or firmware), or a combination of hardware and instructions. In one embodiment, some or all of the components of the audio processing device 202 are implemented by circuitry in an integrated circuit device (i.e., on a single integrated circuit substrate) or a single device package. In an alternative embodiment, the components of the audio processing device 202 are distributed across multiple integrated circuit devices, device packages, or other circuitry.

[0031] Microphone array 220 is used to receive signals such as Figure 1 Sound waves such as sound waves 105 and 107. Each microphone in microphone array 220 includes a transducer or other mechanism (e.g., including a diaphragm) to convert the energy of the sound waves into electronic or digital signals (e.g., audio data). Microphone array 220 may include one or more microphones and is sometimes referred to herein as microphone 220. When sound waves 105 and 107 are received during a common period, the audio data includes components corresponding to both sound waves 105 and 107. In some embodiments, one or more microphones of array 220 may be digital microphones. Microphone array 220 may be part of audio interface 221 or a separate peripheral device external to audio processing device 202 but coupled to bus system 227. In some embodiments, microphone array may include threshold / hysteresis settings and / or processing logic for activity detection and measurement to determine whether sound waves received by microphone array 220 meet or exceed an activation threshold and whether the corresponding audio data should be passed to SOD 223 for processing. In various embodiments, the activity threshold level may be the energy level, amplitude, frequency, or any other property of the sound waves. The microphone array 220 may be coupled to a memory (not shown) that stores an activation threshold, which may be dynamically reprogrammable (e.g., via a threshold calculation module 222).

[0032] The audio interface 221 includes circuitry for processing and analyzing audio data received from the microphone array 220. In embodiments, the audio interface 221 digitizes electronic audio signals. Once digitized, the audio interface 221 can provide signal processing (e.g., demodulation, mixing, filtering) to analyze or manipulate properties of the audio data (e.g., phase, wavelength, frequency). The audio interface 221 can also perform beamforming and / or other noise suppression or signal conditioning methods to improve performance in the presence of noise, reverberation, etc.

[0033] In one embodiment, the audio interface 221 includes a pulse density modulator (PDM) front end connected to the microphone array 220. In the PDM front end, a PDM generates a pulse density modulated bit stream based on electronic signals from the microphone array 220. The PDM provides a clock signal to the microphones 220 that determines an initial sampling rate, and then receives a data signal from the microphones 220 that represents audio captured from the environment. From the data signal, the PDM generates a PDM bit stream, and can provide the bit stream to a decimation filter that generates audio data provided to the bus system 227 by either providing high quality audio data or by reducing the sampling rate of the pulse density modulated bit stream from the PDM to low quality audio data. In alternative embodiments, the audio data source is an auxiliary analog-to-digital converter (AUX ADC) front end. In the AUX ADC front end, an analog-to-digital converter converts analog signals from the microphones 220 to digital audio signals. The digital audio signals can be provided to a decimation filter to generate audio data provided to the bus system 227 by either providing high quality audio data or by reducing the sampling rate of the digital audio signals from the ADC to low quality audio data.

[0034] The audio interface controls 224 are used to control the sampling timing of the audio interface 221 or the microphone array 220 and the sampling rate of the audio interface 221 or the microphone array 220. For example, the audio interface controls 224 can control the audio quality (e.g., sampling rate) of the audio data provided to the SOD 223 and the buffer 225, and can also control the timing of such audio data that should be provided to the bus system 227 periodically or continuously. Although shown as a separate functional block, the functionality of the audio interface controls 224 can be performed by the SOD 223 and / or the buffer 225 or any other functional block.

[0035] SOD 223 is used to determine whether audio data received from audio interface 221 is voice onset. SOD 223 can use any voice onset detection algorithm or technique known to those of ordinary skill in the art. In embodiments, audio data with a reduced sampling rate (e.g., 2-4 kHz) is sufficient for detecting voice onset (or other sound onset events) while allowing SOD 223 to be clocked at a lower frequency, thus reducing the power consumption and complexity of SOD 223. Upon detecting a voice onset event, SOD 223 asserts a status signal on bus 227 to wake up wake-up phrase detector (WUPD) 228 from a low-power state (e.g., sleep state) to a high-power state (e.g., active state) to perform phrase detection, as will be discussed further below. Gating of the WUPD 228 block in this manner reduces the average system processing load and reduces the false acceptance rate (FAR) by minimizing background noise and stray audio considered by WUPD 228.

[0036] Threshold computation module 222 monitors ambient noise to dynamically compute and potentially re-adjust an activation threshold of audio that should trigger voice onset detection to avoid unnecessary processing by SOD 223. In embodiments, audio interface control 224 causes audio interface 221 to periodically provide audio data (e.g., ambient noise) to threshold computation module 222 at intervals. In embodiments, threshold computation module 222 can reset the activation threshold level from below to above the current ambient noise level.

[0037] Buffer 225 is used to store periodically sampled primary noise audio data. In embodiments, buffer 225 is sized to store slightly more than 250 milliseconds of audio data (e.g., 253 milliseconds) to accommodate the combination as discussed below. Alternatively, or in addition, buffer 225 can act as a channel to pass through continuously sampled audio data including a wake-up phrase and a command or query after SOD 223 has detected voice onset. In embodiments, audio interface control 224 causes audio interface 221 to periodically provide primary noise to buffer 225 at intervals. Once SOD 223 has detected voice-like sounds, audio interface control 224 can cause audio interface 221 to continuously provide the remaining audio data to buffer.

[0038] The combiner 226 is used to generate continuous audio data using the periodically captured primary noise and the continuously captured residual audio data. In an embodiment, the combiner 226 stitches a portion of the end of the last periodically captured audio data with a portion of the beginning of the continuously captured audio data. For example, the combiner 226 can use an overlap-add operation to overlap 3 milliseconds of the primary noise with the continuously captured audio data. The combiner 226 can output the continuous audio data to the WUPD 228 via a bus system 227.

[0039] The WUPD 228 can determine whether the continuous audio data output by the combiner 226 includes a wake word or wake phrase. When the WUPD 228 is activated, it can perform higher complexity and more power intensive computations (e.g., relative to the SOD 223) to determine whether a wake word or phrase has been spoken, as discussed in relation to Figures 3A-8 The WUPD 228 can make this determination based on the audio data recorded in the buffer 225 (corresponding to the time before the speech onset) and the high quality audio data received after the speech onset is detected, as discussed in further detail. The WUPD 228 can perform a Viterbi decoding operation on the continuous audio data to determine whether the wake word or phrase has been spoken.

[0040] Figure 3A A diagram is shown of a conventional Gaussian Markov Model- Hidden Markov Model (GMM-HMM) wake word or word recognition method 300, where a whole word model for each wake word is trained using any suitable algorithm, such as the maximum likelihood forward-backward algorithm. The Gaussian observation models are determined by using either a diagonal structure or a full covariance structure, where the diagonal structure is the primary approach. The word model can include a "left-to-right" linear sequence of states, where each phoneme has, for example, about three states. The observation vectors O are obtained by front-end spectral analysis, where the Mel-Frequency Cepstral Coefficients (MFCCs) and their derivatives are the most common. In the standard training approach, the final word model is produced using the utterances of tens to hundreds of wake words in an offline training, as depicted in Figure 3A During decoding, a user can wish to compute the probability of an observation sequence O = O1O2...O T using a given model λ, i.e., P(O | λ). This computation can be performed using any suitable algorithm, such as the Viterbi algorithm.

[0041] Figure 3B A phoneme-based training method 310 is shown, where individual phonemes are trained offline to create a phoneme database. It was found that a set of about 50 most common phonemes was sufficient, where each phoneme is modeled with 3 HMM states and 2 Gaussian mixtures per state. During wake word training, the user utters the custom wake word a few times (e.g., 1-3 times). The wake word model is then created by averaging the phonemes of the utterances, as depicted in Figure 3CThese utterances and optionally an augmented set are used in a subsequent training process as shown.

[0042] As Figure 3C shown, the phoneme recognition block 315 is used to determine the most likely phoneme decomposition of the spoken wake word "abracadabra" as Figure 3C shown. The model of the detected phonemes is then used to build an initial word-based model λ 1 for the wake word using a tied-phoneme model (one transcription per instance of the spoken phrase) based on the phoneme transcription.

[0043] Subsequently, the initial word model can be adjusted again using the training utterances to adapt to the speaker, capture inter-phoneme dependencies, and reduce the tied model dimensionality. This training refines the tied-phoneme based model that is independent of the speaker into a speaker-dependent word-based model. Pruning and state combination also occur during adaptation to reduce the size of the model as Figure 3C shown. State combination involves the merging of two or more similar or identical states with very high transition likelihoods between them. In the example of Figure 3C , the 3rd and 4th states of the initial HMM word model corresponding to "R" and "R" can be combined into a single "R" state, and states 11-13 of the initial HMM word model corresponding to "E" "R" and "R" can be combined into a single "R" state.

[0044] In the standard approach, once the model is obtained, it is used to evaluate a speech observation (O) (also referred to herein as an utterance), and the wake word is detected if the probability of having detected the observation sequence exceeds a threshold TH (given the model λ 1 ) normalized by the probability (given the garbage or background model λ g ). This process is described in Figure 3D .

[0045] The likelihood ratio LR is defined as:

[0046]

[0047] In other words, the LR can be based on how close a match of the observation O is to the garbage model λ 1 compared to how close a match of the observation O is to the model λ 1 . However, this approach does not capture anything about λ 1 beyond the final probabilities.How to model any information about O. In many cases, a falsely accepted word can exhibit partial phoneme matching, or contain a fully matching subset of phonemes. Such utterances often have very high LR because all factors match and the vowel is preserved. In this example, relying on LR alone would lead to false acceptance. Another common false acceptance scenario is a word matching a portion of the wake word. Embodiments of the present disclosure overcome the above problems by incorporating how the model is internally excited over the entire duration of an input observation sequence into the model training and decision process. In the case of HMM-GMM models, one such excitation is the most likely state.

[0048] Figure 4A A model generation training method is shown in accordance with some embodiments of the present disclosure. Upon obtaining a model λ 1 (as discussed above with respect to Figures 3B-3D ), the audio processing device 202 can perform a second phase of training by processing the training utterances with the model λ 1 to obtain, for each training utterance, a state sequence indicating the most likely state (e.g., a particular phoneme model) at each time interval during the course of the training utterance, which will form the basis of the phase 2 model λ 2 . For example, each training utterance can last 420 milliseconds, and can be captured in 10 millisecond frames (e.g., 10 millisecond time intervals). The state sequence can indicate the most likely state (e.g., a particular phoneme model) that the training utterance will be in at each particular frame of the utterance (e.g., as we transition through the model). As shown in Figure 4B , the model λ 1 may be an 11 state model captured at 10 millisecond time intervals over 420 ms. Figure 4B The varying length of time for each state is also shown. As shown in the example of Figure 4B , each state can last 20 ms, except for the 4th state (“A”), which is shown lasting 30 ms. The state sequence can indicate the likelihood of remaining in the current state or transitioning to a new state at each time interval.

[0049] Referring back to Figure 4A , the audio processing device 202 can solve for the individual most likely state q 1 in the N state model λ t at time t as follows:

[0050]

[0051] if u k is the kth training utterance of K total training utterances, and is the corresponding observation sequence, then the audio processing device 202 can determine the most likely state sequence for each training utterance u k during training as follows:

[0052]

[0053] The audio processing device 202 can create a second stage model λ 2 by collecting the state sequence for each of the K training utterances:

[0054]

[0055] The audio processing device 202 can then generate a final model λ by concatenating the stage 1 and stage 2 (λ 1 and λ 2 ) models:

[0056] λ = {λ 1 , λ 2}

[0057] Although the model λ 2 is obtained as described above, any suitable method can be used. For example, the audio processing device 202 can utilize a statistical method to obtain the sum distribution of each state over time. Figure 4A A 2-stage recognition process is also shown in accordance with some embodiments of the disclosure. The audio processing device 202 can perform recognition of a detected utterance using the model λ 1 (e.g., based on LR) in conjunction with the model λ 2 . For example, when a recognition decision is made using the model λ 1 , the audio processing device 202 can subsequently verify the result of the recognition using the model λ 2 using the model λ 1 . To verify, the audio processing device 202 can determine the distance between each of the K state sequences and the state sequence of the detected utterance (e.g., the recognized utterance), thereby generating K distances, and determine the minimum distance among the K distances, which is computed as follows:

[0058]

[0059] The audio processing device 202 can incorporate this minimum distance measurement into the final decision:

[0060] Final Decision = f(Dqmin u , LR(O, λ 1 , λ g ))

[0061] The audio processing device 202 can incorporate this minimum distance measurement into the final decision:u is compared to a threshold, and if Dqmin u exceeds the threshold, it determines that the detected utterance is not a wake word. In some embodiments, the audio processing device 202 can assign weights to Dqmin u so that it has more or less influence on the final decision based on, for example, user preferences. It should be noted that although the above examples are described with respect to the use of Dqmin u and LR(O, λ 1 , λ g ) to determine whether the detected utterance corresponds to a wake word, in some embodiments, the audio processing device 202 can utilize only the model λ 2 (e.g., only Dqmin u ) in determining whether the detected utterance corresponds to a wake word.

[0062] Figure 5 sequences of various different utterances are shown. The reference for one of the training utterances for "abracadabra" is labeled "reference." The "reference" may correspond to the training utterance with the smallest distance (Dqmin u ) from the state sequence of the utterance being recognized. The recognized wake word that is the same as the training utterance (e.g., "abracadabra") is labeled "WW" and follows the "reference" state sequence well, as shown. Another wake word that is the same as the training utterance but is affected by noise is labeled "WW-noise." As can be seen, "WW-noise" exhibits some minor state mismatches, but still follows the "reference" state sequence well overall. In a noisy environment, the LR is typically lower than in a quiet environment, and the state sequence exhibits only a minor increase in distance score, thus enhancing the detection capability in a noisy environment.

[0063] In many cases, false acceptances can exhibit partial phoneme matches, or contain a subset of phonemes that match perfectly. For example, the utterance "Abraaaaaaaaaaaaabra" is shown in Figure 5 and labeled "Abraaaaaaaaaaaaabra." In this case, the state sequence for "Abraaaaaaaaaaaaabra" matches well at the beginning, but then gets stuck around the state representing "a." The state sequence matches well again near the end of the utterance. This utterance typically has a very high LR because all the factors match and the vowel is maintained. In this instance, relying on the LR alone can cause a false acceptance. However, using Dqmin uThis shows a significant distance between the state sequence of "Abraaaaaaaaaaaaaabra" in frames 30-63 and the "reference" state sequence in frames 30-63. Therefore, the audio processing device 202 can refuse to emit "Abraaaaaaaaaaaaaaabra". Another common scenario for incorrect acceptance is when the emitted sound matches part of a wake word. This is achieved through... Figure 5 The state sequence labeled "Updowncadabra" is shown. The mismatch portion of the "Updowncadabra" state sequence exhibits a poor match with the "reference" state sequence at the corresponding frame, resulting in a high distance score and rejection. Finally, the completely mismatched vocalization labeled "California" is shown. This vocalization, in addition to poorly following the "reference" state sequence throughout, will likely exhibit a very low range (LR).

[0064] In some embodiments, the distance measurement between each of the K state sequences and the state sequence being identified as emitting sound is replaced or appended to the model λ. 2 It can be based on other parameters. One such parameter might be the likelihood ratio (LR) and how it changes with each observation O. t And it evolves over time. Having obtained the model λ 1 Time (as mentioned above) Figures 3B-3D (As discussed), the audio processing device 202 can utilize model λ 1 Processing training vocalization u k The second phase of training is conducted to determine how the likelihood ratio (LR) of each training phonation changes with each observation O. t And it evolves over time. Given an observation sequence O t =O1,…,O t and model λ 1 In the case of being in state S at time t i The probability is given by the following formula:

[0065] γ t (i,O t ,λ 1 )=P(q t =S i |O t ,λ 1 )

[0066] At time t, in model λ 1 The maximum probability in any state within is then given by the following formula:

[0067]

[0068] Then, process each training vocalization u k Given:

[0069]

[0070] And so, the audio processing device 202 can determine the likelihood ratio (LR) of the training utterance over time as:

[0071]

[0072] The audio processing device 202 can then generate a second stage model λ 2 as follows:

[0073]

[0074] During recognition of the received utterance, the audio processing device 202 can compute a distance measure D between each and LR t to generate K distance measures. The audio processing device 202 can then use the minimum distance Dmin u among all K distance measures (e.g., among all training utterances) along with the likelihood ratio LR(O, λ 1 , λ g to compute a final decision:

[0075]

[0076] Final Decision = f(Dmin u , LR(O, λ 1 , λ g ))

[0077] The audio processing device 202 can compare Dmin u to a threshold, and if Dmin u exceeds the threshold, it determines that the detected utterance is not a wake word. In some embodiments, the audio processing device 202 can assign a weight to Dmin u so that it has more or less influence on the final decision based on, for example, user preferences. In some embodiments, the audio processing device 202 can incorporate both stage 2 recognition parameters discussed above into the final decision:

[0078] Final Decision = f(Dqmin u , Dmin u , LR(O, λ 1 , λ g ))

[0079] Other parameters can also be used in addition to or instead of likelihood ratios and state sequences over time. Examples of such parameters can include measures of loudness, pitch, and frame energy, among others. Additionally, the models λ 1 and λ 2 may not be limited to HMM-GMM models, but can also include neural networks or any other appropriate model type.

[0080] Figure 6 is a flowchart of a method 600 of detecting a wake word in accordance with some embodiments. The method 600 can be performed by processing logic that can comprise hardware (e.g., circuitry, dedicated logic, programmable logic, a processor, processing device, a central processing unit (CPU), a system on a chip (SoC), etc.), software (e.g., instructions run / executed on a processing device), firmware (e.g., microcode), or a combination thereof. For example, the method 600 can be performed by the audio processing device 202 executing wake word detection firmware.

[0081] Referring also to Figure 2 at block 605, the audio processing device 202 can determine a first model configured to identify a wake word based on a set of training utterances. At block 610, the audio processing device 202 can analyze the set of training utterances using the first model to determine a second model that includes a training state sequence for each of the set of training utterances, and wherein each training state sequence indicates a possible state for each time interval of a corresponding training utterance. At block 615, the audio processing device 202 can determine a state sequence for a detected utterance that indicates a possible state for each time interval of the detected utterance. At block 620, the audio processing device 202 can determine a distance between each training state sequence and the state sequence for the detected utterance to generate a set of distances. At block 625, determine whether the detected utterance corresponds to a wake word based at least in part on a likelihood ratio for the detected utterance and a minimum distance among the set of distances.

[0082] Figure 7 is a flowchart of a method 700 for detecting a wake word in accordance with some embodiments. The method 700 can be performed by processing logic that can comprise hardware (e.g., circuitry, dedicated logic, programmable logic, a processor, processing device, a central processing unit (CPU), a system on a chip (SoC), etc.), software (e.g., instructions run / executed on a processing device), firmware (e.g., microcode), or a combination thereof. For example, the method 700 can be performed by the audio processing device 202 executing wake word detection firmware.

[0083] Referring also to Figure 2At box 705, the audio processing device 202 can determine a first model configured to recognize wake words based on a training set of vocalizations. At box 710, the audio processing device 202 can use the first model to analyze the training set of vocalizations to determine a second model, which includes an indication of the likelihood ratio over time for each training vocalization. At box 715, the audio processing device 202 can determine the likelihood ratio over time for detected vocalizations, and at box 720, it can determine the distance between the likelihood ratio over time for each training vocalization and the likelihood ratio over time for the detected vocalizations to generate a distance set. At box 725, the audio processing device 202 can determine whether a detected vocalization corresponds to a wake word, at least in part, based on the minimum distance between the likelihood ratio of the detected vocalizations and the distance set.

[0084] Figure 8 This is a flowchart of a method 800 for detecting a wake word according to some embodiments. Also refer to... Figure 2 At box 805, the audio processing device 202 can determine a first model configured to recognize wake words based on a training vocalization set. At box 810, the audio processing device 202 can use the first model to analyze the training vocalization set to determine a second model, the second model including a training state sequence for each of the training vocalizations, wherein each training state sequence indicates a possible state for each time interval corresponding to the training vocalization. The second model may further include an indication of the likelihood ratio over time for each training vocalization. At box 815, the audio processing device 202 can determine a state sequence of detected vocalizations, the state sequence indicating a possible state for each time interval of the detected vocalizations. The audio processing device 202 may further determine the likelihood ratio over time for the detected vocalizations. At box 820, the audio processing device 202 can determine the distance between each training state sequence and the state sequence of the detected vocalizations to generate a first distance set. The audio processing device 202 may further determine the distance between the likelihood ratio over time for each training vocalization and the likelihood ratio over time for the detected vocalizations to generate a second distance set. At box 825, the audio processing device 202 can, at least in part, base its processing on the likelihood ratio of the detected sound emission and the minimum distance (Dqmin) in the first distance set. u ), and the minimum distance (Dmin) in the second distance set. u This is used to determine whether the detected sound corresponds to a wake word.

[0085] Figure 9 It shows An embodiment of the core architecture 900 of the processing device, for example, provided by Cypress Semiconductor (San Jose, California). The core architecture 900 used in this product series. In one embodiment, the core architecture 900 includes a microcontroller 1102. The microcontroller 1102 includes a CPU (Central Processing Unit) core 1104 (which may correspond to...). Figure 1 The system includes a processing device 130, a flash program storage device 1106, a DOC (debug on-chip) 1108, a prefetch buffer 1110, a dedicated SRAM (static random access memory) 1112, and a special function register 1114. In an embodiment, the DOC 1108, the prefetch buffer 1110, the dedicated SRAM 1112, and the special function register 1114 are coupled to a CPU core 1104 (e.g., CPU core 1006), while the flash program storage device 1106 is coupled to the prefetch buffer 1110.

[0086] The core architecture 1100 may also include a CHub (core hub) 1116, comprising a bridge 1118 and a DMA controller 1120 coupled to the microcontroller 1102 via a bus 1122. The CHub 1116 provides the primary data and control interface between the microcontroller 1102 and its peripherals (e.g., peripherals) and memory, and the programmable core 1124. The DMA controller 1120 can be programmed to transfer data between system elements without burdening the CPU core 1104. In various embodiments, each of these sub-components of the microcontroller 1102 and the CHub 1116 may differ from each selection or type of the CPU core 1104. The CHub 1116 may also be coupled to a shared SRAM 1126 and an SPC (System Performance Controller) 1128. A dedicated SRAM 1112 is independent of the shared SRAM 1126 accessed by the microcontroller 1102 via the bridge 1118. CPU core 1104 accesses dedicated SRAM 1112 without going through bridge 1118, thus allowing local register and RAM access to occur simultaneously with DMA access to shared SRAM 1126. Although referred to herein as SRAM, in various other embodiments, these memory modules can be any suitable type of various (volatile or non-volatile) memory or data storage modules.

[0087] In various embodiments, programmable core 1124 can include various combinations of subcomponents (not shown), including but not limited to digital logic arrays, digital peripheral devices, analog processing channels, global routing analog peripheral devices, DMA controllers, SRAM and other appropriate types of data storage devices, IO ports, and other suitable types of subcomponents. In one embodiment, programmable core 1124 includes GPIO (general purpose IO) and EMIF (extended memory interface) blocks 1130 (to provide a mechanism for extended external off-chip access by microcontroller 1102), programmable digital blocks 1132, programmable analog blocks 1134, and special function blocks 1136, each configured to implement one or more of the subcomponent functions. In various embodiments, special function blocks 1136 can include dedicated (non-programmable) function blocks and / or include one or more interfaces to dedicated function blocks, such as USB, crystal oscillator drivers, JTAG, etc.

[0088] Programmable digital blocks 1132 can include digital logic arrays including arrays of digital logic blocks and associated routing. In one embodiment, the digital block architecture consists of UDBs (universal digital blocks). For example, each UDB can include an ALU along with CPLD functionality.

[0089] In various embodiments, one or more UDBs of programmable digital blocks 1132 can be configured to perform various digital functions, including but not limited to one or more of the following functions: basic I2C slave device; I2C master device; SPI master or slave device; multi-wire (e.g., 3-wire) SPI master or slave device (e.g., MISO / MOSI multiplexed on a single pin); timers and counters (e.g., a pair of 8-bit timers or counters, a 16-bit timer or counter, an 8-bit capture timer, etc.); PWM (e.g., a pair of 8-bit PWMs, a 16-bit PWM, an 8-bit dead-band PWM, etc.); level sensitive I / O interrupt generator; quadrature encoder, UART (e.g., half-duplex); delay line; and any other suitable type of digital function or combination of digital functions that can be implemented in multiple UDBs.

[0090] In other embodiments, additional functionality can be implemented using a set of two or more UDBs. For purposes of illustration only and not limitation, the following functionality can be implemented using multiple UDBs: an I2C slave device that supports hardware address detection and the ability to handle complete transactions without CPU core (e.g., CPU core 1104) intervention and help prevent forced clock stretching on any bit in the data stream; an I2C multi-master device that can include slave options in a single block; PRS or CRC of arbitrary length (up to 32 bits); SDIO; SGPIO; a digital correlator (e.g., with up to 32 bits, with 4x oversampling and support for configurable thresholds); a LIN bus interface; a delta-sigma modulator (e.g., for a class-D audio DAC with differential output pair); I2S (stereo); LCD drive control (e.g., a UDB can be used to implement timing control for an LCD drive block and provide display RAM addressing); a full-duplex UART (e.g., 7, 8, or 9 bits with 1 or 2 stop bits and parity, and support for RTS / CTS); IRDA (transmit or receive); a capture timer (e.g., 16 bits or similar); a deadband PWM (e.g., 16 bits or similar); an SMbus (including formatting SMbus packets with CRC in software); a brushless motor driver (e.g., to support 6 / 12 step commutation); automatic BAUD rate detection and generation (e.g., automatically determine the BAUD rate for standard rates from 1200 to 115200 BAUD, and after detection generate the required clock to generate the BAUD rate); and any other suitable type of digital functionality or combination of digital functionality that can be implemented in multiple UDBs.

[0091] Programmable analog blocks 1134 can include analog resources including, but not limited to, comparators, mixers, PGAs (programmable gain amplifiers), TIAs (trans- impedance amplifiers), ADCs (analog-to-digital converters), DACs (digital-to-analog converters), voltage references, current sources, sample-and-hold circuits, and any other suitable type of analog resource. Programmable analog blocks 1134 can support various analog functions including, but not limited to, analog routing, LCD drive IO support, capacitive sensing, voltage measurement, motor control, current-to-voltage conversion, voltage-to-frequency conversion, differential amplification, light measurement, inductive position monitoring, filtering, voice coil driving, magnetic card reading, acoustic Doppler measurement, echo ranging, modem transmit and receive encoding, or any other suitable type of analog function.

[0092] The embodiments described herein can be used in various designs of mutual-capacitance sensing systems, self-capacitance sensing systems, or a combination of both. In one embodiment, a capacitive sensing system detects a plurality of sensing elements activated in an array, and can analyze signal patterns on adjacent sensing elements to separate noise from actual signals. As will be appreciated by those of ordinary skill in the art having the benefit of the present disclosure, the embodiments described herein are not tied to a particular capacitive sensing solution, and can also be used with other sensing solutions, including optical sensing solutions.

[0093] In the description above, numerous specific details are set forth. However, it is understood that embodiments of the disclosure might be practiced without these specific details. In some instances, well-known structures and devices are not described in exhaustive detail, to avoid obscuring the description.

[0094] Some portions of the detailed description are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self- consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0095] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the above discussion, it is appreciated that throughout the description, discussions utilizing terms such as "determining", "detecting", "comparing", "resetting", "adding", "calculating" or the like, refer to the action and processes of a computing system, or similar electronic computing device, that manipulates and transforms data represented as physical (e.g., electronic) quantities within the computing system's registers and memories into other data similarly represented as physical quantities within the computing system memories or registers, or other such information storage, transmission or display devices.

[0096] The words “example” or “exemplary” are used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “example” or “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects or designs. Rather, use of the words “example” or “exemplary” is intended to present concepts in a concrete fashion. As used in this application, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless specified otherwise, or as is clear from the context, the statement “X includes A or B” is intended to mean any of the natural inclusive permutations. That is, if X includes A; X includes B; or X includes both A and B, then “X includes A or B” is satisfied under any of the foregoing instances. In addition, the articles “a” and “an” as used in this application and the appended claims should generally be construed to mean “one or more” unless specified otherwise or clear from context to be directed to a singular form. Moreover, use of the term “an embodiment” or “one embodiment” or “an implementation” or “one implementation” throughout is not intended to mean the same embodiment or implementation unless specifically so stated.

[0097] Embodiments described herein can also relate to an apparatus for performing the operations herein. This apparatus can be specially constructed for the required purposes, or it can comprise a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program can be stored in a non-transitory computer-readable storage medium, such as, but not limited to, any type of disk including floppy disks, optical disks, CD-ROMs and magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, flash memories, or any type of media suitable for storing electronic instructions. The term “computer-readable storage medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) that store the one or more sets of instructions. The term “computer-readable medium” should also be taken to include any medium that is capable of storing, encoding or carrying a set of instructions for execution by a machine and that causes the machine to perform any one or more of the methodologies of the present embodiments. The term “computer-readable storage medium” shall accordingly be taken to include, but not be limited to, solid-state memories, optical media, magnetic media, any medium that is capable of storing a set of instructions for execution by a machine and that causes the machine to perform any one or more of the methodologies of the present embodiments.

[0098] The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general purpose systems can be used with programs in accordance with the teachings herein, or it can prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will appear from the description above. In addition, the present embodiments are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages can be used to implement the teachings of the embodiments as described herein.

[0099] The above description sets forth numerous specific details such as examples of specific systems, components, methods and so forth, in order to provide a thorough understanding of several embodiments of the present disclosure. It will be apparent to one skilled in the art, however, that at least some embodiments of the present disclosure can be practiced without these specific details. In other instances, well-known components or methods are not described in exhaustive detail in order to avoid unnecessarily obscuring the present embodiments. Thus, the specific details set forth above are merely exemplary. Particular implementations can vary from these exemplary details and still be contemplated to be within the scope of the present embodiments.

[0100] It is to be understood that the above description is intended to be illustrative, and not restrictive. Many other embodiments will be apparent to those of skill in the art upon reading and understanding the above description. Various modifications and adaptations to the embodiments can occur to those skilled in the art without departing from the spirit and scope of the present disclosure, and it is intended to cover all adaptations and modifications as fall within the scope of the present disclosure.

Claims

1. A method for speech recognition, comprising: determining a first model configured to recognize a phrase based on a set of training utterances; analyzing the set of training utterances using the first model to determine a second model, the second model comprising a sequence of training states for each of the set of training utterances, and wherein each sequence of training states indicates possible states for each time interval of a corresponding training utterance; and determining whether a detected utterance corresponds to the phrase based on a concatenation of the first model and the second model.

2. The method of claim 1, wherein, Determining whether the detected utterance corresponds to the phrase comprises: determining a sequence of states for the detected utterance, the sequence of states indicating possible states for each time interval of the detected utterance; and determining distances between each sequence of training states and the sequence of states for the detected utterance to generate a set of distances.

3. The method of claim 2, wherein, Determining whether the detected utterance corresponds to the phrase further comprises: determining a likelihood ratio for the detected utterance using the first model; and determining whether the detected utterance corresponds to the phrase based at least in part on the likelihood ratio for the detected utterance and a minimum distance among the set of distances.

4. The method of claim 3, wherein, Determining that the minimum distance among the set of distances is below a threshold indicates that the detected utterance corresponds to the phrase.

5. The method of claim 1, wherein, The second model comprises parameters for each of the set of training utterances, the parameters comprising likelihood ratios over time, such that the second model comprises an indication of likelihood ratios over time for each of the set of training utterances.

6. The method of claim 5, wherein, Determining whether the detected utterance corresponds to the phrase comprises: determining a likelihood ratio over time for the detected utterance; and determining distances between the likelihood ratio over time for each training utterance and the likelihood ratio over time for the detected utterance to generate a set of distances.

7. The method of claim 6, wherein, Determining whether the detected utterance corresponds to the phrase further comprises: determining a likelihood ratio for the detected utterance using the first model; and determining whether the detected utterance corresponds to the phrase based at least in part on the likelihood ratio for the detected utterance and a minimum distance among the set of distances.

8. A system for speech recognition, comprising: a memory; and a processing device operatively coupled to the memory, the processing device configured to: determine a first model configured to recognize a phrase based on a set of training utterances; analyze the set of training utterances using the first model to determine a second model, the second model comprising a sequence of training states for each of the set of training utterances, and wherein each sequence of training states indicates possible states for each time interval of a corresponding training utterance; and determine whether a detected utterance corresponds to the phrase based on a concatenation of the first model and the second model.

9. The system of claim 8, wherein, To determine whether the detected utterance corresponds to the phrase, the processing device is configured to: determine a sequence of states for the detected utterance, the sequence of states indicating possible states for each time interval of the detected utterance; and and determining a distance between each training state sequence and the state sequence of the detected utterance to generate a set of distances.

10. The system of claim 9, wherein, To determine whether the detected utterance corresponds to the phrase, the processing device is further configured to: determine a likelihood ratio for the detected utterance using the first model; and determine whether the detected utterance corresponds to the phrase based at least in part on the likelihood ratio for the detected utterance and a minimum distance among the set of distances.

11. The system of claim 10, wherein, The processing device is further configured to: compare the minimum distance among the set of distances to a threshold; and in response to determining that the minimum distance among the set of distances is below the threshold, indicate that the detected utterance corresponds to the phrase.

12. The system of claim 8, wherein, The processing device is further configured to: for each training utterance, determine a likelihood ratio over time for the training utterance, wherein the second model further comprises an indication of the likelihood ratio over time for each training utterance.

13. The system of claim 12, wherein, To determine whether the detected utterance corresponds to the phrase, the processing device is configured to: determine a likelihood ratio over time for the detected utterance; and determine a distance between the likelihood ratio over time for each training utterance and the likelihood ratio over time for the detected utterance to generate a set of distances.

14. The system of claim 13, wherein, To determine whether the detected utterance corresponds to the phrase, the processing device is further configured to: determine a likelihood ratio for the detected utterance using the first model; and determine whether the detected utterance corresponds to the phrase based on the likelihood ratio for the detected utterance and a minimum distance among the set of distances.

15. A non-transitory computer-readable medium having instructions stored thereon that, when executed by a processing device, cause the processing device to: determine a first model configured to recognize a phrase based on a set of training utterances; analyzing the set of training vocalizations using the first model to determine a second model, the second model including a sequence of training states for each of the set of training vocalizations, and wherein, each training state sequence indicates a possible state for each time interval of a corresponding training utterance; and determine whether a detected utterance corresponds to the phrase based on a concatenation of the first model and the second model.

16. The non-transitory computer-readable medium of claim 15, wherein, To determine whether the detected utterance corresponds to the phrase, the processing device is to: determine a state sequence for the detected utterance, the state sequence indicating a possible state for each time interval of the detected utterance; and determine a distance between each training state sequence and the state sequence of the detected utterance to generate a set of distances.

17. The non-transitory computer-readable medium of claim 16, wherein, To determine whether the detected utterance corresponds to the phrase, the processing device is further to: determine a likelihood ratio for the detected utterance using the first model; and determine whether the detected utterance corresponds to the phrase based at least in part on the likelihood ratio for the detected utterance and a minimum distance among the set of distances.

18. The non-transitory computer-readable medium of claim 17, wherein, In response to determining that the minimum distance among the set of distances is below a threshold, the processing device determines that the detected utterance corresponds to the phrase.

19. The non-transitory computer-readable medium of claim 18, wherein, The second model includes parameters for each of the set of training vocalizations, the parameters including likelihood ratios over time, such that the second model includes an indication of the likelihood ratios over time for each of the set of training vocalizations.

20. The non-transitory computer-readable medium of claim 19, wherein, To determine whether the detected vocalization corresponds to the phrase, the processing device is to: determine a likelihood ratio over time for the detected vocalization; and determine a distance between the likelihood ratio over time for each training vocalization and the likelihood ratio over time for the detected vocalization to generate a set of distances.

21. The non-transitory computer-readable medium of claim 20, wherein, To determine whether the detected vocalization corresponds to the phrase, the processing device is further to: determine a likelihood ratio for the detected vocalization using the first model; and determine whether the detected vocalization corresponds to the phrase based at least in part on the likelihood ratio for the detected vocalization and a minimum distance among the set of distances.

Images