Contextual Bias for Speech Recognition
A neural network-based speech recognition model with joint audio and context bias components enhances contextual awareness, improving recognition of out-of-vocabulary words and phrases by up to 68% through dynamic contextual adaptation.
Patent Information
- Application Number
- JP2023025301
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-05-06
- Filing Date
- 2023-02-21
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2040-03-31
AI Technical Summary
Automated speech recognition systems struggle with recognizing out-of-vocabulary words and phrases due to limited contextual awareness, particularly in end-to-end models, leading to increased word error rates and latency.
A neural network-based speech recognition model that incorporates context bias through a joint training of an audio processing component and a context bias component, using context n-gram embeddings to dynamically adjust recognition towards relevant phrases, including out-of-vocabulary terms.
Significantly improves word error rate by up to 68% and enhances recognition of unusual phrases by dynamically adapting to the current context, allowing for personalized and flexible contextual biasing without requiring fixed context information during training.
Smart Images

Figure 0007583080000011 
Figure 0007583080000012 
Figure 0007583080000013
Abstract
Description
[Technical field]
[0001] This disclosure relates to contextual bias for speech recognition. [Background technology]
[0002] Recognizing the context of speech is difficult for automated speech recognition (ASR) systems, given the wide variety of words that people may speak, as well as the many variations in accents and pronunciations. Often, the types of words and phrases a person speaks vary depending on the context in which that person finds themselves.
[0003] Contextual automatic speech recognition (ASR) biases speech recognition towards a given context, such as a user's own playlist, contacts, or names of geographical locations. Contextual information typically includes a list of relevant phrases to be recognized, which often includes rare phrases or even foreign words that are rarely encountered during training. To implement contextual biasing, conventional ASR systems sometimes model the contextual information in an independent contextual language model (LM) using n-gram weighted finite state transducers (WFSTs) and construct the independent contextual LM with a baseline LM for on-the-fly (OTF) rescoring.
[0004] Recently, end-to-end (E2E) models have shown great promise for ASR, demonstrating improved word error rate (WER) and latency metrics compared to traditional on-device models. These E2E models, which convolve an acoustic model (AM), a pronunciation model (PM), and a LM into a single network to directly learn speech-to-text mappings, have shown competitive results compared to traditional ARS systems with separate AM, PM, and LM. Typical E2E models include word-based connectionist temporal classification (CTC) models, recurrent neural network transducer (RNN-T) models, and attention-based models such as Listen, Attend, and Spell (LAS).
[0005] In the grapheme-only model of E2E, the predicted subword units output by the attendant component are graphemes. In the subword-only model of E2E, the predicted subword units output by the attendant component are subwords, i.e., sets of common subword units. The output of the attention module is passed to a speller (or "decoder"), which is similar to an LM and generates a probability distribution over the set of hypothesized words.
[0006] Contextual ASR can be difficult for E2E models because they maintain a limited number of recognition candidates during beam search decoding. Spelling out-of-vocabulary (OOV) words, especially classes of words rarely seen in training, can be particularly difficult for E2E grapheme-only and E2E subword-only models. Summary of the Invention [Means for solving the problem]
[0007] Implementations herein are directed to a speech recognition model configured to bias the speech recognition process towards predefined words and phrases relevant to the current context. The model can be an end-to-end model implemented with a neural network. The model can have a jointly trained audio processing component and a context bias component, where audio-based encoding and contextual n-gram embedding are jointly optimized. In use, the model can be presented with a set of context phrases that may include out-of-vocabulary (OOV) terms not seen during training. Compared to traditional contextualization techniques, the proposed system can provide significant improvements on word error rate (WER), with some tests showing a relative WER improvement of as much as 68%, demonstrating the benefit of joint optimization over individually trained components.
[0008] The set of context phrases used to bias can be dynamically changed as the context changes. For example, the system can use text information (e.g., a sequence of graphemes) that represents the biasing phrases. Different sets of phrases can be used for different situations, and the set of phrases can be personalized to the user. For example, if the user mentions the word "call" or has opened an application for calling or messaging, the system can select a name from the user's contact list as a context phrase to bias. As another example, if the user has opened a media playback application or is currently playing media, the list of context phrases to bias can be a list of available songs or other media items from the user's media library, or a list of related terms. The speech recognition system can determine the appropriate context based on context data indicating, for example, the user's contacts, calendar events, open applications, or location. From this context, the system can select the appropriate set of related phrases.
[0009] As will become apparent, dynamically incorporating contextual bias into a neural network can enable an ASR model to improve recognition of out-of-vocabulary terms and rare phrases. Context phrases can be represented as a set of n-grams of words. In some implementations, a neural network ASR model incorporating contextual bias includes a contextual listen, attend, and spell (CLAS) model. A CLAS model is an encoder-decoder model that jointly optimizes ASR components along with contextual n-gram embeddings. During inference, a CLAS model can be presented with a set of context phrases that may include OOV terms not seen during training. Feeding the model with a set of context phrases provides a significant advantage because the model does not require the number and content of context phrases to be fixed at training time.
[0010] One aspect of the disclosure provides a method for incorporating contextual bias information into a speech recognition model to bias speech recognition toward one or more predefined terms / phrases. The method includes receiving, in data processing hardware, audio data encoding an utterance, and obtaining, by the data processing hardware, a set of bias phrases corresponding to a context of the utterance. Each bias phrase in the set of bias phrases includes one or more words. The method also includes processing, by the data processing hardware, acoustic features derived from the audio data using a speech recognition model to generate an output from the speech recognition model. The speech recognition model includes a first encoder and a corresponding first attention module, a bias encoder and a corresponding bias attention module, and a decoder configured to determine a likelihood of a sequence of speech elements based on an output of the first attention module and an output of the bias attention module. The first encoder is configured to receive the acoustic features, and the bias encoder is configured to receive data indicative of the obtained set of bias phrases. The method also includes determining, by the data processing hardware, a transcript for the utterance based on the likelihood of the sequence of speech elements determined by the decoder.
[0011] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, the bias encoder is configured to encode a corresponding bias context vector for each bias phrase in the set of bias phrases, and the bias attention module is configured to calculate attention for the bias context vector. In these implementations, the bias attention module may be configured to receive an additional bias context vector that does not correspond to any of the bias phrases in the set of bias phrases, where the additional bias context vector represents an option to not bias the likelihood of the sequence of speech elements determined by the decoder toward any of the bias phrases. These implementations may additionally or alternatively include a bias encoder and a bias attention module configured to operate with a variable number of bias phrases in the set of bias phrases that are not specified during training of the speech recognition model.
[0012] A phonetic element may include a word, a subword, or a grapheme. A set of bias phrases may include a set of contact names, or a set of media item names, personalized for a particular user.
[0013] In some examples, the first encoder, the first attention module, the biased encoder, the biased attention module, and the decoder are jointly trained to predict a sequence of graphemes from a sequence of acoustic feature frames. In some configurations, the first encoder includes a stacked recurrent neural network (RNN) and / or the decoder includes a stacked unidirectional RNN configured to calculate probabilities of a sequence of output tokens.
[0014] In some implementations, the first attention module is configured to calculate attention as a function of a previous hidden state of the decoder and the complete sequence of context vectors output by the first encoder for the utterance, and the bias attention module is configured to calculate attention as a function of a previous hidden state of the decoder and the complete sequence of context vectors representing bias phrases in the set of bias phrases. In additional implementations, the decoder is configured to determine the hidden state based on an embedding vector for a previous grapheme output from the speech recognition model, a previous hidden state of the decoder, the audio context vector output by the first attention module, and the bias context vector output by the bias attention module.
[0015] In some examples, the method also includes obtaining, by the data processing hardware, a list of bias prefixes corresponding to bias phrases in the set of bias phrases, and determining, by the data processing hardware, using the speech recognition model, whether the partial transcript for the utterance includes one of the bias prefixes in the list of bias prefixes, where each bias prefix in the list of bias prefixes represents a first portion of one or more of the bias phrases in the set of bias phrases. In these examples, if the partial transcript for the utterance includes one of the bias prefixes, the method also includes identifying, by the data processing hardware, a subset of one or more bias phrases in the set of bias phrases that includes one of the bias prefixes included in the partial transcript, and enabling, by the data processing hardware, only the identified subset of the one or more bias phrases to bias the speech recognition model. In another example, the method includes obtaining, by the data processing hardware, a list of bias prefixes corresponding to bias phrases in the set of bias phrases, where each bias prefix in the list of bias prefixes represents an initial portion of one or more of the bias phrases in the set of bias phrases, and adjusting, by the data processing hardware, a bias attention probability of a bias attention module for encoding an output of a bias encoder for bias phrases in the set of bias phrases based on whether one or more of the bias prefixes in the list of bias prefixes are included in the partial transcription for the utterance, where at least one bias prefix in the list of bias prefixes includes a first word of a corresponding bias phrase in the set of bias phrases and excludes a last word of the corresponding bias phrase.
[0016] Another aspect of the present disclosure provides a system that incorporates contextual bias information into a speech recognition model to bias speech recognition toward one or more predefined terms / phrases. The system includes data processing hardware and memory hardware in communication with the data processing hardware that stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving audio data encoding an utterance and obtaining a set of bias phrases corresponding to a context of the utterance. Each bias phrase in the set of bias phrases includes one or more words. The operations also include processing acoustic features derived from the audio data using a speech recognition model to generate an output from the speech recognition model. The speech recognition model includes a first encoder and a corresponding first attention module, a bias encoder and a corresponding bias attention module, and a decoder configured to determine a likelihood of a sequence of speech elements based on an output of the first attention module and an output of the bias attention module. The first encoder is configured to receive the acoustic features, and the bias encoder is configured to receive data indicative of the obtained set of bias phrases. The operations also include determining a transcript for the utterance based on the likelihood of the sequence of speech elements determined by the decoder.
[0017] This aspect may include one or more of the following optional features: In some implementations, the bias encoder is configured to encode a corresponding bias context vector for each bias phrase in the set of bias phrases, and the bias attention module is configured to calculate attention for the bias context vector. In these implementations, the bias attention module may be configured to receive an additional bias context vector that does not correspond to any of the bias phrases in the set of bias phrases, where the additional bias context vector represents an option to not bias the likelihood of the sequence of speech elements determined by the decoder toward any of the bias phrases. These implementations may additionally or alternatively include a bias encoder and a bias attention module configured to operate with a variable number of bias phrases in the set of bias phrases that are not specified during training of the speech recognition model.
[0018] A phonetic element may include a word, a subword, or a grapheme. A set of bias phrases may include a set of contact names, or a set of media item names, personalized for a particular user.
[0019] In some examples, the first encoder, the first attention module, the biased encoder, the biased attention module, and the decoder are jointly trained to predict a sequence of graphemes from a sequence of acoustic feature frames. In some configurations, the first encoder includes a stacked recurrent neural network (RNN) and / or the decoder includes a stacked unidirectional RNN configured to calculate probabilities of a sequence of output tokens.
[0020] In some implementations, the first attention module is configured to calculate attention as a function of a previous hidden state of the decoder and the complete sequence of context vectors output by the first encoder for the utterance, and the bias attention module is configured to calculate attention as a function of a previous hidden state of the decoder and the complete sequence of context vectors representing bias phrases in the set of bias phrases. In additional implementations, the decoder is configured to determine the hidden state based on an embedding vector for a previous grapheme output from the speech recognition model, a previous hidden state of the decoder, the audio context vector output by the first attention module, and the bias context vector output by the bias attention module.
[0021] In some examples, the operations also include obtaining a list of bias prefixes corresponding to bias phrases in the set of bias phrases, and using the speech recognition model to determine whether the partial transcript for the utterance includes one of the bias prefixes in the list of bias prefixes, where each bias prefix in the list of bias prefixes represents an initial portion of one or more of the bias phrases in the set of bias phrases. In these examples, if the partial transcript for the utterance includes one of the bias prefixes, the operations also include identifying a subset of one or more bias phrases in the set of bias phrases that includes one of the bias prefixes included in the partial transcript, and enabling only the identified subset of the one or more bias phrases to bias the speech recognition model. In another example, the operations include obtaining a list of bias prefixes corresponding to bias phrases in the set of bias phrases, where each bias prefix in the list of bias prefixes represents an initial portion of one or more of the bias phrases in the set of bias phrases, and adjusting a bias attention probability of a bias attention module for encoding an output of a bias encoder for bias phrases in the set of bias phrases based on whether one or more of the bias prefixes in the list of bias prefixes are included in the partial transcription for the utterance. At least one bias prefix in the list of bias prefixes includes a first word of a corresponding bias phrase in the set of bias phrases and excludes a last word of the corresponding bias phrase.
[0022] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]
[0023] [Figure 1]1 is a schematic diagram of an exemplary speech recognizer incorporating contextual bias to improve speech recognition; [Diagram 2] FIG. 1 is a schematic diagram of an example architecture of a speech recognition model for context bias. [Figure 3A] FIG. 1 is a schematic diagram illustrating various techniques for on-the-fly rescoring of context bias phrases. [Figure 3B] FIG. 1 is a schematic diagram illustrating various techniques for on-the-fly rescoring of context bias phrases. [Figure 3C] FIG. 1 is a schematic diagram illustrating various techniques for on-the-fly rescoring of context bias phrases. [Figure 4] 1 is a flowchart of an exemplary arrangement of operations for a method of incorporating contextual bias information into a speech recognition model. [Diagram 5] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0024] Like reference symbols in the various drawings indicate like elements.
[0025] As speech technology becomes more and more prevalent, speech has emerged as one of the primary input modalities in mobile devices and intelligent personal assistants. The accuracy of an automatic speech recognition (ASR) model can be improved if the ASR model incorporates information about the current context in which the utterance is being spoken into the recognition process. Examples of such context include the state of the dialog (e.g., when an alarm is ringing, one may want to make "stop" or "cancel" more likely), the location of the speaker (which may make it more likely to be a nearby restaurant or place), as well as personalized information about the user, such as the user's contacts or song playlist. In particular, providing the ASR model with phrases known to be relevant to the current context can greatly improve the accuracy of recognizing uncommon words, or words that were not included within the training examples. Implementations herein are directed to incorporating context bias functionality into an end-to-end (E2E) speech recognition model that combines the functionality traditionally performed by each of the following separately trained models, namely, the acoustic model (AM), the language model (LM), and the pronunciation model (PM), into a single model. Specifically, an E2E speech recognition model can receive context phrases represented as a set of word n-grams during inference (e.g., use of the model to predict speech content), some of which may contain out-of-vocabulary terms that are not included in the training examples used to train the model.
[0026] In traditional ASR systems, one of the main paradigms for incorporating such information involves the use of an independently trained on-the-fly (OTF) rescoring framework that dynamically adjusts the LM weights of a small number of n-grams that are relevant to a particular recognition context. Extending such techniques to sequence-to-sequence models is important for improving system performance and is an area of active research. While some approaches apply the use of an external, independently trained LM to the sequence-to-sequence model for OTF rescoring, much of the benefit gained from the joint optimization of the components of the sequence-to-sequence model is lost.
[0027] An E2E speech recognition model incorporating contextual bias may include a contextual listening, attending, and spelling (CLAS) model that does not use any external, independently trained model for OTF rescoring, but instead generates a context embedding for each context phrase given to the model. For example, a CLAS model may include a biased encoder that receives as input each context phrase represented as a sequence of graphemes and generates as output a corresponding fixed-dimensional representation for the context phrase separately as a biased embedding. The CLAs model may then use a biased attention mechanism to summarize the available context at each step of the model's output prediction. The biased attention mechanism operates on the complete sequence of biased embeddings representing the relevant context phrases. In this way, the number of context phrases, the length of the context phrases, and the content of the context phrases can change dynamically over time and do not need to be set when the model is trained. This technique allows for easy incorporation of out-of-vocabulary terms using a variable number of context phrases during inference. This technique does not require specific context information to be available at training time or careful tuning of the rescoring weights.
[0028] Examples of types of context that can be used to select context phrases for bias include the dialog state of an application, or a conversation with a digital assistant. Other signals or factors can similarly provide context indicating which terms may be most relevant. For example, when an alarm is ringing, the words "stop" or "cancel" may be more likely. Another exemplary context is the location of the speaker, which may make the name of a nearby store, restaurant, or other place more likely. Additional context information can be personalized information about the user, such as the user's contacts or song playlists.
[0029] Referring to FIG. 1, an exemplary ASR system 100 includes a speech recognition model 200 that incorporates a contextual bias to bias the speech recognition process toward certain words and phrases that are relevant to the current context in which the utterance is being spoken. The speech recognition model 200 includes an audio encoder 210, a biased encoder 220, an audio attention module 218, a biased attention module 228, and a decoder 240 that receives as input the outputs of both attention modules 218, 228. The output of the decoder 240 is further normalized using a softmax layer 242 that may provide a probability distribution over a set of linguistic units, such as a set of graphemes. The output of the softmax layer 242 may be a vector having probability values for each of the graphemes in the language, as well as potentially other symbols (e.g., punctuation, spaces, etc.). The sequence of vectors from the softmax layer 242 is used to generate a transcription 150 of the speech utterance 104 recorded by the device 106. For example, the beam search module 243 may determine the sequence of graphemes that make up the transcription 150. In some examples, the speech recognition model 200 resides on a user device 106 associated with the user 102. In other examples, the speech recognition model 200 resides on a remote server that communicates with the user device 106, or the functionality of the speech recognition model 200 is split between the remote server and the user device 106.
[0030] In the illustrated example, a user 102 speaks an utterance 104 that is captured by one or more microphones of a user device 106. The user device 106 may include a mobile device such as a smartphone, a tablet, smart headphones, a smart watch, etc. The utterance 104 spoken by the user 102 may be a command such as "turn off the alarm," in which case the one or more microphones generate an audio signal 108 from the spoken command. The audio signal 108 may be processed in any of a variety of ways to determine inputs for neural network elements of the speech recognition model 200. For example, a feature extraction module 109 may generate acoustic features 110 from the audio signal, such as mel-frequency cepstral components for different windows or frames of the audio signal 108.
[0031] The speech recognition model includes two different encoders / attention paths / branches that operate on different types of input. For example, the audio encoder 210 and the audio attention module 218 process acoustic information, while the bias encoder 220 and the bias attention module 228 process information about context-relevant terms (e.g., words or phrases). More specifically, the audio encoder 210 receives acoustic features 110 extracted by the feature extraction module 109 that represent acoustic characteristics of the utterance 104, and the audio encoder 220 receives inputs that represent various bias phrases.
[0032] In some implementations, the user device obtains context data 111 indicative of information about a current context, where the context data can indicate factors such as the physical state of the user device 106, the operating state of the user device 106, recent inputs or tasks of the user 102, etc. In one example, the context data 111 indicates the location of the user device 106, applications installed and / or running on the user device 106, and upcoming calendar events of the user 102. For example, the context data 111 can include an open clock app and a time indicating that it is early in the morning. The context data 111 can indicate that the device 106 is at the user's home and that the upcoming calendar event includes a meeting that occurs in an hour. The context data 111 in this example indicates that the user is likely at home early in the morning and waking up to an alarm from the clock app in order to get to the meeting on time.
[0033] From the context data 111, the user device 106 selects a set of contextual bias phrases 114 that are predicted to be relevant to the current context. These phrases 114 may be selected by a bias phrase selector 113, which interprets the context data 111 and compiles a list of phrases. The bias phrase selector 113 may determine a classification or category for the current context, such as selecting whether the context data 111 is most likely to represent playing media, initiating a phone call, dictating a message, issuing a voice command, etc. From these classifications, or directly from the context data 111, the bias phrase selector 113 compiles a list of phrases 114 that are likely to be uttered. In the illustrated example, the bias phrase selector 113 determines that a voice command is likely, so the bias phrase selector 113 provides a set of contextual bias phrases 114, 114a-n that include commands such as "turn on the light," "stop the alarm," etc.
[0034] 1, the speech recognition model 200 receives a set of contextual bias phrases 114 as context information. The speech recognition model 200 allows for the reception and processing of a variable number of contextual bias phrases 114 of variable length. Here, the bias encoder 220 receives each bias phrase 114 in the set as a sequence of corresponding graphemes 116 (e.g., grapheme data) and generates, for each bias phrase 114, a corresponding fixed-dimensional representation called a bias vector 222 or bias embedding. That is, the bias encoder 220 generates each bias vector 222 one by one based on the sequence of graphemes 116 in the corresponding bias phrase 114.
[0035] When a large number of bias phrases 114 are available, selecting meaningful bias information can be difficult. In some examples, to improve the effectiveness of the contextual bias function, the bias conditioning module 115 may selectively activate bias phrases so that only a proper subset of the bias phrases 114 are active at a given time. In this technique, a bias phrase 114 may be enabled only when an associated bias prefix is detected on a partially decoded hypothesis / transcription on a beam search beam. The bias conditioning module 115 may be part of the speech recognition model 200 or a separate component.
[0036] The bias conditioning module 115 may use partial, intermediate, or preliminary transcription information about the utterance 104 to effectively filter out bias phrases 114 that are not meaningful at the current time step, thereby limiting the possibility of distraction or confusion in paying attention to the bias vector 222 in the bias attention module 228. In some examples, for each bias phrase 114, the bias phrase selector 113 provides the bias conditioning module 115 with a corresponding bias prefix 114F that represents the first part of the bias phrase 114. Thus, the bias conditioning module 115 may compare each bias prefix 114F with the current partial transcription 151 and determine whether the current partial transcription 151 includes the bias prefix 114F. If the partial transcription 151 includes the bias prefix 114F, the bias conditioning module 115 may activate / enable the corresponding bias phrase 114. For example, in a given example for the bias phrase 114 "stop the alarm," the bias phrase selector 113 may provide the corresponding bias prefix 114F of "stop" (or "stop the") to the bias conditioning module 115 to condition the activation / enablement of the bias vector 122 for the phrase 114 if the current partial transcription 151 includes the bias prefix 114F.
[0037] The bias conditioning module 115 may also invalidate or penalize bias phrases 114 that correspond to bias prefixes 114F that are not included in the current partial transcription 151. The invalidated or penalized bias phrases 114 may be adjusted by indicating a reduced probability for the bias phrase 114, or by providing an unbiased input 117 that changes the bias vector 222 for the phrase 114 or causes the bias encoder 220 to encode the bias vector 222 for the phrase 114 differently. Other techniques, such as removing the phrase 114 from the set of bias phrases 114 or removing the bias vector 222 from the set computed by the bias attention module 228, may also be used.
[0038] 1 , the bias attention module 228 processes the bias vector 222 output from the bias encoder 220 and the decoder context state 238 from the previous time step to generate the bias attention vector 232. The decoder context state 238 includes the state of the decoder 240 at the previous time step with respect to the previous bias attention vector 232. The decoder context state 238 may represent a portion of the output of the decoder 240, such as a designated subset of the output of the decoder 140. The decoder context state 238 informs the context bias attention module 228 of the relative importance of the bias vector 222 input to the bias attention module 228 at the current time step. Thus, as the decoder context state 238 changes, the bias attention module 228 computes a different summary or aggregation of the bias vector 222, thus changing the probabilities that the decoder exhibits for different bias phrase 114 elements.
[0039] Without reference to the acoustic processing branch of the speech recognition model 200, the audio vectors 212 output by the audio encoder 210 are provided to an audio attention module 218 that generates an audio attention vector 230. The audio attention vector 230 provides a weighted summary of the audio frames that the attention module 218 is currently paying attention to, e.g., a weighted summary of the sequence of audio vectors 212 previous to the utterance 104 so far. The audio encoder 210 can generate an audio vector 212 for each window or frame of the audio data 108 as additional audio features are received, thus continuing to provide additional audio vectors 212 as more audio is received. In some examples, the audio attention module 218 uses the decoder audio state 236 from a previous time step and the encoded audio vectors 212 to calculate attention weights over the sequence of audio vectors 212. The decoder audio state 236 may represent a portion of the state of the decoder 240 when outputting the latest decoder output, e.g., a portion of the decoder state of a previous time step that represents the previous window or frame of audio. The decoder state serves to inform the audio attention module 218 of the relative importance of the audio vectors 212 input to the audio attention module 218 at the current time step. The decoder audio state 336 can be a portion of the output of the decoder 340, such as a designated subset of the outputs of the decoder 240, typically a different set of outputs than those used in the decoder context state 338. In some implementations, the state vectors 236, 238 represent non-overlapping portions of the outputs of the decoder 140. In other implementations, the state vectors 236, 238 include one or more overlapping sections, or both state vectors 236, 238 each represent the entire output of the decoder 240.
[0040] The outputs of both attention modules 218, 228 are provided to the decoder 240. For example, the audio attention vector 230 and the context attention vector 232 may be concatenated to generate a composite vector 234 that serves as an input to the decoder 230. This concatenation injects a bias into the speech recognition model 200, and the decoder 240 processes the bias information together with the acoustic information. The decoder 240 may be a sequence model such as a long short-term memory (LSTM) neural network. As discussed above, state vectors 236, 238 representing respective portions of the state of the decoder 240 are fed back to the audio attention module 218 and the context bias attention module 228 to be used in computing the next attention vectors 230, 232. The output of the decoder 240 is normalized in a softmax layer 242 to generate a probability distribution over a set of output targets, which are graphemes in the illustrated example. In other examples, the set of output targets in the probability distribution may include subwords or phonemes. The various output vectors from the decoder 240 are provided to a beam search module 243 which uses beam searching to generate a transcription 150 for the utterance 104. The decision made about which grapheme represents the audio at the current time step is fed back to the decoder 140 as the grapheme output 244 and used to calculate the next decoder output 241.
[0041] The user device 106 can use the transcription 150 output from the beam search module 243 in any of a variety of ways. For example, depending on how the user 102 invoked the speech recognition function, the user device 106 may identify and execute commands specified by the transcription 150, display the transcription 150, provide the transcription 150 to another system (e.g., a server system on a network, etc.), provide the transcription to an application on the mobile device 106, etc.
[0042] 1, contextual information of the user's location, open apps, and upcoming calendar events biases the speech recognition model 200 toward the bias phrase 114, "stop the alarm," the same phrase spoken by the user in the utterance 104. Thus, the user device 106 (or, alternatively, the server system, if a server system runs the speech recognition model 200) correctly transcribes the user's command as "stop the alarm." The user device 106 can then perform an action based on this transcription, such as turning off the alarm in an alarm clock app.
[0043] FIG. 2 illustrates an exemplary contextual listen, attend, and spell (CLAS) model architecture for a speech recognition model 200 incorporating contextual bias. The CLAS model architecture 200 is an all-neural mechanism that can exploit contextual information provided as a list of contextual bias phrases 114 (FIG. 1) to improve recognition performance. The technique consists of first embedding each phrase, represented as a sequence of graphemes 116, into a fixed-dimensional representation 222, and then using an attention model 228 to summarize the available context 232 at each step of the model's output prediction. This approach allows for a variable number of contextual phrases during inference. Furthermore, the technique disclosed herein does not require specific contextual information to be available at training time, and importantly, unlike some previous systems, the method does not require careful tuning of rescoring weights, while still being able to incorporate out-of-vocabulary (OOV) terms.
[0044] Experimental evaluations show that the CLAS model 200, which trains the context component jointly with the rest of the model, significantly outperforms online rescoring techniques when dealing with hundreds of context phrases, and is comparable to these techniques when dealing with thousands of phrases. The standard LAS model is described along with the standard context approach. Then, modifications to the LAS model that are proposed to obtain the CLAS model are described.
[0045] The LAS architecture includes three main components: an encoder 210, a decoder 240, and an attention network 218. The three main components are jointly trained to predict a sequence of graphemes 244 from a sequence of acoustic feature frames 110. The encoder 210 receives acoustic features 110, x=(x1,...,x K ) and obtain a sequence of 212 high-level features (hidden states),
[0046]
number
[0047] The encoder 210 includes a layered recurrent neural network (RNN) (unidirectional in this work) that outputs: The encoder 210 is similar to an acoustic model in an ASR system.
[0048] The decoder 220 computes the probability y=(y1,...,y T ) is a stacked unidirectional RNN that calculates
[0049]
number
[0050] Encoder state vector 212, h x The conditional dependence on the current decoder hidden state d t and the complete encoder state sequence hx The context vector 230, c, calculated using multi-head attention as a function of t = c x t is modeled using. The hidden state d of the decoder that captures the previous character context y<t t is
[0051]
Number
[0052] given by, where d t-1 is the previous hidden state of the decoder,
[0053]
Number
[0054] is the embedding vector for y t-1 . The posterior distribution of the output 241 at time step t is P(y t |h x , y<t) = softmax(W s [c t ; d t + b s ) (3) given by, where W s and b s are again learnable parameters, and [c t ; d t represents the concatenation of two vectors. The model is trained to minimize the discriminative loss. L LAS = -log P(y|x) (4)
[0055] On-the-fly (OTF) rescoring is a technique for adjusting the weights of n-grams associated with a particular context. This technique can be used in the LAS or CLAS models 200 discussed below in each of its variations below. In OTF rescoring, a set of word-level bias phrases is assumed to be known a priori and compiled into a weighted finite state transducer (WFST). The word-level WFST, G, is then left-combined with a "speller" FST, S, that converts sequences of graphemes / subwords into corresponding words. The contextual language model is
[0056]
number
[0057] It is obtained by
[0058] Contextualized Language Model P C The scores from (y) can then be incorporated into the decoding criterion by augmenting the standard log-likelihood term with the scaled contribution from the contextualized language model:
[0059]
number
[0060] Here, λ is a tunable hyperparameter that controls how much the context language model influences the overall model score during beam search.
[0061] 3A-3C show various techniques for OTF rescoring context bias phrases 114 by assigning weights to subword units. FIG. 3A shows an example OTF rescoring where no weight imposition is applied, resulting in the application of a global score using Equation 5 at word boundaries. Thus, this technique improves OTF rescoring performance only when a related word first appears in the beam. Furthermore, while this technique works fairly well when the number of context bias phrases is small (e.g., yes, no, cancel), this technique is inhibited when the set of context phrases includes a large number of proper nouns (e.g., song titles, contacts, etc.).
[0062] 3B illustrates an exemplary OTF rescoring technique that applies weight imposition, which results in the application of an overall score to the first subword unit of each word. Applying the score only to the first subword unit can lead to problems with excessive bias, as words can be artificially boosted early on.
[0063] Figure 3C shows a third OTF rescoring technique that imposes weights on each subword unit of a word. To avoid artificially weighting prefixes that are boosted early but do not match the whole phrase, a subtraction cost is included, as indicated by the negative weights shown in Figure 3C. By imposing weights on each subword unit of a word, the OTF rescoring technique of Figure 3C aims to help keep the word on the beam.
[0064] Referring back to FIG. 2, the CLAS model 200 modifies the LAS model by using additional context through a list of provided bias phrases 114, z, thus effectively modeling P(y|x;z). Individual elements in z represent phrases, such as personalized contact names, song lists, etc., that are relevant to a particular recognition context. In the example shown, the CLAS model 200 modifies the LAS model by using additional context through a list of provided bias phrases, z, such as z=z1,...,z NOne can access a list of additional sequences of bias phrase 114, shown as. Individual elements within z represent phrases such as personalized contact names, lists of songs, etc., related to a particular recognition context. The purpose of bias phrase 114 is to bias the model in the direction of outputting a particular phrase. However, considering the current utterance, not all bias phrases are necessarily relevant, and if any, it depends on the model to determine which phrases might be relevant and use them to modify the target distribution P(y t |h x ,y<t).
[0065] Bias encoder 220 embeds bias phrase 114 (z = z1,...,z N ) into a set of vectors 222, h z ={h0 z ,h1 z ,...,h n z}. The superscript z is used to distinguish between bias attention variables and audio-related variables. Vector h i z is the embedding of z i when i > 0. Since bias phrase 114 may not be relevant to the current utterance, an additional learnable vector h z 0=h z nb is added. The learnable vector h z 0 corresponds to the unbiased input 117 (Figure 1) and does not use any of the bias phrases to generate the output. This option allows the model to revert to an "unbiased" decoding approach when none of the bias phrases match the audio and enables the model to completely ignore the bias phrases.
[0066] In some examples, bias encoder 220 is the same writing element or subword unit used by the decoder to obtain the embedding by using the last state of the LSTM as the embedding for the entire phrase, which is zi The method includes receiving a sequence of embeddings of subwords in a multi-layer LSTM network configured to receive ...
[0067] The biased attention module 228 is z We compute the biased attention over t. The second-order context vector summarizes z at time step t as
[0068]
number
[0069] The biased attention module 228 calculates the concatenated context vector 234 (FIG. 1), c t to provide the audio context vector 230, c t x The bias context vector 232 concatenated with c t z Outputs c t =[c t x ;c t z ] is the concatenation of the context vectors obtained for x and z. In some implementations, other components of the CLAS module 200 that are not related to context bias (e.g., the decoder and audio encoder) are identical to the corresponding components in a standard LAS model, but these components will have significantly different training states as a result of joint training with the context bias element.
[0070] Thus, the CLAS model 200 explicitly models the probability of seeing a particular bias phrase 114 given the audio features 110 and previous outputs 241, 244, where the probability
[0071]
number
[0072] corresponds to the biased attention probability, which can be expressed as:
[0073]
number
[0074] Furthermore, the CLAS model 200 may be trained to minimize the loss based on the following formula: L CLAS =-logP(y|x,z) (Eq. 9)
[0075] 1 and 2, since the model 200 does not make any assumptions about what bias phrases 114 will be used during inference, during training of each batch, the CLAS model 200 randomly generates a list of bias phrases z to allow for flexibility during inference. In some examples, the CLAS model 200 randomly generates a list of bias phrases z for the reference transcripts r1,...,r2 corresponding to the audio features 110 in the training batch. Nbatch Randomly create / generate a list of training bias phrases 114 from the reference transcripts associated with the utterances in the training batch by taking a list of z, and randomly select a list of bias phrases 114, z of n-gram phrases 116 that appear as substrings in some of the reference transcripts.
[0076] In some examples, to train the unbiased option, training a CLAS model 200 where a particular bias phrase 114, z, does not match some of the utterances in the batch has probability P keep This involves discarding / excluding each reference from the production process using P. When a reference is discarded, the utterance continues to be kept in the batch, but the bias phrases are not extracted from the transcript. keep P keep If set to 0, then no unbiased phrases are presented in the training batch.keep If = 1, then each utterance in the batch has at least one matching bias phrase. Then, from each retained reference, the training process calculates the number of matching bias phrases in [1,N phrases ] and randomly select k uniformly from [1,N order ], where P keep , N phrases , and N order are the hyperparameters of the training process. For example, P keep = 1.0, N phrases =1, N order If =1, one unigram is selected from each reference transcript. In one example, P keep =0.5, N phrases = 1, and N order Setting = 4 promotes robustness to unbiasedness and leads to a list of bias phrases 214 with an expected size of 17 (e.g., half the shard size plus 1 for unbiasedness).
[0077] In some implementations, the training process randomly generates a list of bias phrases 114, z, and then computes the intersection of z with each reference transcript r, such that the training process inserts a special symbol every time a match is found. If the reference transcript is "play a song" and the matching bias phrase is "play", the target sequence is changed to "play a song". The purpose of the symbols is to introduce training errors that can only be corrected by considering the correct bias phrases. To be able to predict , the model 200 must take note of the correct bias phrases and therefore must ensure that the bias encoder 220 receives updates during the training process.
[0078] During inference, when user 102 provides the model 200 with a sequence of audio feature vectors 110, x, and a set of context bias phrase sequences z that are likely not seen during training, the model 200 uses the bias encoder 220 to embed z into h z This embedding can be done before audio streaming begins. The sequence of audio feature vectors 110, x can be represented as a sequence of frames fed to the audio encoder 210, and the decoder 240 is executed to generate N-best hypotheses using beam search decoding (e.g., in the beam search module 243 of FIG. 1) to generate the transcript 150.
[0079] As described above, the bias context vector 232 corresponds to a weighted sum of many different bias embeddings z, and thus may be far from any context vector seen during training. So, if user 102 presents the CLAS model 200 with thousands of phrases 114, it may be difficult to search for a meaningful bias context vector 232, cz. To mitigate this problem, bias conditioning provides the model 200 with a list of bias phrases z = z1,..., z N and a list of bias prefixes 114F p = p1,..., p N both during inference. Here, bias conditioning enables the bias phrase z i only when the corresponding bias prefix p i is detected in a partially decoded hypothesis (e.g., partial transcript 151) on the beam y<t. This is achieved by updating the bias attention probability by setting
[0080]
Number
[0081] where ⊆ is string inclusion.
[0082] Bias conditioning can arbitrarily construct a list of bias prefixes 114F. For example, a bias phrase such as "the cat sat" can be conditioned on the bias prefix 114F of "the cat". Here, in bias conditioning, when the model 200 detects the bias prefix 114F of "the cat" in the partial transcript 151 on the beam y<t, only the embedding z for the bias phrase 114 of "the cat sat" is enabled. A good selection of the bias prefix 114F reduces the overload of bias attention in the bias attention module 228 by minimizing the number of bias phrases 114 sharing the same prefix, and at the same time enables characteristic bias embeddings z by preventing the splitting of each bias phrase 114 into too many segments. In some implementations, the algorithm starts from an empty prefix (p i =ε) and selects the bias prefix 114F by repeatedly expanding each prefix by only one word (from i z).
[0083] An exemplary training setting uses a training set of approximately 25,000 hours consisting of 33 million English utterances. The training utterances are anonymized and handwritten transcripts. This dataset can be augmented by artificially corrupting clean utterances using a room simulator and adding various degrees of noise and reverberation such that the overall SNR is between 0 dB and 30 dB and the average SNR is 12 dB.
[0084] The model is trained on 8x8 tensor processing unit (TPU) slices with a global batch size of 4096. Each training core works on a shard size of 32 utterances at each training step. From this shard, the bias phrases are randomized, so each shard sees a maximum of 32 bias phrases during training.
[0085] The exemplary training process uses 80-dimensional log-mel acoustic features 100 computed every 10 ms over a 25 ms window. Three consecutive frames are stacked and strided by a factor of 3. This downsampling allows the use of a simpler encoder architecture.
[0086] The encoder 210 architecture may include 10 unidirectional LSTM layers, each with 256 nodes. In some examples, the encoder attention module 218 uses four attention heads to compute attention across 512 dimensions. In addition, the biased encoder 220 may include a single LSTM layer with 512 nodes, and the biased attention module 228 may compute attention across 512 dimensions. The decoder 240 may include four LSTM layers with 256 nodes. In total, the model 200 may have approximately 58 million trainable parameters.
[0087] In a test set of utterances with hundreds of bias phrases 114 associated with a high rate of out-of-vocabulary (OOV) words, the CLAS model 200 functions significantly better than a non-contextual ASR model. However, for test sets with thousands of bias phrases 114, the CLAS model 200 degrades as correlations begin to appear between the respective embeddings z. This scalability problem can be addressed using bias conditioning by providing a list of bias prefixes 114F such that the CLAS model 200 "activates" the bias phrases 114 only if the model 200 detects the corresponding prefix 114F in a partially decoded hypothesis on the beam y<t (e.g., partial transcript 151).
[0088] Tests that included the use of a class CLAS model with bias conditioning created bias prefixes in a rule-based manner. For the "talk-to-chatbot" set, the prefixes were generated as "talk to" + the next word (e.g., the phrase "talk to pharmacy flashcards" was split into prefix p i = "talk to pharmacy" and suffix z i = "flashcards"). Additionally, it was useful to condition the first word after "talk to" on its first letter (e.g., "pharmacy" was conditioned in "talk to p"). This structure limits the number of phrases sharing the same prefix to 255 (vs. 3255) while increasing the total number of bias phrase segments by only 10 percent (10%).
[0089] CLAS model 200 benefits from both bias conditioning and OTF rescoring approaches, as well as their combination, by allowing scalability to large numbers of phrases without any loss of accuracy. Without these features, the benefits of contextual bias may be lost if a very large number of bias phrases are used. Nevertheless, conditioning and / or OTF rescoring can allow a large set of bias phrases, e.g., thousands or tens of thousands, to be used without loss of accuracy.
[0090] Thus, the CLAS model 200 includes an all-neural contextualized ASR model that incorporates context information by embedding full context bias phrases 114. The CLAS model 200 may use bias conditioning to enable it to handle a large set of context phrases. The model 200 outperforms standard bias techniques by sixty-eight percent (68%) in relative WER.
[0091] FIG. 4 is a flowchart of an exemplary arrangement of operations for a method 400 of incorporating contextual bias information into a speech recognition model 200 to bias speech recognition toward predefined words and phrases related to a current context in which an utterance is being spoken. The method 400 may be described with reference to FIGS. 1 and 2. A user device 106 (and / or a remote server) may perform operations for the method 400. At operation 402, the method 400 includes receiving audio data 108 encoding an utterance 104. For example, one or more microphones of the user device 106 associated with the user 102 may record and encode the utterance 104 spoken by the user 102. At operation 404, the method 400 includes obtaining a set of bias phrases 114 corresponding to the context of the utterance 104. Each bias phrase 114 in the set of bias phrases includes one or more words.
[0092] At operation 406, the method 400 includes processing the acoustic features 110 derived from the audio data 108 using a speech recognition model 200 to generate an output 241 from the speech recognition model 200. The speech recognition model 200 includes a first encoder 210, a first attention module 218 corresponding to the first encoder 210, a bias encoder 220, a bias attention module 228 corresponding to the bias encoder 220, and a decoder 240 configured to determine a likelihood of a sequence of speech elements 224 based on an output of the first attention module 218 and an output of the bias attention module 228. The first encoder 210 is configured to receive the acoustic features 110 derived from the audio data 108, and the bias encoder is configured to receive data 116 indicative of a set of obtained bias phrases 114.
[0093] At operation 408, the method 400 includes determining a transcript 150 for the utterance 104 based on the likelihood of the sequence of phonetic elements 244 determined by the decoder 240. The phonetic elements 244 may include words, subwords, or graphemes. Further, the first encoder 210, the first attention module 218, the biased encoder 220, the biased attention module 228, and the decoder 240 are jointly trained to predict a sequence of graphemes from a sequence of acoustic feature frames.
[0094] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," an "app," or a "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.
[0095] Non-transient memory may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by a computing device. Non-transient memory may be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disks or tapes.
[0096] 5 is a schematic diagram of an exemplary computing device 500 that may be used to implement the systems and methods described in this document. The computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit the implementation of the invention described and / or claimed in this document.
[0097] Computing device 500 includes a processor 510, a memory 520, a storage device 530, a high-speed interface / controller 540 that connects to memory 520 and a high-speed expansion port 550, and a low-speed interface / controller 560 that connects to a low-speed bus 570 and storage device 530. Each of components 510, 520, 530, 540, 550, and 560 are interconnected using various buses and may be mounted on a common motherboard or otherwise as desired. Processor 510 can process instructions for execution within computing device 500, including instructions stored in memory 520 or on storage device 530, to display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 580 coupled to high-speed interface 540. In other implementations, multiple processors and / or multiple buses may be used as desired, along with multiple memories and types of memories. Also, multiple computing devices 500 may be connected (eg, as a server bank, a group of blade servers, or a multi-processor system), with each device providing a portion of the required operations.
[0098] The memory 520 stores information non-transiently within the computing device 500. The memory 520 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-transient memory 520 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disks or tapes.
[0099] The storage device 530 can provide mass storage for the computing device 500. In some implementations, the storage device 530 is a computer-readable medium. In various different implementations, the storage device 530 can be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices including devices in a storage area network or other configuration. In additional implementations, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as the methods described above. The information carrier is a computer-readable medium or a machine-readable medium, such as the memory 520, the storage device 530, or a memory on the processor 510.
[0100] The high-speed controller 540 manages bandwidth-intensive operations for the computing device 500, and the low-speed controller 560 manages less bandwidth-intensive operations. Such role assignments are merely exemplary. In some implementations, the high-speed controller 540 is coupled to the memory 520, the display 580 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 550 that may accept various expansion cards (not shown). In some implementations, the low-speed controller 560 is coupled to the storage device 530 and the low-speed expansion port 590. The low-speed expansion port 590, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device, such as a switch or a router, for example, via a network adapter.
[0101] The computing device 500 may be implemented in a number of different forms, as shown in the figure: for example, it may be implemented as a standard server 500a, or multiple times within a group of servers 500a, as a laptop computer 500b, or as part of a rack server system 500c.
[0102] Various implementations of the systems and techniques described herein may be realized in digital electrical and / or optical circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transmit data and instructions to the storage system, at least one input device, and at least one output device.
[0103] These computer programs (also known as programs, software, software applications, or codes) include machine language for programmable processors and can be implemented in high-level procedural and / or object-oriented languages, and / or assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.
[0104] The processes and logic flows described herein can be executed by one or more programmable processors, also referred to as data processing hardware, which execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be executed by special purpose logic circuitry, for example, an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Processors suitable for executing computer programs include, by way of example, both general purpose and special purpose microprocessors, as well as any one or more processors of any kind of digital computer. In general, a processor receives instructions and data from a read-only memory, or a random access memory, or both. The essential elements of a computer are a processor for executing instructions, and one or more memories for storing instructions and data. In general, a computer also includes one or more mass storage devices, for example, magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively coupled to receive data from or transmit data to, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.
[0105] To provide for user interaction, one or more aspects of the present disclosure can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, for displaying information to the user, and optionally a keyboard and pointing device, e.g., a mouse or trackball, by which the user can provide input to the computer. Other types of devices can be used to provide for user interaction as well, and the feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and the input from the user can be received in any form, including acoustic input, speech input, or tactile input. In addition, the computer can interact with the user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to the web browser in response to a request received from the web browser on the user's client device.
[0106] Although several implementations have been described, it will be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, other implementations are within the scope of the following claims. [Explanation of symbols]
[0107] 100 ASR System 102 users 104 Speech, speech 106 Devices, User Devices 108 Audio signal, audio data 109 Feature Extraction Module 110 Acoustic features, Acoustic feature frames, Audio features, Audio feature vectors 111 Context Data 113 Bias Phrase Selector 114 Contextual Bias Phrases, Phrases, Bias Phrases, Training Bias Phrases 114F Bias prefix, prefix 115 Bias Conditioning Module 116 graphemes, n-gram phrases, data 117 No bias input 140 Decoder 150 Transcription, Transcript 151 Partial Transcription, Partial Transcript 200 Speech Recognition Model, CLAS Model Architecture, CLAS Model, CLAS Module, Model 210 Audio encoder, encoder, first encoder 212 Audio vector, Encoded audio vector, High level features (hidden states), Encoder state vector 218 Audio Attention Module, Attention Module, First Attention Module 220 Bias Encoder, Audio Encoder, Decoder 222 Bias Vector, Fixed Dimensional Representation, Vector 228 Biased Attention Module, Attention Module 230 Audio Attention Vector, Attention Vector, Context Vector, Audio Context Vector 232 Bias Attention Vector, Context Attention Vector, Attention Vector, Context, Bias Context Vector 234 Composite Vectors, Concatenated Context Vectors 236 Decoder Audio State, Decoder State, State Vector 238 Decoder context state, state vector 240 Decoder 241 Decoder output, output 242 Softmax Layer 243 Beam Search Module 244 grapheme output, grapheme, output, phonetic element 500 computing devices 500a Server 500b Laptop Computer 500c Rack Server System 510 Processor, components 520 Memory, components 530 Storage devices, components 540 High Speed Interface / Controller, Components, High Speed Interface, High Speed Controller 550 High Speed Expansion Port, Components 560 Low Speed Interface / Controller, Components, Low Speed Controller 570 Slow Bus 580 Display 590 Low Speed Expansion Port
Claims
1. When executed on data processing hardware, it causes the data processing hardware to: receiving audio data encoding speech; an operation of obtaining a set of bias phrases corresponding to a context of an utterance, each bias phrase in the set of bias phrases including one or more words; an act of using a speech recognition model to determine that a partial transcript for an utterance includes a bias prefix, the bias prefix representing an initial portion of one or more of the bias phrases in the set of bias phrases; identifying a subset of one or more bias phrases within a set of bias phrases that include the bias prefix included in the partial transcript; enabling only the identified subset of the one or more bias phrases to bias the speech recognition model; and processing acoustic features derived from the audio data to generate an output from the speech recognition model using the speech recognition model biased by an identified subset of the one or more bias phrases that are enabled, the speech recognition model comprising: an audio encoder configured to receive the acoustic features and to output encoded audio vectors from the acoustic features; a bias encoder configured to receive data indicative of the obtained set of bias phrases and to output, for each bias phrase in the set of bias phrases, a corresponding bias embedding; a bias attention module configured to receive the corresponding bias embedding output from the bias encoder for each bias phrase in the set of bias phrases; a decoder configured to determine a likelihood of a sequence of speech elements based on the audio vectors output from the audio encoder and an output of the biased attention module; and determining a transcript for the utterance based on the output from the speech recognition model; A computer-implemented method for performing operations including:
2. The method of claim 1 , wherein each word of the one or more words of each bias phrase in the set of bias phrases is represented by a sequence of subword units.
3. The method of claim 1 , wherein the speech elements include subwords.
4. 2. The method of claim 1 , wherein the biased attention module is further configured to receive each corresponding biased embedding output from the biased encoder and a decoder context state from a previous time step indicative of a sequence of non-blank symbols output by the decoder.
5. 5. The method of claim 4, wherein the bias attention module is configured to process the corresponding bias embeddings output from the bias encoder and the decoder context state to generate a corresponding bias context vector for each bias phrase in the set of bias phrases.
6. 2. The method of claim 1, wherein the biased encoder and the biased attention module are configured to operate with a variable number of bias phrases in the set of bias phrases that are not specified during training of a speech recognition model.
7. The method of claim 1 , wherein the audio encoder, the biased encoder, the biased attention module, and the decoder are jointly trained to predict a sequence of subword units from a sequence of acoustic feature frames.
8. The method of claim 1 , wherein the phonetic element is a word, a subword, or a grapheme.
9. The method of claim 1 , wherein the biased encoder comprises a multi-layer long short-term memory (LSTM) network.
10. The method of claim 1 , wherein the set of bias phrases comprises a set of contact names personalized for a particular user.
11. Data processing hardware; a memory hardware in communication with the data processing hardware, the memory hardware, when executed on the data processing hardware, receiving audio data encoding speech; an operation of obtaining a set of bias phrases corresponding to a context of an utterance, each bias phrase in the set of bias phrases including one or more words; an act of using a speech recognition model to determine that a partial transcript for an utterance includes a bias prefix, the bias prefix representing an initial portion of one or more of the bias phrases in the set of bias phrases; identifying a subset of one or more bias phrases within a set of bias phrases that include the bias prefix included in the partial transcript; enabling only the identified subset of the one or more bias phrases to bias the speech recognition model; and processing acoustic features derived from the audio data to generate an output from the speech recognition model using the speech recognition model biased by an identified subset of the one or more bias phrases that are enabled, the speech recognition model comprising: an audio encoder configured to receive the acoustic features and to output encoded audio vectors from the acoustic features; a bias encoder configured to receive data indicative of the obtained set of bias phrases and to output, for each bias phrase in the set of bias phrases, a corresponding bias embedding; a bias attention module configured to receive the corresponding bias embedding output from the bias encoder for each bias phrase in the set of bias phrases; a decoder configured to determine a likelihood of a sequence of speech elements based on the audio vectors output from the audio encoder and an output of the biased attention module; and determining a transcript for the utterance based on the output from the speech recognition model; and a memory hardware for storing instructions for performing operations including: A system comprising:
12. 12. The system of claim 11, wherein each word of the one or more words of each bias phrase in the set of bias phrases is represented by a sequence of subword units.
13. The system of claim 11 , wherein the speech elements include subwords.
14. 12. The system of claim 11, wherein the biased attention module is further configured to receive each corresponding biased embedding output from the biased encoder and a decoder context state from a previous time step indicative of a sequence of non-blank symbols output by the decoder.
15. 15. The system of claim 14, wherein the bias attention module is configured to process the corresponding bias embeddings output from the bias encoder and the decoder context state to generate a corresponding bias context vector for each bias phrase in the set of bias phrases.
16. 12. The system of claim 11, wherein the bias encoder and the bias attention module are configured to operate with a variable number of bias phrases in the set of bias phrases that are not specified during training of a speech recognition model.
17. 12. The system of claim 11, wherein the audio encoder, the biased encoder, the biased attention module, and the decoder are jointly trained to predict a sequence of subword units from a sequence of acoustic feature frames.
18. The system of claim 11 , wherein the phonetic element is a word, a subword, or a grapheme.
19. 12. The system of claim 11, wherein the biased encoder comprises a multi-layer long short-term memory (LSTM) network.
20. 12. The system of claim 11, wherein the set of bias phrases comprises a set of contact names personalized for a particular user.
Citation Information
Patent Citations
Voice and Connection Platform
JP2017535823A
Cited By
Automatic speech recognition
US12658191B1
