Contextual Bias for Speech Recognition

Abstract

Providing contextual bias for speech recognition. A method (500) for biasing speech recognition includes receiving audio data (108) encoding an utterance (104) and obtaining a set of one or more bias phrases (116) corresponding to a context (202) of the utterance. Each bias phrase in the set of one or more bias phrases includes one or more words. The method also includes processing acoustic features (109) obtained from the audio data and grapheme and phoneme data (118, 120) obtained from the set of one or more bias phrases using a speech recognition model (300) to generate a speech recognition model output. The method also includes determining a transcription (146) of the utterance based on the speech recognition model output.

Description

[Technical field]

[0001] This disclosure relates to contextual bias for speech recognition. [Background technology]

[0002] Recognizing the context of speech is the goal of an automatic speech recognition (ASR) system. However, this ability to recognize the context of speech is challenging, given the variety of words people may speak and the many variations in accents and pronunciations. Often, the types of words and phrases a person speaks will vary depending on the context in which they are placed.

[0003] Contextual automatic speech recognition (ASR) involves biasing speech recognition towards a given context, such as towards a user's own playlist, contacts, or geographic place names. The contextual information typically includes a list of relevant phrases to be recognized, which often includes rare phrases or even foreign words rarely seen in training. To perform contextual biasing, traditional ASR systems may model the contextual information in an independent contextual language model (LM) using an N-gram weighted finite state transducer (WFST) and construct the independent contextual LM using a baseline LM for on-the-fly (OTF) rescoring.

[0004] Recently, end-to-end (E2E) models have shown great promise for ASR, showing improved word error rate (WER) and latency metrics compared to traditional on-device models. These E2E models, which collapse the acoustic model (AM), pronunciation model (PM), and LM into a single network to directly learn speech-to-text mapping, have shown competitive results compared to traditional ASR systems with separate AM, PM, and LM. Representative E2E models include word-based connectionist temporal classification (CTC) models, recurrent neural network transducer (RNN-T) models, and attention-based models such as listen, attend, and spell (LAS). Contextual ASR can be challenging for E2E models because they maintain a limited number of recognition candidates during decoding for beam search. Summary of the Invention [Means for solving the problem]

[0005] Implementations herein aim to bias a speech recognition model towards a set of phrases relevant to the current context. The set of phrases is dynamic in that the phrases may change as the context changes. In addition to the written representation of the phrase, the speech recognition model can further take into account the pronunciation of the phrase used for bias. For example, the model can use textual information (e.g., grapheme sequence) and pronunciation information (e.g., phoneme sequence) representing the bias phrase to select the phrase that best suits the speech being recognized. This allows the contextual bias aspect of the speech model to be aware of the pronunciation of individual bias phrases, thereby enabling the model to achieve much higher accuracy in recognizing rare words and words with unusual pronunciations. The contextual bias techniques described herein can be used with various types of speech recognition models, including end-to-end models that can generate transcription data without separate acoustic, pronunciation, and language models.

[0006] To provide high speech recognition accuracy, general purpose automatic speech recognition (ASR) systems may use information indicative of the context in which the speech occurs. The context may relate to the user's contacts, calendar events, open apps, and location, for example. One example ASR system includes separate acoustic, pronunciation, and language models. Other ASR systems combine the acoustic, pronunciation, and language models as a single neural network. The single neural network model improves simplicity and quality, optimizing word error rate (WER).

[0007] For traditional neural network-based ASR models, it is difficult to recognize words that occur rarely in a language. The distribution of words in a language usually follows a Zipf distribution, where a small number of words are used very frequently and a huge number of words are used rarely. It can be difficult to obtain a sufficient amount of training data, including audio and corresponding text, to effectively provide examples of rare words. Even if sufficient training data is available, adding more and more training examples often results in less and less improvement. Furthermore, it is difficult for these models to recognize words that have unusual pronunciations compared to their spellings.

[0008] Incorporating contextual bias into neural network-based ASR models can improve recognition of rare and unusually pronounced words. One useful application is to better recognize unique names (i.e., proper nouns such as people's names, song names, city names, etc.), which may be relatively rare in the training data and / or may not follow common pronunciation rules. For example, smartphones or other user devices often store the user's contacts. When the user is using a messaging or phone calling application, this information can be used as context to assist the ASR system in recognizing the name spoken by the user. Similarly, for example, a list of songs in the user's music library can be used to bias speech recognition when the user is using a media player application. Implementations herein aim to apply contextual bias to an ASR model by injecting information associated with both the written form and pronunciation of the bias phrase into the ASR model. Contextual biasing an ASR model incorporates knowledge about the pronunciation of rare words, even if the word was not observed during training of the ASR model.

[0009] One aspect of the disclosure provides a method for biasing speech recognition that includes receiving, in data processing hardware, audio data encoding an utterance, and obtaining, by the data processing hardware, a set of one or more bias phrases corresponding to a context of the utterance, where each bias phrase in the set of one or more bias phrases includes one or more words. The method also includes processing, by the data processing hardware, acoustic features obtained from the audio data and grapheme and phoneme data obtained from the set of one or more bias phrases using a speech recognition model to generate an output of a speech recognition model. The method also includes determining, by the data processing hardware, a transcription of the utterance based on the output of the speech recognition model.

[0010] Implementations of the present disclosure may include one or more of any of the following features. In some implementations, the speech recognition model includes a first encoder, a first attention module, a grapheme encoder, a phoneme encoder, a second attention module, and a decoder. The first encoder is configured to receive acoustic features as input and generate an audio encoding from the acoustic features as output. The first attention module is configured to receive an audio encoding output from the first encoder as input and generate a first attention output as output. The grapheme encoder is configured to receive grapheme data indicative of graphemes of each word in a set of one or more bias phrases as input and generate a grapheme encoding as output. The phoneme encoder is configured to receive phoneme data indicative of phonemes of each word in a set of one or more bias phrases as input and generate a phoneme encoding as output. The second attention module is configured to receive as input the representations of the grapheme encoding output from the grapheme encoder and the phoneme encoding output from the phoneme encoder and to generate as output a second attention output. The decoder is configured to determine likelihoods of sequences of phonetic elements based on the first attention output and the second attention output.

[0011] In some examples, for each particular word of each bias phrase in the set of one or more bias phrases, the grapheme encoder is configured to generate a grapheme encoding corresponding to the particular word, the phoneme encoder is configured to generate a phoneme encoding corresponding to the particular word, and the second attention module is configured to encode a corresponding second attention output comprising a corresponding context bias vector of the particular word based on the grapheme and phoneme encoding corresponding to the particular word. Further, by way of example, the representation of the grapheme encoding output from the grapheme encoder and the phoneme encoding output from the phoneme encoder includes a projection vector representing a concatenation between the grapheme encoding and the phoneme encoding. Furthermore, the first encoder, the first attention module, the grapheme encoder, the phoneme encoder, the second attention module, and the decoder may be jointly trained to predict a sequence of graphemes from a sequence of acoustic feature frames.

[0012] In some implementations, the first attention module is configured to calculate attention as a function of a previous hidden state of the decoder and a complete sequence of audio encodings output by the first encoder, and the second attention module is configured to calculate attention as a function of a previous hidden state of the decoder and a complete sequence of projection vectors representing characteristics of both grapheme and phoneme data obtained from bias phrases in a set of one or more bias phrases. In additional implementations, the second attention module is configured to receive a contextual bias vector that does not correspond to any of the bias phrases in the set of one or more bias phrases, the contextual bias vector representing an option to not bias the output of the speech recognition model. In some examples, the speech elements include graphemes. In other examples, the speech elements include words or wordpieces. Optionally, the speech elements may include phonemes.

[0013] The set of one or more bias phrases may include one or more contact names personalized for the particular user. Additionally or alternatively, the set of one or more bias phrases may include one or more calendar events personalized for the particular user. In some examples, the method also includes determining, by the data processing hardware, a context of the utterance based on at least one of a location of the user who spoke the utterance, one or more applications open on a user device associated with the user who spoke the utterance, or a current date and / or time of the utterance. The context of the utterance may additionally or alternatively be based on one or more previous commands issued by the user.

[0014] In some implementations, the speech recognition model includes a decoder configured to determine a hidden state and an output of the speech recognition model based on an embedding vector, a previous hidden state of the decoder, a first vector, and a second vector, where the embedding vector is for a previous grapheme output by the speech recognition model, the first vector is output by a first attention module, and the second vector is output by a second attention module.

[0015] Another aspect of the present disclosure provides a system for biasing speech recognition based on a context of an utterance. The system includes data processing hardware and memory hardware in communication with the data processing hardware and storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving audio data encoding the utterance and obtaining a set of one or more bias phrases corresponding to a context of the utterance, each bias phrase in the set of one or more bias phrases including one or more words. The operations also include processing acoustic features obtained from the audio data and grapheme and phoneme data obtained from the set of one or more bias phrases using a speech recognition model to generate an output of the speech recognition model. The operations also include determining a transcription of the utterance based on the output of the speech recognition model.

[0016] This aspect may include one or more of any of the following features. In some implementations, the speech recognition model includes a first encoder, a first attention module, a grapheme encoder, a phoneme encoder, a second attention module, and a decoder. The first encoder is configured to receive acoustic features as input and generate an audio encoding from the acoustic features as output. The first attention module is configured to receive audio encodings output from the first encoder as input and generate a first attention output as output. The grapheme encoder is configured to receive grapheme data indicative of graphemes of each word in a set of one or more bias phrases as input and generate a grapheme encoding as output. The phoneme encoder is configured to receive phoneme data indicative of phonemes of each word in a set of one or more bias phrases as input and generate a phoneme encoding as output. The second attention module is configured to receive representations of the grapheme encodings output from the grapheme encoder and the phoneme encodings output from the phoneme encoder as input and generate a second attention output as output. The decoder is configured to determine likelihoods of sequences of speech elements based on the first attention output and the second attention output.

[0017] In some examples, for each particular word of each bias phrase in the set of one or more bias phrases, the grapheme encoder is configured to generate a grapheme encoding corresponding to the particular word, the phoneme encoder is configured to generate a phoneme encoding corresponding to the particular word, and the second attention module is configured to encode a corresponding second attention output comprising a corresponding context bias vector of the particular word based on the grapheme and phoneme encoding corresponding to the particular word. Further, by way of example, the representation of the grapheme encoding output from the grapheme encoder and the phoneme encoding output from the phoneme encoder includes a projection vector representing a concatenation between the grapheme encoding and the phoneme encoding. Furthermore, the first encoder, the first attention module, the grapheme encoder, the phoneme encoder, the second attention module, and the decoder may be jointly trained to predict a sequence of graphemes from a sequence of acoustic feature frames.

[0018] In some implementations, the first attention module is configured to calculate attention as a function of a previous hidden state of the decoder and a complete sequence of audio encodings output by the first encoder, and the second attention module is configured to calculate attention as a function of a previous hidden state of the decoder and a complete sequence of projection vectors representing characteristics of both grapheme and phoneme data obtained from bias phrases in a set of one or more bias phrases. In additional implementations, the second attention module is configured to receive a contextual bias vector that does not correspond to any of the bias phrases in the set of one or more bias phrases, the contextual bias vector representing an option to not bias the output of the speech recognition model. In some examples, the speech elements include graphemes. In other examples, the speech elements include words or word pieces. Optionally, the speech elements may include phonemes.

[0019] The set of one or more bias phrases may include one or more contact names personalized for the particular user. Additionally or alternatively, the set of one or more bias phrases may include one or more calendar events personalized for the particular user. In some examples, the operations also include determining a context for the utterance based on at least one of a location of the user who spoke the utterance, one or more applications open on a user device associated with the user who spoke the utterance, or a current date and / or time of the utterance. The context of the utterance may additionally or alternatively be based on one or more previous commands issued by the user.

[0020] In some implementations, the speech recognition model includes a decoder configured to determine a hidden state and an output of the speech recognition model based on an embedding vector, a previous hidden state of the decoder, a first vector, and a second vector, where the embedding vector is for a previous grapheme output by the speech recognition model, the first vector is output by a first attention module, and the second vector is output by a second attention module.

[0021] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0022] [Figure 1] FIG. 1 is a schematic diagram of an example automatic speech recognition system including a speech recognition model that biases speech recognition results based on context associated with an utterance. [Diagram 2] FIG. 2 is a schematic diagram of an exemplary selection process for selecting bias phrases. [Diagram 3] FIG. 2 is a schematic diagram of an example architecture of the speech recognition model of FIG. 1. [Figure 4]FIG. 2 is a schematic diagram of an exemplary training process for training the speech recognition model of FIG. [Diagram 5] 1 is a flowchart of an exemplary arrangement of operations for a method of context biasing a speech recognition model. [Figure 6] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] Like reference symbols in the various drawings indicate like elements.

[0024] Traditional automatic speech recognition (ASR) systems rely on three models: the acoustic model, the pronunciation model, and the language model. Design decisions in the pronunciation model affect both the acoustic model and the language model. For the pronunciation model, a set of phonemes is chosen to represent all possible different sound units, determining the set of labels that the acoustic model needs to predict. Every word in the language model has a pronunciation.

[0025] Accurate pronunciation models have wide coverage. In a given variant of a language, the set of phonemes varies from region to region. For example, in American English, native speakers do not agree on whether words like "pen" and "pin" are pronounced the same way. Furthermore, certain words may have truncated pronunciations, where speakers may or may not pronounce the letter "t" in words like "twenty". For very common words like "the", it is unnatural to hard-code the pronunciation.

[0026] End-to-end (E2E) speech recognition models combine acoustic, pronunciation, and language models into a single neural network. The single neural network model improves simplicity and quality, optimizing word error rate (WER). However, a challenge in E2E speech recognition models is optimizing performance in recognizing words that occur rarely in the language and / or words that have unusual pronunciation compared to their spelling. Although training data can include both human-transcribed voice data and text-only data, it is inefficient to use large training datasets to train these E2E speech recognition models. The distribution of words in a language typically follows a Zipf distribution, where a small number of words are used very frequently and a huge number of words are used rarely, so increasing the number of training examples typically results in less and less improvement.

[0027] Incorporating contextual bias into neural network ASR models can improve the recognition of rare words and words with unusual pronunciations. For example, a user's contacts are often stored on a smartphone and can be used as context to assist the ASR system in recognizing the names of the contacts spoken by the user. Contextual bias can be applied to ASR models by injecting both the biasing context and the pronunciation into the model. Contextually biased models retain the advantages of neural network models, including simple, unified training and implicit learning of the pronunciation of rare words. Contextually biased models incorporate knowledge of the pronunciation of rare words even if the word has never been present during training.

[0028] Referring to FIG. 1, an exemplary ASR system 100 includes a speech recognition model 300 incorporating context bias to improve speech recognition accuracy. The speech recognition model 300 includes an audio encoder 110, a grapheme encoder 122, a phoneme encoder 126, an audio attention module 132, a context bias attention module 134, and a decoder 142 that receives as inputs outputs from both the audio attention module 132 and the context bias attention module 134. In some examples, the output of the decoder 142 is normalized in a softmax layer 144, whereby the softmax layer 144 outputs a probability distribution over a sequence of output graphemes. The softmax layer 144 may provide a probability distribution over a set of linguistic units (e.g., phonetic elements), such as a set of graphemes. For example, the output of the softmax layer 144 may be a vector having a probability value for each of the graphemes in the language, as well as potentially other symbols (e.g., punctuation, spaces, etc.). The sequence of vectors from the softmax layer 144 is used to generate a transcription 150 of the speech utterance 104 recorded by the user device 106. In some examples, the speech recognition model 300 resides on the user device 106 associated with the user 102. In other examples, the speech recognition model 300 resides on a remote server that communicates with the user device 106, or the functionality of the speech recognition model 300 is split between the remote server and the user device 106.

[0029] In the illustrated example, a user 102 speaks an utterance 104 that is captured by one or more microphones of a user device 106. The user device 106 may include a mobile device such as a smartphone, a tablet, smart headphones, a smart watch, etc. The utterance 104 spoken by the user 102 may be a command to "Navigate to Bexar Courthouse," where "Bexar" is the name of a county pronounced / ber / , similar to the words "bear" and "bare." The one or more microphones of the user device 106 generate an audio signal 108 from the spoken command. The audio signal (interchangeably referred to as "audio data") encodes the utterance 104 and may be processed in any of a variety of ways to determine inputs for neural network elements of the speech recognition model 300. For example, a feature extraction module (not shown) may generate acoustic features 109 derived from the audio signal 108 , such as mel-frequency cepstral components of different windows or frames of the audio signal 108 .

[0030] The user device 106 also provides context information 114 to the speech recognition model 300. The context information 114 may include, for example, the user's location, apps open on the device 106, the user's contacts, and upcoming calendar events. In this case, the context information 114 may include the user's current GPS location within Bexar County, a navigation application 107 open on the user device 106, and the user's upcoming calendar events at the courthouse. The calendar events may be obtained by accessing a calendar application 107 executable on the user device 106. The user 102 may explicitly opt in to consent to sharing each type of context information 114 individually, and may opt out at any time to revoke previous consent to sharing one or more types of context information 114.

[0031] The speech recognition model 300 uses contextual information 114 received from the user device 106 to select particular bias phrases 116, each of which may include both grapheme data 118 and phoneme data 120. The grapheme data 118 is input to a grapheme encoder 122, and the phoneme data 120 is input to a phoneme encoder 126. Additional details regarding the selection of bias phrases 116 are provided in FIG.

[0032] In some implementations, the audio encoder 110, the grapheme encoder 122, and the phoneme encoder 126 include sequence models having long short-term memory (LSTM) elements or other recurrent neural network (RNN) elements. The audio encoder 110 may include a first encoder configured to generate an encoded audio vector (e.g., an audio encoding) 112 based on the audio signal 108. For example, the audio encoder 110 may receive acoustic features 109 derived from the audio signal 108 encoding the utterance 104 and generate the encoded audio vector 112 from the acoustic features 109. The encoded vector 112 is input to the audio attention module 132. The grapheme encoder 122 is configured to generate the encoded grapheme vector 124 from the grapheme data 118 associated with the selected bias phrase 116, while the phoneme encoder 126 is configured to generate the encoded phoneme vector 128 from the phoneme data 120 associated with the selected bias phrase 116. In the illustrated example, the speech recognition model 300 concatenates the encoded grapheme vector 124 and the encoded phoneme vector 128 into a projection vector 130 that is input to a context-biased attention module 134. Thus, the projection vector 130 may correspond to a representation of the encoded grapheme vector 124 and the phoneme vector 128.

[0033] The audio attention module 132 is configured to calculate attention weights and generate a weighted audio encoding vector 136 that includes a vector summary of the audio frame that the model 300 is currently attending. For example, the audio encoder 110 can generate an audio vector 112 for each window or frame of the audio signal 108 as additional acoustic features 109 are received, and thus continue to provide additional audio vectors 112 as more audio is received. In some examples, the audio attention module 132 uses the decoder audio state 141 and the encoded audio vectors 112 from a previous time step to calculate attention weights for a sequence of audio vectors 112. The decoder audio state 141 may represent a portion of the state of the decoder 142 when outputting the latest decoder output, e.g., a portion of the decoder state of a previous time step that represents the immediately preceding window or frame of audio. The decoder state 141 serves to inform the audio attention module 132 of the relative importance of the audio vectors 112 input to the audio attention module 132 in the current time step. The decoder audio state 141 may be a portion of the output of the decoder 142, such as a designated subset of the outputs of the decoder 142, typically a different set of outputs than those used in the decoder context state 143. In some implementations, the state vectors 141, 143 represent non-overlapping portions of the outputs of the decoder 142. In other implementations, the state vectors 141, 143 include one or more overlapping sections, or both state vectors 141, 143 each represent the entire output of the decoder 142.

[0034] At the same time, the contextual bias attention module 134 is configured to calculate attention weights using both the encoded grapheme vector 124 and the phoneme vector 128 included in the projection vector 130. The contextual bias attention module 134 uses these attention weights to calculate only a weighted sum of grapheme embeddings, which is output as a contextual biasing vector 138. More specifically, the contextual bias attention module 134 processes the projection vector 130 from the previous time step and a decoder context state 143 to generate the contextual biasing vector 138. The decoder context state 143 includes the state of the decoder 142 at the previous time step with respect to the previous contextual biasing vector 138. The decoder context state 143 may represent a portion of the output of the decoder 142, such as, for example, a specified subset of the output of the decoder 142. The decoder context state 143 informs the contextual bias attention module 134 of the relative importance of the projection vector 130 input to the contextual biasing attention module 134 at the current time step. Thus, as the decoder context state 143 changes, the context-biased attention module 134 computes different summaries or aggregations of the projection vectors 130, thus changing the probabilities that the decoder 142 exhibits for different bias phrase 116 elements.

[0035] By having the context-biased attention module 134 compute only the weighted sum of grapheme embeddings, the model 300 does not learn a priori relationships between spelling and pronunciation, thereby allowing the model to assign any sequence of phonemes to any word. However, because the context-biased attention module 134 uses the encoded phoneme vectors 128 in addition to the encoded grapheme vectors 124 to compute the attention weights, the weighted sum of grapheme embeddings computed from the attention weights contains phoneme information in addition to grapheme information.

[0036] In particular, the context bias vector 138 output from the context bias attention module 134 represents the context bias of the speech recognition model 300 and includes both grapheme and phoneme information. Thus, when the context bias vector 138 is concatenated with the weighted audio encoding vector 136 into a weight vector, the context bias is "injected" into the speech recognition model 300. The weight vector 140 collectively represents the audio, grapheme, and phoneme information. The weight vector 140 is input to the decoder 142.

[0037] In some configurations, the decoder 142 includes a sequence model, such as an LSTM network, and is configured to extract an output sequence from the weight vector 140. The output of the decoder 142 is also normalized with a softmax layer 144 to generate a probability distribution over a set of output targets, which in the illustrated example are graphemes. In other examples, the set of output targets of the probability distribution may include words, word pieces, or phonemes. A beam search process may use various output vectors from the decoder 142 to generate a transcription 146 of the utterance 104. The decision made about which grapheme represents the audio at the current time step is fed back to the decoder 142 as a speech element output 148 and used to calculate the next output from the decoder 142. In the illustrated example, the speech element output 148 includes a grapheme output. In other examples, the speech element output 148 includes a word output or a word piece output. In yet other examples, the speech element output 148 includes a phoneme output.

[0038] 1, contextual information 114 of user's 102 location, open apps, and upcoming calendar appointments biases speech recognition model 300 toward the bias phrase "Bear" and the pronunciation / ber / . This allows speech recognition model 300 to correctly transcribe user's 102 command as "Navigate to Bear Court." User device 106 can then perform actions based on this transcription, such as mapping a route to the courthouse in a navigation app.

[0039] 2 illustrates a schematic diagram of an example process 200 for generating a set of one or more bias phrases 116 associated with a context of an utterance 104. The process 200 includes identifying a context 202 of the speech utterance 104 and selecting context information 114 from a data source 210 to generate bias phrases 116 that include both grapheme data 118 and phoneme data 120. Each bias phrase 116 in the set of one or more bias phrases includes one or more words such that the grapheme data 118 includes graphemes associated with each word of the bias phrase 116 and the phoneme data 118 includes phonemes associated with each word of the bias phrase 116.

[0040] More specifically, the process 200 begins when a user speaks an utterance 104. When the ASR system receives audio input from the spoken utterance 104, one or more contexts 202 of the utterance 104 are determined. The context 202 determines the bias phrases 116 that are selected for use in the context bias.

[0041] The context 202 may be, for example, the location of the user. The user's location may be determined by GPS coordinates. For example, the context may be that the user is at home, at work, or at a geographic location near the user's home or work. The user's location may also be identified as a venue, such as a restaurant, a concert hall, or a stadium. If the user is traveling, the context may be that the user is traveling for vacation or work, perhaps to an unfamiliar or foreign city.

[0042] Another example of a context 202 is an app that is open on a user's mobile device. If a user opens a phone app, they may be attempting to call a contact. If a user opens a navigation app, they may be attempting to navigate to a destination.

[0043] An example of a context 202 can also be the date and time of a spoken utterance. For example, if it is early Monday morning, the user may be preparing to drive to work. Thus, the user may speak utterances 104 that help him or her prepare for the workday, such as asking about the weather or traffic conditions, navigating to the office, etc.

[0044] The selection module 203 references the context 202 to select information 114 from various data sources 210. The data sources 210 may include, for example, contacts 204, calendar events 206, and previously spoken commands 208. In one example, if the context 202 includes an open phone app, the selection module 203 may select information 114 from stored contacts 204. If the context 202 includes an open navigation app, the selection module 203 may select information 114 related to the location of an upcoming calendar event. The selection module 203 may also reference previous commands 208. For example, a user may request to navigate to work every weekday morning. Thus, an utterance 104 spoken on Monday morning is likely to include words recently spoken by the user related to the user's workplace. The context information 114 may also specify one or more contexts 202 associated with the spoken utterance 104.

[0045] The information 114 obtained / selected by the selection module 203 is used to create a bias phrase 116. The bias phrase 116 includes both grapheme data 118 and phoneme data 120. The phoneme data 120 may be generated using a pronunciation dictionary 216.

[0046] As an example, a user may leave home most weekday mornings, drive to Bowie, Maryland, drop off their child at daycare, and then go to work in Baltimore, Maryland. When the user speaks the utterance 104 "Navigate to Bowie, Maryland," the context 202 of the utterance 104 may include the user's location at home in Maryland. The context 104 may also include a navigation app that is open, and a date and time, which may be Monday morning. The selection module 203 accesses a data source 210. A calendar event 206 indicates that the user has a meeting in Baltimore, Maryland later that day. A previous command 208 indicates that most weekday mornings the user requests to navigate to Bowie and then Baltimore. The selection module 203 uses this context information 114 to create a bias phrase 212.

[0047] In this example, since the user has a navigation app open, the grapheme bias data 118 may include the words "Baltimore," "Maryland," and "Bowie," three words associated with the user's possible destinations. The pronunciation dictionary 216 is used to create the phoneme bias data 120. The user device 106 accesses the pronunciation dictionary 216 and searches for pronunciations, such as sequences of phonemes, that represent common pronunciations of a word. The pronunciation dictionary may include variations in the pronunciation of a word. For example, a person from Baltimore may pronounce "Baltimore" differently than a person visiting from another state. Similarly, if a traveler is visiting a country where the language spoken is not the native language, the traveler's pronunciation of a place name may differ from that of a native speaker.

[0048] This process 200 generates grapheme bias data 118 and phoneme bias data 120. The bias phrases 116 are then provided to a grapheme encoder 122 and a phoneme encoder 126 for injection into an ASR model 300, as shown in FIG.

[0049] FIG. 3 illustrates an exemplary architecture of a speech recognition model 300 of the ASR system 100 of FIG. 1 for context bias using graphemes and phonemes. The exemplary architecture of the model 300 illustrated in FIG. 3 includes a listen, attend, and spell (LAS) model. The LAS architecture includes three main components: an audio encoder (e.g., a first encoder) 110, an audio attention module (e.g., a first attention module) 132, and a decoder 142. The audio encoder 110 is configured to receive a time-frequency representation (i.e., acoustic features) 109 of an input speech audio signal 108 and uses a set of neural network layers to map the input 109 to a higher-level feature representation, i.e., an encoded audio vector (e.g., an audio encoding) 112. The encoded audio vectors 112 output of the audio encoder 110 are passed to an audio attention module 132, which uses the encoded audio vectors 112 to learn alignments between the input acoustic features 109 and predicted graphemes 148. The output 136 of the attention audio module 132 is passed to a decoder 142, which generates a probability distribution over the set of hypothesized graphemes. The components of the LAS model are trained jointly as a single end-to-end neural network.

[0050] The exemplary LAS architecture is extended to include additional components that enable the model 300 to bias the speech recognition results 146, 148 toward bias phrases 116 associated with a given context 202. These additional components include a grapheme encoder (e.g., a second encoder) 122, a phoneme encoder (e.g., a third encoder) 126, and a biased attention module (e.g., a second attention module) 134.

[0051] The extended architecture of the speech recognition model 300 is based on audio, grapheme, and phoneme input x u , g n , and p n where the subscripts correspond to U audio frames 109 and N bias phrases 116. The encoders 110, 122, 126 encode their respective inputs as h x , h g , and h p which are then fed to the corresponding attention modules 132, 134. The attention by the attention modules 132, 134 is then converted into the corresponding context attention vectors ct x and ct g The decoder 142 outputs dt x and dt g The output of the exemplary LAS model 300 is the vector y t It is.

[0052] In some implementations, the audio encoder 110 may generate audio frames x1...x UThe audio encoder 110 includes a bidirectional LSTM configured to receive as input the acoustic features 109 (e.g., log-mel features, etc.). In one example, the audio encoder 110 includes 10 LSTM layers, each with 256 nodes, with a hidden size of 1400. Furthermore, the grapheme encoder 122 and the phoneme encoder 126 may each include a single-layer LSTM of 512 dimensions. The grapheme encoder 122 and the phoneme encoder 126 may each receive as input respective grapheme data 118 and phoneme data 120 associated with the N randomly shuffled bias phrases 116. Thus, the number of phoneme inputs (e.g., phoneme data) 120 conveying pronunciation information may be the same as the number of grapheme inputs (e.g., grapheme data) 118, and the corresponding embeddings / encodings 124, 128 output from the encoders 122, 126 are calculated independently of each other.

[0053] In some examples, the audio attention module 132 includes four heads with coordinates of 512. In the illustrated example, the audio attention module 132 includes an audio encoding 112, which includes an encoding of the audio frame 109, h x and the output 141 of the decoder 142 at the previous time step, d x t-1 The superscript x informs the audio attention module 132 that the output 141 corresponds to a decoder audio state that represents a portion of the state of the decoder 142 when it outputs the most recent decoder output, e.g., a portion of the decoder state of a previous time step that represents the previous window or frame of audio. Thus, the output 141 associated with the decoder audio state represents the audio vector 112, h x, to inform audio attention module 132 of the relative importance of the decoder audio state. Output 141 corresponding to the decoder audio state may be a portion of the outputs of decoder 142, such as a designated subset of the outputs of decoder 142, and is typically a different set of outputs than those used in output 143 corresponding to the decoder's context state.

[0054] On the other hand, the biased attention module 134 is configured to generate a grapheme encoding 124, h, based on the grapheme input 118 and the phoneme input 120. g and the phoneme encoding 128 output from the phoneme encoder 126, h p The projection layer 132 may concatenate the encodings 124, 128 and output the corresponding encoded projections 130. More specifically, the biased attention module 134 attends to a weighted sum of the encoded projections 130 at every decoding step t. Here, the biased attention module 134 may use both the grapheme encoding 124 and the phoneme encoding 128 to compute the attention weights. Once the attention weights are computed, they are calculated based on the weighted sum of the grapheme embeddings c t g Thus, the output 138 of the biased attention module 134 is used to calculate the grapheme encoding 124, h g The weighted sum of c t g Contains only.

[0055] More specifically, the output 138 of the biased attention module 134 is computed in three steps. In a first step, the biased attention module 134 computes the grapheme encoding h i g , phoneme encoding h i p , and the previous decoding step, d g t-1The biased attention module 134 receives as input the output 143 from u. The superscript g informs the biased attention module 134 that the output 143 corresponds to a decoder grapheme state that represents a portion of the state of the decoder 142 when it outputs the most recent decoder output. In some examples, the biased attention module 134 receives as input the output 143 from u. g it We calculate the set of weights for

[0056]

number

[0057] In the second step, α t g To obtain,the output is normalized with softmax144 as follows:

[0058]

number

[0059] In the third step, the output 138 is i g It is calculated as a weighted sum of

[0060]

number

[0061] Bias attention output 138, c t g represents the embedding of the grapheme. Thus, in some examples, the decoder 142 is configured to operate only on graphemes, and thus the decoder input c t gis grapheme-only. This allows the injection of words where the pronunciation is difficult to predict from the spelling. With this approach, there is no a priori relationship between spelling and pronunciation, and any sequence of phonemes can be assigned to any word.

[0062] However, in configurations in which the decoder 142 is configured to generate probability distributions over a set of output targets other than graphemes, such as word pieces, the biased attention output 138 may be with respect to the type of output target that the decoder 142 is configured to generate. Optionally, the output of the biased attention module may represent a phoneme embedding, e.g., a weighted sum of phonemes, and the decoder 142 may instead operate only on phonemes to generate a set of output targets associated with the phonemes.

[0063] In some examples, the decoder 142 includes four LSTM layers, each with a size of 256, with a hidden dimension of 1024. In the example shown, the decoder 142 receives three inputs. The first input is the previous grapheme output 138, y t-1 When the model is used to make predictions, y t-1 is the output of softmax144. When the model is in training, y t-1 may be either a one-hot encoding of the true graphemes, or a sampling of the output and a one-hot encoding of the true graphemes. t x contains a summary of the audio frame 109 that the model 300 is currently attending to. The third input 138 to the decoder 142 is the biased attention output c t g Includes.

[0064] Continuing with reference to FIG. 3, the output of the exemplary LAS model 300 is the vector y for all decoding steps t from 1 to T. t In ASR, the vector y tcorresponds to one-hot encoding of the output symbols (graphemes, spaces, and the end of the sequence). The model 300 outputs T-1 symbols until it predicts the end symbol of the sequence at the Tth step.

[0065] In some implementations, the training process trains the speech recognition model 300 on multiple batches of training examples, where for each batch, a reference transcript is generated with probability P keep For the remaining reference transcripts, n-grams are randomly selected, where n is in the range [1:N order ]. The number of n-grams selected is N, where N is the sum of [1:N phrases ]. Example values ​​are P keep =0:5, N phrases = 1, and N order = 4. Thus, the amount of bias phrases 116 in each batch is variable and not specified when training the model 300. Additionally, one or more bias phrases 116 applied during training may have a different length than other bias phrases 116. The model 300 may be trained using log-mel acoustic features every 10 ms, computed over 25 ms windows and stacked in groups of three. An exemplary acoustic feature vector has 80 coordinates.

[0066] If the biased graphemes and phonemes are empty, an unbiased option can be added. The unbiased option can be applied by applying a contextual bias vector 138 that does not contain graphemes and bias data 118, 120 associated with any bias phrases 116. This sampling technique ensures that some examples do not have matching words, but other examples have matching words. To help the model converge, special tokens are introduced. For each matching bias token, a special character is added. This introduces bias errors that can only be corrected by using bias phrases.

[0067] For phoneme training, a dictionary and grapheme-to-phoneme (G2P) combination can be used. If the word is in the dictionary, the corresponding pronunciation can be used. If the word is not in the dictionary, G2P can be used to predict the pronunciation.

[0068] Injecting a bias context, including pronunciation knowledge, into a neural network model results in a 14% relative improvement over when no bias context is injected. Contextually biased models retain the advantages of neural network models, including simple, unified training and implicit learning of rare word pronunciations. At the same time, contextually biased models incorporate knowledge of rare word pronunciations, even if they were never present during training. However, disambiguating the correct phrase is still difficult when the context includes similar-sounding phrases (e.g., when both "Joan" and "John" are in the user's contact list). Disambiguation can be improved by training the ASR context mechanism with difficult negative examples.

[0069] In the training example above, training bias phrases are created by randomly sampling n-grams from the reference transcripts and other utterances in the training data. At test time, the bias phrases are pre-corrected and are from the same semantic category (e.g., contact names). In this case, the testing task is more difficult than the training task because distinguishing first names such as Joan and John is typically more difficult than distinguishing random unrelated n-grams.

[0070] 4 shows an example method 400 that can be used to close train-test discrepancies that arise from training with randomly sampled N-grams. Instead of sampling random N-grams, method 400 detects proper nouns from reference transcripts and augments them with phonetically similar or "fuzzy" alternatives to create a set of bias phrases for training.

[0071] Method 400 focuses on proper nouns (e.g., unique entities such as names of people and places) in the reference transcripts and uses phonetically similar phrases as negative examples to encourage the neural network model to learn more discriminatory representations. This technique can be applied to neural network contextual ASR models that jointly learn to transcribe and select correct context items to improve WER by up to 53.1%.

[0072] In ASR, two phrases may be very similar to each other phonetically (i.e., in the way they are pronounced), yet arguably different (e.g., "call Joan" and "call John"). For a neural network ASR model, the learned representations of these names may be very similar, causing the model to predict the wrong representation. This problem is particularly difficult for ASR models that are presented with rare, hard-to-spell words, because the model may never observe these words during training and would never be able to spell these words correctly at test time.

[0073] Training with negative examples is a method to teach a model to distinguish these rare and hard-to-spell phrases from phonetically similar phrases. Using this technique, the model learns more robust representations that help it perform better at test time. The method for training with negative examples includes (i) detecting rare, hard-to-transcribe words in the input utterance as focus targets, and (ii) training the model more rigorously on these words by providing the model with difficult negative examples.

[0074] When detecting phrases that focus on (i), proper nouns (also tagged as "NNP") are a common category of phrases that, while relatively easy to detect, are rare and usually difficult to transcribe. To train these phrases (ii) more rigorously, phonetically similar alternative phrases are extracted and fed to the model as negative examples.

[0075] This technique can be thought of as data augmentation for speech. Whereas data augmentation is typically used in machine learning to generate mutated positive examples, this technique is used to generate mutated negative examples. This technique is applied to a contextually biased LAS model to improve the WER.

[0076] 4, the utterance "Call Joan's mobile" is provided to the ASR model. Given the example utterance 410, the method 400 detects 415 proper nouns from the reference transcript 405.

[0077] In this example, the word "Joan" 420 is selected as the bias phrase. Because "Joan" 420 is a person's name, it is easily identified as a proper noun. To identify proper nouns, the reference transcript is analyzed with a part-of-speech tagger.

[0078] The method 400 then adds (425) phonetically similar or fuzzy phrases as alternatives. In this case, the words "John" and "Jean" 430 are selected as alternative bias phrases. A word N-gram is identified as a fuzzy alternative to a second word N-gram if both are phonetically similar and frequently co-occur in different decoding beams of the same utterance in the training data. The process of identifying fuzzy alternatives is described below.

[0079] First, a fuzzy inventory is created that stores per N-gram, a set of alternatives for fuzzy N-grams. The fuzzy inventory is built in an unsupervised way by using an external conventional model to decode a large corpus of utterances. Each decoded utterance becomes a set of hypotheses. All co-occurrences of pairs of word N-grams that appear in different hypotheses where the remaining hypotheses are identical are counted, and each N-gram pair is scored according to its count. For example, "John Lemon" gets a high score against the target N-gram "John Lennon" if it is frequently found in different hypotheses of the same utterance.

[0080] The fuzzy inventory is then used during contextually biased LAS training. Specifically, given a word n-gram (representing a bias phrase in this case), fuzzy alternatives are selected from the fuzzy inventory and sorted by co-occurrence score. The top candidates are filtered by retaining only those that are phonetically similar to the target n-gram, where phonetic similarity is measured using the Hixon metric. The selection of fuzzy bias alternatives is done as part of the data preparation phase.

[0081] The set of extracted proper names and their fuzzy alternatives are fed to the grapheme and phoneme encoders as bias phrases 435. The bias phrases 435 are thus used to train the ASR model on examples in the reference transcript 405. This approach allows the model to be trained on the difficult task of distinguishing between phonetically similar names.

[0082] From each training example, up to three proper nouns are randomly selected and three fuzzy alternatives are added for each proper noun. Other random proper nouns are added from the remaining training data, generating up to 64 bias phrases per example.

[0083] When presented with phonetically similar phrases at test, the fuzzy model captures subtle phonetic differences better than the non-fuzzy model, as manifested by both more accurate predictions and, as a consequence, more attention is paid to the bias phrases that actually appear in the reference transcript, rather than to their fuzzy alternatives. Thus, training with fuzzy distractors allows the model to better discriminate between phonetically similar phrases. Moreover, the fuzzy model attends more sharply to the misleading phrases in its attention, and its distribution is much cleaner, compared to the non-fuzzy model.

[0084] Training a contextualized neural speech recognition model with challenging negative examples results in improved WER. The central idea is to detect and focus on proper nouns ("NNPs") in the reference transcripts and present the model with phonetically similar ("fuzzy") phrases as their negative examples. When applied to a speech bias task, the technique improves WER by up to 53.1%.

[0085] 5 is a flowchart of an exemplary arrangement of operations for a method 500 of biasing a speech recognition model 300 toward one or more bias phrases 116 associated with a context 202 of a spoken utterance 104. The method 500 may be executed on a data processing hardware 610 (FIG. 6) located on a user device 106 associated with a user 102 who spoke the utterance 104. Optionally, the data processing hardware 610 may reside on a remote device (e.g., a server of a cloud-based computing environment) that communicates with the user device 106, for example, via a network. At operation 502, the method 500 includes receiving, at the data processing hardware 610, audio data 106 encoding the utterance 104, and at operation 504, the method 500 includes obtaining, by the data processing hardware 610, a set of one or more bias phrases 116 corresponding to the context 202 of the utterance 104. Here, each bias phrase 116 in the set of one or more bias phrases 116 includes one or more words. As used herein, the one or more "words" of each bias phrase 116 may include, but are not limited to, proper nouns such as the name of a person in the user's contact list 204, a city name, a musical artist name, the title of an album / movie (or other media content type), etc. The context 202 may be determined based on at least one of the current date and / or time of the utterance, one or more applications open on the user device 106, and the location of the user 102.

[0086] At operation 506, the method 500 includes processing, by the data processing hardware 610, the acoustic features 109 obtained from the audio data and the grapheme and phoneme data 118, 120 obtained from the set of one or more bias phrases 116 using the speech recognition model 300 to generate an output 148 of the speech recognition model 300. The output 148 may include a grapheme output for each frame of the acoustic features 109. At operation 508, the method 500 includes determining, by the data processing hardware 610, a transcription 146 of the utterance 104 based on the output 148 of the speech recognition model 300. The transcription 146 may be displayed on a graphical user interface 106 of the user device 106 and / or one or more other devices in communication with the data processing hardware 610. In additional examples, the transcription 146 is provided as a command to a search engine or software application to perform an operation.

[0087] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," an "app," or a "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0088] Non-transient memory may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by a computing device. Non-transient memory may be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), as well as disks or tapes.

[0089] 6 is a schematic diagram of an exemplary computing device 600 that may be used to implement the systems and methods described herein. Computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components, their connections and relationships, and their functions shown herein are meant to be merely illustrative and are not meant to limit the implementation of the inventions described and / or claimed herein.

[0090] Computing device 600 includes a processor 610, a memory 620, a storage device 630, a high-speed interface / controller 640 that connects to memory 620 and a high-speed expansion port 650, and a low-speed interface / controller 660 that connects to a low-speed bus 670 and storage device 630. Each of the components 610, 620, 630, 640, 650, and 660 are interconnected using various buses and may be attached to a common motherboard or otherwise as desired. Processor 610 can process instructions for execution within computing device 600, including instructions stored in memory 620 or storage device 630, to display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 680 coupled to high-speed interface 640. In other implementations, multiple processors and / or multiple buses may be used as desired, along with multiple memories and types of memories. Also, multiple computing devices 600 may be connected, with each device providing a portion of the required operations (eg, as a bank of servers, a group of blade servers, or a multi-processor system).

[0091] The memory 620 stores information non-temporarily within the computing device 600. The memory 620 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-transient memory 620 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), as well as disks or tapes.

[0092] The storage device 630 can provide mass storage for the computing device 600. In some implementations, the storage device 630 is a computer-readable medium. In various different implementations, the storage device 630 can be an array of devices, including a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or a device in a storage area network or other configuration. In additional implementations, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable medium or a machine-readable medium, such as the memory 620, the storage device 630, or a memory on the processor 610.

[0093] The high-speed controller 640 manages the bandwidth-intensive operations of the computing device 600, while the low-speed controller 660 manages the lower bandwidth-intensive operations. Such a role assignment is merely exemplary. In some implementations, the high-speed controller 640 is coupled to the memory 620, the display 680 (e.g., through a graphics processor or accelerator), and the high-speed expansion port 650 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 660 is coupled to the storage device 630 and the low-speed expansion port 690. The low-speed expansion port 690, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a network device such as a switch or router through a network adapter.

[0094] The computing device 600 may be implemented in a number of different forms, as shown in the drawing: For example, it may be implemented as a standard server 600a, or multiple times within a group of such servers 600a, as a laptop computer 600b, or as part of a rack server system 600c.

[0095] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system that includes at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0096] These computer programs (also known as programs, software, software applications, or codes) contain machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine code. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0097] The processes and logic flows described herein can be executed by one or more programmable processors, also referred to as data processing hardware, which execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by dedicated logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits). Processors suitable for executing computer programs include, by way of example, both general purpose and dedicated microprocessors, and any one or more processors of any type of digital computer. In general, a processor receives instructions and data from a read-only memory or a random access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. In general, a computer also includes or is operatively coupled to one or more mass storage devices, e.g., magnetic disks, magneto-optical disks, or optical disks, for storing data, for receiving data or for transferring data, or both. However, a computer does not require such devices. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, such as, for example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0098] To provide for user interaction, one or more aspects of the present disclosure can be implemented in a computer having a display device, e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or a touch screen for displaying information to a user, and optionally a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide for user interaction, e.g., feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, tactile input, and the like. Additionally, the computer can interact with the user by sending and receiving documents to and from a device used by the user, e.g., by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0099] Although several implementations have been described, it will be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, other implementations are within the scope of the following claims. [Explanation of symbols]

[0100] 100 ASR System 102 users 104 Voice utterances 106 User Devices 107 Navigation Applications 107 Calendar Application 108 Audio Signals 109 Acoustic Features 109 Audio Frames 110 Audio Encoder 112 Audio Vectors 114 Context Information 116 Bias Phrases 118 Grapheme Data 118 Grapheme Input 118 Grapheme Bias Data 120 Phoneme Data 120 Phoneme Bias Data 120 Phoneme Input 122 Grapheme Encoder 124 grapheme vectors 124 Grapheme Encoding 126 Phoneme Encoder 128 Phoneme Vectors 128 Phoneme Encoding 130 Projection Vector 130 Projection 132 Audio Attention Module 132 Projection Layer 134 Context-Biased Attention Module 136 Weighted Audio Encoding Vectors 136 Output 138 Context Bias Vector 138 Biased Attention Output 138 Output 140 Weight Vector 141 State Vector 141 Decoder Audio Status 141 Output 142 Decoder 143 Decoder Context State 143 State Vector 143 Output 144 Softmax Layer 146 Transcription 146 Voice Recognition Results 148 Audio Element Output 148 Grapheme 148 Voice Recognition Results 150 Transcription 200 processes 202 Context 203 Selection Module 204 Contact 204 Contact List 206 Calendar Events 208 Command 210 Data Sources 212 Bias Phrases 216 Pronunciation Dictionary 300 Voice Recognition Model 300 LAS model 400 ways 405 Reference Transcript 410 utterances 420 "Joan" 430 words "John" and "Jean" 435 Bias Phrases 500 ways 502 operation 506 operation 600 computing devices 600a Standard Server 600b Laptop Computer 600c Rack Server System 610 Data Processing Hardware 610 Processor 620 Memory 630 Storage Devices 640 High Speed ​​Interface / Controller 650 High Speed ​​Expansion Port 660 Low Speed ​​Interface / Controller 670 Slow Bus 680 Display 690 Low Speed ​​Expansion Port

Claims

1. 1. A computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform a number of operations, the number of operations comprising: an act of receiving audio data encoding one or more words of an utterance; an operation of obtaining grapheme data and phoneme data for a bias phrase of the one or more words of the utterance, the bias phrase corresponding to a context of the utterance and including a proper noun; generating an audio encoding using an audio encoder configured to receive acoustic features of the audio data; generating a grapheme encoding using a grapheme encoder configured to receive the grapheme data; generating a phoneme encoding using a phoneme encoder configured to receive the phoneme data; an operation of generating a first attention output using a first attention module configured to receive the audio encoding, the first attention output comprising a weighted audio encoding vector comprising a vector summary of the audio data; generating a second attention output using a second attention module configured to receive a representation of the grapheme encoding output from the grapheme encoder and a representation of the phoneme encoding output from the phoneme encoder; processing said first and second attention outputs using a decoder to determine a set of likely output graphemes corresponding to said audio data; 4. A computer-implemented method comprising:

2. The computer-implemented method of claim 1 , wherein the plurality of operations further comprises an operation of generating a transcription of the utterance based on the probabilities of the set of output graphemes.

3. 10. The computer-implemented method of claim 1, wherein the grapheme encoder and the phoneme encoder each comprise a neural network.

4. the grapheme encoder configured to generate a grapheme encoding corresponding to a particular word; the phoneme encoder configured to generate a phoneme encoding corresponding to the particular word; 2. The computer-implemented method of claim 1, wherein the second attention module is configured to encode the second attention output comprising a context bias vector corresponding to the particular word based on the grapheme and phoneme encoding corresponding to the particular word.

5. 2. The computer-implemented method of claim 1, wherein the representation of the grapheme encoding output from the grapheme encoder and the phoneme encoding output from the phoneme encoder comprises a concatenation between the grapheme encoding and the phoneme encoding.

6. 6. The computer-implemented method of claim 5, wherein the link between the grapheme encoding and the phoneme encoding is represented by a projection vector that is input to the second attention module.

7. 2. The computer-implemented method of claim 1, wherein the grapheme encoder, the phoneme encoder, the first and second attention modules, and the decoder are jointly trained to determine likelihoods of a set of output graphemes that correspond to audio data.

8. The computer-implemented method of claim 1 , wherein the set of output graphemes comprises graphemes.

9. The computer-implemented method of claim 1 , wherein the set of output graphemes includes words or pieces of words.

10. The plurality of operations include: the location of the user who spoke the utterance; one or more applications open on a user device associated with the user who spoke the utterance; and The current date and / or time of the utterance 10. The computer-implemented method of claim 1, further comprising the act of determining the bias phrase based on at least one of:

11. 1. A system comprising: Data processing hardware; memory hardware in communication with said data processing hardware; the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform a plurality of operations, the plurality of operations comprising: an act of receiving audio data encoding one or more words of an utterance; an operation of obtaining grapheme data and phoneme data for a bias phrase of the one or more words of the utterance, the bias phrase corresponding to a context of the utterance and including a proper noun; generating an audio encoding using an audio encoder configured to receive acoustic features of the audio data; generating a grapheme encoding using a grapheme encoder configured to receive the grapheme data; generating a phoneme encoding using a phoneme encoder configured to receive the phoneme data; an operation of generating a first attention output using a first attention module configured to receive the audio encoding, the first attention output comprising a weighted audio encoding vector comprising a vector summary of the audio data; generating a second attention output using a second attention module configured to receive a representation of the grapheme encoding output from the grapheme encoder and a representation of the phoneme encoding output from the phoneme encoder; processing said first and second attention outputs using a decoder to determine a set of likely output graphemes corresponding to said audio data; Including, the system.

12. The system of claim 11 , wherein the plurality of operations further comprises an operation of generating a transcription of the utterance based on the probabilities of the set of output graphemes.

13. The system of claim 11 , wherein the grapheme encoder and the phoneme encoder each comprise a neural network.

14. the grapheme encoder configured to generate a grapheme encoding corresponding to a particular word; the phoneme encoder configured to generate a phoneme encoding corresponding to the particular word; 12. The system of claim 11, wherein the second attention module is configured to encode the second attention output comprising a context bias vector corresponding to the particular word based on the grapheme and phoneme encoding corresponding to the particular word.

15. 12. The system of claim 11, wherein the representation of the grapheme encoding output from the grapheme encoder and the phoneme encoding output from the phoneme encoder comprises a concatenation between the grapheme encoding and the phoneme encoding.

16. 16. The system of claim 15, wherein the link between the grapheme encoding and the phoneme encoding is represented by a projection vector that is input to the second attention module.

17. 12. The system of claim 11, wherein the grapheme encoder, the phoneme encoder, the first and second attention modules, and the decoder are jointly trained to determine likelihoods of a set of output graphemes corresponding to audio data.

18. The system of claim 11 , wherein the set of output graphemes comprises graphemes.

19. The system of claim 11 , wherein the set of output graphemes includes words or pieces of words.

20. The plurality of operations include: the location of the user who spoke the utterance; one or more applications open on a user device associated with the user who spoke the utterance; and The current date and / or time of the utterance The system of claim 11 , further comprising the operation of determining the bias phrase based on at least one of:

Images