Contextual text injection for neural biasing in automatic speech recognition models

Abstract

Various implementations include contextual text injection (CTI) for neural biasing of automatic speech recognition (ASR) models to improve accuracy and / or robustness of ASR. In many implementations, a CTI-ASR model can be trained using text based natural language (NL) phrases and audio-transcript pairs. In several implementations, the text based NL phrases include bias phrase(s) that are infrequent and / or absent in the audio-transcript pairs.

Description

Attorney Docket No. GOOG-0520-WO-01 CONTEXTUAL TEXT INJECTION FOR NEURAL BIASING IN AUTOMATIC SPEECH RECOGNITIONMODELS Background

[0001] Computing systems can convert audio data, corresponding to a spoken utterance of auser, into corresponding text (or other semantic representation). For example, audio data can be generated based on the detection of a spoken utterance of a user via one or more microphones of a client device. A speech recognition engine can attempt to recognize various characteristics of the spoken utterance captured in the audio data, such as the sounds produced (e.g., phonemes) by the spoken utterance, the order of the pronounced sounds, rhythm of speech, intonation, etc. Further, the speech recognition engine can identify text words or phrases represented by such characteristics. The text can then be further processed by the one or more additional components of the computing system, such as using a natural language understanding engine and / or a dialog state engine.

[0002] Contextual biasing can be used in automatic speech recognition (ASR) systems toimprove the recognition of uncommon words and / or phrases, such as word(s) and / or phrase(s) that are relevant to context associated with a received spoken utterance. For example, word(s) and / or phrases(s) that are relevant to a user providing the spoken utterance, to an application executing during or before receiving the spoken utterance, to action(s) performed before receiving the spoken utterance, etc. Contextual biasing can seek to mitigate misrecognition of word(s) and / or phrase(s) that infrequently, or never, appear in training data utilized to train an ASR system. Some contextual biasing techniques include an external biasing component (e.g., an external language model, a contextual finite state transducer, etc.), and a score generated using the biasing component can be utilized by the ASR system at inference. However, such external biasing components are optimized separately from an ASR model utilized by the ASR system.

[0003] Neural biasing can integrate the biasing component into the ASR model to allow forjoint optimization of the biasing component and the ASR model. For example, neural biasing techniques can use attention mechanisms to associate bias phrases with a portion of the ASR model (e.g., the encoder portion of the ASR model, the decoder portion of the ASR model,Attorney Docket No. GOOG-0520-WO-01 etc.), where the neural biasing component can be used to generate a context vector that adapts the ASR model towards the biasing content. However, as the number of biasing phrases increases, both computational latency and word error rate tend to increase in neural biasing ASR models.

[0004] In some implementations, a neural associate memory (NAM) can be used to reduce thecomputational latency and / or the word error rate of a neural biasing ASR model. In some implementations, the NAM includes a two-pass hierarchical biasing scheme with a first pass portion of the neural biaser that identifies the top-K phrases that are most likely to occur in a speech signal via correlation. Additionally or alternatively, the NAM includes a second pass portion of the neural biaser that restricts the encoder portion of the ASR model to only the K phrases selected in the first pass. However, training neural biasing models (including a NAM neural biasing model) requires paired audio-transcript data. Paired audio-transcript data can be computationally expensive to generate (e.g., requiring significant client device resources of human reviewers), can take significant clock on the wall time to generate, can include inaccuracies, and / or can be non-existent for various word(s) and / or phrase(s) such as those that recently entered a lexicon. Additionally or alternatively, bias phrases sampled from such transcripts are generally not rare phrases for the ASR model and / or are generally not contextually relevant to the test utterances. Summary

[0005] Techniques described herein are directed towards contextual text injection (CTI) forneural biasing of automatic speech recognition (ASR) models to improve accuracy and / or robustness of ASR. In some implementations, a CTI-ASR model can be trained using text based natural language (NL) phrases and audio-transcript pairs, where the text based NL phrases include bias phrase(s) that are infrequent and / or absent in traditional audio-transcript pairs. In training the CTI-ASR model, the text based NL phrases can be transformed into speech-like representations by processing a phoneme representation of the text based NL phrases using one or more layers of an encoder portion of the CTI-ASR model.

[0006] These speech-like representations, and a set of bias phrases sampled from the textbased NL phrases, are then processed using a neural biaser portion of the CTI-ASR model toAttorney Docket No. GOOG-0520-WO-01 generate a biased speech-like representation which is biased towards the sampled bias phrases. The encoder portion of the CTI-ASR model can process these biased speech-like representations to generate an ASR generated text representation of the text based NL phrase. Additionally or alternatively, a text based ASR loss can be generated based on processing the ASR generated text representation and the text based NL phrase. The set of bias phrases, sampled from the text based NL phrases, can include bias phrase(s) that are infrequent and / or absent in training the CTI-ASR model using traditional audio-transcript pairs. Accordingly, processing the text based NL phrases and the set of bias phrases exposes the CTI-ASR model to phrase(s) that are otherwise infrequent and / or absent from training of the CTI-ASR model, thereby increasing the recognition robustness and / or accuracy of the CTI-ASR model.

[0007] In some implementations, a given speech-like representation (e.g., one or more textbased acoustic embeddings) can be generated by injecting a given text based NL phrase into at least a portion of the encoder portion of the CTI-ASR model. For instance, the one or more text based acoustic embeddings can be injected into a layer of the encoder which precedes a biasing layer of the encoder to generate the one or more text based acoustic embeddings.

[0008] In some implementations, the neural biaser portion of the CTI-ASR model includes afirst pass correlation portion and a second pass attention portion. The first pass correlation portion can process the one or more text based acoustic embeddings (e.g., generated using the encoder portion of the CTI-ASR model based on the given text based NL phrase) and a set of bias phrases, sampled from the set of text based NL phrases, to identify a subset of the bias phrases that correlate with the given text based NL phrase. For example, the first pass correlation portion of the CTI-ASR model can identify the top-K bias phrases, in the set of bias phrases, with the highest likelihood of being included in the given text based NL phrase.

[0009] After the subset of bias phrases (e.g., the top-K bias phrases) are identified, the secondpass attention portion of the neural biaser can process the subset of bias phrases (e.g., identified using the first pass portion of the neural biaser) and the one or more text based acoustic embeddings (generated using layer(s) of the encoder portion of the CTI-ASR model preceding the biasing layer of the encoder portion) to generate CTI-ASR cross attention output. The CTI-ASR cross attention output can be combined with the one or more text based acousticAttorney Docket No. GOOG-0520-WO-01 embeddings to generate one or more biased text based acoustic embeddings. The one or more biased text based acoustic embeddings focus the attention of the CTI-ASR model on the subset of bias phrases.

[0010] The biased text based acoustic embeddings can be injected into the biasing layer(s) ofthe encoder portion of the CTI-ASR model to generate an ASR generated text representation of the given text based NL phrase. Additionally or alternatively, one or more portions of the CTI- ASR model can be updated based on processing the ASR generated text representation and the given text based NL phrase.

[0011] In some implementations, the CTI-ASR model can be trained to jointly minimize the textbased ASR loss and an audio based ASR loss, where the text based ASR loss is generated based on the text based NL phrases and the audio based ASR loss is generated based on traditional audio-transcript pairs.

[0012] Accordingly, various implementations are directed towards training a CTI-ASR modelusing text based NL phrases and audio transcript pairs and / or utilization of such a CTI-ASR model. The text based NL phrases include bias phrase(s) that are infrequent and / or absent in traditional audio-transcript pairs. Accordingly, training the CTI-ASR model using the infrequent and / or absent bias phrase(s) found in the text based NL phrases increases the recognition robustness and / or accuracy of the CTI-ASR model. Computing resources (e.g., power, battery power, memory, processor cycles, etc.) can be conserved by increasing the recognition robustness and / or accuracy of the ASR model. For example, when a traditional ASR model generates an inaccurate text representation of an utterance spoken by the user, the user generally repeats the utterance and / or speaks an additional utterance. Further, action(s) may inadvertently be performed based on the inaccurate text representation. Computing resources used to process the repeated utterance, process the additional utterance, and / or perform the inadvertent action(s) are conserved when the CTI-ASR model generates an accurate text representation of the spoken utterance.Attorney Docket No. GOOG-0520-WO-01 Brief Description of the Drawings

[0013] FIG. 1 depicts a block diagram of an example of generating one or more biased textbased acoustic embeddings in accordance with various implementations.

[0014] FIG. 2 depicts a block diagram of an example of generating one or more biased audiobased acoustic embeddings in accordance with various implementations.

[0015] FIG. 3 depicts a block diagram of an example of generating a text based ASR loss inaccordance with various implementations.

[0016] FIG. 4 is a block diagram illustrating an example of generating an audio based ASRloss in accordance with various implementations.

[0017] FIG. 5 depicts a flowchart illustrating an example process of training a CTI-ASR modelin accordance with various implementations.

[0018] FIG. 6 depicts a flowchart illustrating an example process of generating a text basedASR loss in accordance with various implementations.

[0019] FIG. 7 depicts a flowchart illustrating an example process of generating an audiobased ASR loss in accordance with various implementations.

[0020] FIG. 8 depicts a block diagram of an example environment in accordance withvarious implementations.

[0021] FIG. 9 depicts an example architecture of a computing device, in accordance withvarious implementations. Detailed Description

[0022] Turning to the figures, FIG. 1 illustrates an example of generating one or more biasedtext based acoustic embeddings in accordance with various implementations. In some implementations, the example 100 includes a set of text based NL phrases 102 and a set of bias phrases 106 sampled from the set of text based NL phrases. The system can select a given text based NL phrase 104 from the set of text based NL phrases 102. In some implementations, the given text based NL phrase 104 includes one or more of the bias phrases in the set of bias phrases 106. In some other implementations, the given text based NL phrase 104 does not include any of the bias phrases in the set of bias phrases 106.Attorney Docket No. GOOG-0520-WO-01

[0023] In some implementations, the given text based NL phrase 104 can be processed togenerate one or more phoneme embeddings (not depicted) corresponding to the phrase (not depicted). The one or more phoneme embeddings can be processed using one or more layers of an encoder 108 portion of a CTI-ASR model to generate one or more text based acoustic embeddings 110.

[0024] In some implementations, the one or more text based acoustic embeddings 110 andthe set of bias phrases 106 can be processed using a first pass correlation portion 112 of a neural biaser to identify a subset of the bias phrases 114. The subset of the bias phrases 114 indicates a correlation between the one or more acoustic embeddings and the selected subset of bias phrases. For example, the subset of bias phrases 114 can identify the top-K bias phrases with the greatest likelihood of being correlated with the one or more acoustic embeddings.

[0025] In some implementations, the one or more text based acoustic embeddings 110 andthe subset of the bias phrases 114 can be processed using a second pass attention portion 116 of the neural biaser to generate cross attention output. The cross attention output and the one or more text based acoustic embeddings can be processed to generate one or more biased text based acoustic embeddings 118, where the cross attention vector focuses attention on the selected subset of bias phrases.

[0026] Fig. 2 illustrates an example of generating one or more biased audio based acousticembeddings in accordance with various implementations. In some implementations, the example 200 includes a set of audio-transcript pairs 202. Each audio-transcript pair includes audio data capturing a spoken utterance and a text representation of the spoken utterance. Additionally or alternatively, a set of audio-transcript bias phrases 206 can correspond to the set of audio-transcript pairs. For example, the set of audio-transcript bias phrases 206 can be sampled from the set of audio-transcript pairs 202.

[0027] In some implementations, a given instance of audio data capturing a spokenutterance 204 can be selected from the set of audio-transcript pairs 202. The given instance of audio data capturing the spoken utterance 204 can be processed using one or more layers of an encoder 108 to generate one or more acoustic embeddings corresponding to the spoken utterance 210.Attorney Docket No. GOOG-0520-WO-01

[0028] In some implementations, the one or more acoustic embeddings corresponding tothe spoken utterance 210 and the set of audio-transcript bias phrases 206 can be processed using the first pass correlation portion 112 of the neural biaser to identify a subset of the audio-transcript bias phrases 214. In some implementations, the subset of the audio-transcript bias phrases indicate a correlation between the one or more acoustic embeddings of the spoken utterance and the selected subset of the audio-transcript bias phrases. For example, the subset of the audio-transcript bias phrases can be the top-K bias phrases most likely to be included in the one or more acoustic embeddings of the spoken utterance.

[0029] In some implementations, the one or more acoustic embeddings corresponding tothe spoken utterance 210 and the subset of the audio-transcript bias phrases 214 can be processed using a second pass attention portion 116 of the neural biases to generate cross attention output. The cross attention output can be processed with the one or more acoustic embeddings of the spoken utterance to generate the one or more biased acoustic embeddings 218.

[0030] FIG. 4 illustrates an example of a speech transducer with a neural biaser. In someimplementations, the speech transducer 400 with the neural biaser 382 uses deferred NAM. In the illustrated example, the transducer 400 includes an encoder 108, a decoder 334, and a joint network 336. The encoder 108 transforms input speech features ^^ into acoustic embeddings. To recognize named entities in speaker’s context, each speech utterance ^^ is assigned ^^ biasphrases ^^ = {^^1, … ^^^^}. Any particular bias phrase ^^^^ may or may not exist in ^^. In someimplementations, the deferred NAM system performs 2-pass hierarchical biasing at layer ^^ (i.e., conformer layer 314) of the acoustic encoder. In the first pass the system can identify the ^^ most relevant biasing phrases based on the correlations between acoustic embeddings ^^^^and bias phrase embeddings ^^, followed by a max-pooling over ^^:

[0031] ^^ = ^^^^^^^^^^^^^^^^^^=1[^^^^^^^^^^^^^^(ℎ^^,^^^^^^ / √^^ )] (1)

[0032] where ^^ ∈ ^^^^+1, ^^^^ = {ℎ^^,1, … , ℎ^^,^^}, ℎ^^,^^ ∈ ^^^^is the sequence of acousticembeddings at the biasing layer ^^ (i.e., conformer layer 314). ^^ ∈ ^^(^^+1)×^^ is generated byfeeding ^^ through a phrase encoder 324 and appending the output with a ^^-dim ^^^^_^^^^^^^^Attorney Docket No. GOOG-0520-WO-01 embedding. In some implementations, the system selects the top-^^ phrases from ^^ corresponding to the ^^ largest values of the correlation vector ^^, and tokenize them into word-pieces ^^^^ = {^^ ^^1 , … , ^^ ^^1 },^^ ≥ ^^.

[0033] In the second pass, the system generates the cross -attention between acousticembeddings ^^^^ (as queries) and the word-piece embeddings ^^ ∈ ^^^^×^^of ^^^^(as keys and values ):

[0034] ^^ = ^^^^^^^^^^^^^^^^^^^^^^^^^^^^(^^ = ^^^^;^^, ^^ = ^^)(2)

[0035] Where ^^ ∈ ^^^^×^^ contains the ^^ context vectors generated by √ the cross-attention

[0036] The biased acoustic embeddings can be obtained by ^^ ^^^^^^^^^^ = ^^^^ + ^^^^^^, ^^^^ ∈ ^^,which are then propagated through the upper acoustic encoder layers. The decoder takes in previous labels to generate the current label embedding. In some implementations, the joint network combines the top-layer acoustic and label embeddings via a feed-forward network,and estimates posteriors of the ground-truth transcript ^^. A transducer loss ^^^^^^^^(^^) =−is calculated on supervised audio-transcript data ^^. Inaddition, to ensure the ^^ phrases selected in the first pass are most relevant to ^^, minimize a retrieval loss ^^^^^^^^^^computed by a cross-entropy between ^^ and the ground-truth label ^^ can be minimized. In some implementations, ^^ indicates the longest bias phrase present in ^^, or^^^^_^^^^^^^^ if none exists. Likewise, the system can calculate a word-piece retrieval loss basedon the correlations between ^^^^and ^^, and it can be added add to ^^^^^^^^^^(^^). In some implementations, the system can jointly minimize the transducer loss and the retrieval lossare loss weights.

[0037] FIG. 3 illustrates an example 300 of an example of a speech transducer with a neuralbiaser. In some implementations, the speech transducer 400 with the neural biaser 382 uses deferred NAM. In the illustrated example, the transducer 400 includes an encoder 108, a decoder 334, and a joint network 336. In some implementations, contextual text injection caninject text based NL phrases ^^^^^^^^^^ ∈ ^^^^^^^^^^ along with corresponding contextual biases phrases^^^^^^^^^^into a CTI-ASR model and its neural biasing component 328 to enhance its biasingAttorney Docket No. GOOG-0520-WO-01 capability. In some implementations, the system first tokenizes text based NL phrases into phoneme sequences and then replaces each token a fixed or a random number of times to getan unsampled sequence ^^^^ℎ^^ = {^^ ^^ℎ^^1 , … , ^^^^^^ℎ^^}. ^^^^ℎ^^can be fed into a phoneme encoder togenerate phoneme embeddings ^^^^ℎ^^ ∈ ^^^^×^^. To utilize unpaired text for neural biasing, thephoneme embeddings need to be injected to an acoustic encoder layer ^^^^^^^^^^that precedes thebiasing layer ^^, i.e., ^^^^^^^^^^ < ^^. In some implementations, the masking in CTI has to beperformed after the biasing layer instead of before the phoneme encoder. The system can propagate ^^^^ℎ^^through layer ^^^^^^^^^^to layer ^^ of the acoustic encoder to generate text-based acoustic embeddings ^^^^^^^^^^^^ ∈ ^^^^×^^.

[0038] In some implementations, a given text based NL phrase ^^^^^^^^^^ is associated with ^^bias phrases ^^^^^^^^^^ = {^^ ^^^^^^^^1 , … , ^^^^^^^^^^^^}, sampled from sentences within ^^^^^^^^^^’s training mini- batch. ^^^^^^^^^^^^may or may not exist in ^^^^^^^^^^. ^^^^^^^^^^^^and ^^^^^^^^^^undergo the two-pass contextual biasing procedure. In the first pass, correlations between ^^^^^^^^^^^^and ^^^^^^^^^^are used to select the top-K bias phrases for each of the text based NL phrases and to compute a text based retrieval loss. In the second pass, cross-attention between ^^^^^^^^^^^^and only the selected bias phrases is computed to generate text-based biased acoustic embeddings ^^^^^^^^^^^^,^^^^^^^^.

[0039] In some implementations, in the first pass, the system generates phrase embeddings^^^^^^^^^^ ∈ ^^(^^+1)×^^ by feeding ^^^^^^^^^^ into the phrase encoder and appending a ^^-dim ^^^^_^^^^^^^^embedding to the output. The system can then calculate ^^^^^^^^^^ ∈ ^^^^+1 by max-pooling thecorrelations between ^^^^^^^^^^^^and ^^^^^^^^^^over ^^:

[0041] Where ^^^^^^^^^^ signifies the correlations between ^^^^^^^^^^^^and each bias phrase (and ^^^^_^^^^^^^^) in ^^. From set ^^^^^^^^^^, the system can select the top-^^ bias phrases that have the ^^ largest correlation values in ^^^^^^^^^^, and tokenize them into word-pieces ^^^^^^^^^^,^^= ^^. In the second pass, ^^first cross- attends into the word- of ^^^^^^^^^^,^^ to compute text-based context vectors ^^^^^^^^^^ ∈ ^^^^×^^.^^^^^^^^obtained by adding ^^ to ^^^^^^^^^^Attorney Docket No. GOOG-0520-WO-01

[0044] Where ^^^^ ∈ ^^ is the biasing strength.

[0045] To make the task sufficiently difficult for the model with text input, the systemmasks a portion of text-based biased acoustic embeddings ^^^^^^^^^^,^^^^^^^^at layer ^^^^^^^^^^^^^^^^^^^^after thebiasing layer ^^, i.e., ^^^^^^^^^^ > ^^. The masked embeddings are then propagated through the restof the transducer, contributing to the calculation of a text-based ASR loss:

[0047] To identify the ^^ most relevant bias phrases from the text based NL phrases, thesystem can minimize a retrieval loss ^^^^^^^^^^(^^^^^^^^^^) computed as a cross-entropy between the correlation ^^^^^^^^^^and ground-truth label ^^^^^^^^^^. In some implementations, ^^^^^^^^^^can be assigned the longest bias phrase from ^^^^^^^^^^that appears in ^^^^^^^^^^. Otherwise, ^^^^^^^^^^is assigned ^^^^_^^^^^^^^ if none of the bias phrases are present in ^^^^^^^^^^. Similarly, a word-piece retrieval loss based on the same unpaired text is computed and then added to ^^^^^^^^^^.

[0048] The text-injected neural biasing model can be trained to jointly minimize the twoASR losses and two retrieval losses computed from audio-transcript paired data ^^ and the text based NL phrases ^^^^^^^^^^:

[0050] Where ^^ ^^^^^^^^^^ , ^^ ^^^^^^^^^^ ∈ ^^ are weights for the text-based losses. With CTI, the modelcan learn from significantly larger pool of text based NL phrases the ability to extract relevant bias phrases and / or improve the accuracy of named entity recognition based on them.

[0051] Conventional minimum word error rate (MWER) training minimizes the expectednumber of word errors on supervised training data, optionally including unpaired text. However, these methods reply on ^^-best hypotheses produced by a non-contextualized ASR system, limiting its ability to correct errors caused by contextual biasing. In some implementations, contextual text-injected (CTI) MWER training can generate the ^^-best hypotheses by feeding both audio-transcript pairs ^^ and text based NL phrases ^^^^^^^^^^through an ASR model with neural biasing, and then determining MWER losses based on these hypotheses.Attorney Docket No. GOOG-0520-WO-01

[0052] With CTI MWER, the system can generate the ^^-best hypotheses {^^1, … , ^^^^} byinputting speech ^^ and the bias phrases ^^ into the model and applying the two-pass neural contextual biasing. This can align the ^^-best generation with the actual inference, empowering the MWER training to rectify errors introduced by contextual biasing. The contextual MWER loss on supervised data is computed as the expected number of word errors on these ^^-best hypotheses:

[0054] Where ^^(^^^^,^^) is the number of word errors in a hypothesis ^^^^ compared to theground-truth transcript ^^.

[0055] Further, the system can inject text based NL phrases ^^^^^^^^^^ and bias phrases ^^^^^^^^^^into the CTI-ASR and its neural biasing model to calculate a text-based MWER loss:

[0057] Where {^^ ^^^^^^^^1 , … , ^^^^^^^^^^^^} are ^^-best hypotheses generated via the text-based two- pass neural biasing process described herein. Furthermore, the system can minimize the CTI- MWER loss across both ^^ D and ^^^^^^^^^^:

[0058] ^^^^^^^^−^^^^^^^^ = ^^^^^^ ^^^^^^^^(^^) + ^^^^^^^^^^^^^^^^^^^^^^(^^^^^^^^^^), (9)

[0059] are loss

[0060] To enhance the ASR performance in the absence of bias phrases, the system canselectively activate the biasing component during CTI MWER training. For example, the system can feed in empty bias phrases for a specific percentage of the training data.

[0061] FIG. 5 is a flowchart illustrating an example process 500 of training a CTI-ASR modelin accordance with various implementations disclosed herein. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. This system may include various components of various computer systems, such as one or more components of computing device 802 and / or computing system 910. Moreover, while operations of process 500 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, and / or added.Attorney Docket No. GOOG-0520-WO-01

[0062] At block 502, the system processes one or more text based NL phrases using the CTI-ASR model to generate a text based ASR loss. In some implementations, the system processes the one or more text based NL phrases using the CTI-ASR model to generate the text based ASR loss in accordance with process 600 described herein with respect to FIG.6.

[0063] At block 504, the system processes one or more audio-transcript pairs using the CTI-ASR model to generate an audio based ASR loss. In some implementations, the system processes the one or more audio-transcript pairs using the CTI-ASR model to generate the audio based ASR loss in accordance with process 700 described herein with respect to FIG.7.

[0064] At block 506, the system trains the CTI-ASR model to jointly minimize the text basedASR loss and the audio based ASR loss. Additionally or alternatively, the system can train the CTI-ASR model to jointly minimize the text based ASR loss, the audio based ASR loss, the text based retrieval loss 330, and the audio based retrieval loss 406.

[0065] FIG. 6 is a flowchart illustrating an example process 600 of generating a text basedASR loss in accordance with various implementations disclosed herein. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. This system may include various components of various computer systems, one or more components of computing device 802 and / or computing system 910. Moreover, while operations of process 600 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, and / or added.

[0066] At block 602, the system selects a given text based NL phrase from a set of textbased NL phrases. In some implementations, the set of text based NL phrases has a corresponding set of bias phrases sampled from the set of text based NL phrases. In some implementations, the given text based NL phrase includes one or more bias phrases in the set of bias phrases. In some other implementations, the given text based NL phrase does not include any of the bias phrases in the set of bias phrases.

[0067] At block 604, the system processes the given text based NL phrase to generate a setof speech features. In some implementations, the system can generate a phoneme representation of the given text based NL phrase. Additionally or alternatively, the system can process the phoneme representation using a phoneme encoder to generate one or more textAttorney Docket No. GOOG-0520-WO-01 based phoneme embeddings, where the one or more text based phoneme embeddings are the set of speech features. Additional or alternative speech features can be used in accordance with many implementations.

[0068] At block 606, the system propagates the set of speech features through one or morelayers of an encoder portion of the CTI-ASR model to generate one or more text based acoustic embeddings (e.g., one or more speech-like representations of the text based NL phrase). In some implementations, the system can inject the set of speech features into one or more layers of the encoder that precede one or more biasing layers of the encoder.

[0069] At block 608, the system processes the one or more text based acoustic embeddingsand the set of bias phrases using a first pass correlation portion of a neural biaser to generate a subset of the bias phrases, where the neural biaser is a portion of the CTI-ASR mode. In some implementations, the subset of bias phrases indicates a correlation between the one or more text based acoustic embeddings and the given text based NL phrase. For example, the system can select the top-K bias phrases (in the set of bias phrases) most likely to correspond to the one or more text based acoustic embeddings.

[0070] At block 610, the system processes the one or more text based acoustic embeddingsand the subset of bias phrases (e.g., the top-K bias phrases selected at block 608) using a second pass attention portion of the neural biaser to generate one or more biased text based acoustic embeddings. In some implementations, the second pass attention portion of the neural biaser can be used to restrict the attention of the encoder portion of the ASR model to only the subset of bias phrases.

[0071] At block 612, the system propagates the one or more biased text based acousticembeddings through the encoder portion of the CTI-ASR model to generate text based encoder output.

[0072] At block 614, the system processes the given text based NL phrase using a decoderportion of the CTI-ASR model to generate text based decoder output.

[0073] At block 616, the system generates a text based ASR loss based on processing thetext based encoder output and the text based decoder output.Attorney Docket No. GOOG-0520-WO-01

[0074] FIG. 7 is a flowchart illustrating an example process 700 of generating an audio basedASR loss in accordance with various implementations disclosed herein. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. This system may include various components of various computer systems, one or more components of computing device 802 and / or computing system 910. Moreover, while operations of process 700 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, and / or added.

[0075] At block 702, the system selects a given instance of audio data capturing a spokenutterance from a set of audio-transcript pairs. In some implementations, the set of audio- transcript pairs has a corresponding set of audio-transcript bias phrases. For example, the set of audio-transcript bias phrases can be sampled from the set of audio-transcript pairs.

[0076] At block 704, the system propagates the instance of audio data capturing the spokenutterance through one or more layers of an encoder portion of a CTI-ASR model to generate one or more audio based acoustic embeddings.

[0077] At block 706, the system processes the one or more audio based acousticembeddings and the set of audio-transcript bias phrases using a first pass correlation portion of a neural biaser to generate a subset of the audio-transcript bias phrases, where the neural biaser is a portion of the CTI-ASR model. In some implementations, the subset of audio- transcript bias phrases indicates a correlation between the one or more audio based acoustic embeddings and the given spoken utterance. For example, the system can select the top-K audio-transcript bias phrases (in the set of audio-transcript bias phrases) most likely to correspond to the one or more audio based acoustic embeddings.

[0078] At block 708, the system processes the one or more audio based acousticembeddings and the subset of the audio-transcript bias phrases (e.g., the top-K bias phrases selected at block 706) using a second pass attention portion of the neural biaser to generate one or more biased audio based acoustic embeddings. In some implementations, the second pass attention portion of the neural biaser can be used to restrict the attention of the encoder portion of the ASR model to only the subset of audio-transcript bias phrases.Attorney Docket No. GOOG-0520-WO-01

[0079] At block 710, the system propagates the one or more biased audio based acousticembeddings through the encoder portion of the CTI-ASR model to generate audio based encoder output.

[0080] At block 712, the system processes the NL text based representation of the spokenutterance using a decoder portion of the CTI-ASR model to generate audio based decoder output.

[0081] At block 714, the system generates an audio based ASR loss based on processing theaudio based encoder output and the audio based decoder output.

[0082] FIG. 8 is a block diagram of an example environment 800 that demonstrates variousaspects of the present disclosure, and in which implementations disclosed herein can be implemented is depicted. The example environment 800 includes a computing device 802, user interface input / output device(s) (not depicted), a CTI-ASR model 808, a set of text based NL phrases 804, a set of audio-transcript pairs 806, one or more additional or alternative components (not depicted), and / or combinations thereof. The computing device 802 includes CTI-ASR engine 808, CTI-ASR training engine 810 more additional or alternative engines (not depicted), and / or combinations thereof.

[0083] In some implementations, the computing device 802 and / or additional or alternativecomponents may be communicatively coupled with each other via one or more networks, such as one or more wired or wireless local area networks (“LANs,” including Wi-Fi LANs, mesh networks, Bluetooth, near-field communication, etc.) or wide area networks (“WANs”, including the Internet). In some implementations, the computing device 802 may include one or more user interface input / output devices (not depicted), which may include, for example, a physical keyboard, a touch screen (e.g., implementing a virtual keyboard or other textual input mechanisms), a microphone, a camera, a display screen, and / or speaker(s). The user interface input / output device(s) may be incorporated with one or more computing devices 802 of a user. For example, a mobile phone of the user may include the user interface input output devices; a standalone digital assistant hardware device may include the user interface input / output device; a first computing device may include the user interface input device(s) and a separate computing device may include the user interface output device(s); etc. In someAttorney Docket No. GOOG-0520-WO-01 implementations, all or aspects of computing device 802 may be implemented on a computing system that also contains the user interface input / output devices.

[0084] Some non-limiting examples of computing device 802 include one or more of: adesktop computing device, a laptop computing device, a standalone hardware device at least in part dedicated to an automated assistant, a tablet computing device, a mobile phone computing device, a computing device of a vehicle (e.g., an in-vehicle communications system, and in-vehicle entertainment system, an in-vehicle navigation system, an in-vehicle navigation system), or a wearable apparatus of the user that includes a computing device (e.g., a watch of the user having a computing device, glasses of the user having a computing device, a virtual or augmented reality computing device). Additional and / or alternative computing systems may be provided. Computing device 802 may include one or more memories for storage of data and software applications, one or more processors for accessing data and executing applications, and other components that facilitate communication over a network. The operations performed by computing device 802 may be distributed across multiple computing devices. For example, computing programs running on one or more computers in one or more locations can be coupled to each other through a network.

[0085] In some implementations, the set of text based NL phrases 804 can contain textwhich includes bias phrases which are infrequent or absent in the set of audio-transcript pairs 806.

[0086] In some implementations, CTI-ASR engine 808 can process audio data capturing anutterance spoken by a user, using the CTI-ASR model 802 to generate a text representation of the utterance. In some of those implementations, a set of contextual information corresponding to the user and / or the computing device 802 and the audio data capturing the utterance can be processed using the CTI-ASR model 802 to generate the text representation of the utterance.

[0087] In some implementations, CTI-ASR training engine 810 can process the set of textbased NL phrases 804 to generate a text based ASR loss. For example, the CTI-ASR engine 810 can generate a text based ASR loss in accordance with process 600 described herein with respect to FIG.6. Additionally or alternatively, the CTI-ASR training engine 810 can process theAttorney Docket No. GOOG-0520-WO-01 set of audio-transcript pairs 806 to generate an audio based ASR loss. For example, the CTI- ASR engine 810 can generate an audio based ASR loss in accordance with process 700 described herein with respect to FIG.7. Furthermore, CTI-ASR training engine 810 can update one or more portions of CTI-ASR model 802 based on the text based ASR loss and / or the audio based ASR loss. In some implementations, the CTI-ASR training engine 810 can train the CTI- ASR model 802 to jointly minimize the text based ASR loss and the audio based ASR loss. In some of those implementations, the CTI-ASR training engine 810 can train the CTI-ASR model 802 in accordance with process 500 described herein with respect to FIG.5.

[0088] Although FIG.8 is described with respect to a single client device having a singleuser, it should be understood that is for the sake of example and is not meant to be limiting. For example, one or more additional client devices of a user can also implement the techniques described herein. For instance, the computing device 802, the one or more additional client devices, and / or any other computing devices of the user can form an ecosystem of devices that can employ techniques described herein. These additional client devices and / or computing devices may be in communication with the computing device 802 (e.g., over one or more network(s)). As another example, a given client device can be utilized by multiple users in a shared setting (e.g., in a household environment, in an enterprise or work environment, in a hospitality environment, etc.).

[0089] FIG. 9 is a block diagram of an example computing device 910 that may optionally beutilized to perform one or more aspects of techniques described herein. In some implementations, one or more of a client computing device, and / or other component(s) may comprise one or more components of the example computing device 910.

[0090] Computing device 910 typically includes at least one processor 914 whichcommunicates with a number of peripheral devices via bus subsystem 912. These peripheral devices may include a storage subsystem 924, including, for example, a memory subsystem 925 and a file storage subsystem 926, user interface output devices 920, user interface input devices 922, and a network interface subsystem 916. The input and output devices allow user interaction with computing device 910. Network interface subsystem 916 provides anAttorney Docket No. GOOG-0520-WO-01 interface to outside networks and is coupled to corresponding interface devices in other computing devices.

[0091] User interface input devices 922 may include a keyboard, pointing devices such as amouse, trackball, touchpad, or graphics tablet, a scanner, a touchscreen incorporated into the display, audio input devices such as voice recognition systems, microphones, and / or other types of input devices. In general, use of the term “input device” is intended to include all possible types of devices and ways to input information into computing device 910 or onto a communication network.

[0092] User interface output devices 920 may include a display subsystem, a printer, a faxmachine, or non-visual displays such as audio output devices. The display subsystem may include a cathode ray tube (“CRT”), a flat-panel device such as a liquid crystal display (“LCD”), a projection device, or some other mechanism for creating a visible image. The displaysubsystem may also provide non-visual display such as via audio output devices. In general,use of the term "output device" is intended to include all possible types of devices and ways to output information from computing device 910 to the user or to another machine or computing device.

[0093] Storage subsystem 924 stores programming and data constructs that provide thefunctionality of some or all of the modules described herein. For example, the storage subsystem 924 may include the logic to perform selected aspects of the process of FIG.5, FIG. 6, and / or FIG.7, as well as to implement various components depicted in FIG.8.

[0094] These software modules are generally executed by processor 914 alone or incombination with other processors. Memory 925 used in the storage subsystem 924 can include a number of memories including a main random access memory (“RAM”) 930 for storage of instructions and data during program execution and a read only memory (“ROM”) 932 in which fixed instructions are stored. A file storage subsystem 926 can provide persistent storage for program and data files, and may include a hard disk drive, a floppy disk drive along with associated removable media, a CD-ROM drive, an optical drive, or removable media cartridges. The modules implementing the functionality of certain implementations may beAttorney Docket No. GOOG-0520-WO-01 stored by file storage subsystem 926 in the storage subsystem 924, or in other machines accessible by the processor(s) 914.

[0095] Bus subsystem 912 provides a mechanism for letting the various components andsubsystems of computing device 910 communicate with each other as intended. Although bus subsystem 912 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0096] Computing device 910 can be of varying types including a workstation, server,computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computing device 910 depicted in FIG.9 is intended only as a specific example for purposes of illustrating some implementations. Many other configurations of computing device 910 are possible having more or fewer components than the computing device depicted in FIG.9.

[0097] In situations in which the systems described herein collect or otherwise monitorpersonal information about users, or may make use of personal and / or monitored information), the users may be provided with an opportunity to control whether programs or features collect user information (e.g., information about a user’s social network, social actions or activities, profession, a user’s preferences, or a user’s current geographic location), or to control whether and / or how to receive content from the content server that may be more relevant to the user. Also, certain data may be treated in one or more ways before it is stored or used, so that personal identifiable information is removed. For example, a user’s identity may be treated so that no personal identifiable information can be determined for the user, or a user’s geographic location may be generalized where geographic location information is obtained (such as to a city, ZIP code, or state level), so that a particular geographic location of a user cannot be determined. Thus, the user may have control over how information is collected about the user and / or used.

[0098] These and other implementations of technology disclosed herein can optionallyinclude one or more of the following features.Attorney Docket No. GOOG-0520-WO-01

[0099] In addition, some implementations include one or more processors (e.g., centralprocessing unit(s) (CPU(s)), graphics processing unit(s) (GPU(s), and / or tensor processing unit(s) (TPU(s)) of one or more computing devices, where the one or more processors are operable to execute instructions stored in associated memory, and where the instructions are configured to cause performance of any of the aforementioned methods. Some implementations also include one or more computer readable storage media (e.g., transitory or non-transitory) storing computer instructions executable by one or more processors to perform any of the aforementioned methods. Some implementations also include a computer program product including instructions executable by one or more processors to perform any of the aforementioned methods.

Claims

Attorney Docket No. GOOG-0520-WO-01 CLAIMS What is claimed is:

1. A method implemented by one or more processors, the method comprising: injecting a given text based natural language (NL) phrase into an encoder portion of a contextual text injection (CTI) automatic speech recognition (ASR) model to generate one or more text based acoustic embeddings corresponding to the given text based NL phrase, wherein the given text based NL phrase is selected from a set of text based NL phrases, and injecting a set of bias phrases and the one or more text based acoustic embeddings into a first pass correlation portion of a neural biaser to select a subset of the bias phrases indicating a correlation between the selected subset of bias phrases and the one or more text based acoustic embeddings, wherein the set of bias phrases are sampled from the set of text based NL phrases, and wherein the neural biaser is a portion of the CTI-ASR model; injecting the subset of bias phrases and the one or more text based acoustic embeddings into a second pass attention portion of the neural biaser to generate one or more biased text based acoustic embeddings; processing the one or more biased text based acoustic embeddings using the encoder portion of the CTI-ASR model to generate text based encoder output; generating a text based ASR loss based on processing the text based encoder output and the given text based NL phrase; and updating one or more portions of the CTI-ASR model based on the text based ASR loss.

2. The method of claim 1, further comprising: processing audio data capturing a spoken utterance using the encoder portion of the CTI-ASR model to generate one or more acoustic embeddings corresponding to the spoken utterance,Attorney Docket No. GOOG-0520-WO-01 wherein the audio data has a corresponding NL text representation of the spoken utterance, and wherein the audio data capturing the spoken utterance and the NL text representation of the spoken utterance is selected from a set of audio-transcript pairs; processing a set of audio-transcript bias phrases and the one or more acoustic embeddings corresponding to the spoken utterance using the first pass correlation portion of the neural biaser to select a subset of the audio-transcript bias phrases indicating a correlation between the selected subset of audio-transcript bias phrases and the one or more acoustic embeddings, wherein the set of audio-transcript bias phrases corresponds to the set of audio- transcript pairs; processing the subset of audio-transcript bias phrases and the one or more acoustic embeddings using the second pass attention portion of the neural biaser to generate one or more biased acoustic embeddings; propagating the one or more biased acoustic embeddings using the encoder portion of the CTI-ASR model to generate audio based encoder output; generating an audio based ASR loss based on processing the audio based encoder output and the NL text representation of the spoken utterance; updating one or more portions of the CTI-ASR model based on the audio based ASR loss.

3. The method of claim 2, wherein the CTI-ASR model is trained to jointly minimize the text based ASR loss and the audio based ASR loss.

4. The method of claim 1, wherein selecting the subset of bias phrases, using the first pass correlation portion of the neural biaser comprises: for each bias phrase in the set of bias phrases, determining a probability measure indicating the likelihood the text based NL phrase includes the bias phrase; andAttorney Docket No. GOOG-0520-WO-01 selecting the subset of bias phrases based on one or more bias phrases with a corresponding probability measure that satisfies a threshold value.

5. The method of claim 1, wherein the at least one of the bias phrases, in the set of bias phrases, is included in the given text based NL phrase.

6. The method of claim 1, wherein none of the bias phrases, in the set of bias phrases, are included in the given text based NL phrase.

7. The method of claim 1, wherein the CTI-ASR model is a recurrent neural network transducer (RNN-T).

8. The method of claim 1, wherein injecting the given text based NL phrase into the encoder portion of the CTI-ASR model to generate the one or more text based acoustic embeddings comprises: generating one or more phoneme embeddings based on processing the given text based NL phrase; and injecting the one or more phoneme embeddings into a layer of the CTI-ASR encoder preceding a biasing layer of the CTI-ASR encoder.

9. The method of claim 8, wherein generating the one or more phoneme embeddings based on processing the given text based NL phrase comprises: generating a phoneme representation of the given text based NL phrase; and processing the phoneme representation using a phoneme encoder to generate the one or more phoneme embeddings.

10. The method of claim 1, wherein injecting the set of bias phrases and the one or more text based acoustic embeddings into the first pass correlation portion of the neural biaser to select the subset of the bias phrases comprises:Attorney Docket No. GOOG-0520-WO-01 processing the set of bias phrases using a phrase encoder to generate one or more phrase embeddings; and processing the one or more phrase embeddings and the one or more text based acoustic embeddings using the first past correlation portion of the neural biaser to select the subset of the bias phrases.

11. The method of claim 1, wherein injecting the subset of bias phrases and the one or more text based acoustic embeddings into the second pass attention portion of the neural biaser to generate the one or more biased text based acoustic embeddings comprises: processing the subset of bias phrases using a word-piece encoder to generate one or more word-piece embeddings; processing the one or more word-piece embeddings and the one or more acoustic embeddings to generate cross attention output; and processing the cross attention output and the one or more text based acoustic embeddings to generate the one or more biased text based acoustic embeddings.

12. The method of claim 1, wherein processing the one or more biased text based acoustic embeddings using the encoder portion of the CTI-ASR model to generate text based encoder output comprises: processing the one or more biased text based acoustic embeddings using a masking layer of the encoder portion of the CTI-ASR model; masking at least a portion of the one more biased text based acoustic embeddings; and generating the text based encoder output based on the one or more masked biased text based acoustic embeddings.

13. A method implemented by one or more processors, the method comprising: capturing audio data via one or more microphones of a client device, wherein the audio data captures an utterance spoken by a user;Attorney Docket No. GOOG-0520-WO-01 determining a set of contextual information corresponding to the user and / or the client device; processing the audio data using one or more layers of an encoder portion of a contextual text injection (CTI) automatic speech recognition (ASR) model to generate one or more acoustic embeddings; processing the one or more acoustic embeddings and the set of contextual information using a neural biaser to generate cross attention output; processing the cross attention output and the one or more acoustic embeddings to generate one or more biased acoustic embeddings, where the one or more biased acoustic embeddings are biased towards the contextual information corresponding to the user and / or the client device; processing the one or more biased acoustic embeddings using one or more additional layers of the encoder portion of the CTI-ASR model to generate encoder output, wherein the one or more additional layers of the encoder are distinct from the one or more layers of the encoder; and generating a text representation of the utterance based on the encoder output.

14. The method of claim 13, further comprising: processing the text representation of the utterance to generate action output; and causing the client device to perform one or more actions based on the action output.

15. The method of claim 13, wherein the CTI-ASR model is a recurrent neural network transducer (RNN-T).

16. The method of claim 13, wherein processing the one or more acoustic embeddings and the set of contextual information using the neural biaser to generate cross attention output comprises:Attorney Docket No. GOOG-0520-WO-01 processing the set of contextual information and the one or more acoustic embeddings using a first pass correlation portion of the neural biaser to identify a subset of the contextual information; and processing the subset of the contextual information and the one or more acoustic embeddings to generate the cross attention output.

17. The method of claim 13, further comprising: causing the client device to perform one or more actions based on processing the text representation of the utterance.

Images