Two-pass end-to-end speech recognition

Abstract

To perform two pass end-to-end speech recognition.SOLUTION: A two pass automatic speech recognition (ASR) model is used for performing a streaming on-device ASR to generate text representation of utterance captured in audio data. Various implementations include a first pass portion of the ASR model used for generating streaming candidate recognition of the utterance captured in audio data. The first pass portion may include a Recurrent Neural Network-transformer (RNN-T) decoder, for example. The various implementations include a second pass portion of the ASR model which is used for correcting streaming candidate recognition of utterance and generating text representation of utterance. The second pass portion may include a Listen Attend Spell (LAS) decoder, for example. The various implementations include a shared encoder shared between the RNN-T decoder and the LAS decoder.SELECTED DRAWING: Figure 2

Description

[Background technology]

[0001] Automated assistants (also known as "personal assistants," "mobile assistants," etc.) may be interacted with by users through a variety of client devices, such as smartphones, tablet computers, wearable devices, automotive systems, standalone personal assistant devices, etc. The automated assistant may receive input from the user, including verbal natural language input (i.e., speech), and respond by performing an action to control another device and / or by providing responsive content (e.g., visual and / or audible natural language output). An automated assistant interacted with through a client device may be implemented through the client device itself and / or through one or more remote computing devices (e.g., computing devices in the cloud) in network communication with the client device.

[0002] The automated assistant can convert audio data corresponding to the user's verbal utterance into corresponding text (or other semantic representation). For example, the audio data can be generated based on detection of the user's verbal utterance via one or more microphones of a client device that includes the automated assistant. The automated assistant can include a speech recognition engine that attempts to recognize various characteristics of the verbal utterance captured in the audio data, such as sounds (e.g., phonemes) produced by the verbal utterance, the order of sounds pronounced, speech rhythm, intonation, etc. Additionally, the speech recognition engine can identify text words or phrases represented by such characteristics. The text can then be further processed by the automated assistant (e.g., using a natural language understanding engine and / or a dialogue state engine) in determining response content for the verbal utterance. The speech recognition engine can be implemented by the client device and / or by one or more automated assistant components that are remote from the client device but in network communication with the client device. Summary of the Invention [Means for solving the problem]

[0003] Techniques described herein are directed to generating text representations of oral utterances using an end-to-end (E2E) two-pass automatic speech recognition (ASR) model. The two-pass ASR model includes a streaming first pass portion along with a non-streaming second pass portion. In numerous implementations, the first pass portion can include a recurrent neural network-transformer (RNN-T) decoder and generate candidate text representations of the oral utterances in a streaming manner. In various implementations, the non-streaming second pass portion (e.g., one that includes a listen attend spell (LAS) decoder) can be used to improve the initial candidate text representations generated in the first pass portion. For example, the LAS decoder can be used to rerank the candidate text representations generated using the RNN-T decoder. In numerous implementations, the two-pass ASR model can include a shared encoder, where the RNN-T decoder and the LAS decoder share a shared encoder. The use of a shared encoder can reduce the model size and / or provide improved computational efficiency of the two-pass ASR model, for example, compared to using a dedicated encoder for the RNN-T decoder and a dedicated encoder for the LAS decoder. In other words, the use of a shared encoder can enable efficient utilization of memory and / or computational resources when the two-pass ASR model is utilized in generating a textual representation of an oral utterance. This savings in memory and / or computational resources can be particularly impactful when the two-pass ASR model is stored and utilized by a client device that often has limited memory and / or computational resources. For example, the use of a shared encoder can enable on-device ASR to be performed on a client device whose limited resources may prevent on-device ASR using other models (at least in some circumstances, such as low battery conditions).

[0004] As an example, a client device may capture an oral utterance of "Turn on the living room lights" using one or more microphones of the client device. The oral utterance of "Turn on the living room lights" may be processed in a streaming manner using a shared encoder to generate a shared encoder output, which may be processed using an RNN-T decoder to generate a streaming first pass candidate text representation of "Turn on the living room lights." In response to determining that the user has finished speaking, the first pass candidate text representation along with the shared encoder output may be processed using an LAS decoder to generate a text representation of "Turn on the living room lights."

[0005] In some implementations, the two-pass model may be trained using a joint loss function that includes both the RNN-T loss and the LAS loss. In some of those implementations, the training process may include (1) training an encoder and an RNN-T decoder, (2) freezing the encoder trained in (1) and using the frozen encoder in training the LAS decoder, and (3) training the shared encoder, the RNN-T decoder, and the LAS decoder simultaneously using the joint loss function. In some versions of those implementations, the LAS decoder may be further trained using a minimum word error rate (MWER) training process.

[0006] Thus, various implementations illustrate techniques for using a two-pass ASR model in on-device streaming ASR. A conventional ASR system may require, for example, capturing audio data at a client device, transmitting the audio data and / or a representation of the audio data to a remote server, processing the audio data and / or the representation of the audio data at the remote server to generate a text representation of the speech captured in the audio data, and transmitting the text representation of the speech back to the client device. In contrast, an on-device ASR system generates a text representation of the speech locally at the client device without the need to transmit data to or from a remote server. In many implementations, the use of an on-device ASR including a two-pass ASR model can conserve computational resources (e.g., battery power, processor cycles, memory, etc.) when compared to conventional ASR by eliminating the need to transmit data to and receive data from a remote server. Additionally, on-device ASR can provide the benefit of user privacy by processing the audio data to generate a text representation locally without transmitting the audio data to a remote server. Additionally, on-device ASR systems provide improved reliability over traditional ASR models. For example, on-device ASR systems can generate a text representation of the audio data when a network connection is unavailable to transmit the audio data to a remote server, such as when a wireless network is down. In contrast, traditional ASR systems cannot generate a text representation of the audio data when a network connection is unavailable to transmit data to or from a remote server.

[0007] It should be appreciated that all combinations of the above concepts and additional concepts described in more detail herein are contemplated as being part of the subject matter disclosed herein, for example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 illustrates an example of generating a textual representation of an utterance according to various implementations disclosed herein. [Diagram 2] FIG. 2 illustrates an example of generating a text representation of an utterance using a two-pass ASR model, according to various implementations disclosed herein. [Diagram 3] FIG. 1 is a block diagram of an example environment in which implementations disclosed herein may be implemented. [Figure 4] 1 is a flowchart illustrating an example process of generating training instances for training a two-pass ASR model according to various implementations disclosed herein. [Diagram 5] 1 is a flowchart illustrating an example process for training a first pass portion of a two-pass ASR model according to implementations disclosed herein. [Figure 6] 1 is a flowchart illustrating an example process for training a second pass portion of a two-pass ASR model according to implementations disclosed herein. [Figure 7] 1 is a flowchart illustrating an example process for training a two-pass ASR model according to implementations disclosed herein. [Figure 8] 1 is a flowchart illustrating an example process for generating a textual representation of an utterance using a two-pass ASR model, according to implementations disclosed herein. [Figure 9] FIG. 1 is a block diagram of another example environment in which implementations disclosed herein may be implemented. [Figure 10] FIG. 1 illustrates an example architecture of a computing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Requirements for many applications of state-of-the-art speech recognition systems may include not only low word error rate (WER) but also low latency. Specifically, in many use cases, the system must be able to decode speech in a streaming manner and faster than real-time. Recently, streaming recurrent neural network transducer (RNN-T) end-to-end (E2E) models have been shown to be good candidates for on-device speech recognition, with improved WER and latency metrics compared to traditional on-device models. However, the models can still lag behind large state-of-the-art traditional models in quality. On the other hand, non-streaming E2E Listen, Attend and Spell (LAS) models have shown quality comparable to large traditional models. The techniques described herein bring the quality of E2E streaming models closer to that of traditional systems by incorporating a LAS network as a second pass component, while still complying with latency constraints.

[0010] E2E models for speech recognition weave an acoustic model (AM), a pronunciation model (PM), and a language model (LM) into a single network and have shown competitive results compared to traditional ASR systems with separate AM, PM, and LM. E2E models are particularly attractive for on-device ASR because they can outperform on-device traditional models of comparable size.

[0011] In many implementations, there are many challenges to having ASR run on-device with direct user interaction. First, the recognition results must be streaming; that is, words should appear on the screen as soon as they are spoken. Second, the model must have low latency (i.e., the delay between the user speaking and the text appearing), and therefore must run in real-time or faster than real-time on a mobile device. Third, the model must be able to take advantage of user context (e.g., list of contacts, song titles, etc.) to improve recognition accuracy. The RNN-T E2E model can meet these constraints.

[0012] Non-streaming E2E models such as Listen, Attend, and Spell (LAS) have shown competitive performance against large, conventional models. However, the LAS model must attend to the entire audio segment, making it difficult to use in interactive applications because it is non-streaming.

[0013] In two-pass decoding, a second pass model is often used to improve the initial output from the first pass model by using lattice rescoring or n-best reranking. Keeping the latency perceived by the user low while obtaining quality gains is the main challenge of applying the second pass model. Language model rescoring is commonly used for multi-pass decoding, but more recently has been used with LAS models to rescore hypotheses from the first pass traditional models. LAS decoders, which take acoustic information from the encoder and language model information from the previous prediction, can be considered strictly more powerful than second pass language models. Thus, the techniques described herein explore using LAS models for second pass processing.

[0014] Specifically, a two-pass architecture is explored in which the RNN-T and LAS decoders share an encoder network. Sharing the encoder allows for reduced model size and computational cost compared to having a dedicated encoder for the RNN-T decoder and a different dedicated encoder for the LAS decoder. During inference, the RNN-T decoder produces streaming predictions, while the LAS decoder finalizes the predictions. Various implementations explore tradeoffs by running the LAS decoder as a beam search versus rescoring hypotheses generated by the RNN-T decoder. In some implementations, computational cost can be reduced by running a first pass RNN-T model with adaptive beams and pruning the first pass lattice before rescoring.

[0015] In some implementations, the two-pass ASR architecture is as shown in Figure 2. Let a parameterized input acoustic frame be x=(x1...x T ), except that

[0016]

number

[0017] is the logarithmic mel-filterbank energy. In some implementations, for (d=80), T denotes the number of frames in x. In the first pass, for each acoustic frame x t is passed through a shared encoder, which includes a multi-layer LSTM, and the output is e t The output is e t At each time step in the streaming fashion, y r In the second pass, the output of the shared encoder for all frames e = (e1...e T ) is passed to the LAS decoder. During training, the LAS decoder calculates the output y lDuring decoding, the LAS decoder computes y r may further be used.

[0018] The use of the LAS decoder in two different decoding modes is explored in this work. Specifically, in the “second beam search” mode, the LAS decoder searches y r Ignore and output y from e only l Additionally or alternatively, when the LAS decoder is in "rescoring" mode, the top K hypotheses from the RNN-T decoder may be selected. The LAS decoder may process each sequence in supervised mode, paying attention to e, to compute a score, which combines the log-probability of the sequence and an attention coverage penalty. The sequence with the highest LAS score is chosen as the output sequence.

[0019] In some implementations, a two-pass model may be trained from a random initialization with the following combination loss, where y* represents the ground truth transcript:

[0020]

number

[0021] In the above formula, λ is a hyperparameter. In many implementations, λ can be set to 0.5 to equally weight the RNN-T loss and the LAS loss. In practice, training a two-pass model directly from scratch can be unstable, mainly because the losses for RNN-T and LAS are in significantly different ranges when training from scratch. Therefore, a multi-step process for training the model is used, namely, (1) train the encoder and the RNN-T model, (2) use the encoder trained in (1) to train the LAS decoder, where the trained encoder is frozen while training the LAS decoder, and (3) "deep finetuning", i.e., train a shared encoder and both decoders simultaneously with the joint loss.

[0022] One of the drawbacks of the loss in Equation 1 is that the second pass LAS decoder is optimized independently of the RNN-T decoder. This means that there may be a mismatch between the training and decoding strategies. To address this, an additional training step may be used to further improve the LAS decoder to minimize errors using a mean word error rate (MWER) training process. Specifically, for an input x, a ground truth transcript y*, and a given target sequence y with teacher enforcement, m The probability P(y m |x) (where y m is given by RNN-T, then m=r and y m Given m=l, where m is given by LAS, the pre-trained two-pass model can be improved as follows:

[0023] First, the system performs a beam search with one of the decoders m from the two-pass model to find a set of hypotheses H m={h1,...,h b}, where b is the beam size. To make the MWER training consistent with the decoding, m The generation of H depends on the target decoding mode. For the LAS decoder to be used in the "second beam search" mode, the system derives H by performing a beam search in x(m=l) together with the LAS decoder itself. m Meanwhile, for the LAS decoder to be used in the “rescoring” mode, the system computes H by performing a beam search with the first pass RNN-T decoder (m=r). m Calculate (x).

[0024] For each sequence y m ∈H m For W(y*,y m ) is y m Let be the number of word errors in

[0025]

number

[0026] But, H m Let be the average number of word errors for

[0027]

number

[0028] But, H m In m Let the relative word error rate be

[0029]

number

[0030] But the LAS decoder is H m Hypothesis y among all hypotheses in mLet denote the conditional probability of assigning,to,. The MWER loss is defined as:

[0031]

number

[0032] The LAS decoder can be trained to minimize the combination of the MWER loss and the maximum-likelihood cross-entropy loss.

[0033] L MWER (x,y*)+λ MLE logP(y*|x) (3)

[0034] where λ MLE is a hyperparameter. In many implementations, λ MLE may be set equal to 0.01.

[0035] Turning now to the figures, FIG. 1 illustrates an example of a two-pass ASR process according to numerous implementations. FIG. 1 includes a horizontal axis 100 representing time, including time points 102, 104, 106, and 108. A first time point 102 indicates where a user begins speaking an utterance, followed by a second time point 104 indicating that the user has finished speaking the utterance. Time point 106 provides an indication of when the two-pass ASR system determines that the user has finished speaking the utterance, which follows time 104. In the illustrated example, the duration between time 104 and time 106 (i.e., the amount of time it takes for the system to determine that the user has finished speaking) is much less than the duration between time 102 and time 104 (i.e., the amount of time the user has been speaking the utterance). Additionally or alternatively, time point 108 indicates when the system generates a textual representation of the utterance, where time 108 follows time 106.

[0036] In some implementations, the first pass streaming portion 110 of the ASR model is of duration between time 102 and time 106 (i.e., the time from when the user begins speaking the utterance to when the system determines that the user has finished speaking the utterance). In the illustrated example, the first pass streaming portion 110 of the ASR model begins when the user begins speaking. However, this is merely exemplary and the first pass streaming portion 110 can begin a short time after the user begins speaking (e.g., the first pass portion 110 can begin 0.01 seconds, 0.05 seconds, 0.1 seconds, 0.5 seconds, 1 second, and / or additional durations after the user begins speaking). Similarly, in the illustrated example, the first pass streaming portion 110 ends when the system determines 106 that the user has finished speaking the utterance. However, this is merely exemplary, and the first pass streaming portion 110 can end shortly after the system determines 106 that the user has finished speaking the utterance (e.g., the first pass portion 110 can end 0.01 seconds, 0.05 seconds, 0.1 seconds, 0.5 seconds, 1 second, and / or additional durations after the system determines that the user has finished speaking). In numerous implementations, the ASR system generates one or more candidate streaming text representations of the utterance in a streaming manner while the user is speaking the utterance. In some implementations, the first pass streaming portion 110 of the ASR model includes a shared encoder and an RNN-T decoder. Additionally or alternatively, the first pass streaming portion 110 of the ASR model may include additional encoders in addition to the shared encoder and the RNN-T decoder.

[0037] In some implementations, the second pass portion 114 of the ASR model is the duration between time 106 and time 108 (i.e., the time from when the system determines that the user has finished speaking the utterance to when the system generates the final textual representation of the utterance). In the illustrated example, the second pass portion 114 begins when the system determines that the user has finished speaking the utterance at 106. However, this is merely illustrative and the second pass portion 114 can begin a short time after the system determines that the user has finished speaking (e.g., the second pass portion 114 can begin 0.01 seconds, 0.05 seconds, 0.1 seconds, 0.5 seconds, 1 second, and / or additional durations after the system determines that the user has finished speaking). In some implementations, the second pass portion 114 includes a LAS decoder, where the LAS decoder can improve the candidate text representations generated in a streaming manner during the first pass by processing output generated in the first pass portion, such as output generated using a shared encoder and / or output generated using an additional encoder.

[0038] FIG. 2 is a block diagram illustrating an example process 200 for generating a text representation of audio data using a two-pass ASR model. Audio data 202 capturing speech spoken by a human is processed using a shared encoder 204 to generate a shared encoder output 206. In numerous implementations, the audio data 202 is captured using one or more microphones of a client device. A first pass portion of the ASR model may include processing the audio data 202 using the shared encoder 204 to generate the shared encoder output 206, as well as processing the shared encoder output 206 using an RNN-T decoder 208 to generate an RNN-T output 210. In some implementations, the first pass portion is streaming ASR, and the RNN-T output 210 may include one or more candidate text representations of the speech captured in the audio data 202 generated in a streaming manner.

[0039] In many implementations, the second pass portion is non-streaming and may include processing the shared encoder output 206 along with the RNN-T output 210 using a LAS decoder 212 to generate a LAS output 214. The LAS output 214 may be used to generate a final text representation 216 of the audio data 202. In many implementations, the LAS decoder 212 may be used to improve a candidate text representation of the audio data (i.e., the LAS decoder 212 may improve the RNN-T output 210) to generate the text representation 216 of the audio data.

[0040] 3 illustrates an example environment 300 in which implementations disclosed herein may be implemented. FIG. 3 includes a client device 302. In many implementations, the client device 302 may run an instance of an automated assistant (not shown). The client computing device 302 may be, for example, a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device in a user's vehicle (e.g., an in-vehicle communication system, and an in-vehicle entertainment system, an in-vehicle navigation system), a standalone interactive speaker, a smart appliance such as a smart TV, and / or a user's wearable device that includes a computing device (e.g., a user's watch with a computing device, a user's glasses with a computing device, a virtual or augmented reality computing device). Additional and / or alternative client computing devices may be provided.

[0041] The exemplary environment 300 includes a two-pass ASR engine 304, a shared encoder module 306, an RNN-T decoder module 310, an LAS decoder module 312, a training engine 316, a training instance engine 320, a two-pass ASR model 314, a training instance 318, and / or additional or alternative engines and / or models (not shown). The two-pass ASR engine 304, the shared encoder module 306, the RNN-T decoder module 310, and the LAS decoder module 312 are exemplary components with which the techniques described herein may interface. In some implementations, aspects of one or more of the engines and / or modules 304, 306, 310, 312, and 316 of FIG. 3 may be combined. For example, aspects of the shared encoder module 306 may be combined with aspects of the LAS decoder module 312.

[0042] The training instance engine 320 can generate the training instances 318. For example, the training instance engine 320 can generate one or more training instances, where each training instance includes audio data capturing an utterance and a ground truth text representation of the utterance. In some implementations, the training instances 318 can be generated by the training instance engine 320 according to the process 400 of FIG. 4 described herein.

[0043] The training engine 316 can train the two-pass ASR model 314 using the training instances 318. In some implementations, the two-pass ASR model 314 can include a shared encoder portion, an RNN-T decoder portion, and an LAS decoder portion (e.g., the two-pass ASR model can follow the model shown in FIG. 2 as described herein). For example, the training engine 316 can train the two-pass ASR model 314 by (1) training the shared encoder portion and the RNN-T decoder portion according to process 500 of FIG. 5, (2) training the LAS decoder portion using the shared encoder trained in step (1) according to process 600 of FIG. 6, where the shared encoder is frozen during the training of the LAS decoder, and (3) improving the trained shared encoder, the RNN-T decoder, and the LAS decoder using a common loss according to process 700 of FIG. 7.

[0044] Generating a text representation of an utterance is described herein with respect to process 800 of FIG. 8. The two-pass ASR engine 304 can generate a text representation of the captured audio data using the two-pass ASR model 314. In some implementations, the shared encoder module 306 of the two-pass ASR engine 304 can process the captured audio data using the shared encoder portion of the two-pass ASR model 314 to generate a shared encoder output in a streaming manner. The RNN-T decoder module 310 can process the shared encoder output using the RNN-T decoder portion of the two-pass ASR model 314 to generate one or more candidate text representations of the utterance. In some implementations, the RNN-T decoder module 310 can process the shared encoder output generated by the shared encoder module 306 in a streaming manner. The LAS decoder module 312 can generate a text representation of the utterance using the shared encoder module 306 by processing the candidate text representations of the utterance generated using the RNN-T decoder module along with any shared encoder outputs stored in a buffer.

[0045] 4 is a flowchart illustrating a process 400 for generating one or more training instances that may be used to train a two-pass ASR model, according to various implementations. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. The system may include one or more components, such as one or more processors (e.g., CPUs, GPUs, and / or TPUs). Although the operations of process 400 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0046] In block 402, the system selects an instance of audio data capturing speech. For example, the instance of audio data may capture a human speaking the utterance "Turn the thermostat up three degrees."

[0047] At block 404, the system determines a textual representation of the utterance. In many implementations, the textual representation of the utterance may be determined by a human reviewer of the selected instance of the audio data. Additionally or alternatively, the textual representation of the utterance may be determined using an additional ASR system. In many implementations, the textual representation of the audio data may be determined using an additional ASR system, where the generated textual representation is reviewed for accuracy by a human reviewer.

[0048] At block 406, the system stores training instances that include (1) instances of audio data capturing speech and (2) textual representations of the speech.

[0049] At block 408, the system determines whether to generate any additional training instances. If so, the system proceeds back to block 402 to select additional instances of audio data to capture speech before proceeding to blocks 404 and 406, which use the additional instances of audio data. In numerous implementations, the system can determine whether to generate additional training instances based on whether one or more conditions are met, such as whether a threshold number of training instances have been generated, whether there are any remaining unprocessed instances of audio data, and / or whether additional conditions are met. If at block 408 the system determines not to generate any additional training instances, the process ends.

[0050] 5 is a flowchart illustrating a process 500 for training a shared encoder and an RNN-T decoder for a two-pass ASR model, according to various implementations. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. The system may include one or more components, such as one or more processors (e.g., a CPU, a GPU, and / or a TPU). Although the operations of process 500 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0051] At block 502, the system selects training instances that include (1) instances of audio data that capture speech and (2) a ground truth representation of the speech. In some implementations, the training instances may be generated according to process 400 of FIG.

[0052] At block 504, the system may process the audio data portions of the training instances using a shared encoder to generate a shared encoder output, and process the shared encoder output using an RNN-T decoder to generate a predicted RNN-T output.

[0053] In block 506, the system generates an RNN-T loss based on the predicted RNN-T outputs and the ground truth text representation portions of the training instances.

[0054] At block 508, the system updates one or more portions of the shared encoder and / or one or more portions of the RNN-T decoder based on the RNN-T loss.

[0055] At block 510, the system determines whether to perform additional training. If so, the system proceeds back to block 502 and then selects additional training instances before proceeding to blocks 504, 506, and 508, which use the additional training instances before performing additional iterations of block 510. In some implementations, the system may decide to perform further training if there are one or more additional unprocessed training instances and / or if one or more other criteria have not yet been met. The other one or more criteria may include, for example, whether a threshold number of epochs have occurred and / or a threshold duration of training has occurred. Although process 500 is described with respect to non-batch learning techniques, batch learning may additionally and / or alternatively be utilized. If at block 510, the system determines not to perform additional training, the process ends.

[0056] 6 is a flowchart illustrating an example process 600 for training a LAS decoder for a two-pass ASR model, according to various implementations. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. The system may include one or more components, such as one or more processors (e.g., CPUs, GPUs, and / or TPUs). Although the operations of process 600 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0057] At block 602, the system selects a training instance that includes (1) audio data capturing speech and (2) a ground truth representation of the speech. In some implementations, the selected training instance is separate from the training instance used to train the shared encoder and / or RNN-T decoder portion of the two-pass ASR model. In some other implementations, the selected training instance is further utilized to train the shared encoder and / or RNN-T decoder portion of the two-pass ASR model. In some implementations, the training instance may be generated according to process 400 of FIG.

[0058] At block 604, the system processes the audio data portions of the training instances using a shared encoder and a LAS decoder to generate a predicted LAS output. In many implementations, the shared encoder has been previously trained by the system with an RNN-T decoder, such as the shared encoder trained with the RNN-T decoder in process 500 of FIG.

[0059] In block 606, the system generates a LAS loss based on the ground truth representations of the speech parts of the training instances and the predicted LAS outputs.

[0060] In block 608, the system does not update the shared encoder, but updates one or more portions of the LAS decoder based on the LAS loss. In other words, the previously trained shared encoder is frozen while training the LAS decoder.

[0061] At block 610, the system determines whether any additional training should be performed. If so, the system proceeds back to block 602, selects additional training instances, then performs the iterations of blocks 604, 606, and 608 based on the additional training instances, and then performs the additional iterations of block 610. In some implementations, the system may decide to perform further training if there are one or more additional unprocessed training instances and / or if one or more other criteria have not yet been met. The other one or more criteria may include, for example, whether a threshold number of epochs have occurred and / or a threshold duration of training has occurred. Although process 600 is described with respect to non-batch learning techniques, batch learning may additionally and / or alternatively be utilized. If at block 610, the system determines not to perform additional training, the process ends.

[0062] FIG. 7 is a flowchart illustrating an example process 700 for jointly training a shared encoder, an RNN-T decoder, and / or an LAS decoder to improve a trained two-pass ASR model according to various implementations. For example, the shared encoder and / or the RNN-T decoder may be trained according to the process 500 of FIG. 5, and / or the LAS decoder may be trained according to the process 600 of FIG. 6. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. The system may include one or more components, such as one or more processors (e.g., a CPU, a GPU, and / or a TPU). Although the operations of the process 700 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0063] At block 702, the system selects a training instance that includes (1) audio data capturing speech and (2) a ground truth text representation of the speech. In some implementations, the selected training instance is separate from the training instance used to initially train the shared encoder, RNN-T decoder, and / or LAS decoder of the two-pass ASR model. In some other implementations, the selected training instance is further utilized to initially train the shared encoder, RNN-T decoder, and / or LAS decoder of the two-pass ASR model. In some implementations, the training instance may be generated according to process 400 of FIG.

[0064] At block 704, the system processes the audio data using the initially trained shared encoder to generate a shared encoder output. For example, the shared encoder may be initially trained according to process 500 of FIG.

[0065] At block 706, the system processes the shared encoder output using an initially trained RNN-T decoder to generate a predicted RNN-T output. For example, the RNN-T decoder may be initially trained according to process 500 of FIG.

[0066] At block 708, the system processes the shared encoder output using the initially trained LAS decoder to generate a predicted LAS output. For example, the LAS decoder may be initially trained according to process 600 of FIG. 6.

[0067] At block 710, the system can generate a combined loss based on the predicted RNN-T outputs, the predicted LAS outputs, and the ground truth text representation portion of the training instances. For example, the system can:

[0068]

number

[0069] where x is an instance of audio data, y* is the ground truth text representation part of the training instance, and L RNNT is the RNN-T loss based on the predicted RNN-T output and the ground truth text representation of the training instances, and L LAS is the LAS loss, which is based on the predicted LAS output and the ground truth text representation portion of the training instances. Additional and alternative combination losses may be utilized according to numerous implementations.

[0070] In block 712, the system updates (1) one or more portions of the shared encoder, (2) one or more portions of the RNN-T decoder, and / or (3) one or more portions of the LAS decoder based on the combination loss.

[0071] At block 714, the system determines whether any additional training should be performed. If so, the system proceeds back to block 702 and selects additional training instances, then performs the iterations of blocks 704, 706, 708, 710, and 712 based on the additional training instances, and then performs the additional iterations of block 714. In some implementations, the system may decide to perform further training if there are one or more additional unprocessed training instances and / or if one or more other criteria have not yet been met. The other one or more criteria may include, for example, whether a threshold number of epochs have occurred and / or a threshold duration of training has occurred. Although process 700 is described with respect to non-batch learning techniques, batch learning may additionally and / or alternatively be utilized. If at block 714, the system determines not to perform additional training, the process ends.

[0072] 8 is a flowchart illustrating a process 800 for generating a textual representation of an utterance captured in audio data using a two-pass ASR model, where the two-pass ASR model includes a shared encoder, an RNN-T decoder, and an LAS decoder, according to various implementations. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. The system may include one or more components, such as one or more processors (e.g., a CPU, a GPU, and / or a TPU). Although the operations of process 800 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0073] At block 802, the system receives streaming audio data, where the audio data captures speech spoken by a human. In many implementations, the audio data is captured using one or more microphones of a client device. For example, the audio data may be captured using a microphone of a mobile phone.

[0074] At block 804, before the human finishes speaking, and in a streaming manner, the system generates one or more candidate text representations of the utterance by processing the audio data using a first pass portion of a two-pass ASR model. In numerous implementations, the system processes the audio data using the first pass portion of the two-pass ASR model by (1) processing the audio data using a shared encoder to generate a shared encoder output, and (2) processing the shared encoder output using an RNN-T decoder to generate one or more candidate text representations of the utterance.

[0075] In block 806, the system determines that the human has finished speaking. In some implementations, the system can determine that the human has finished speaking based on the end of a query token generated using an RNN-T decoder. In some implementations, the system can determine that the human has finished speaking based on the end of a query token generated by processing the audio data using an end pointer model (not shown).

[0076] In block 808, in response to determining that the human has finished speaking, the system uses a second pass portion of the two-pass ASR model to generate a text representation of the utterance by processing (1) the shared encoder output together with (2) one or more candidate text representations of the utterance using an LAS decoder to generate a text representation of the utterance. In some implementations, the LAS decoder may be used to rescore the top K hypotheses (i.e., the top K candidate text representations generated using the RNN-T decoder) generated using the RNN-T decoder. For example, the LAS decoder may be used to rescore the top three hypotheses, top five hypotheses, top 20 hypotheses, and / or an additional number of top hypotheses generated using the RNN-T decoder. For example, for each of the top hypotheses, the LAS decoder may be run in a teacher-enforced mode, attending to the shared encoder output, to generate an LAS score that combines the probability of the candidate hypothesis with an attention parameter. The system may select the hypothesis with the highest LAS score as the text representation of the utterance. Additionally or alternatively, a LAS decoder may be used to rescore the tree-based lattice, where the top candidate text representations of the utterance are represented as a lattice. For example, a LAS decoder may be used to process each lattice arc in a teacher-enforced mode, heeding the shared encoder output to update the probabilities in the arc. The system may identify the text representation of the utterance as the candidate text representation with the highest probability after the probabilities have been updated using the LAS decoder.

[0077] At block 810, the system performs one or more actions based on the textual representation of the utterance. For example, the system may render an output for a speaker based on the textual representation on a screen of a client device. In some implementations, the system may render an output based on one or more of the candidate textual representations, such as rendering an output based on the candidate textual representation with the highest probability while the human was speaking the utterance. In some such implementations, the system may render an output based on the textual representation by modifying the output rendered while the human was speaking the utterance. Additionally or alternatively, the system may render one or more responses to the textual representation of the utterance. For example, the system may render the response "75 degrees and sunny" in response to a human speaking the utterance "What's the weather like today?". Additionally or alternatively, the system may control one or more client devices based on the textual representation of the utterance. For example, the system may change the temperature on a networked smart thermostat to 72 degrees in response to the utterance "Set my thermostat temperature to 72 degrees." In various implementations, the system can perform additional and / or alternative actions based on the textual representation of the utterance.

[0078] Referring now to Figure 9, an example environment is shown in which various implementations may be implemented. Figure 9 is first described and includes a client computing device 902 that runs an instance of an automated assistant client 904. One or more cloud-based automated assistant components 910 may be implemented on one or more computing systems (collectively "cloud" computing systems) that are communicatively coupled to the client device 902 via one or more local area networks and / or wide area networks (e.g., the Internet), generally shown at 908.

[0079] An instance of an automated assistant client 904, through its interaction with one or more cloud-based automated assistant components 910, may form what appears from a user's perspective to be a logical instance of an automated assistant 900 with which the user may engage in human-computer interaction. Such an instance of an automated assistant 900 is illustrated in FIG. 9. Thus, it should be understood that in some implementations, a user engaging with an automated assistant client 904 running on a client device 902 may actually be engaging with their own logical instance of the automated assistant 900. For brevity and simplicity, the term "automated assistant" as used herein as "serving" a particular user will more likely refer to a combination of an automated assistant client 904 running on a client device 902 operated by the user and one or more cloud-based automated assistant components 910 (which may be shared among multiple automated assistant clients of multiple client computing devices). It should also be understood that in some implementations, the automated assistant 900 may respond to requests from any user, regardless of whether the user is actually "served" by that particular instance of the automated assistant 900.

[0080] The client computing device 902 may be, for example, a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device of the user's vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a standalone interactive speaker, a smart appliance such as a smart television, and / or a user's wearable device including a computing device (e.g., a user's watch having a computing device, a user's glasses having a computing device, a virtual or augmented reality computing device). Additional and / or alternative client computing devices may be provided. In various implementations, the client computing device 902 may operate one or more other applications, possibly in addition to the automated assistant client 904, such as a messaging client (e.g., SMS, MMS, online chat), a browser, etc. In some of those various implementations, one or more of the other applications may possibly interface with the automated assistant 900 (e.g., via an application programming interface) or include its own instance of an automated assistant application (which may also interface with the cloud-based automated assistant component 910).

[0081] The automated assistant 900 engages in a human-computer interaction session with the user via the user interface input and output devices of the client device 902. To protect user privacy and / or conserve resources, in many situations, the user must often explicitly invoke the automated assistant 900 before the automated assistant can fully process the verbal utterance. Explicit invocation of the automated assistant 900 may occur in response to some user interface input received at the client device 902. For example, the user interface input that may invoke the automated assistant 900 via the client device 902 may include, in some cases, the actuation of hardware and / or virtual buttons of the client device 902. Moreover, the automated assistant client may include one or more local engines 906, such as an invocation engine operable to detect the presence of one or more verbal invocation phrases. The invocation engine may invoke the automated assistant 900 in response to the detection of one of the verbal invocation phrases. For example, the paging engine may invoke the automated assistant 900 in response to detecting a verbal invocation phrase such as "Hey, assistant," "OK, assistant," and / or "assistant." The paging engine may continuously process a stream of audio data frames based on output from one or more microphones of the client device 602 to monitor for the occurrence of the verbal invocation phrase (e.g., when not in "inactive" mode). While monitoring for the occurrence of the verbal invocation phrase, the paging engine discards (e.g., after temporarily storing in a buffer) any audio data frames that do not contain the verbal invocation phrase. However, when the paging engine detects the occurrence of the verbal invocation phrase in the processed audio data frames, the paging engine may invoke the automated assistant 900.As used herein, "invoking" the automated assistant 900 may include causing one or more previously inactive features of the automated assistant 900 to be activated. For example, invoking the automated assistant 900 may include causing one or more local engines 906 and / or cloud-based automated assistant component 910 to further process the audio data frame based on which the invocation phrase was detected and / or one or more subsequent audio data frames (whereas prior to the invocation, no further processing of the audio data frame had occurred). For example, the local and / or cloud-based component may process the captured audio data using a two-pass ASR model in response to the invocation of the automated assistant 900.

[0082] The one or more local engines 906 of the automated assistant 900 are optional and may include, for example, the invocation engine described above, a local speech-to-text ("STT") engine (that converts captured audio to text), a local text-to-speech ("TTS") engine (that converts text to speech), a local natural language processor (that determines the semantic meaning of the audio and / or the text converted from the audio), and / or other local components. Because the client device 902 is relatively constrained in terms of computing resources (e.g., processor cycles, memory, batteries, etc.), the local engines 906 may have limited capabilities compared to any counterparts included in the cloud-based automated assistant component 910.

[0083] The cloud-based automated assistant component 910 leverages the virtually limitless resources of the cloud to perform more robust and / or more accurate processing of audio data and / or other user interface inputs compared to any counterpart of the local engine 906. Again, in various implementations, the client device 902 can provide audio data and / or other data to the cloud-based automated assistant component 910 in response to detection of a spoken invocation phrase by the invocation engine or some other explicit invocation of the automated assistant 900.

[0084] The illustrated cloud-based automated assistant component 910 includes a cloud-based TTS module 912, a cloud-based STT module 914, a natural language processor 916, a dialogue state tracker 918, and a dialogue manager 920. In some implementations, one or more of the engines and / or modules of the automated assistant 900 may be omitted, combined, and / or implemented in a component that is separate from the automated assistant 900. Additionally, in some implementations, the automated assistant 900 may include additional and / or alternative engines and / or modules. The cloud-based STT module 914 may convert the audio data to text, which may then be provided to the natural language processor 916.

[0085] The cloud-based TTS module 912 can convert text data (e.g., natural language responses formulated by the automated assistant 900) into computer-generated voice output. In some implementations, the TTS module 912 may provide the computer-generated voice output to the client device 902, for example, to be output directly using one or more speakers. In other implementations, the text data (e.g., natural language responses) generated by the automated assistant 900 may be provided to one of the local engines 906, which may then convert the text data into computer-generated speech that is output locally.

[0086] The natural language processor 916 of the automated assistant 900 processes the free-form natural language input and generates annotated output based on the natural language input for use by one or more other components of the automated assistant 900. For example, the natural language processor 916 can process the natural language free-form input, which is a text input, which is a conversion by the STT module 914 of audio data provided by a user via the client device 902. The generated annotated output may include one or more annotations of the natural language input and possibly one or more (e.g., all) of the terms of the natural language input.

[0087] In some implementations, the natural language processor 916 is configured to identify and annotate various types of grammatical information in the natural language input. In some implementations, the natural language processor 916 may additionally and / or alternatively include an entity tagger (not shown) configured to annotate entity references in one or more segments, such as references to people (including, for example, literary characters, celebrities, public figures, etc.), organizations, locations (real and imaginary), and the like. In some implementations, the natural language processor 916 may additionally and / or alternatively include a coreference resolver (not shown) configured to group or "cluster" references to the same entity based on one or more contextual cues. For example, the coreference resolver may be utilized to resolve the term "there" to "Hypothetical Cafe" in the natural language input "I liked the Hypothetical Cafe where we ate last time." In some implementations, one or more components of the natural language processor 916 may rely on annotations from one or more other components of the natural language processor 916. In some implementations, when processing a particular natural language input, one or more components of the natural language processor 916 may use related previous inputs and / or other relevant data other than the particular natural language input to determine one or more annotations.

[0088] In some implementations, the dialogue state tracker 918 may be configured to track "dialogue state," including, for example, the belief state of one or more users' goals (or "intent") during a human-computer interaction session and / or across multiple interaction sessions. In determining the dialogue state, some dialogue state trackers may seek to determine the most likely values ​​for slots instantiated in the interaction based on user and system utterances in the interaction session. Some techniques utilize a fixed ontology that defines a set of slots and a set of values ​​associated with those slots. Some techniques may additionally or alternatively be tailored to individual slots and / or domains. For example, some techniques may require training a model for each slot type in each domain.

[0089] The dialogue manager 920 may be configured to map, for example, the current dialogue state provided by the dialogue state tracker 918 to one or more "response actions" of a plurality of candidate response actions that are then executed by the automated assistant 900. The response actions may come in a variety of forms, depending on the current dialogue state. For example, early and intermediate dialogue states corresponding to turns in the dialogue session that occur before the final turn (e.g., when the user's desired task is finally performed) may be mapped to various response actions, including the automated assistant 900 outputting additional natural language dialogue. This response dialogue may include, for example, a request for the user to provide parameters (i.e., fill a slot) for some action that the dialogue state tracker 918 believes the user intends to perform. In some implementations, the response actions may include actions such as "request" (e.g., asking for parameters to fill a slot), "offer" (e.g., proposing an action or set of actions for the user), "select", "inform" (e.g., providing the user with the requested information), "no match" (e.g., informing the user that the user's last input was not understood), a command to a peripheral device (e.g., to turn off a light bulb), etc.

[0090] 10 is a block diagram of an example computing device 1010 that may be utilized to perform one or more aspects of the techniques described herein. In some implementations, one or more of the client computing devices and / or other components may comprise one or more components of the example computing device 1010.

[0091] The computing device 1010 typically includes at least one processor 1014 that communicates with a number of peripheral devices via a bus subsystem 1012. These peripheral devices may include, for example, a storage subsystem 1024 including a memory subsystem 1025 and a file storage subsystem 1026, user interface output devices 1020, user interface input devices 1022, and a network interface subsystem 1016. The input and output devices enable user interaction with the computing device 1010. The network interface subsystem 1016 provides an interface to external networks and is coupled to corresponding interface devices in other computing devices.

[0092] The user interface input devices 1022 may include a keyboard and pointing device such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touch screen integrated into a display, a voice recognition system, an audio input device such as a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computing device 1010 or onto a communications network.

[0093] The user interface output devices 1020 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a cathode ray tube ("CRT"), a liquid crystal display ("LCD"), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computing device 1010 to a user or to another machine or computing device.

[0094] The storage subsystem 1024 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 1024 may include logic for executing selected aspects of one or more of the processes of Figures 4, 5, 6, 7, and / or 8, as well as for implementing the various components shown in Figures 3 and / or 9.

[0095] These software modules are generally executed by the processor 1014 alone or in combination with other processors. The memory 1025 used within the storage subsystem 1024 may include several memories, including a main random access memory ("RAM") 1030 for storing instructions and data during program execution, and a read-only memory ("ROM") 1032 in which fixed instructions are stored. The file storage subsystem 1026 may provide persistent storage for program and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules implementing the functionality of some implementations may be stored by the file storage subsystem 1026 within the storage subsystem 1024 or within other machines accessible by the processor 1014.

[0096] The bus subsystem 1012 provides a mechanism for allowing the various components and subsystems of the computing device 1010 to communicate with each other as intended. Although the bus subsystem 1012 is shown diagrammatically as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0097] The computing device 1010 can be of different types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of the computing device 1010 shown in Figure 10 is only a specific example to illustrate some implementations. Many other configurations of the computing device 1010 are possible, having more or fewer components than the computing device shown in Figure 10.

[0098] In situations where the systems described herein may collect or utilize personal information about users (or as is often referred to herein as "participants"), the user may be provided with an opportunity to control whether a program or feature collects user information (e.g., information about the user's social network, social actions or activities, occupation, user preferences, or the user's current geographic location) or to control whether and / or how to receive content from a content server that may be more relevant to the user. Also, some data may be handled in one or more ways before being stored or used such that personally identifiable information is removed. For example, the user's identification information may be handled such that personally identifiable information cannot be determined about the user, or the user's geographic location may be generalized, in which case geographic location information is obtained (such as to the city, zip code, or state level) such that the user's specific geographic location cannot be determined. Thus, the user may have control over how information is collected and / or used about the user.

[0099] In some implementations, a processor-implemented method is provided, comprising receiving audio data including a sequence of segments and capturing an utterance spoken by a human speaker. The method further comprises, for each of the segments and in the sequence, processing the segment using a first pass portion of an automatic speech recognition ("ASR") model to generate a recurrent neural network transformer ("RNN-T") output. Processing each of the segments using the first pass portion of the ASR model comprises: processing the segment using the shared encoder portion to generate a shared encoder output; adding the shared encoder output as a next item in a shared encoder buffer; and processing the shared encoder output using the RNN-T decoder portion to generate a corresponding portion of the RNN-T output. The method further comprises determining one or more first pass candidate text representations of the utterance based on the RNN-T output; and determining that the human speaker has finished speaking the utterance. The method further includes, in response to determining that the human speaker has finished speaking the utterance, generating a LAS output using a second pass, listen attention spell ("LAS") decoder portion of the ASR model based on processing a shared encoder output from the shared encoder buffer with (a) the RNN-T output, and / or (b) one or more first pass candidate text representations of the utterance. The method further includes generating a final text representation of the utterance based on the LAS output.

[0100] These and other implementations of the technology may include one or more of the following features.

[0101] In some implementations, the method further includes rendering an output in a streaming manner based on at least one of the one or more first pass candidate text representations of the utterance, and updating the rendered output based on a final text representation of the utterance.

[0102] In some implementations, the method further includes controlling one or more devices based on the final textual representation of the utterance.

[0103] In some implementations, the method further includes determining content responsive to the final textual representation, and rendering the output based on the determined content.

[0104] In some implementations, a processor-implemented method is provided, comprising: receiving an utterance; processing the utterance using a multi-pass decoder including a first pass, a recurrent neural network transducer (RNN-T), end-to-end (E2E) decoder and a second pass, a listen-attend-and-spell (LAS) decoder; and outputting a final result of processing the utterance using the multi-pass decoder.

[0105] These and other implementations of the technology may include one or more of the following features.

[0106] In some implementations, processing the speech using a multi-pass decoder includes: transmitting the frame of audio data representing the speech to an RNN-T, E2E decoder for processing; providing one or more streaming results of processing the transmitted frame of audio data using the RNN-T, E2E decoder for output before a result of processing the speech is determined; transmitting the frame of audio data representing the speech to a LAS decoder for processing after the RNN-T, E2E decoder completes processing the transmitted frame; and determining a result of processing the speech based on a result of processing the transmitted frame of audio data using the LAS decoder. In some of those implementations, the method further includes transmitting one or more of the streaming results of processing the transmitted frame of audio data using the RNN-T, E2E decoder to the LAS decoder. The result of processing the speech may be determined further based on a result of processing the one or more streaming results using the LAS decoder. In some of these implementations, determining the result of processing the speech includes using a LAS decoder to select a particular streaming result selected from among one or more streaming results transmitted from the RNN-T, E2E decoder to the LAS decoder.

[0107] In some implementations, the speech is processed by the LAS decoder while the LAS decoder is in the second beam search mode.

[0108] In some implementations, the utterance is processed by the LAS decoder while the LAS decoder is in rescoring mode.

[0109] In some implementations, the multi-pass decoder further includes a single encoder that is shared with both the LAS decoder and the RNN-T, E2E decoder.

[0110] In some implementations, a multi-pass decoder is trained to optimize the minimum word error rate (MWER).

[0111] In some implementations, the multi-pass decoder is trained using a multi-step approach that involves training the RNN-T, E2E decoders independently, then stopping training the RNN-T, E2E decoders, and then training the LAS decoder independently.

[0112] Additionally, some implementations include one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) of one or more computing devices, where the one or more processors are operable to execute instructions stored in associated memory, where the instructions are configured to cause performance of any of the methods described herein. Some implementations also include one or more non-transitory computer-readable storage media that store computer instructions executable by the one or more processors to perform any of the methods described herein.

[0113] Although several implementations have been described and illustrated herein, various other means and / or structures for performing the functions and / or obtaining the results and / or one or more of the advantages described herein may be utilized, and each such variation and / or modification is deemed to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on the particular application or applications for which the teachings are used. Those skilled in the art will recognize, or be able to ascertain, at most using routine experimentation, many equivalents to the specific implementations described herein. Thus, the above implementations are presented by way of example only, and it should be understood that within the scope of the appended claims and their equivalents, implementations may be practiced other than as specifically described and claimed. Implementations of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent. [Explanation of symbols]

[0114] 100 horizontal axis 102 time point, first time point, time 104 Time, second time, time 106 Time 108 Time 110 First pass streaming part of the ASR model, first pass streaming part, first pass part 114 ASR model second pass part, second pass part 202 Audio Data 204 Shared Encoder 206 Shared Encoder Output 208 RNN-T Decoder 210 RNN-T Output 212 LAS Decoder 214 LAS Output 216 Final text representation, text representation of audio data 300 Example Environment 302, 902 Client device, client computing device 304 2-pass ASR engines, engines and / or modules 306 Shared Encoder Modules, Engines and / or Modules 310 RNN-T Decoder Modules, Engines and / or Modules 312 LAS Decoder Modules, Engines and / or Modules 314 2-pass ASR model 316 Training Engines, Engines and / or Modules 318 training instances 320 Training Instance Engine 900 Automated Assistant 904 Automated Assistant Client 906 Local Engine 910 Cloud-Based Automated Assistant Components 912 Cloud-based TTS module, TTS module 914 Cloud-based STT module, STT module 916 Natural Language Processor 918 Dialogue State Tracker 920 Dialogue Manager 1010 Computing devices 1012 Bus Subsystem 1014 Processor 1016 Network Interface Subsystem 1020 User interface output device 1022 User Interface Input Device 1024 Storage Subsystem 1025 Memory Subsystem, Memory 1026 File Storage Subsystem 1030 Main Random Access Memory ("RAM") 1032 Read-Only Memory ("ROM")

Claims

1. 1. A method implemented by one or more processors, comprising: training an automatic speech recognition (ASR) model, the step of training the ASR model comprising: selecting training instances comprising training audio data capturing training utterances and textual representations of said training utterances; processing portions of the training audio data of the selected training instances using a portion of a shared encoder of the ASR model to generate a training encoder output; processing the training encoder output using a recurrent neural network transformer (RNN-T) decoder portion of the ASR model to generate a training RNN-T output; generating candidate text representations of the training utterances based on the training RNN-T outputs; generating a first path loss based on comparing the candidate text representations of the training utterances with portions of the text representations of the training instances; updating the shared encoder and / or the RNN-T decoder based on the first path loss; using the shared encoder in training a listen-attention-spell (LAS) decoder portion of the ASR model; A method comprising:

2. using the shared encoder and the RNN-T decoder in training the LAS decoder portion of the ASR model, selecting further training instances comprising further training audio data capturing further training utterances and further textual representations of said further training utterances; processing the further training audio data using the shared encoder portion of the ASR model to generate further training encoder outputs; processing the further training encoder outputs using the RNN-T decoder portion of the ASR model to generate further training RNN-T outputs; processing the further training encoder output and the further training RNN-T output using the LAS decoder to generate further training LAS outputs; generating further training candidate text representations based on the further training LAS output; generating a further loss based on comparing portions of the further text representations of the further training instances with the further training candidate text representations; updating the LAS decoder based on the further loss.

2. The method of claim 1, comprising:

3. the further training RNN-T output includes a plurality of candidate text representations of the further training utterances; generating the further training candidate text representations based on the further training LAS output, re-ranking the plurality of candidate text representations based on the further training LAS output; selecting the further training candidate text representations of the further training utterances based on the reranking; 3. The method of claim 2, comprising:

4. the RNN-T decoder is part of a first pass decoder of the ASR model; 4. The method of claim 1, wherein the LAS decoder is part of a second pass decoder of the ASR model.

5. After training the ASR model, the method further includes using the ASR model to generate a text representation of a spoken utterance; The step of using the ASR model includes: receiving an instance of audio data capturing the spoken utterance; During the reception of said instance of audio data, prior to receiving a complete instance of audio data in a streaming manner, processing at least a portion of the instances of audio data using the shared encoder to generate an encoder output; processing the encoder output using the RNN-T decoder to generate an RNN-T output; 5. The method according to claim 1 , comprising:

6. determining whether the complete instance of audio data has been received; in response to determining that the complete instance of audio data has been received, processing the encoder output and the RNN-T output using the LAS decoder to generate a LAS output; generating the text representation of the spoken utterance based on the LAS output; 6. The method of claim 5, further comprising:

7. 1. A method implemented by one or more processors, comprising: training an automatic speech recognition (ASR) model, the step of training the ASR model comprising: selecting training instances comprising training audio data capturing training utterances and textual representations of said training utterances; processing portions of the training audio data of the selected training instances using a portion of a shared encoder of the ASR model to generate a training encoder output; processing the training encoder output using a recurrent neural network transformer (RNN-T) decoder portion of the ASR model to generate a training RNN-T output; processing the training RNN-T output and the training encoder output using a LAS decoder to generate a training listen attention spelling (LAS) output; generating a training text representation of the training utterance based on the training LAS output; generating a loss based on comparing the training LAS output with portions of the text representations of the selected training instances; updating the shared encoder, the RNN-T decoder, and / or the LAS decoder based on the loss; A method comprising:

8. training the ASR model, selecting further training instances comprising further training audio data capturing further training utterances and further textual representations of said further training utterances; processing portions of the further training audio data of the selected further training instances using a portion of the shared encoder of the ASR model to generate further training encoder outputs; processing the further training encoder outputs using the RNN-T decoder portion of the ASR model to generate further training RNN-T outputs; processing (a) the further training RNN-T output and (b) the training encoder output using a LAS decoder portion of the ASR model to generate further training LAS outputs; generating further training text representations of the further training utterances based on the further training LAS output; generating a further loss based on comparing the further training LAS output with portions of the further text representations of the selected further training instances; updating one or more further parts of the shared encoder, the RNN-T decoder, and / or the LAS decoder based on the further loss; 8. The method of claim 7, further comprising:

9. A computer program comprising instructions which, when executed by one or more processors of a computing system, cause the computing system to perform the method of any one of claims 1 to 8.

10. A client device including one or more processors, the one or more processors executing instructions stored in a memory of the client device to perform the method of any one of claims 1 to 8.

11. A computer-readable storage medium storing instructions executable by one or more processors of a computing system to perform the method of any one of claims 1 to 8.

Images