Efficient streaming non-recurrent on-device end-to-end model

Abstract

To provide an Automatic Speech Recognition (ASR) system.SOLUTION: An ASR system (109) includes a first encoder (210) configured to receive a sequence of acoustic frames (110) and generate a first high-level feature representation for corresponding acoustic frames within the sequence. The ASR system further includes a second encoder (220) configured to receive the first high-level feature representation generated by the first encoder at each of multiple output steps and to generate a second high-level feature representation for the corresponding first high-level feature frame. The ASR system further includes a decoder (204) configured to receive a second high-level feature representation generated by the second encoder at each of the multiple output steps and to generate a first probability distribution over possible speech recognition hypotheses. The ASR system further includes a language model (206) configured to receive the first probability distribution over possible speech recognition hypotheses and to generate a re-scored probability distribution (120).SELECTED DRAWING: Figure 2A

Description

[Technical Field]

[0001] The present disclosure relates to an efficient streaming non-recurrent on-device end-to-end model. [Background technology]

[0002] Automatic speech recognition (ASR) systems have evolved from multiple models, each with a specialized purpose, to an integrated model that uses a single neural network to directly map an audio waveform (i.e., an input sequence) to an output sentence (i.e., an output sequence). This integration has established a sequence-to-sequence approach, which generates a sequence of words (or graphemes) given a sequence of audio features. In an integrated structure, all components of the model may be trained together as a single end-to-end (E2E) neural network. Here, an E2E model refers to a model whose architecture is composed entirely of neural networks. A complete neural network functions without external and / or manually designed components (e.g., finite-state transducers, dictionaries, or text normalization modules). Furthermore, when training E2E models, these models generally do not require bootstrapping from decision trees or clocking from a separate system. These E2E automatic speech recognition (ASR) systems have made great progress, outperforming traditional ASR systems in several common benchmarks, including word error rate (WER). The architecture of an E2E ASR model is primarily application dependent. For example, some applications involving user interaction, such as voice search or on-device dictation, require the model to perform recognition in a streaming manner. Other applications, such as offline video captioning, do not require the model to stream and can fully utilize future context to improve performance. Additionally, existing E2E models are trained with only a small number of audio-text pairs compared to the over 100 billion text utterances used to train traditional models. Summary of the Invention [Means for solving the problem]

[0003] One aspect of the present disclosure provides an automatic speech recognition (ASR) model, including a first encoder configured to receive as input a sequence of acoustic frames and generate, at each of a plurality of output steps, a first high-order feature representation for a corresponding acoustic frame in the sequence of acoustic frames. The ASR model also includes a second encoder configured to receive as input the first high-order feature representation generated by the first encoder at each of the plurality of output steps and generate, at each of the plurality of output steps, a second high-order feature representation for the corresponding first high-order feature frame. The ASR model also includes a decoder configured to receive as input the second high-order feature representation generated by the second encoder at each of the plurality of output steps and generate, at each of a plurality of time steps, a first probability distribution over possible speech recognition hypotheses. The ASR model also includes a language model configured to receive as input the first probability distribution over the possible speech recognition hypotheses and generate, at each of the plurality of time steps, a rescored probability distribution over the possible speech recognition hypotheses.

[0004] Implementations of the present disclosure may include one or more of the following features: In some implementations, the second encoder generates the second high-order feature representation without receiving any acoustic frames as input; In some examples, the decoder is further configured to receive as input the first high-order feature representation generated by the first encoder at each of a plurality of output steps and to generate a second probability distribution over possible speech recognition hypotheses at each of a plurality of time steps; In these examples, the decoder may include a prediction network configured to receive as input the sequence of N previous non-blank symbols output by the final softmax layer at each time step, generate a respective embedding for each non-blank symbol in the sequence of N previous non-blank symbols, and generate an average embedding by averaging the respective embeddings. Here, the decoder also includes a joint network configured to receive as input the mean embedding generated by the prediction network at each of the plurality of output steps and one of the first high-order feature representation generated by the first encoder at each of the plurality of output steps when the ASR model is operating in streaming mode or the second high-order feature representation generated by the second encoder at each of the plurality of output steps when the ASR model is operating in non-streaming mode. The joint network is also configured to generate, at each of the plurality of output steps, one of a second probability distribution over possible speech recognition hypotheses when the ASR model is operating in streaming mode or a second probability distribution over possible speech recognition hypotheses when the ASR model is operating in non-streaming mode.

[0005] The predictive network may include a V2 embedded lookup table. In some cases, the first encoder may include a causal encoder including an initial stack of conformer layers. In some examples, the second encoder includes an acausal encoder including a final stack of conformer layers superimposed on the initial stack of conformer layers. In some implementations, the language model includes a neural language model. In these implementations, the neural language model may include a stack of conformer layers or transformer layers. The first encoder and the second encoder may be trained using hybrid autoregressive transducer factorization to facilitate the integration of language trained on text-only data.

[0006] Another aspect of the present disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations. The operations include receiving a sequence of acoustic frames as input to an ASR model. The operations also include performing streaming and non-streaming speech recognition on the sequence of acoustic frames using the ASR model, by generating, by a first encoder, a first high-order feature representation of a corresponding acoustic frame in the sequence of acoustic frames at each of a plurality of output steps, receiving the first high-order feature representation generated by the first encoder at each of the plurality of output steps as input to a second encoder, generating, by the second encoder, a second high-order feature representation for the corresponding first high-order feature frame at each of the plurality of output steps, receiving the second high-order feature representation generated by the second encoder at each of the plurality of output steps as input to a decoder, and generating, at each of a plurality of time steps, a first probability distribution over possible speech recognition hypotheses. The operations also include rescoring the first probability distribution over the possible speech recognition hypotheses using an external language model to generate a transcription of the utterance.

[0007] Implementations of the present disclosure may include one or more of the following features: In some implementations, the second encoder generates the second high-order feature representation without receiving any acoustic frames as input; In some examples, the operations of performing streaming and non-streaming speech recognition on a sequence of acoustic frames further include receiving the first high-order feature representation generated by the first encoder as input to a decoder at each of a plurality of output steps, and generating a second probability distribution over possible speech recognition hypotheses at each of a plurality of time steps; In these examples, at each of the plurality of time steps, the operations may further include receiving the sequence of N previous non-blank symbols output by the final softmax layer as input to a prediction network; and, for each non-blank symbol in the sequence of N previous non-blank symbols, generating, by the prediction network, a respective embedding; and generating, by the prediction network, an average embedding by averaging the respective embeddings. Here, the operations further include receiving as inputs to the joint network the mean embedding generated by the predictive network at each of the plurality of output steps and one of a first high-order feature representation generated by the first encoder at each of the plurality of output steps when the ASR model is operating in streaming mode or a second high-order feature representation generated by the second encoder at each of the plurality of output steps when the ASR model is operating in non-streaming mode; and generating, at each of the plurality of output steps, one of a second probability distribution over possible speech recognition hypotheses when the ASR model is operating in streaming mode or a first probability distribution over possible speech recognition hypotheses when the ASR model is operating in non-streaming mode.

[0008] The predictive network may include a V2 embedded lookup table. In some cases, the first encoder may include a causal encoder including an initial stack of conformer layers. In some examples, the second encoder includes an acausal encoder including a final stack of conformer layers superimposed on the initial stack of conformer layers. In some implementations, the language model includes a neural language model. In these implementations, the neural language model may include a stack of conformer layers or transformer layers. The first encoder and the second encoder may be trained using hybrid autoregressive transducer factorization to facilitate integration of language models trained on text-only data.

[0009] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0010] [Figure 1A] FIG. 1 is a schematic diagram of an exemplary audio environment using a cascade encoder and language model architecture for automatic speech recognition. [Figure 1B] FIG. 1 is a schematic diagram of an exemplary audio environment using a cascade encoder and language model architecture for automatic speech recognition. [Figure 2A] FIG. 1 is a schematic diagram of a cascade encoder and language model architecture. [Figure 2B] FIG. 1 is a schematic diagram of a cascade encoder and language model architecture. [Figure 2C] FIG. 1 is a schematic diagram of a cascade encoder and language model architecture. [Figure 3] FIG. 1 is a schematic diagram of an exemplary training process for training a cascade encoder model to learn consistent prediction and a language model to learn consistent rescoring. [Figure 4]1 is a flowchart of an exemplary configuration of operations for a method of implementing a cascade encoder and language model for streaming and non-streaming automatic speech recognition. [Figure 5] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE INVENTION

[0011] Like reference numbers in the various drawings indicate like elements.

[0012] End-to-end (E2E) automatic speech recognition (ASR) models are traditionally configured to operate in either streaming or non-streaming modes. Traditionally, E2E ASR models include an encoder and a decoder as their primary components. Applications involving end-user interaction, such as voice search or on-device dictation, may require the model to perform recognition in a streaming manner, where words are expected to be output as they are spoken with as little latency as possible. This precludes the use of models that use future context to improve accuracy, such as bidirectional LSTMs. In contrast, applications such as offline video captioning do not require streaming recognition and may fully utilize any available future context to improve performance. Furthermore, traditional E2E ASR models are trained with only a few audio-text pairs, compared to the over 100 billion text utterances used to train traditional models, and therefore perform poorly on long-tail proper nouns and rare words.

[0013] The implementations herein are directed to a single E2E ASR model that uses a cascade encoder capable of operating in both streaming and non-streaming modes combined with an on-device neural language model trained on text-only data. The cascade encoder includes a streaming encoder and a non-streaming encoder, while the single decoder of the ASR model is configured to learn to decode either the output from the streaming encoder or the output from the non-streaming encoder. In addition to ASR models, this architecture can be applied to other models, such as machine translation, that implement both streaming and non-streaming modes.

[0014] 1A and 1B are examples of voice environments 100, 100a-100b. In the voice environment 100, a user 104's way of interacting with a computing device, such as a user device 10, may be via voice input. The user device 10 (also commonly referred to as a device 10) is configured to capture voice (e.g., streaming audio data) from one or more users 104 in the voice environment 100. Here, streaming audio data may refer to utterances 106 by the users 104 that act as audible queries, commands for the device 10, and audible communications captured by the device 10. A voice-enabled system of the device 10 may process the queries or commands by answering the queries and / or executing / completing the commands through one or more downstream applications.

[0015] The user device 10 may represent any computing device associated with a user 104 and capable of receiving audio data. Some examples of the user device 10 include, but are not limited to, a mobile device (e.g., a mobile phone, a tablet, a laptop, etc.), a computer, a wearable device (e.g., a smart watch), a smart appliance, an Internet of Things (IoT) device, a vehicle infotainment system, a smart display, a smart speaker, etc. The user device 10 includes data processing hardware 12 and memory hardware 14 in communication with the data processing hardware 12, storing instructions that, when executed by the data processing hardware 12, cause the data processing hardware 12 to perform one or more operations. The user device 10 further includes an audio system 16 having audio capture devices (e.g., microphones) 16, 16a for capturing and converting speech 106 in an audio environment 100 into electrical signals, and audio output devices (e.g., speakers) 16, 16b for communicating audible audio signals (e.g., as output audio data from the device 10). Although the user device 10 implements a single audio capture device 16a in the illustrated example, the user device 10 may implement an array of audio capture devices 16a without departing from the scope of the present disclosure, and one or more capture devices 16a in the array may communicate with the audio system 16 without being physically present on the user device 10.

[0016] In the speech environment 100, an ASR system 109 implementing an automatic speech recognition (ASR) model 200 (also referred to as model 200) integrated with a language model (LM) 206 resides on the user device 10 of the user 104 and / or on a remote computing device 60 (e.g., one or more remote servers of a distributed system running in a cloud computing environment) that communicates with the user device 10 via a network 40. The user device 10 and / or the remote computing device 60 also include an audio subsystem 108 configured to receive utterances 106 by the user 104 captured by the audio capture device 16a and convert the utterances 106 into a corresponding digital format associated with input acoustic frames 110 that can be processed by the ASR system 109. In the example shown in FIG. 1A , the user 104 utters each utterance 106, and the audio subsystem 108 converts the utterances 106 into corresponding audio data (e.g., acoustic frames) 110 that are input to the ASR system 109. The model 200 then receives as input audio data 110 corresponding to the utterance 106 and generates / predicts as output a corresponding transcription 120 (also referred to as a recognition result / hypothesis 120) of the utterance 106. As described in more detail below (e.g., in FIG. 3), the model 200 may be trained in a single training stage to simplify the process of training the model 200 to operate in streaming and non-streaming modes. The model 200 also includes a decoder 204 (also referred to as a shared decoder 204) that is shared between the model's encoders, allowing the model 200 to be a single model capable of operating in streaming and non-streaming modes (as opposed to, e.g., two separate models, each dedicated to either streaming or non-streaming modes). For example, as shown in FIG. 1A, a digital assistant application 50 running on a user device 10 may need to stream speech recognition so that words, word pieces, and / or individual characters are displayed on a screen as they are spoken.Additionally, the user 104 of the user device 10 may have a low tolerance for latency when issuing queries to be executed by the digital assistant application 50. In these scenarios where the application requires minimal latency, the model 200 may operate in a streaming mode, providing streaming transcription capabilities as the user 104 utters the utterance 106. On the other hand, when the user 104 has a higher tolerance for speech recognition latency and / or the utterance 106 to be recognized relates to long-form speech (i.e., speech consisting of a complete paragraph or multiple sentences), the same model 200 may operate in a non-streaming mode, utilizing a predictive network to provide an accurate transcription 120 but with increased latency. Additionally, the user 104 requires that the ASR system 109 of the user device 10 be able to accurately identify rare words or long-tail proper nouns, which can be achieved by using the LM 206 with the model 200 to help bias the output of the model 200 when rare words or proper nouns are detected. Thus, the ASR system 109 may implement a single ASR model that includes cascaded encoders 210, 220 for multiple different speech recognition tasks, provides both streaming and non-streaming transcription capabilities without having to utilize a separately trained ASR model for each task, and uses the LM 206 to improve the accuracy of the transcription 120 when the utterance 106 contains long-tail proper nouns.

[0017] In some implementations, the model 200 first performs streaming encoding on the audio data 110 and then performs non-streaming encoding on the output of the streaming encoder. For example, in the illustrated example, the model 200 performs streaming speech recognition on the audio data 110 using a first encoder (i.e., a low-latency encoder (FIG. 2B)) to generate partial speech recognition results 120, 120a, and performs non-streaming speech recognition on the encoded audio data 110 using a second encoder (i.e., a high-latency encoder (FIG. 2C)) to generate final speech recognition results 120, 120b. In particular, the first encoder generates the partial speech recognition result 120a, and the second encoder waits for the output of the first encoder to generate the final speech recognition result 120b. Thus, the final speech recognition result 120b for the input utterance 106 may be delayed by a certain duration from the partial speech recognition result 120a for the input utterance.

[0018] The user device 10 and / or the remote computing device 60 also execute a user interface generator 107 configured to present a representation of a transcription 120 of the utterance 106 to the user 104 of the user device 10. As described in more detail below, the user interface generator 107 may display partial speech recognition results 120a in a streaming manner for time 1, followed by a final speech recognition result 120b for time 2. In some configurations, the transcription 120 output from the ASR system 109 is processed by a natural language understanding (NLU) module executing, for example, on the user device 10 or the remote computing device 60 to execute the user command / query specified by the utterance 106. Additionally or alternatively, a text-to-speech system (not shown) (e.g., executing on any combination of the user device 10 or the remote computing device 60) may convert the transcription 120 into synthesized speech for audible output by the user device 10 and / or another device.

[0019] In the example of FIG. 1A, a user in voice environment 100a interacts with a program or application 50 (e.g., digital assistant application 50a) of user device 10 that uses ASR system 109. For example, FIG. 1A shows user 104 communicating with digital assistant application 50, with digital assistant application 50a displaying digital assistant interface 18 on the screen of user device 10, illustrating a conversation between user 10 and the digital assistant of digital assistant application 50a. In this example, user 104 asks digital assistant application 50a, "What year was Serendipity released?" This question from user 104 is utterance 106 that is captured by voice capture device 16a and processed by audio system 16 of user device 10. In this example, audio system 16 receives and converts utterance 106 into acoustic frames 110 that are input to ASR system 109.

[0020] Continuing with the example, while model 200 receives acoustic frames 110 corresponding to utterance 106 as user 104 speaks, it encodes the acoustic frames 110 using a first encoder 210 (i.e., FIG. 2A ) and then decodes the encoded representation of acoustic frames 110 into partial speech recognition results 120 a using a decoder 204 (FIG. 2A ). During time 1, user interface generator 107 presents a representation of partial speech recognition results 120 a of utterance 106 to user 104 of user device 10 in a streaming manner via digital assistant interface 18, whereby words, word pieces, and / or individual characters are displayed on the screen as they are spoken.

[0021] After all (or a certain amount of) the acoustic frames 110 corresponding to the utterance 106 have been received and the first encoder 210 has encoded these acoustic frames 110, the second encoder 220 (i.e., FIG. 2A ) encodes the encoding output from the first encoder 210 to generate an encoding output for the set of acoustic frames 110 corresponding to the utterance 106 already encoded by the first encoder 210. The decoder 204 then decodes the acoustic frames 110 encoded by the second encoder 220 and processes the decoded acoustic frames 110 using the LM 206, which rescores the decoded acoustic frames to generate the final speech recognition result 120b. For example, once the first encoder 210 has encoded all of the acoustic frames 110 corresponding to the utterance 106 (as the acoustic frames 110 were received), the second encoder 220 encodes all of the acoustic frames 110 encoded by the first encoder 210. In this regard, the second encoder 220 can achieve greater context awareness by encoding across multiple encoded acoustic frames 110 (e.g., by receiving all representations of the acoustic frames 110 for the utterance 106) in a non-streaming manner, which may potentially adjust for or compensate for aspects of the utterance 106 that are lost or misinterpreted due to the streaming nature of the first encoder 210. In some examples, an indication such as an endpoint indicating that a user has finished the utterance 106 serves to trigger the encoder 220 of the model 200 to encode all acoustic frames 110. In other examples, the second encoder 220 encodes the acoustic frames 110 in parallel with the first encoder 210, which identifies the final endpoint of the utterance 106 and thereby triggers the second encoder 220 to output the final speech recognition result 120b. The endpoint identified by the first encoder 210 may simultaneously trigger a microphone closure event.During time 2, user interface generator 107 presents a representation of final speech recognition result 120b of utterance 106 to user 104 of user device 10 via digital assistant interface 18. In some implementations, user interface generator 107 replaces (or modifies) the representation of partial speech recognition result 120a with a representation of final speech recognition result 120b. In this example, user 104's utterance 106 includes the rare word "serendipity," for which model 200 was not trained. Thus, partial speech recognition result 120a, output by model 200 and displayed on-screen at time 1, incorrectly predicts that user 104's utterance 106 is "What year was serene released?" With the increased latency, the final speech recognition result 120b output by model 200 and displayed on the screen at time 2 improves speech recognition quality in terms of accuracy by making it clear that user 104 spoke "Serendipity." However, because user interface generator 107 displays the partial speech recognition results as the user makes utterance 106, the higher latency involved in generating and ultimately displaying final recognition result 120b is unlikely to be noticeable to user 104.

[0022] In some implementations, model 200 utilizes a prefetching technique that reduces latency by fetching speech recognition results before final speech recognition results 120b are available, where if partial speech recognition results 120a match the final speech recognition results, the response captured for the partial speech recognition results can be output immediately, eliminating the execution latency that typically occurs after the final speech recognition results are complete.

[0023] In the example shown in FIG. 1A , digital assistant application 50 a may use natural language processing to respond to questions posed by user 104. Natural language processing generally refers to the process of interpreting written language (e.g., partial speech recognition result 120 a and / or final speech recognition result 120 b) and determining whether the written language prompts an action. In this example, digital assistant application 50 a uses natural language processing to recognize that the question from user 104 is about the user's environment, and more specifically, about a song playing near the user. By recognizing such detailed information through natural language processing, the automated assistant returns a response 19 to the user's query, which in this case presents "Serendipity was released in 2001." In some configurations, natural language processing occurs on a remote computing device 60 that communicates with data processing hardware 12 of user device 10.

[0024] FIG. 1B illustrates another example of speech recognition by the ASR system 109 in a speech environment 100b. As shown in the example, a user 104 interacts with a voicemail application 50, 50b, which displays a voicemail application interface 18, 18b on the screen of the user device 10, to transcribe a voicemail left for the user 104 by Jane Doe. In this example, latency is not critical; transcription accuracy is critical when processing long-tail proper nouns or rare words. The ASR system 109 and the model 200 of the LM 206 can utilize the entire audio context by waiting until all of the acoustic frames 110 corresponding to the voicemail are generated. This voicemail scenario also illustrates how the model 200 can handle long utterances, since voicemails are often multiple sentences or even several paragraphs. The ability to handle long speech is particularly advantageous over other ASR models, such as two-pass models with LAS decoders. This is because such two-pass models often suffer from long-sentence problems (e.g., higher word deletion rates for long speech) when applied to long-sentence conditions. For example, by using an RNN-T decoder as decoder 204 in combination with cascaded encoder 202 (e.g., first encoder 210 and second encoder 220), model 200 works for both long and short speech without long-sentence suppression.

[0025] 1B, as described with respect to FIG. 1A, while model 200 receives acoustic frame 110, it encodes acoustic frame 110 using first encoder 210. After model 200 receives and encodes all of acoustic frame 110 using first encoder 210, model 200 provides the first encoder output as input to second encoder 220. Second encoder 220 encodes the first encoder output before decoder 204 generates the embedding, and LM 206 rescores decoder 204 output to generate final speech recognition result 120b. During time 3, user interface generator 107 presents a representation of final speech recognition result 120b via digital assistant interface 18b without first displaying partial speech recognition result 120a. For example, final speech recognition result 120b is a transcript of a long voicemail from Jane Doe that reads, "Do you want to watch Serendipity tonight? Give me a call back when you get this."

[0026] 2A-2C include exemplary models 200a-200c operating in various combinations of streaming and non-streaming modes. Specifically, each of the models 200a-200c includes a cascade encoder 202, a decoder 204, and an LM 206. The cascade encoder 202 refers to a model structure in which the encoding path includes two encoders 210, 220, which are cascaded such that the output of one encoder 210 provides the input of the other encoder 220 before decoding. Note that the encoders 210, 220 can be cascaded regardless of the underlying architecture for each encoder. In some examples, the encoders 210, 220 include a stack of 512-dimensional conformer layers. Causal convolutions and left-context attention layers may be used in each conformer layer to strictly restrict the model from using future inputs. A multi-head (e.g., eight-head) attention mechanism may be used in the self-attention layer. The cascade encoders 210, 220 may include 17 conformer layers, where the causal encoder 210 may include 15 conformer layers, while the non-causal encoder 210 may include two conformer layers that incorporate additional right context (e.g., 5.04 seconds). In some cases, transformer layers may be used instead of conformer layers.

[0027] In other implementations, one encoder is configured with an LSTM structure, and the other encoder is configured using a bidirectional LSTM layer or a conformal layer (e.g., a conformal transducer). In other words, the encoders 210, 220 may have different or similar architectures. For example, the cascade encoder 202 may be roughly similar to an acoustic model (AM) in a conventional ASR system and may include a recurrent network of stacked long short-term memory (LSTM) layers. Here, the first encoder 210 is a streaming encoder including a unidirectional long short-term memory (LSTM) layer, while the second encoder 220 is a non-streaming encoder including a bidirectional LSTM layer or a conformal layer. In the cascade encoder 202, both encoders 210, 230 include LSTM layers, and the second encoder 220, which receives the output of the first encoder 210, may utilize the LSTM layers of the first encoder, such that the second encoder 220 includes fewer LSTM layers than the first encoder 210 (and fewer LSTM layers than a fully non-streaming model). By having fewer LSTM layers, the cascade encoder 202 may reduce the number of more computationally expensive bidirectional layers, making the model 200 more streamlined than simply combining a traditional streaming model and a traditional non-streaming model.

[0028] Referring to FIG. 2A, a first encoder generates a sequence of d-dimensional feature vectors x = (x1, x2, ..., x T ), where

[0029]

number

[0030] The first encoder generates a first high-dimensional feature representation at each time step. This first high-dimensional feature representation is e sSimilarly, the second encoder 220 is cascaded to the first encoder 210 and outputs the first high-dimensional feature representation e s It is trained to receive as input and output a second higher-level representation, e a Both the first encoder 210 and the second encoder 220 are directly connected to and shared by the decoder 204. Thus, the decoder 204 receives the first high-level feature representation e s and the second higher-level feature representation e a It receives both as input.

[0031] The decoder 204 may include a recurrent neural network-transducer (RNN-T) architecture having a joint layer 230 and a prediction network 240. The prediction network 300 may be a non-recurrent prediction network 240. In some implementations, the prediction network includes a V2 embedding lookup table 240. The V2 embedding lookup table 240 is used to generate a V2 embedding lookup table (V2 lookup table) for the N previous non-blank subword unit prediction y i-1 , ..., y i-N Given the output, we define the embedding of each of these outputs as {d1, d2, ... d n}. In some examples, the N previous non-blank subword unit predictions are equal to the last 5 non-blank subword unit predictions. The V2 embedding lookup table 240 then uses SWISH activation to compute the embedding {d1, d2, ...d n} to the projection layer 242, generating the output l that is provided to the joint layer 230. In particular, the joint layer and the embedding lookup table 240 share the same dimensions, and therefore parameters may be shared between the joint layer 230 and the table 240, whereby the joint layer 230 is represented as the inverse of the lookup table 240. In non-streaming mode, the decoder 204 uses the joint layer 230 to embed the first and second high-dimensional feature representations e output by the cascade encoder 202. s , ea , as well as the average embedding d from the V2 embedding lookup table 240 to generate the decoder output. The decoder output is the unit before N previous non-blank symbols {y i-1 ,...,y i-N} and input x, the current subword unit y i The probability distribution P(y i |y i-1 ,...,y0,x). In non-streaming mode, the decoder output is then passed to an external language model (LM) 206, which rescores / improves the initial output from the decoder 204 using techniques such as lattice rescoring or n-best reranking. In other words, the decoder 204 generates predictions and the LM 206 refines the predictions.

[0032] In some implementations, the LM 206 includes a one-way conformer that examines a predetermined number of tokens (e.g., 31 tokens) for each output wordpiece model prediction. The conformer LM 206 may have a stack of layers (e.g., 12 layers), each with a model dimension of 768, a feedforward layer dimension of 2048, and 6 head attention. In these implementations, the conformer LM 206 is trained to predict 4,096 wordpieces.

[0033] When integrating an ASR model with an external LM, shallow fusion is generally required. However, overconfidence in the cascade encoder 202 and decoder 204 can make weighting difficult and often result in the deletion of many words. Therefore, a hybrid autoregressive transducer (HAT) model may be utilized to filter out the internal loss scores of the cascade encoder 202 and decoder 204, making integration with the LM 206 easier.

[0034] Although not shown, the model 200 may include a softmax layer that receives the output of the decoder 204. In some implementations, the softmax layer is separate from the decoder 204 and receives the output y r The output of the softmax layer is then used in a beam search process to select the orthographic element. In some implementations, the softmax layer is integrated with the decoder 204, whereby the output y r represents the output of the softmax layer.

[0035] The decoder 204 is configured to generate a probability distribution over possible speech recognition hypotheses at each output step. In other words, the joint layer 230 generates a probability distribution over possible speech recognition hypotheses at each output step (e.g., time step). Here, a "probable speech recognition hypothesis" corresponds to a set of output labels / symbols (also called "phonetic units"), each representing a grapheme (e.g., a symbol / character) or wordpiece in a specified natural language. For example, when the natural language is English, the set of output labels may include 27 symbols, e.g., one label for each of the 26 letters in the English alphabet and one label specifying a space. Thus, the joint network 230 may output a set of values ​​indicating the likelihood of occurrence of each of a given set of output labels. This set of values ​​may be a vector (e.g., a one-hot vector) and may indicate a probability distribution over the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, and possibly punctuation and other symbols), although the set of output labels is not so limited. For example, the set of output labels may include wordpieces and / or entire words in addition to or instead of graphemes. The output labels may also be other types of speech units, such as phonemes or subphonemes. The output distribution of the joint network 230 may include a posterior probability value for each of the different output labels. Thus, if there are 100 different output labels, each representing a different grapheme or other symbol, the output of the joint network 230 may include 100 different probability values, one for each output label. The probability distribution may then be used in a beam search process (e.g., with a softmax layer) to select and assign scores to candidate orthogonal elements (e.g., graphemes, wordpieces, and / or words) to determine the transcription 120.

[0036] In some examples, the cascade encoder 202 is configured with a stack of conformer layers. For example, the first causal encoder 210 may include an initial stack of 15 conformer layers, and the second acausal encoder 220 may include two additional conformer layers on top of the initial stack of 15 conformer layers. The two acausal conformer layers may accommodate an additional predefined duration of the right context (e.g., 5.04 seconds). The conformer layers of the cascade encoder may include 512-dimensional conformer layers and may use causal convolutional layers and left-context attention layers to severely restrict the model from using future inputs. Eight-head attention may be used in the self-attention layer, and the convolution kernel size may be equal to 15.

[0037] Within the decoder 204, the V2 embedding lookup table 240 may be a non-recurrent embedding prediction network with approximately 2 million parameters. In contrast, an LSTM-based prediction network includes approximately 23.4 million parameters. In some examples, the prediction network 240 includes an LSTM-based prediction network. Finally, the joint network 230 may include a single feedforward layer with 640 hidden units. The softmax layer may consist of a unified word piece or grapheme set generated using all unique word pieces or graphemes in the multiple training datasets 132, 132a-132n (FIG. 3).

[0038] The external LM 206 may include a conformer LM using one-way attention and a lookback attention context of 31 tokens for each output wordpiece model prediction. Here, the LM 206 may include 12 layers, each with a model dimension of 768 and a feedforward layer dimension of 2,048. The number of attention heads may be six. The conformer LM 206 may be trained to predict 4,096 wordpieces.

[0039] 2A, in some implementations, the model 200a operates in both streaming and non-streaming modes in parallel. When the model 200a operates in both streaming and non-streaming modes simultaneously, it first performs streaming speech recognition on the audio data 110 using the first encoder 210 to generate a first high-order representation e for both the second encoder 220 and the decoder 204. s The decoder 204 then generates partial speech recognition results 120, 120a. The model 200b also performs non-streaming speech recognition on the encoded audio data 110, and the second encoder 200 generates the first high-level representation e received from the first encoder 210. s Using the second higher-order representation e a The decoder 204 then generates a speech recognition result, which is rescored by the LM 206 to generate the final speech recognition result 120, 120b. As indicated by time, the first encoder 210 generates the partial speech recognition result 120a, and the second encoder 220 waits for the output of the first encoder 210 to generate the final speech recognition result 120b. Thus, the final speech recognition result 120b for the input utterance 106 may be delayed relative to the partial speech recognition result 120a for the input utterance. As described above, the first encoder may identify an endpoint of the utterance 106 that triggers a microphone closure event and triggers output of the final speech recognition result 120b.

[0040] 2B, in some implementations, the model 200b operates only in streaming mode. This may occur, for example, when the user 104 uses applications such as voice search or on-device dictation, where the lowest possible latency is required. Here, the model 200b performs streaming speech recognition on the audio data 110 using only the first encoder 210 to generate a first high-order representation e for the decoder 204. sThe decoder 204 then generates speech recognition results, which are rescored by the LM 206 to generate partial speech recognition results 120, 120a. Because the streaming mode of the cascade encoder model 200b quickly generates the partial speech recognition results 120, 120a, the inaccuracy of the word "playing" is generally acceptable to the user 104.

[0041] Referring to FIG. 2C, in some implementations, the model 200c operates only in non-streaming mode. Non-streaming mode may occur, for example, when a user 104 views a transcription of a voicemail left on their phone (FIG. 1B). As explained above, in this type of application, it is advantageous to use future context to improve performance at the expense of increased processing time. Here, the cascade encoder model 200c first uses a first encoder 210 to generate a first higher-order representation e for a second encoder 220. s , whereas the decoder 204 generates the first high-level representation e s The cascade encoder model 200 then performs non-streaming speech recognition on the encoded audio data 110, and the second encoder 220 decodes the first high-order representation e received from the first encoder 210. s Using the second higher-order representation e a The decoder 204 then generates speech recognition results, which are rescored by the LM 206 to generate the final speech recognition results 120, 120b. Because the non-streaming mode of the model 200c accurately generates the partial speech recognition results 120, 120b, the time delay to display the accurate transcription is generally acceptable to the user 104.

[0042] FIG. 3 shows an example of a training process 300 for training the cascade encoder and language model 200 to be operable for both streaming and / or non-streaming. In some configurations, the training process 300 is executed on the remote computing device 60 of FIGS. 1A and 1B. The training process 300 retrieves a plurality of training utterances 132, 132a-132n stored in a sample database 130 and trains the model 200 on the training utterances 132. The training process 300 also retrieves a plurality of text-only training samples 142, 142a-142n stored in a sample database 140 to train the LM 206 of the model 200. The sample databases 130, 140 may reside on the memory hardware of the remote computing device 60. As described above with respect to FIG. 2A, the first encoder 210 and the second encoder 220 can share the same decoder 204 and be trained in a single stage, simplifying the training process 300. This means that the output of the direct streaming encoder 210 (e.g., the first high-level representation e s ), which means that we may train the non-streaming encoder 220 on

[0043] As shown in Figure 3, there are two processing paths for model 200: one path for the streaming mode of model 200b (shown in Figure 2B) and one path for the non-streaming mode of model 200c (shown in Figure 2C). Because there are two input processing paths in the training process 300, the loss of the cascade encoder model includes two loss functions. Specifically, the loss for the streaming mode of model 200b is generally defined as the sum of the negative logarithms of the probabilities corresponding to the probability distributions over the possible speech recognition hypotheses given the training utterance 132 as input. That is, the cascade encoder model loss from the first encoder 210 connection to the decoder 204 can be expressed as follows:

[0044]

number

[0045] The cascade encoder model loss for non-streaming mode is also generally defined as the sum of negative logarithms of probabilities corresponding to probability distributions over possible speech recognition hypotheses given the training utterance 132. Thus, the cascade encoder model loss from the second encoder 220 connection to the decoder 204 can be expressed as:

[0046]

number

[0047] Based on these expressions in Equation (1) and Equation (2), the total loss between two input paths is calculated as the weighted sum of each input path loss: L = λL s + (1 - λ)L a (3) In this case, λ is a weighting term. In the training process 300, training the cascade encoders together involves minimizing the weighted sum of the losses between both input processing paths.

[0048] At each step time during the training process 300, training can be performed either in a streaming or non-streaming manner for each training utterance 132. In other words, the input processing path is probabilistically selected to either train cascade encoder model 200b or train cascade encoder model 200c. By sampling the training utterances 132, the training process only needs to calculate the loss once per training utterance 132 at each training step, significantly speeding up the training process 300. In some implementations, if longer training times are acceptable, an alternative training process is used to train each input processing path with each training utterance and calculate the losses of both cascade encoder model 200b and cascade encoder model 200c for each training utterance 132 at each training step.

[0049] In the illustrated example, training utterances 132b and 132c are selected as training utterances for training a first processing path represented by cascade encoder model 200b. Cascade encoder model 200b receives training utterances 132b and 132c, and first encoder 210 converts training utterances 132b and 132c into a first high-level feature representation (e.g., audio embedding) as output. Decoder 204 then receives the first high-level feature representation of training utterances 132b and 132c as input and generates an output that is tested for accuracy. Similarly, training utterances 132a and 132d are selected as training utterances for training a second processing path represented by cascade encoder model 200c. The cascade encoder model 200c receives training utterances 132a, 132d, with a first encoder converting the training utterances 132a, 132d into a first high-level feature representation (e.g., audio embedding) as output. A second encoder 220 receives the first high-level feature representation of the training utterances 132a, 132d as input and generates a second high-level feature representation of the training utterances 132a, 132d as output. A decoder 204 then receives the second high-level feature representation of the training utterances 132a, 132d as input and generates an output that is tested for accuracy. This ensures that the model 200 learns to operate in either streaming or non-streaming mode during inference.

[0050] As mentioned above, integrating the training of the cascade encoder 202 and decoder 204 with the LM 206 during the training process 300 may result in a higher deletion rate when performing shallow fusion using the following equation: y * = arg max_ y [log p(y|x) + λ1log plm(y)] (4) where λ1 includes the weights assigned to the LM 206, and plm(y) denotes the external LM 206. To avoid high deletion rates caused by shallow fusion, techniques such as coverage penalty and blank scaling are used. Furthermore, HAT factorization reduces the internal language model score p of the model 200 so that the effective score of the model 200 can be expressed as ILM Suggest a way to exclude (y). log p(x|y)≒log p(y|x) - log plm(y) (5) Thus, the HAT factorization allows the model 200 to be integrated with an external LM 206 without requiring a coverage penalty, as follows: y * = arg max _y [λ1log p(y|x) - λ2log pilm(y) + log plm(y)] (6) In the above equation, λ1 and λ2 denote the weights assigned to the external LM 206 and the internal language model, respectively. By using HAT factorization during the training process 300, the LM 206 is better integrated with the cascaded encoder 202 and decoder 204.

[0051] The LM206 may be trained with text-only data containing over 100 billion utterances across multiple domains. Rare words in the text-only data may be identified. For example, words that occur five or fewer times may be identified as rare words. Additionally, words with unusual pronunciations given their spelling may be identified. These rare words and words with unusual pronunciations may be combined to form a long-tail set of audio-text pairs for training the ASR model 200.

[0052] 4 includes a flowchart of an example configuration of operations for a method 400 of performing streaming and non-streaming speech recognition using the cascade encoder model 200. The method 400 includes, at operation 402, receiving a sequence of acoustic frames 110 as input to the cascade encoder model 200. The method 400 further includes, at operation 404, performing streaming and non-streaming speech recognition on the sequence of acoustic frames 110 using the cascade encoder model.

[0053] The method 400 includes generating, by the first encoder 210, a first high-order feature representation for a corresponding acoustic frame 110 in the sequence of acoustic frames 110 at each of a plurality of output steps, at operation 406. The method 400 further includes receiving, as input to the second encoder 220, the first high-order feature representation generated by the first encoder 210 at each of the plurality of output steps, at operation 408. The method 400 also includes generating, by the second encoder 220, a second high-order feature representation for the corresponding first high-order feature frame at each of the plurality of output steps, at operation 410. At operation 414, the method 400 further includes generating, at each of the plurality of time steps, a first probability distribution over possible speech recognition hypotheses, and then rescoring the first probability distribution over the possible speech recognition hypotheses using the external language model 206 to generate a transcription 120 of the utterance 106.

[0054] 5 is a schematic diagram of an exemplary computing device 500 that may be used to implement the systems (e.g., audio subsystem 108, ASR system 109, user interface generator 107, and / or model 200) and methods (e.g., method 400) described herein. Computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The illustrated components, their connections and relationships, and their functionality are intended to be exemplary only and are not intended to limit the implementation of the invention(s) described and / or claimed herein.

[0055] Computing device 500 includes a processor 510 (e.g., data processing hardware), a memory 520 (e.g., memory hardware), a storage device 530, a high-speed interface / controller 540 that connects to memory 520 and a high-speed expansion port 550, and a low-speed interface / controller 560 that connects to a low-speed bus 570 and storage device 530. Each of components 510, 520, 530, 540, 550, and 560 are interconnected using various buses and may be mounted on a common motherboard or otherwise mounted as needed. Processor 510 can process instructions executed within computing device 500, including instructions stored in memory 520 or on storage device 530 to display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 580 coupled to high-speed interface 540. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and multiple types of memory, as needed. Also, multiple computing devices 500 may be connected such that each device performs a portion of the required operations (eg, a bank of servers, a group of blade servers, or a multi-processor system).

[0056] The memory 520 stores information non-temporarily within the computing device 500. The memory 520 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-transient memory 520 may be a physical device used to temporarily or persistently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0057] The storage device 530 can provide mass storage for the computing device 500. In some implementations, the storage device 530 is a computer-readable medium. In various different implementations, the storage device 530 may be a floppy disk device, a hard disk device, an optical disk device, or an array of devices including a tape device, a flash memory or other similar solid-state memory device, or a device in a storage area network or other configuration. In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory 520, the storage device 530, or memory on the processor 510.

[0058] The high-speed controller 540 manages bandwidth-intensive operations for the computing device 500, while the low-speed controller 560 manages lower-bandwidth-intensive operations. Such allocation of duties is merely exemplary. In some implementations, the high-speed controller 540 is coupled to the memory 520, the display 580 (e.g., via a graphics processor or accelerator), and the high-speed expansion port 550. The high-speed expansion port 550 may accept various expansion cards (not shown). In some implementations, the low-speed controller 560 is coupled to the storage device 530 and the low-speed expansion port 590. The low-speed expansion port 590 may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) and may be coupled to one or more input / output devices, such as a keyboard, pointing device, scanner, or networking device, such as a switch or router, via a network adapter, for example.

[0059] Computing device 500 may be implemented in a number of different ways as shown, for example, computing device 500 may be implemented as a standard server 500a or multiple times within a group of such servers 500a, as a laptop computer 500b, or as part of a rack server system 500c.

[0060] Various implementations of the systems and techniques described herein may be realized in digital electrical and / or optical circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor. The programmable processor may be dedicated or general-purpose and may be coupled to receive and transmit data and instructions from a storage system, at least one input device, and at least one output device.

[0061] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0062] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0063] The processes and logic flows described herein may be implemented by one or more programmable processors, also referred to as data processing hardware, that execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be implemented by special-purpose logic circuitry, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). Processors suitable for executing computer programs include, by way of example, both general-purpose and special-purpose microprocessors, as well as any one or more processors of any type of digital computer. Generally, a processor receives instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, or is operably coupled to receive data from or transfer data to one or more mass storage devices, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, by way of example only, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0064] To enable user interaction, one or more aspects of the present disclosure can be implemented on a computer having a display device, e.g., a cathode ray tube (CRT), liquid crystal display (LCD) monitor, or touch screen, for displaying information to the user, and possibly a keyboard and pointing device, e.g., a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to enable user interaction. For example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic input, voice input, or tactile input. Also, a computer can interact with a user by sending documents to and receiving documents from a device being used by the user, e.g., by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0065] Although several implementations have been described, it will be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, other implementations are within the scope of the following claims. [Explanation of symbols]

[0066] 1-2 hours 10 User Devices 12 Data Processing Hardware 14 Memory Hardware 16 Audio System 16a Audio Capture Device 16b Audio Output Device 18, 18b Digital Assistant Interface, Voicemail Application Interface 50 Voicemail applications, digital assistant applications, VM applications 50a Digital Assistant Applications 50b Voicemail application, VM application 60 Remote Computing Devices 100, 100a, 100b Audio Environment 104 users 106 utterances 107 User Interface Generator 108 Audio Subsystem 109 Automatic Speech Recognition (ASR) System 110 Acoustic Frames, Audio Data 120 Transcription 120a Partial speech recognition result 120b Final speech recognition result 130 Sample Database 132, 132a-132n Training utterances 140 Sample Database 142, 142a-142n Training samples 200 ASR Model, Model 200a, 200b, 200c models, cascade encoder models 202 Cascade Encoder 204 decoder, shared decoder 206 Language Model (LM), External Language Model 210 first encoder, first causal encoder, streaming encoder, cascade encoder 220 second encoder, second non-causal encoder, non-streaming encoder, cascaded encoder 230 Encoder, Joint Layer, Joint Network 240 Prediction Network 242 Projection Layer 300 Prediction Network, Training Process 400 ways 500 computing devices 500a Server 500b laptop computer 500c Rack Server System 510 processor 520 memory 530 Storage Devices 540 High-Speed ​​Interface / Controller 550 High-Speed ​​Expansion Port 560 Low-Speed ​​Interface / Controller 570 Slow Bus 580 Display 590 Low-Speed ​​Expansion Port

Claims

1. An automatic speech recognition (ASR) system (109), comprising: A first encoder (210), receiving as input a sequence of acoustic frames (110); In each of a plurality of output steps, a first high-level feature representation is generated for a corresponding acoustic frame (110) in the sequence of acoustic frames (110). a first encoder (210) configured as follows: A second encoder (220), receiving as input the first high-dimensional feature representation generated by the first encoder (210) in each of the plurality of output steps; In each of the plurality of output steps, a second high-level feature representation is generated for the corresponding first high-level feature frame. a second encoder (220) configured as follows: A decoder (204), receiving as input the second high-dimensional feature representation generated by the second encoder (220) in each of the plurality of output steps; generating a first probability distribution over possible speech recognition hypotheses (120b) at each of the plurality of time steps; a decoder (204) configured as follows: A language model (206), receiving as input the first probability distribution (120b) over possible speech recognition hypotheses; generating a rescored probability distribution over the possible speech recognition hypotheses (120) at each of the plurality of time steps; and a language model (206) configured as described above.

2. 2. The ASR system of claim 1, wherein the second encoder generates the second high-level feature representation without receiving any of the acoustic frames as input.

3. The decoder (204) receiving as input the first high-dimensional feature representation generated by the first encoder (210) in each of the plurality of output steps; generating a second probability distribution over the possible speech recognition hypotheses (120a) at each of the plurality of time steps; 3. The ASR system (109) of claim 1 or 2, further configured to:

4. The decoder (204) a prediction network (240) for each of the plurality of time steps; receives as input the sequence of N previous non-blank symbols output by the final softmax layer, generating a respective embedding for each non-blank symbol in the sequence of N previous non-blank symbols; The average embedding (d avg ) A prediction network (240) configured as follows: A joint network (230), The mean embedding (d avg )and, the first high-level feature representation generated by the first encoder (210) in each of the plurality of output steps when the ASR system (109) is operating in a streaming mode; or and one of the second high-level feature representations generated by the second encoder (220) in each of the plurality of output steps when the ASR system (109) is operating in a non-streaming mode; In each of the plurality of output steps, the second probability distribution over possible speech recognition hypotheses (120a) when the ASR system (109) is operating in the streaming mode; or generating one of the first probability distributions over possible speech recognition hypotheses when the ASR system is operating in the non-streaming mode; and a joint network configured to:

5. 5. The ASR system of claim 4, wherein the prediction network comprises a V2 embedded lookup table.

6. 6. The ASR system of claim 1, wherein the first encoder comprises a causal encoder comprising an initial stack of conformer layers.

7. 7. The ASR system of claim 6, wherein the second encoder comprises a non-causal encoder comprising a final stack of conformer layers superimposed on the initial stack of conformer layers.

8. 8. The ASR system (109) of claim 1, wherein the language model (206) comprises a neural language model (206).

9. 10. The ASR system of claim 8, wherein the neural language model comprises a stack of conformer or transformer layers.

10. the language model (206) is trained on text-only data; 10. The ASR system of claim 1, wherein the first encoder and the second encoder are trained using hybrid autoregressive transducer factorization to facilitate integration of the language model trained on the text-only data.

11. A computer-implemented method (400) that, when executed on data processing hardware (12), causes the data processing hardware (12) to perform operations, the operations comprising: receiving a sequence of acoustic frames (110) as input to an automatic speech recognition (ASR) model (200); performing streaming and non-streaming speech recognition on the sequence of acoustic frames (110) using the ASR model (200), said performing including: generating, by a first encoder (210), in each of a plurality of output steps, a first high-level feature representation of a corresponding acoustic frame in the sequence of acoustic frames (110); receiving the first high-dimensional feature representation generated by the first encoder (210) in each of the plurality of output steps as input to a second encoder (220); generating, by the second encoder (220), a second high-level feature representation for the corresponding first high-level feature frame in each of the plurality of output steps; receiving the second high-level feature representation generated by the second encoder (220) in each of the plurality of output steps as input to a decoder (240); performing, at each of the plurality of time steps, by generating a first probability distribution over possible speech recognition hypotheses; and rescoring the first probability distribution over possible speech recognition hypotheses using an external language model to generate a transcription of the utterance.

12. 12. The computer-implemented method of claim 11, wherein the second encoder generates the second high-level feature representation without receiving any of the acoustic frames as input.

13. The operations, when performing streaming and non-streaming speech recognition on the sequence of acoustic frames (110), include: receiving the first high-level feature representation generated by the first encoder (210) in each of the plurality of output steps as input to the decoder (204); generating a second probability distribution over possible speech recognition hypotheses at each of the plurality of time steps; 13. The computer-implemented method (400) of claim 11 or 12, further comprising:

14. The operation includes, at each of the plurality of time steps: receiving as input to a prediction network (240) the sequence of N previous non-blank symbols output by the final softmax layer; generating, for each non-blank symbol in the sequence of N previous non-blank symbols, a respective embedding by the predictive network (240); The prediction network (240) calculates the average embedding (d) by averaging the individual embeddings. avg ) and The mean embedding (d avg )and, the first high-level feature representation generated by the first encoder in each of the plurality of output steps when the ASR model is operating in streaming mode; or receiving, as input to a joint network (230), one of the second high-level feature representations generated by the second encoder (220) in each of the plurality of output steps when the ASR model (200) is operating in a non-streaming mode; In each of the plurality of output steps, the second probability distribution over possible speech recognition hypotheses when the ASR model (200) is operating in the streaming mode; or generating one of the first probability distributions over possible speech recognition hypotheses when the ASR model is operating in the non-streaming mode; and 14. The computer-implemented method (400) of claim 13, further comprising:

15. 15. The computer-implemented method of claim 14, wherein the predictive network comprises a V2 embedded lookup table.

16. 16. The computer-implemented method (400) of any one of claims 11 to 15, wherein the first encoder (210) comprises a causal encoder (210) comprising an initial stack of conformer layers.

17. 17. The computer-implemented method of claim 16, wherein the second encoder comprises a non-causal encoder comprising a final stack of conformer layers superimposed on the initial stack of conformer layers.

18. 18. The computer-implemented method (400) of any one of claims 11 to 17, wherein the language model (206) comprises a neural language model (206).

19. 20. The computer-implemented method of claim 18, wherein the neural language model comprises a stack of conformer or transformer layers.

20. the language model (206) is trained on text-only data; 20. The computer-implemented method (400) of any one of claims 11 to 19, wherein the first encoder (210) and the second encoder (220) are trained using hybrid autoregressive transducer factorization to facilitate integration of the language model (206) trained on the text-only data.

Images