Unified Cascade Encoder ASR Model for Dynamic Model Size

Abstract

The automatic speech recognition (ASR) model (200) includes first and second encoders (210), (220) and first and second decoders (204), (206). The first encoder receives as input a sequence of acoustic frames (104) and generates, at each of a plurality of output steps, a first high-order feature representation of a corresponding acoustic frame in the sequence of acoustic frames. The first decoder receives as input the first high-order feature representation generated by the first encoder and generates a first probability distribution for possible speech recognition hypotheses. The second encoder receives as input the first high-order feature representation generated by the first encoder and generates a second high-order feature representation of the corresponding first high-order feature frame. The second decoder receives as input the second high-order feature representation generated by the second encoder and generates a second probability distribution for possible speech recognition hypotheses.

Description

[Technical field]

[0001] This disclosure relates to an on-device end-to-end model for simplified streaming and non-streaming automatic speech recognition (ASR). [Background technology]

[0002] Automatic speech recognition (ASR) systems have evolved from multiple models, each with a dedicated purpose, to unified models that use a single neural network to directly map speech waveforms (i.e., input sequence) to output sentences (i.e., output sequence). This unification has resulted in a sequence-to-sequence approach, where a sequence of words (or graphemes) is generated given a sequence of speech features. With the unified structure, all components of the model can be trained jointly as a single end-to-end (E2E) neural network. Here, an E2E model refers to a model whose architecture is built entirely of neural networks. A complete neural network works without external and / or manually designed components (e.g., finite state transducers, lexicons, or text normalization modules). Furthermore, when training E2E models, these models generally do not require bootstrapping from decision trees or time alignment from separate systems. These E2E automatic speech recognition (ASR) systems have made great progress and have outperformed traditional ASR systems in several common metrics, including word error rate (WER). The architecture of E2E ASR models is largely application dependent. For example, many applications with user interaction, such as voice search or on-device voice input, require the model to perform recognition in a streaming manner. Other applications, such as offline video captioning, do not require streaming the model and can utilize future context to improve performance. Furthermore, the model size of an on-device ASR system depends heavily on the hardware constraints of the device. For example, a vehicular device with processing hardware with larger computational power may support a larger ASR model size, while a mobile device with limited processing hardware may be able to benefit from a smaller ASR model size. Existing E2E approaches typically include separate ASR models for streaming and non-streaming applications.Managing multiple models for different applications presents unique challenges and can slow down the run time of an ASR system. Summary of the Invention

[0003] One aspect of the present disclosure provides an automatic speech recognition (ASR) model including a first encoder, a first decoder, a second encoder, and a second decoder. The first encoder receives as input a sequence of acoustic frames and generates, at each of a plurality of output steps, a first high-order feature representation of a corresponding acoustic frame in the sequence of acoustic frames. The first decoder receives as input the first high-order feature representation generated by the first encoder at each of the plurality of output steps and generates, at each of the plurality of output steps, a first probability distribution for a possible speech recognition hypothesis. The second encoder receives as input the first high-order feature representation generated by the first encoder at each of the plurality of output steps and generates, at each of the plurality of output steps, a second high-order feature representation of a corresponding first high-order feature frame. The second decoder receives as input the second high-order feature representation generated by the second encoder at each of the plurality of output steps and generates, at each of the plurality of output steps, a second probability distribution for a possible speech recognition hypothesis.

[0004] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, the first decoder generates a partial speech recognition result based on a first probability distribution over possible speech recognition hypotheses. In some examples, the first encoder includes a causal encoder including one of multiple unidirectional long short-term memory (LSTM) layers, multiple conformer layers, or multiple transformer layers. Additionally or alternatively, the second encoder includes a non-causal encoder including one of multiple unidirectional long short-term memory (LSTM) layers, multiple conformer layers, or multiple transformer layers.

[0005] In some examples, the first decoder includes a prediction network and a combination network. The prediction network receives as input the sequence of non-blank symbols output by the final softmax layer and generates a dense representation at each of the multiple output steps. The combination network receives as input the dense representation generated by the prediction network at each of the multiple output steps and the first high-level feature representation generated by the first encoder at each of the multiple output steps and generates a first probability distribution over possible speech recognition hypotheses at each of the multiple output steps. In these examples, the prediction network includes a long short-term memory (LSTM) based prediction network, or a V2 embedded lookup table.

[0006] In some implementations, the second decoder includes a prediction network and a combination network. The prediction network receives as input the sequence of non-blank symbols output by the final softmax layer and generates a dense representation at each of the multiple output steps. The combination network receives as input the dense representation generated by the prediction network at each of the multiple output steps and the second higher-level feature representation generated by the second encoder at each of the multiple output steps and generates a second probability distribution over the possible speech recognition hypotheses at each of the multiple output steps. In these implementations, the prediction network includes a long short-term memory (LSTM) based prediction network or a V2 embedded lookup table.

[0007] In some examples, the first encoder includes a greater number of parameters than the second encoder. In some implementations, the ASR model further includes a third encoder and a third decoder. The third encoder receives as input the second high-order feature representation generated by the second encoder at each of the multiple output steps and generates a third high-order feature representation of the corresponding second high-order feature frame at each of the multiple output steps. The third decoder receives as input the third high-order feature representation generated by the third encoder at each of the multiple output steps and generates a third probability distribution for the possible speech recognition hypotheses at each of the multiple output steps.

[0008] Another aspect of the disclosure provides a computer-implemented method that, when executed on the data processing hardware, causes the data processing hardware to perform operations including receiving a sequence of acoustic frames as an input to an automatic speech recognition (ASR) model and performing streaming and non-streaming speech recognition using the ASR model on the sequence of acoustic frames. The operations also include generating, by a first encoder, a first high-order feature representation of a corresponding acoustic frame in the sequence of acoustic frames at each of a plurality of output steps, and receiving the first high-order feature representation generated by the first encoder at each of the plurality of output steps as input to both a first decoder and a second encoder. The operations also include generating, by the second encoder, a second high-order feature representation for the corresponding first high-order feature representation at each of the plurality of output steps, and generating, by the first decoder, a first probability distribution over possible speech recognition hypotheses at each of the plurality of output steps. The operations further include receiving as input to a second decoder a second higher-order feature representation generated by the second encoder at each of the plurality of output steps, and generating, by the second decoder, a second probability distribution over possible speech recognition hypotheses at each of the plurality of output steps.

[0009] This aspect may include one or more of the following optional features: In some implementations, the operations further include generating a partial speech recognition result based on the first probability distribution over the possible speech recognition hypotheses. In some examples, the first encoder includes a causal encoder including one of a plurality of unidirectional long short-term memory (LSTM) layers, a plurality of conformer layers, or a plurality of transformer layers. Additionally or alternatively, the second encoder includes a non-causal encoder including one of a plurality of unidirectional long short-term memory (LSTM) layers, a plurality of conformer layers, or a plurality of transformer layers.

[0010] In some implementations, when the ASR model is operating in streaming mode, the operations further include receiving as input to a prediction network of the first decoder the sequence of non-blank symbols output by the final softmax layer, and generating a dense representation by the prediction network of the first decoder at each of the multiple output steps, where the operations further include receiving as input to a combination network of the first decoder the dense representation generated by the prediction network at each of the multiple steps and the first high-level feature representation generated by the first encoder at each of the multiple output steps, and generating by the combination network of the first decoder a first probability distribution over possible speech recognition hypotheses at each of the multiple output steps. In these implementations, the prediction network of the first decoder includes a long short-term memory (LSTM) based prediction network, or a V2 embedded lookup table.

[0011] In some embodiments, when the ASR model is operating in a non-streaming mode, the operations also include receiving the sequence of non-blank symbols output by the final softmax layer as an input to a prediction network of the second decoder, and generating a dense representation by the prediction network of the second decoder at each of the multiple output steps. Here, the operations further include receiving the dense representation generated by the prediction network at each of the multiple output steps and the second high-level feature representation generated by the second encoder at each of the multiple output steps as an input to a combination network of the second decoder, and generating a second probability distribution for possible speech recognition hypotheses by the combination network of the second decoder at each of the multiple output steps. In these examples, the prediction network of the second decoder includes a long short-term memory (LSTM)-based prediction network, or a V2 embedded lookup table.

[0012] In some implementations, the first encoder includes a greater number of parameters than the second encoder. In some examples, the operations also include receiving, as an input to a third encoder, the second high-order feature representation generated by the second encoder at each of the plurality of output steps while performing streaming and non-streaming speech recognition on the sequence of acoustic frames. In these implementations, the operations further include generating, by the third encoder at each of the plurality of output steps, a third high-order feature representation for the corresponding second high-order feature representation, receiving, as an input to a third decoder, the third high-order feature representation generated by the third encoder at each of the plurality of output steps, and generating, by the third decoder at each of the plurality of output steps, a third probability distribution over the possible speech recognition hypotheses.

[0013] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0014] [Figure 1A] FIG. 1 is a schematic diagram of a system using a cascade encoder model architecture for automatic speech recognition. [Figure 1B] FIG. 1 is a schematic diagram of a system using a cascade encoder model architecture for automatic speech recognition. [Figure 2A] FIG. 1 is a schematic diagram of a cascade encoder model architecture. [Figure 2B] FIG. 1 is a schematic diagram of a cascade encoder model architecture. [Figure 2C] FIG. 1 is a schematic diagram of a cascade encoder model architecture. [Figure 2D] FIG. 1 is a schematic diagram of a cascade encoder model architecture. [Figure 3A] FIG. 1 is a schematic diagram of an example training process for facilitating a cascade encoder model to learn consistent predictions for both streaming and non-streaming automatic speech recognition. [Figure 3B] FIG. 1 is a schematic diagram of an example training process for facilitating a cascade encoder model to learn consistent predictions for both streaming and non-streaming automatic speech recognition. [Figure 4] 1 is a flowchart of an example operational procedure for a method of implementing a cascade encoder model for both streaming and non-streaming automatic speech recognition. [Diagram 5] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Like reference symbols in the various drawings indicate like elements. End-to-end (E2E) automatic speech recognition (ASR) models are traditionally structured to operate in either streaming or non-streaming modes. Traditionally, E2E ASR models include an encoder and a decoder as major components. Applications involving end-user interaction, such as voice search or on-device voice input, may require the model to perform recognition in a streaming manner, with the expectation that words are output at the same time as they are spoken, with minimal latency. This precludes the use of models that use future context to improve accuracy, such as bidirectional LSTMs. In contrast, applications such as offline video captioning do not require streaming recognition and can fully utilize any available future context to improve performance. Furthermore, the model size of an on-device ASR system depends heavily on the hardware constraints of the device. For example, a vehicle device with processing hardware with larger computational power may support a larger ASR model size, while a mobile device with limited processing hardware may be able to benefit from a smaller ASR model size. On the same device, a large ASR model size may be used for short-form applications such as voice search, while a medium or small ASR model size may be better suited for long-running applications such as dictation or video captioning to maintain low power consumption.

[0016] The embodiments herein are directed to a single E2E ASR model that uses a cascaded encoder and two separate decoders to enable operation in both streaming and non-streaming modes with various model parameters. The cascaded encoder includes a streaming encoder and a non-streaming encoder, and the two separate decoders include a streaming decoder and a non-streaming decoder that are configured to learn to decode the output from the respective streaming encoder or the output from the respective non-streaming encoder. In addition to ASR models, the architecture can be applied to other models, such as machine translation, that implement both streaming and non-streaming modes.

[0017] 1A and 1B, in some embodiments, a system 100, 100a-b includes a user 104 interacting with a voice-enabled user device 10 (also referred to as device 10 or user device 10) in a voice environment. In the system 100, the manner in which the user 104 interacts with the user device 10 may be through voice input. The user device 10 is configured to obtain sounds (e.g., streaming voice data) from one or more users 104 in the voice environment. Here, the streaming voice data may refer to utterances 106 by the users 104 that function as audible queries, commands to the device 10, or audible communications obtained by the device 10. The voice-enabled system of the device 10 may address the queries or commands by responding to the queries and / or causing the commands to be executed / fulfilled by one or more downstream applications.

[0018] The user equipment 10 may correspond to any computing device associated with a user 104 and capable of receiving voice data. Some examples of the user equipment 10 include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smart watches), smart appliances, Internet of Things (IoT) devices, in-vehicle infotainment systems, smart displays, smart speakers, etc. The user equipment 10 includes data processing hardware 12 and memory hardware 14 in communication with the data processing hardware 12, storing instructions that, when executed by the data processing hardware 12, cause the data processing hardware 12 to perform one or more operations. The user equipment 10 further includes an audio system 16 with audio capture devices (e.g., microphones) 16, 16a for capturing and converting speech utterances 106 in an audio environment into electrical signals, and audio output devices (e.g., speakers) 16, 16b for communicating audible audio signals (e.g., as output audio data from the device 10). In the illustrated example, the user device 10 implements a single audio capture device 16a, however, the user device 10 may implement an array of audio capture devices 16a without departing from the scope of this disclosure, in which case one or more capture devices 16a of the array may communicate with the audio system 16 without being physically resident on the user device 10.

[0019] In the system 100, an automatic speech recognition (ASR) system 109 implementing the ASR model 200 (also referred to as model 200) resides on the user equipment 10 of the user 104 and / or on a remote computing device 60 (e.g., one or more remote servers of a distributed system executing in a cloud computing environment) that communicates with the user equipment 10 via the network 40. The remote computing device 60 may include remote data processing hardware 62 (e.g., a remote server or CPU) and / or remote memory hardware 64 (e.g., a remote database or other storage hardware). The user equipment 10 and / or the remote computing device 60 also include a speech subsystem 108 configured to receive speech 106 produced by the user 104 and captured by the speech capture device 16a, and to convert the speech 106 into a corresponding digital form associated with input acoustic frames 110 that can be processed by the ASR system 109. In the embodiment shown in FIG. 1A, a user 104 makes a respective utterance 106, and a speech subsystem 108 converts the utterance 106 into corresponding speech data (e.g., acoustic frames) 110 for input to an ASR system 109. The model 200 then receives the speech data 110 corresponding to the utterance 106 as input and generates / predicts a corresponding transcription 120 (e.g., also referred to as a recognition result / hypothesis 120) of the utterance 106 as output. As described in more detail below (e.g., in FIGS. 3A and 2B), the model 200 may be trained in two stages of training to simplify the process of training the model 200 and operate in streaming and non-streaming modes. The model 200 also includes a first decoder 204 (also referred to as a streaming decoder 204) and a second decoder 206 (also referred to as a non-streaming decoder 206), which allows the model 200 to operate in streaming and non-streaming modes (e.g., as opposed to two separate models dedicated to streaming or non-streaming modes).

[0020] For example, as shown in 1A, a digital assistant application 50 running on a user device 10 may require streaming speech recognition, whereby words, parts of words, and / or individual characters appear on the screen as soon as they are spoken. Additionally, a user 104 of the user device 10 may have a low tolerance for latency when issuing queries for the digital assistant application 50 to execute. In these scenarios where the application requires minimal latency and benefits from a small model size, the model 200 may operate in a streaming mode and provide streaming transcription capabilities in real time as the user 104 speaks the utterance 106. On the other hand, if the user 104 has a higher tolerance for speech recognition latency and / or the utterance 106 to be recognized is associated with long speech (i.e., referring to speech consisting of a complete paragraph or multiple sentences) and a larger model capacity, the same model 200 may operate in a non-streaming mode and leverage a predictive network to provide an accurate transcription 120, but with potentially increased latency. Additionally, in scenarios where the user 104 has a higher tolerance for speech recognition latency and / or where accuracy is paramount, the model 200 can operate in a non-streaming mode that uses additional context to improve accuracy, but at the expense of increased speech recognition latency. Thus, the ASR system 109 can implement the ASR model 200 for a variety of speech recognition tasks and provide both streaming and non-streaming transcription capabilities, without the need to use a separately trained ASR model for each task. Additionally, using a single ASR model 200 for a variety of speech recognition tasks reduces the computational requirements for running separate ASR models.

[0021] In some implementations, the model 200 first performs streaming speech recognition on the speech data 110 and then performs non-streaming speech recognition on the output of the streaming encoder. For example, in the illustrated example, the model 200 performs streaming speech recognition on the speech data 110 using a first encoder (i.e., a low-latency encoder (FIG. 2B)) to generate partial speech recognition results 120, 120a, and performs non-streaming speech recognition on the encoded speech data 110 using a second encoder (i.e., a high-latency encoder (FIG. 2C)) to generate final speech recognition results 120, 120b. In particular, the model 200 outputs the partial speech recognition result 120a at the same time that the speech data 110 is received, and outputs the final speech recognition result 120b once all of the speech data 110 has been received after the user 104 has finished speaking 106. Thus, the output of the final speech recognition result 120b for the input utterance 106 may be delayed from the partial speech recognition result 120a.

[0022] The user device 10 and / or the remote computing device 60 also execute a user interface generator 107 configured to present a representation of the transcription 120 of the utterance 106 to the user 104 of the user device 10. As described in more detail below, the user interface generator 107 may display partial speech recognition results 120a in a streaming manner during time 1, and then display final speech recognition results 120b during time 2. In some configurations, the transcription 120 output from the ASR system 109 is processed by, for example, a natural language understanding (NLU) module executing on the user device 10 or the remote computing device 60 to execute the user command / query specified by the utterance 106. Additionally or alternatively, a text-to-speech system (not shown) (e.g., executing on any combination of the user device 10 or the remote computing device 60) can convert the transcription 120 into synthetic speech and output audibly on the user device 10 and / or other devices.

[0023] In the exemplary system 100a of FIG. 1A, a user 104 in a speech environment interacts with a program or application 50 (e.g., a digital assistant application 50a) of a user device 10 that uses an ASR system 109. For example, FIG. 1A depicts a user 104 communicating with the digital assistant application 50a, which displays a digital assistant interface 18 on the screen of the user device 10 to represent a conversation between the user 104 and the digital assistant of the digital assistant application 50a. In this example, the user 104 asks the digital assistant application 50a, "What song are you playing now?" This question from the user 104 is an utterance 106, which is captured by the speech capture device 16a and processed by the speech system 16 of the user device 10. In this embodiment, the speech system 16 receives the utterance 106 and converts it into acoustic frames 110 for input to the ASR system 109.

[0024] Continuing with the example, the model 200 receives an acoustic frame 110 corresponding to the utterance 106 as the user 104 speaks it, encodes the acoustic frame 110 using a first encoder 210 (i.e., FIGS. 2A and 2B), and then decodes the encoded representation of the acoustic frame 110 into a partial speech recognition result 120a using a first decoder 204 (FIGS. 2A and 2B). During time 1, the user interface generator 107 presents a representation of the partial speech recognition result 120a of the utterance 106 to the user 104 of the user device 10 in a streaming manner via the digital assistant interface 18, such that words, parts of words, and / or individual letters are displayed on the screen as they are spoken.

[0025] After all (or some) of the acoustic frames 110 corresponding to the utterance 106 are received and the first encoder 210 encodes these acoustic frames 110, the second encoder 220 (i.e., FIGS. 2A and 2C) encodes the encoding output from the first encoder 210 to generate an encoding for the set of acoustic frames 110 corresponding to the utterance 106 already encoded by the first encoder 210. The second decoder 206 (i.e., FIGS. 2A and 2C) then decodes the acoustic frames 110 encoded by the second encoder 220 into the final speech recognition result 120b. For example, when the first encoder 210 encodes all the acoustic frames 110 corresponding to the utterance 106 (e.g., at the same time as the acoustic frames 110 are received), the second encoder 220 encodes all the acoustic frames 110 encoded by the first encoder 210. In this regard, by encoding multiple encoded acoustic frames 110, the second encoder 220 can provide greater context awareness in a non-streaming manner (e.g., by receiving a representation of all acoustic frames 110 for an utterance 106), potentially allowing it to adjust for or correct an aspect(s) of the utterance 106 that was missed or misinterpreted due to the streaming nature of the first encoder 210. In some examples, an indication that the user 104 has finished speaking 106, e.g., an endpoint, triggers the second encoder 220 of the model 200 to encode all acoustic frames 110.

[0026] During time 2, the user interface generator 107 presents a representation of the final speech recognition result 120b of the utterance 106 to the user 104 of the user device 10 via the digital assistant interface 18. In some implementations, the user interface generator 107 replaces (or modifies) the representation of the partial speech recognition result 120a with a representation of the final speech recognition result 120b. For example, the final speech recognition result 120b is deemed more accurate than the partial speech recognition result 120a, and the final speech recognition result 120b is finally displayed as the transcription 120 to correct terms that may have been misrecognized in the partial speech recognition result 120a. In this embodiment, the partial speech recognition result 120a output by the model 200 (i.e., the first encoder 210 and the first decoder 204) and displayed on the screen of the user device 10 at time 1 is associated with low latency and provides responsiveness to the user 104 indicating that their query is being processed. Thus, the partial speech recognition result 120a may incorrectly predict the user's 104 utterance 106 to be "What song are you playing now?". The final speech recognition result 120b output by the model 200 (i.e., the cascade encoder 202 and the second decoder 206) and displayed on the screen at time 2, with increased latency, identifies that the user 104 said "playing", improving the speech recognition quality in terms of accuracy. However, because the user interface generator 107 displays the partial speech recognition results while the user is making the utterance 106, the high latency associated with generating and ultimately displaying the final recognition result 120b is not noticeable to the user 104.

[0027] In the example shown in FIG. 1A, the digital assistant application 50a can use natural language processing to respond to a question posed by the user 104. Natural language processing generally refers to the process of interpreting written language (e.g., partial speech recognition results 120a and / or final speech recognition results 120b) and determining whether the written language prompts some action. In this example, the digital assistant application 50a uses natural language processing to recognize that the question from the user 104 is about the user's environment, more specifically, the song playing near the user. By recognizing these details using natural language processing, the automated assistant returns a response 19 to the user's query, which response 19 indicates "Cloud Play is now playing." In some configurations, the natural language processing is performed on a remote computing device 60 that communicates with the data processing hardware 12 of the user device 10.

[0028] FIG. 1B is another example of speech recognition using the ASR system 109 in a speech environment 100b. As shown in this example, a user 104 interacts with a voicemail application 50, 50b that displays a voicemail application interface 18, 18b on the screen of the user device 10 to transcribe a voicemail left by Jane Doe for the user 104. In this example, latency is not important. Without considering latency, the model 200 of the ASR system 109 can exploit the full context of the speech by waiting until all acoustic frames 110 corresponding to the voicemail are generated. This voicemail scenario also illustrates how the model 200 can handle long speech, since voicemails often span multiple sentences or even paragraphs. The ability to handle long speech is particularly advantageous over other ASR models, such as multi-path models with LAS decoders, because these models often suffer from performance degradation (e.g., high word deletion rate for long speech) when recognizing long speech. For example, using a dedicated second decoder 206 trained with features of the second encoder 220 in combination with a cascaded encoder 202 (e.g., a first encoder 210 and a second encoder 220) allows the model 200 to exploit the full context of the input speech when performing speech recognition on long speech without incurring performance degradation. As described in more detail below, the cascaded encoder 202 of the model 200 may optionally include a third encoder 230 cascaded with the first encoder 210 and the second encoder 220, and a dedicated third decoder 208 for decoding the acoustic frames 110 encoded by the third encoder 230 into the final speech recognition result 120b.

[0029] Continuing with reference to FIG. 1B, as described with respect to FIG. 1A, the model 200 uses the first encoder 210 to encode the acoustic frames 110 as they are received. After the model 200 receives all the acoustic frames 110 and encodes them with the first encoder 210, the model 200 provides the first encoder output as input to the second encoder 220. The second encoder 220 encodes the first encoder output before the second decoder 206 generates the final speech recognition result 120b. The user interface generator 107 then presents a representation of the final speech recognition result 120b via the digital assistant interface 18b without first displaying the partial speech recognition result 120a. For example, the final speech recognition result 120b is a transcription of a long voicemail from Jane Doe that reads, "Call me back when you get this. I'm trying to plan for the new year."

[0030] 2A-2D include exemplary models 200a-d operating in various combinations of streaming and non-streaming modes. Specifically, the models 200a-d each include a cascaded encoder 202, a first decoder 204, and a second decoder 206. The cascaded encoder 202 refers to a model structure including encoders 210 and 220 whose encoding paths are cascaded, whereby the output of one encoder 210 is fed to the input of a second encoder 220 before decoding. Here, the encoders 210 and 220 can be cascaded regardless of the underlying architecture of each encoder. As shown in FIG. 2D and described in more detail below, the model 200 can be generalized to a three-pass model 200 (i.e., a large model), where the cascaded encoder 202 includes a third encoder 230 cascaded with the encoders 210, 220 and connected to a dedicated third decoder 208. Here, the output of the second encoder 220 is provided to the input of a third encoder 230 before decoding by the third decoder 208 .

[0031] In some examples, the encoders 210, 220, 230 include a stack of 512-dimensional conformer layers. In other examples, the first encoder 210 (i.e., first pass) includes a stack of 256-dimensional causal conformer layers (e.g., 6 layers), the second encoder 220 (i.e., second pass) includes a stack of 512-dimensional non-causal conformer layers (e.g., 6 layers), and the third encoder 230 (i.e., third pass) includes a stack of 640-dimensional non-causal conformer layers (e.g., 6 layers). Causal convolutional layers and left contextual attention layers can be used for each conformer layer to strictly constrain the model to not use future inputs. A multi-head (e.g., 8 head) attention mechanism can be used in the self-attention layer.

[0032] The encoder 210 includes six conformer layers with approximately 20 million parameters when the model 200 operates as a first pass model. The cascade encoders 210, 220 may include 12 conformer layers with approximately 50 million parameters when the model 200 operates as a second pass model. Here, the first encoder 210 may include six conformer layers with approximately 20 million parameters, with the first three layers having no self-attention (e.g., no left and right context for 92 frames) to prevent the model 200 from using future inputs, while the second encoder 220 may include six conformer layers with approximately 30 million parameters and incorporating an additional right context (e.g., 5.04 seconds). The cascade encoders 210, 220, 230 include 18 layers with approximately 110 million parameters when the model 200 operates as a third pass model, where the third encoder 230 includes six non-causal conformer layers with an additional right context (e.g., a 900 millisecond (ms) 30-frame right context with approximately 60 million parameters). Optionally, other types of layers incorporating self-attention mechanisms, such as transformer layers, can be used instead of the conformer layers. The first encoder 210 may be referred to as a causal encoder, and the second encoder 220 and the third encoder 230 may each be referred to as a non-causal encoder.

[0033] In other implementations, one encoder is constructed using an LSTM structure, and the other encoder is constructed using a bidirectional LSTM layer or a conformer layer (e.g., a conformer-transducer). In other words, the encoders 210, 220, 230 may have different or similar architectures. For example, the cascade encoder 202 may be roughly similar to an acoustic model (AM) of a conventional ASR system and may include a recurrent network of stacked long short-term memory (LSTM) layers. Here, the first encoder 210 is a streaming encoder that includes a unidirectional long short-term memory (LSTM) layer, while the second encoder 220 is a non-streaming encoder that includes a bidirectional LSTM layer or a conformer layer. In the cascade encoder 202, the encoders 210, 220, 230 include LSTM layers, and the second encoder 220, which receives the output of the first encoder 210, can utilize the LSTM layer of the first encoder 210 such that the second encoder 220 includes fewer LSTM layers than the first encoder 210 (and fewer LSTM layers than a full non-streaming model). By having fewer LSTM layers, the cascade encoder 202 can reduce the number of more computationally intensive bidirectional layers, making the model 200 more streamlined than simply combining a traditional streaming model with a traditional non-streaming model.

[0034] Referring to Figure 2A, a first encoder 210 reads a sequence of d-dimensional feature vectors (e.g., the acoustic frame 110 shown in Figures 1A and 1B). ( x1,x2,···,x T ) where xt∈Rd) is 、 At each output step, we generate a first high-level representation, which is s Similarly, the second encoder 220 is cascaded to the first encoder 210 and outputs the first high-order feature e s The trained model receives as input e and outputs a second higher-level representation, which is aOptionally, a third encoder 230 is cascaded to the second encoder 220 to generate a second high-level feature representation e a The trained model receives as input e and outputs a third higher-level representation, which is r The first encoder 210, the second encoder 220, and the third encoder 230 are directly connected to dedicated smaller decoders 204, 206, 208, respectively. Specifically, the first encoder 210 is connected to the first decoder 204, the second encoder 220 is connected to the second decoder 206, and the third encoder 230 is connected to the third decoder 208. Thus, the first decoder 204 generates a first high-level feature representation e s and the second decoder 206 generates a second high-level feature representation e a The third decoder 208 receives as input a third high-dimensional feature representation e r Takes as input.

[0035] The first decoder 204, the second decoder 206, and the third decoder 208 may each include a recurrent neural network transducer (RNN-T) architecture with 4.4 million parameters based on a 384-dimensional fully connected joint layer 240, 240a-c, and a 320-dimensional embedding prediction network 250, 250a-c, respectively. When the model 200 operates in streaming mode (i.e., FIG. 2B), the joint layer 240a of the first decoder 204 receives the embedding output (the previous prediction y) from the prediction network 250a. r-1 ) and the first high-level feature representation e output from the first encoder 210 s When the model 200 operates in a two-pass non-streaming mode (FIG. 2C), the second decoder 206 uses the joint layer 240b to combine the second high-level feature representation e output by the cascade encoder 202. a and the embedding output from the prediction network 250b (the previous prediction y r-1) to generate the output of the second decoder 206. Similarly, when the model 200 operates in a three-pass non-streaming mode (FIG. 2D), the third decoder 208 uses a joint layer 240c to combine the third high-level feature representation e r and the embedding output from the prediction network 250c (the previous prediction y r-1 ) to generate the output of the third decoder 208.

[0036] The output of each of the decoders 204, 206, 208 is the current subword unit y i Probability distribution P(y i |y i-1 ,...,y0,x), which is the sequence of N previous non-blank units {y i-1 ,...,y i-N} and the input x. Although not shown, the model 200 may include a softmax layer that receives the outputs of the decoders 204, 206, 208. In some implementations, the softmax layer is separate from the decoders 204, 206, 208 and receives the outputs y r The output of the softmax layer is then used in the beam search process to select the orthographic element. In some implementations, a softmax layer is integrated inside each of the decoders 204, 206, 208, and the output y r represents the output of the softmax layer.

[0037] The decoders 204, 206, 208 are configured to generate a probability distribution for possible speech recognition hypotheses at each output step. In other words, for each of the decoders 204, 206, 208, the respective joint layer 240 generates a probability distribution for possible speech recognition hypotheses at each output step (e.g., time step). Here, a "probable speech recognition hypothesis" corresponds to a set of output labels / symbols (also called "speech units"), each of which represents a grapheme (e.g., a symbol / character) or part of a word of a particular natural language. For example, if the natural language is English, the set of output labels may include 27 symbols, e.g., one label for each of the 26 letters of the English alphabet and one label representing a space. Thus, each joint layer 240 may output a set of values ​​indicating the likelihood of occurrence of each of a given set of output labels. This set of values ​​may be a vector (e.g., a one-hot vector) and may indicate a probability distribution of the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, potential punctuation marks or other symbols), but the set of output labels is not limited thereto. For example, the set of output labels can include parts of words and / or whole words in addition to or instead of graphemes. The output labels can also be other types of speech units, such as phonemes or sub-phonemes. The output distribution of the joint layer 240 can include posterior probability values ​​for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, the output of the joint layer 240 can include 100 different probability values, one for each output label. The probability distributions can then be used to select candidate orthographic elements (e.g., graphemes, parts of words, and / or words) in a beam search process (e.g., by a softmax layer) and assign scores to determine the transcription 120.

[0038] Within each of the decoders 204, 206, 208, each prediction network 250 may have two 2048-dimensional LSTM layers, each followed by a 640-dimensional projection layer, so that the LSTM-based prediction network may have approximately 23.4 million parameters. In other words, each prediction network 250 may include an LSTM layer. In other configurations, the prediction network 250 may include a conformer layer or a transformer layer instead of an LSTM layer. In yet other configurations, the prediction network 250 includes a V2 embedding lookup table that includes an embedding prediction network. At each time step, the V2 embedding lookup table receives as input the two previous predictions (e.g., 1-hot vectors) output by the prediction network 250, calculates the respective embeddings d1, d2 for each of the two previous predictions, and provides a concatenated output [d1, d2] to the joint layer 240. In comparison, the V2 embedding lookup table may only have about 2 million parameters, while the LSTM-based predictive network may include about 23.4 million parameters. Finally, the predictive network 250 may also be a one-layer neural network with 640 hidden units. The softmax layer may be composed of a unified word fragment or grapheme set that is generated using all unique word fragments or graphemes in the multiple training utterances 132, 132a-n (FIGS. 3A-3B). In some implementations, to limit the amount of future context the ASR model 200 sees, the second encoder 220 uses a certain amount of right context (e.g., 5 seconds of right context) and several conformer layers (e.g., 2 layers), while the first encoder 210 continues to use an LSTM layer. In these implementations, each conformer layer in the second encoder 220 may have 640 units consistent with the LSTM layer, adding about 10 million additional parameters.

[0039] In some implementations, model 200 operates in streaming mode (i.e., a one-pass model) and funnel pooling is used to downsample the input acoustic frames 110, thereby reducing the number of parameters of model 200 while maintaining the performance of model 200. For example, the self-attention layer of the first encoder 210 receives as input a feature map x∈R T×D where T is the length of the original sequence and D is the feature dimension. Then, we receive the downsampled sequence x'∈R T×D is generated by average pooling, which can be expressed as x' = Avg Pool(x) (1) where T' = T / 2. Rather than providing x' as the input to the self-attention layer, we use it as a query vector q, where the key vector k and value vector v are based on the input feature map x, y∈R TD This is mapped to the output features of the self-attention layer as follows: y=self-attention(q=x',kv=x) (2) 2A, in some implementations, the model 200a operates in both streaming and non-streaming modes in parallel. When operating simultaneously in both streaming and non-streaming modes, the model 200a first performs streaming speech recognition on the audio data 110 using the first encoder 210, and generates a first high-level feature representation e for both the second encoder 220 and the first decoder 204. s Next, the first decoder 204 generates the first high-level feature representation e s The model 200b also performs non-streaming speech recognition on the encoded speech data 110, and the second encoder 220 generates partial speech recognition results 120, 120a based on the first high-level feature representation e received from the first encoder 210. s The second high-level feature representation e a The second decoder 206 then generates a second high-level feature representation e a, and generates the final speech recognition result 120, 120b based on the input utterance 106. As indicated by "time", the first decoder 204 outputs the partial speech recognition result 120a in a streaming manner using the output from the first encoder 210, and then waits for the model 200a utterance to be completed and operates in a non-streaming mode, and the second decoder 206 uses the output from the second encoder 220 to generate the final speech recognition result 120b. Thus, the final speech recognition result 120b for the input utterance 106 may be delayed from the partial speech recognition result 120a.

[0040] 2B, in some implementations, the model 200b operates only in streaming mode (i.e., single-pass mode). This may occur, for example, when the user 104 is using an application that requires minimal latency, such as voice search or on-device dictation. Here, the model 200b performs streaming speech recognition of the audio data 110 using only the first encoder 210 to generate a first high-level feature representation e to the first decoder 204. s The first decoder 204 then generates the partial speech recognition results 120, 120a. Because the streaming mode of the model 200b generates the partial speech recognition results 120, 120a quickly, the imprecision of the term "playback" is generally acceptable to the user 104.

[0041] Referring to FIG. 2C, in some implementations, model 200c operates only in a non-streaming mode (i.e., a two-pass model). Non-streaming mode may occur in non-latency intensive applications, such as when user 104 is viewing a transcription of a voicemail left on his / her phone (e.g., FIG. 1B). As noted above, this type of application would benefit from using future context to improve performance, at the expense of increased processing time. Here, model 200c uses a first encoder 210 to generate a first high-dimensional feature representation e at each time step for input to a second encoder 220. swhereas the first decoder 204 generates a first high-level feature representation e s The model 200c then performs non-streaming speech recognition on all of the audio data 110, and the second encoder 220 decodes the first high-level feature representation e received from the first encoder 210. s Using the second high-level feature representation e a The second decoder 206 then generates the final speech recognition results 120, 120b. Because generating streaming speech recognition in real-time is of little value to the user and latency is not a factor, the model 200c may simply operate in a non-streaming mode only to generate the final speech recognition results 120, 120b.

[0042] Referring to FIG. 2D, the model 200 can be generalized to a three-pass model 200d (i.e., a large model) that operates only in a non-streaming mode. This non-streaming mode may occur, for example, in non-latency intensive applications where the user 104 expects higher accuracy in speech recognition from the model 200. As mentioned above, this type of application benefits from increasing the amount of future context at the expense of increased latency to the final speech recognition results 120, 120b. Here, the model 200d uses a first encoder 210 to generate a first high-dimensional feature representation e at each time step for input to the second encoder 220. s whereas the first decoder 204 generates a first high-level feature representation e s The model 200d then performs non-streaming speech recognition on all of the audio data 110, and the second encoder 220 decodes the first high-level feature representation e received from the first encoder 210. s Using the second high-level feature representation e a Unlike the model 200c, the second decoder 206 generates a second high-level feature representation e aInstead, the model 200d also performs non-streaming speech recognition on all of the audio data 110, and the third encoder 230 decodes the second high-level feature representation e received from the second encoder 220. a Using the third higher-level feature representation, e r The third decoder 208 then generates the final speech recognition results 120, 120b.

[0043] 3A and 3B show an example of a two-stage training process 300, 300a-b for training the model 200 to operate dynamically between streaming and / or non-streaming modes for various model sizes. In some configurations, the training process 300 is executed on the remote computing device 60 of FIGS. 1A and 1B. The training process 300 takes a number of training utterances 132, 132a-n stored in a sample database 130 and trains the model 200 with the training utterances 132 using a two-stage process. The sample database 130 may reside on the memory hardware of the remote computing device 60. As described above with respect to FIG. 2A, the first encoder 210, the second encoder 220, and the third encoder 230 may be trained jointly or together, simplifying the training process 300. Jointly training the encoders 210, 220, 230 means that the non-streaming encoder 220 can be directly trained on the output of the streaming encoder 210 (e.g., the first high-level feature representation e) instead of the input acoustic features (e.g., the input acoustic frames 110). s ) On the other hand, the non-streaming encoder 230 may use the second high-level feature representation e instead of the input acoustic features (e.g., the input acoustic frame 110). a ) Advantageously, this ensures that model 200 learns to operate in either streaming or non-streaming mode during inference.

[0044] As shown in Figures 3A and 3B, the model 200 has three processing paths, one for the streaming mode of model 200b (shown in Figure 2B), one for the non-streaming mode of model 200c (shown in Figure 2C), and one for the non-streaming mode of model 200d (shown in Figure 2D). Referring to Figure 3A, the training process 300 employs a first-stage training process 300a. In the first stage, the training process 300a uses cross-entropy training by forwarding the same mini-batch of training utterances 132 through each of the cascaded encoder models 200b, 200c, and 200d, and linearly combines the losses of each cascaded encoder model 200b, 200c, and 200d with weights that sum to one. In the first stage of the training process 300a, jointly training the cascaded encoders 210, 220, and 230 includes minimizing a weighted sum of losses between the input processing paths.

[0045] Since there are three input processing paths in the training process 300, the model loss includes three loss functions. Specifically, the loss in streaming mode of the model 200b is generally defined as the sum of negative log probabilities corresponding to the probability distribution over the possible speech recognition hypotheses based on the input training utterance 132. That is, the model loss for the connection from the first encoder 210 to the decoder 204 is:

[0046]

number

[0047] The model loss in the non-streaming mode (e.g., model 200c) is also commonly defined as the sum of negative log probabilities corresponding to the probability distribution over the possible speech recognition hypotheses based on the input training utterances 132. Thus, the model loss of the connection from the second encoder 220 to the decoder 204 is

[0048]

number

[0049] The model loss in the non-streaming mode (e.g., model 200b) is also commonly defined as the sum of negative log probabilities corresponding to the probability distribution over possible speech recognition hypotheses based on the input training utterance 132. Thus, the model loss connecting the third encoder 230 to the third decoder 206 is

[0050]

number

[0051] In some implementations, when the model 200 includes only the encoders 210, 220 and the decoders 204, 206, the total loss between the two input paths of the first decoder 204 and the second decoder 206 is defined as

[0052]

number

[0053] is computed as a weighted sum of each input path, where λ is a weighting term. In the second stage, the training process 300b performs discriminative training to fine-tune the model 200 initialized during the first stage training 300a using cross-entropy training. Here, discriminative training includes fine-tuning the model 200 using a minimum word error rate (MWER) criterion. During the second stage of the training process 300b, for each training utterance 132, the training may be performed in either a streaming or non-streaming path / path. In other words, the input processing path is probabilistically selected to train either the model 200b, or the model 200c, or the model 200d, and the training process 300b randomly samples each model 200b, 200c, 200d with a probability equal to its loss weight. Here, the loss weight of the model 200b may be 0.8, the loss weight of the model 200c may be 0.15, and the loss weight of the model 200d may be 0.05. Spending the most time training the path of model 200b improves the accuracy of the model without sacrificing performance in the later paths 200c, 200d. Then, each decoder 204, 206, 208 of the sampled path performs a beam search on the training utterance 132 to generate an n-best list of hypotheses, and the negative log-likelihoods for all hypotheses are calculated and renormalized in the n-best space to approximate the word error loss for loss minimization in model 200. Thus, for each hypothesis in the n-best list generated by each decoder for the corresponding training utterance, a respective number of word errors for the ground truth transcription of the corresponding training utterance is identified, and the training process uses the MWER criterion to minimize the word error rate based on the respective number of word errors identified for each hypothesis in the n-best list of the corresponding training utterance 132. As used herein, the n-best list includes the n highest ranked hypotheses, and each hypothesis in the n-best list includes a candidate transcription for the same corresponding training utterance 132.In some implementations, n is equal to 4, so that each decoder 204, 206, 208 of the sampled path performs a beam search on the training utterances 132 to generate the top 4 hypotheses, and the negative log-likelihoods are calculated for all hypotheses (more than 4) and renormalized in the top 4 space. Furthermore, by sampling the training utterances 132, the training process only needs to calculate the loss once per training utterance 132 on each path / path, which significantly speeds up the second stage of the training process 300b. In some implementations, if longer training times are acceptable, an alternative training process is employed to train each input processing path / path with each training utterance and calculate the losses of both models 200b and 200c for each training utterance 132.

[0054] In the example shown in FIG. 3A, the training utterances 132b, 132c are selected to train each of the processing paths represented by the cascade encoder models 200b, 200c, 200d in the first stage of the training process 300a. The cascade encoder model 200b receives the training utterances 132b, 132c, and the first encoder 210 converts the training utterances 132b, 132c into a first high-level feature representation (e.g., speech embedding) as output. The first decoder 204 then receives the first high-level feature representation of the training utterances 132b, 132c as input and generates an output that is tested for its accuracy. Similarly, the training utterances 132b, 132c are selected to train the second processing path represented by the cascade encoder model 200c. The cascade encoder model 200c receives the training utterances 132b, 132c, and the first encoder 210 converts the training utterances 132b, 132c into a first high-level feature representation (e.g., speech embedding) as an output. The second encoder 220 receives the first high-level feature representation of the training utterances 132b, 132c as an input and generates a second high-level feature representation of the training utterances 132b, 132c as an output. The second decoder 206 then receives the second high-level feature representation of the training utterances 132b, 132c as an input and generates an output that is tested for its accuracy. Similarly, the cascade encoder model 200d receives the training utterances 132b, 132c, and the first encoder 210 converts the training utterances 132b, 132c into a first high-level feature representation (e.g., speech embedding) as an output. The second encoder 220 receives as input the first high-level feature representation of the training utterances 132b, 132c and generates as output a second high-level feature representation of the training utterances 132b, 132c. The third encoder 230 receives as input the second high-level feature representation of the training utterances 132b, 132c and generates as output a third high-level feature representation of the training utterances 132b, 132c. The third decoder 208 then receives as input the third high-level feature representation of the training utterances 132b, 132c and generates an output that is tested for its accuracy.

[0055] As shown in FIG. 3B, in the second stage of the training process 300b, the training utterances 132 used by the training process 300b are randomly selected, and each path of the model 200 is randomly selected with a probability equal to the loss weight assigned to each path of the model 200. Based on the loss weight, the cascade encoder model 200b receives the training utterances 132a, 132b, 132c, and the first encoder 210 converts the training utterances 132a, 132b, 132c into a first high-level feature representation (e.g., speech embedding) as output. The first decoder 204 then receives the first high-level feature representation of the training utterances 132a, 132b, 132c as input and generates an output that is tested for its accuracy. Based on the loss weights, the cascade encoder model 200c receives the training utterances 132b, 132d, and the first encoder 210 converts the training utterances 132b, 132d into a first high-level feature representation (e.g., speech embedding) as an output. The second encoder 220 receives the first high-level feature representation of the training utterances 132b, 132d as an input and generates a second high-level feature representation of the training utterances 132b, 132d as an output. The second decoder 206 then receives the second high-level feature representation of the training utterances 132b, 132d as an input and generates an output that is tested for its accuracy. Based on the loss weights, the cascade encoder model 200d receives the training utterances 132c, and the first encoder 210 converts the training utterances 132c into a first high-level feature representation (e.g., speech embedding) as an output. The second encoder 220 receives as input the first high-level feature representation of the training utterance 132c and generates as output a second high-level feature representation of the training utterance 132c. The third encoder 230 receives as input the second high-level feature representation of the training utterance 132c and generates as output a third high-level feature representation of the training utterance 132c. The third decoder 208 then receives as input the third high-level feature representation of the training utterance 132c and generates an output that is tested for its accuracy.

[0056] 4 includes a flowchart of an example operational procedure of a method 400 for performing streaming and non-streaming speech recognition using the ASR model 200. In operation 402, the method 400 includes receiving a sequence of acoustic frames 110 as an input to the ASR model 200. In operation 404, the method 400 also includes generating, by the first encoder 210, a first high-level feature representation of a corresponding acoustic frame 110 in the sequence of acoustic frames 110 at each of a plurality of output steps. In operation 406, the method 400 includes receiving, as an input to both the first decoder 204 and the second encoder 220, the first high-level feature representation generated by the first encoder 210 at each of a plurality of output steps.

[0057] The method 400 also includes generating, by the second encoder 220 at each of the plurality of output steps, a second high-order feature representation for the corresponding first high-order feature representation, at operation 408. At operation 410, the method 400 includes generating, by the first decoder 204 at each of the plurality of output steps, a first probability distribution for the possible speech recognition hypotheses. The method 400 further includes receiving, as an input to the second decoder 206, the second high-order feature representation generated by the second encoder 220 at each of the plurality of output steps, at operation 412. At operation 414, the method 400 also includes generating, by the second decoder 206 at each of the plurality of output steps, a second probability distribution for the possible speech recognition hypotheses.

[0058] 5 is a schematic diagram of an exemplary computing device 500 that may be used to implement the systems (e.g., speech subsystem 108, ASR system 109, user interface generator 107, and / or model 200) and methods (e.g., method 400) described herein. Computing device 500 is intended to represent various types of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components depicted herein, their connections and relationships, and their functions are for illustrative purposes only and are not intended to limit the embodiments of the invention described and / or claimed in this document.

[0059] The computing device 500 includes a processor 510 (e.g., data processing hardware), a memory 520 (e.g., memory hardware), a storage device 530, a high-speed interface / controller 540 that connects to the memory 520 and a high-speed expansion port 550, and a low-speed interface / controller 560 that connects to a low-speed bus 570 and the storage device 530. Each of the components 510, 520, 530, 540, 550, and 560 are interconnected using various buses and may be mounted on a common motherboard or may exist in other manners as desired. The processor 510 (e.g., data processing hardware 12, 62 of FIGS. 1A-1B) may process instructions for execution within the computing device 500, including instructions stored in the memory 520 or the storage device 530, to display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 580 connected to the high-speed interface 540. In other implementations, multiple processors and / or multiple buses may be used as needed, along with multiple memories and multiple memory types. Also, multiple computing devices 500 may be connected together (eg, as a server bank, a group of blade servers, or a multi-processor system) with each device performing some of the required operations.

[0060] The memory 520 (i.e., memory hardware 14, 64 in FIG. 1A-1B) stores information non-transiently within the computing device 500. The memory 520 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-transient memory 520 may be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by the computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0061] The storage device 530 can provide mass storage to the computing device 500. In some implementations, the storage device 530 is a computer-readable medium. In various different implementations, the storage device 530 can be a device array, including a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or a storage area network or other configuration of devices. In additional implementations, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods as described above. The information carrier is a computer-readable medium or a machine-readable medium, such as the memory 520, the storage device 530, or a memory on the processor 510.

[0062] The high-speed controller 540 manages the more bandwidth-intensive operations of the computing device 500, while the low-speed controller 560 manages the less bandwidth-intensive operations. Such an allocation of roles is merely an example. In some implementations, the high-speed controller 540 is coupled to the memory 520, the display 580 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 550 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 560 is coupled to the storage device 530 and a low-speed expansion port 590. The low-speed expansion port 590 may include various communication ports (USB, Bluetooth, Ethernet, wireless Ethernet, etc.) that can connect to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a network device such as a switch or router, via a network adapter or the like.

[0063] The computing device 500 can be implemented in many different forms, as shown in the figure: for example, it may be implemented as a standard server 500a, or multiple times within a cluster of such servers 500a, as a laptop computer 500b, or as part of a rack server system 500c.

[0064] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may be specialized or general purpose and may include implementations in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, at least one input device, and at least one output device coupled to receive data and instructions from and transmit data and instructions to the storage system.

[0065] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," an "app," or a "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0066] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in a high-level procedural and / or object-oriented programming language, and / or assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0067] The processes and logic flows described herein can be implemented by one or more programmable processors, also referred to as data processing hardware, executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be implemented by special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Processors suitable for executing computer programs include, for example, both general purpose and special purpose processors, as well as any one or more processors of any type of digital computer. Generally, a processor receives instructions and data from a read-only memory, a random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively connected to receive data from or transmit data to them, or both. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0068] To interact with a user, one or more aspects of the invention may be implemented in a computer having a display device (e.g., a CRT or LCD monitor, or a touch screen) for displaying information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user. For example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic, verbal, or tactile input. Additionally, the computer may interact with the user by sending and receiving documents to a device used by the user, e.g., by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0069] Although several embodiments have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. An automatic speech recognition (ASR) model (200), A first encoder (210), receiving as input a sequence of a plurality of acoustic frames (110); the first encoder (210) configured to generate, in each of a plurality of output steps, a first high-level feature representation for a corresponding acoustic frame (110) in the sequence of the plurality of acoustic frames (110); A first decoder (204), receiving as input the first high-level feature representation generated by the first encoder (210) at each of the plurality of output steps; the first decoder (204) configured to generate a first probability distribution for possible speech recognition hypotheses at each of the plurality of output steps; A second encoder (220), receiving as input the first high-level feature representation generated by the first encoder (210) at each of the plurality of output steps; the second encoder (220) configured to generate a second high-level feature representation for a corresponding first high-level feature frame at each of the plurality of output steps; A second decoder (206), receiving as input the second high-level feature representation generated by the second encoder (220) at each of the plurality of output steps; and the second decoder (206), configured to generate a second probability distribution for possible speech recognition hypotheses at each of the plurality of output steps.

2. 2. The ASR model of claim 1, wherein the first decoder is further configured to generate a partial speech recognition result based on the first probability distribution for possible speech recognition hypotheses.

3. The first encoder (210) A plurality of unidirectional long short-term memory (LSTM) layers; A plurality of conformer layers; The ASR model (200) of claim 1 or 2, further comprising a causal encoder including one of a plurality of transformer layers.

4. The second encoder (220) A plurality of unidirectional long short-term memory (LSTM) layers; A plurality of conformer layers; The ASR model (200) of any one of claims 1 to 3, further comprising a non-causal encoder including one of a plurality of transformer layers.

5. The first decoder (204) A prediction network (250), It receives as input the sequence of non-blank symbols output by the final softmax layer, the prediction network (250) configured to generate dense representations at each of the plurality of output steps; A combining network (240), receiving as input the dense representation produced by the prediction network (250) at each of the plurality of output steps and the first high-level feature representation produced by the first encoder (210) at each of the plurality of output steps; and the combination network (240) configured to generate a first probability distribution for possible speech recognition hypotheses at each of the plurality of output steps.

6. The prediction network (250): a long short-term memory (LSTM) based prediction network (250), or 6. The ASR model (200) of claim 5, comprising a V2 embedded lookup table.

7. The second decoder (206) A prediction network (250), It receives as input the sequence of non-blank symbols output by the final softmax layer, the prediction network (250) configured to generate dense representations at each of the plurality of output steps; A combining network (240), receiving as input the dense representation produced by the prediction network (250) at each of the plurality of output steps and the second higher-level feature representation produced by the second encoder (220) at each of the plurality of output steps; and the combination network (240) configured to generate a second probability distribution for possible speech recognition hypotheses at each of the plurality of output steps.

8. The prediction network (250): a long short-term memory (LSTM) based prediction network (250), or 8. The ASR model (200) of claim 7, comprising a V2 embedded lookup table.

9. The ASR model (200) of any one of claims 1 to 8, wherein the first encoder (210) includes more parameters than the second encoder (220).

10. A third encoder (230), receiving as input the second high-level feature representation generated by the second encoder (220) at each of the plurality of output steps; a third encoder (230) configured to generate a third high-level feature representation for a corresponding second high-level feature representation at each of the plurality of output steps; A third decoder (208), receiving as input the third high-level feature representation generated by the third encoder (230) at each of the plurality of output steps; and a third decoder (208) configured to generate a third probability distribution for possible speech recognition hypotheses at each of the plurality of output steps.

11. A computer-implemented method (400), when executed by data processing hardware (610), causing the data processing hardware (610) to: Receiving a sequence of a plurality of acoustic frames (110); generating, by a first encoder (210), at each of a plurality of output steps, a first high-level feature representation for a corresponding acoustic frame (110) in the sequence of the plurality of acoustic frames (110); generating, by a second encoder (220), a second high-level feature representation for each of the plurality of output steps, for the corresponding first high-level feature representation; generating, by a first decoder (204), a first probability distribution over possible speech recognition hypotheses at each of said plurality of output steps; generating, by a second decoder (206), at each of said plurality of output steps, a second probability distribution over the possible speech recognition hypotheses.

12. 12. The computer-implemented method (400) of claim 11, wherein the operations further comprise generating a partial speech recognition result (120) based on the first probability distribution over possible speech recognition hypotheses.

13. The first encoder (210) comprises a causal encoder, the causal encoder comprising: Multiple unidirectional long short-term memory (LSTM) layers, Multiple conformer layers, or 13. The computer-implemented method (400) of claim 11 or 12, comprising one of a plurality of transformer layers.

14. The second encoder (220) comprises a non-causal encoder, the non-causal encoder comprising: Multiple unidirectional long short-term memory (LSTM) layers, Multiple conformer layers, or The computer-implemented method (400) of any one of claims 11 to 13, comprising one of a plurality of transformer layers.

15. The operation includes, at each of the plurality of output steps, generating a dense representation by a prediction network (250) of said first decoder (204) based on the sequence of non-empty symbols output by a final softmax layer; generating, by a combination network (240) of the first decoder (204), the first probability distribution over possible speech recognition hypotheses based on the dense representation generated by the prediction network (250).

16. The prediction network (250) of the first decoder (204) a long short-term memory (LSTM) based prediction network (250), or 20. The computer-implemented method (400) of claim 15, comprising a V2 embedded lookup table.

17. The operation includes, at each of the plurality of output steps, generating a dense representation at each of the plurality of output steps by a prediction network (250) of the second decoder (206); receiving the second high-level feature representations generated by the second encoder (220) at each of the plurality of output steps and the dense representations generated by the prediction network (250) at each of the plurality of output steps as inputs to a combination network (240) of the second decoder (206); 17. The computer-implemented method (400) of any one of claims 11 to 16, further comprising generating, by the combination network of the second decoder (206) at each of the plurality of output steps, the second probability distributions over possible speech recognition hypotheses.

18. The prediction network (250) of the second decoder (206) a long short-term memory (LSTM) based prediction network (250), or 20. The computer-implemented method (400) of claim 17, comprising a V2 embedded lookup table.

19. The computer-implemented method (400) of any one of claims 11 to 18, wherein the first encoder (210) includes a greater number of parameters than the second encoder (220).

20. The operations include performing streaming and non-streaming speech recognition on the sequence of acoustic frames (110), receiving the second high-level feature representations produced by the second encoder (220) at each of the plurality of output steps as input to a third encoder (230); generating, by the third encoder (230), a third high-level feature representation for a corresponding second high-level feature representation at each of the plurality of output steps; receiving the third high-level feature representations produced by the third encoder (230) at each of the plurality of output steps as input to a third decoder (208); 20. The computer-implemented method (400) of any one of claims 11 to 19, further comprising generating, by the third decoder (208) at each of the plurality of outputting steps, a third probability distribution over possible speech recognition hypotheses.

Images