Cascade encoder for simplified streaming and non-streaming speech recognition

Abstract

An automatic speech recognition (ASR) model (200) includes a first encoder (210), a second encoder (220), and a decoder (204). The first encoder receives a series of acoustic frames (110) as input and generates, at each of a plurality of output steps, a first higher-order feature representation (203) of a corresponding acoustic frame. The second encoder receives, as input, the first higher-order feature representation generated by the first encoder at each of the plurality of output steps and generates, at each of the plurality of output steps, a second higher-order feature representation (205) of a corresponding first higher-order feature frame. The decoder receives, as input, the second higher-order feature representation generated by the second encoder at each of the plurality of output steps and generates, at each of a plurality of time steps, a first probability distribution over possible speech recognition hypotheses.

Description

Technical Field

[0001] This disclosure relates to an end-to-end device model for simplified streaming and non-streaming automatic speech recognition (ASR). Background Technology

[0002] Automatic speech recognition (ASR) systems have evolved from multiple models, each with a specific purpose, to ensemble models where a single neural network is used to directly map audio waveforms (i.e., input sequences) to output sentences (i.e., output sequences). This ensemble produces a sequence-to-sequence approach that, given a set of audio features, generates a sequence of words (or word elements). Through this ensemble structure, all components of the model can be jointly trained as a single end-to-end (E2E) neural network. Here, an E2E model refers to a model whose architecture is entirely built from neural networks. Fully neural networks operate without externally and / or manually designed components (e.g., finite-state permuters, dictionaries, or text normalization modules). Furthermore, when training E2E models, these models typically do not require bootstrapping from decision trees or temporal alignment from separate systems. These E2E automatic speech recognition (ASR) systems have made significant progress, outperforming traditional ASR systems in several common benchmarks, including word error rate (WER). The architecture of an E2E ASR model is highly application-dependent. For example, many applications involving user interaction (e.g., voice search or on-device dictation) require models to perform recognition in a streaming manner. Other applications, such as offline video captioning, do not require models to be streaming and can leverage future context to improve performance. Existing end-to-end (E2E) approaches typically include separate ASR models for streaming and non-streaming applications. Managing multiple models for different applications presents unique challenges and can potentially slow down the execution time of ASR systems. Summary of the Invention

[0003] One aspect of this disclosure provides an automatic speech recognition (ASR) model, the ASR model including a first encoder, a second encoder, and a decoder. The first encoder receives a series of acoustic frames as input and generates a first high-order feature representation of a corresponding acoustic frame in the series of acoustic frames at each of a plurality of output steps. The second encoder receives the first high-order feature representation generated by the first encoder at each of the plurality of output steps as input and generates a second high-order feature representation of the corresponding first high-order feature frame at each of the plurality of output steps. The decoder receives the second high-order feature representation generated by the second encoder at each of the plurality of output steps as input and generates a first probability distribution over possible speech recognition hypotheses at each of a plurality of time steps.

[0004] Implementations of this disclosure may include one or more of the following optional features. In some implementations, the second encoder generates a second high-order feature representation without receiving any audio frames as input. In some examples, the decoder is further configured to receive the first high-order feature representation generated by the first encoder at each of a plurality of output steps as input, and to generate a second probability distribution over possible speech recognition hypotheses at each of a plurality of time steps. In these examples, the model further includes generating a final speech recognition result by modifying an initial speech recognition result from a first probability distribution based on the second probability distribution. Alternatively or additionally, the decoder may further include a prediction network and a joint network. The prediction network is configured to receive a series of non-whitespace symbols output by the final Softmax layer as input, and to generate a dense representation at each of a plurality of output steps. The joint network is configured to receive the dense representation generated by the prediction network at each of a plurality of output steps as input. When the ASR model operates in streaming mode, the joint network is also configured to receive the first high-order feature representation generated by the first encoder at each of a plurality of output steps, and to generate a second probability distribution over possible speech recognition hypotheses at each of a plurality of output steps. When the ASR model operates in non-streaming mode, the joint network is configured to receive a second high-order feature representation generated by the second encoder at each of the multiple output steps, and to generate a first probability distribution over possible speech recognition hypotheses at each of the multiple output steps.

[0005] In some implementations, the decoder includes a Long Short-Term Memory (LSTM) based decoder. Alternatively, the decoder includes a V2 embedding lookup table. In some examples, the first encoder includes a causal encoder comprising multiple unidirectional LSTM layers. Additionally or alternatively, the second encoder includes a non-causal encoder comprising one or more bidirectional LSTM layers. In some implementations, the model is configured to receive a set of training utterances and jointly train the first and second encoders using said set of training utterances. Here, the model jointly trains the first and second encoders by generating a first loss for the first encoder, a second loss for the second encoder, and minimizing a weighted sum of the first and second losses.

[0006] Another aspect of this disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the hardware to perform operations including receiving a series of acoustic frames as input to an automatic speech recognition (ASR) model, and performing streaming and non-streaming speech recognition on the series of acoustic frames using the ASR model. The operations further include generating a first high-order feature representation of a corresponding acoustic frame in the series of acoustic frames by a first encoder at each of a plurality of output steps, and receiving the first high-order feature representation generated by the first encoder at each of the plurality of output steps as input to a second encoder. The operations also include generating a second high-order feature representation of the corresponding first high-order feature frame by the second encoder at each of the plurality of output steps; receiving the second high-order feature representation generated by the second encoder at each of the plurality of output steps as input to a decoder; and generating a first probability distribution over possible speech recognition hypotheses at each of a plurality of time steps.

[0007] This aspect may include one or more of the following optional features. In some implementations, the second encoder generates a second higher-order feature representation without receiving any audio frames as input. In some examples, when the ASR model operates in streaming mode, the operation further includes receiving a first higher-order feature representation generated by the first encoder at each of multiple output steps as input to the decoder, and the decoder generating a second probability distribution over possible speech recognition hypotheses at each of multiple time steps. In these examples, the operation may further include generating a final speech recognition result by modifying the initial speech recognition result from a first probability distribution based on the second probability distribution. Alternatively or additionally, when the ASR model operates in streaming mode, the operation further includes receiving a series of non-whitespace symbols output by the final Softmax layer as input to the prediction network; generating a dense representation by the prediction network at each of multiple output steps; receiving the dense representation generated by the prediction network at each of multiple steps and the first higher-order feature representation generated by the first encoder at each of multiple output steps as input to the joint network; and generating a second probability distribution over possible speech recognition hypotheses by the joint network at each of multiple output steps. When the ASR model operates in non-streaming mode, the operation further includes receiving a series of non-whitespace symbols output by the final Softmax layer as input to the prediction network; generating a dense representation by the prediction network at each of the multiple output steps; receiving the dense representation generated by the prediction network at each of the multiple output steps and a second high-order feature representation generated by the second encoder at each of the multiple output steps as input to the joint network; and generating a first probability distribution on possible speech recognition hypotheses by the joint network at each of the multiple output steps.

[0008] In some implementations, the decoder includes a Long Short-Term Memory (LSTM) based decoder. Alternatively, the decoder includes a V2 embedding lookup table. In some examples, the first encoder includes a causal encoder comprising multiple unidirectional LSTM layers. Additionally or alternatively, the second encoder includes a non-causal encoder comprising one or more bidirectional LSTM layers. In some implementations, the operation further includes receiving a set of training utterances and jointly training the first encoder and the second encoder using the set of training utterances. Here, the model jointly trains the first encoder and the second encoder by generating a first loss for the first encoder, a second loss for the second encoder, and minimizing a weighted sum of the first and second losses.

[0009] Details of one or more implementations of this disclosure are set forth below in the accompanying drawings and detailed description. Other aspects, features, and advantages will become apparent from the detailed description, the accompanying drawings, and the claims. Attached Figure Description

[0010] Figure 1A and 1B This is a schematic diagram of an example speech environment for automatic speech recognition using a cascaded encoder model architecture.

[0011] Figures 2A to 2C This is a schematic diagram of the cascade encoder model architecture.

[0012] Figure 3 This is a schematic diagram of an example training process used to facilitate the cascade encoder model's learning of consistent predictions for both non-synthetic and synthetic speech.

[0013] Figure 4 This is a flowchart of an example layout for implementing a method for a cascade encoder model used for streaming and non-streaming automatic speech recognition.

[0014] Figure 5 This is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0015] Similar reference numerals in the various figures indicate similar elements. Detailed Implementation

[0016] End-to-end (E2E) automatic speech recognition (ASR) models are traditionally constructed to operate in either streaming or non-streaming modes. Typically, E2E ASR models include an encoder and a decoder as the main components. Applications involving end-user interaction (e.g., voice search or on-device dictation) may require the model to perform recognition in a streaming manner, where words are expected to be output with the least possible latency as they are spoken. This prevents the use of models that utilize future context to improve accuracy (e.g., bidirectional LSTM). In contrast, applications such as offline video captioning do not require streaming recognition and can fully utilize any available future context to improve performance.

[0017] The implementation described in this paper involves a single E2E ASR model that uses a cascaded encoder capable of operating in both streaming and non-streaming modes. The cascaded encoder comprises a streaming encoder and a non-streaming encoder, and a single decoder of the ASR model is configured to learn to decode either the output from the streaming encoder or the output from the non-streaming encoder. Besides the ASR model, the architecture can also be applied to other models, such as implementing machine translation in both streaming and non-streaming modes.

[0018] Figure 1A and 1B Examples of voice environments 100, 100a-b are provided. In voice environment 100, user 104 interacts with a computing device, such as user device 10, via voice input. User device 10 (also commonly referred to as device 10) is configured to capture the voice (e.g., streaming audio data) of one or more users 104 within voice environment 100. Here, streaming audio data may refer to spoken words 106 of user 104, which serve as audible queries, commands from device 10, or audible communications captured by device 10. The voice-enabled system of device 10 can process queries or commands by answering queries and / or causing commands to be executed / completed by one or more downstream applications.

[0019] User device 10 may correspond to any computing device associated with user 104 and be capable of receiving audio data. Some examples of user device 10 include, but are not limited to, mobile devices (e.g., mobile phones, tablet computers, laptop computers, etc.), computers, wearable devices (e.g., smartwatches), smart appliances, Internet of Things (IoT) devices, vehicle infotainment systems, smart displays, smart speakers, etc. User device 10 includes data processing hardware 12 and memory hardware 14, the memory hardware communicating with data processing hardware 12 and storing instructions that, when executed by data processing hardware 12, cause data processing hardware 12 to perform one or more operations. User device 10 further includes an audio system 16 having: audio capture devices (e.g., microphones) 16, 16a for capturing spoken words 106 within a speech environment 100 and converting the spoken words into electrical signals; and voice output devices (e.g., speakers) 16, 16b for transmitting audible audio signals (e.g., output audio data from device 10). Although user device 10 implements a single audio capture device 16a in the illustrated example, user device 10 may implement an array of audio capture devices 16a without departing from the scope of this disclosure, whereby one or more capture devices 16a in the array may not physically reside on user device 10, but communicate with audio system 16.

[0020] In the speech environment 100, an Automatic Speech Recognition (ASR) system 109 implementing ASR model 200 (also referred to as model 200) resides on user device 10 of user 104 and / or on a remote computing device 60 (e.g., one or more remote servers of a distributed system executing in a cloud computing environment) communicating with user device 10 via network 40. User device 10 and / or remote computing device 60 also include an audio subsystem 108 configured to receive utterance 106 spoken by user 104 and captured by audio capture device 16a, and to convert utterance 106 into a corresponding digital format associated with an input audio frame 110 that can be processed by ASR system 109. Figure 1A In the example shown, user 104 utters a corresponding utterance 106, and audio subsystem 108 converts utterance 106 into corresponding audio data (e.g., sound frames) 110 for input to ASR system 109. Subsequently, model 200 receives the audio data 110 corresponding to utterance 106 as input and generates / predicts a corresponding transcription 120 (also referred to as recognition result / hypothesis 120) of utterance 106 as output. This is described in more detail below (e.g., Figure 3This allows model 200 to be trained in a single training phase, simplifying the process of training model 200 to operate in both streaming and non-streaming modes. Model 200 also includes a decoder 204 shared among its encoders (also referred to as a shared decoder 204), enabling model 200 to be a single model capable of operating in both streaming and non-streaming modes (e.g., in contrast to two separate models where each model is dedicated to either streaming or non-streaming mode). For example, as... Figure 1A As shown, the digital assistant application 50 running on user device 10 may require speech recognition to be streaming, so that words, word fragments, and / or individual characters appear on the screen as soon as they are spoken. Furthermore, the user 104 of user device 10 may also have a low tolerance for latency when issuing a query to be performed by the digital assistant application 50. In these scenarios where the application requires minimal latency, model 200 operates in streaming mode, where model 200 can provide streaming transcription capabilities in real time as user 104 speaks utterance 106. On the other hand, when user 104 has a higher tolerance for speech recognition latency and / or the utterance 106 to be recognized is associated with long-form speech (i.e., speech consisting of complete paragraphs or multiple sentences), the same model 200 can operate in non-streaming mode and can utilize a prediction network to provide accurate transcription 120, but this results in increased latency. Therefore, ASR system 109 can implement only a single ASR model 200 for multiple different speech recognition tasks to provide both streaming and non-streaming transcription capabilities, rather than utilizing separately trained ASR models in a task-by-task manner.

[0021] In some implementations, model 200 first performs streaming speech recognition on the audio data 110, and then performs non-streaming speech recognition on the output of the streaming encoder. For example, in the example shown, model 200 uses a first encoder (i.e., a low-latency encoder). Figure 2B )) Perform streaming speech recognition on audio data 110 to produce partial speech recognition results 120, 120a (e.g., a second probability distribution of speech recognition result 120a), and use a second encoder (i.e., a high-latency encoder) Figure 2C Non-streaming speech recognition is performed on the encoded audio data 110 to produce final speech recognition results 120, 120b (e.g., a first probability distribution of speech recognition result 120b). It is noteworthy that model 200 outputs a partial speech recognition result 120a upon receiving audio data 110, and subsequently outputs the final speech recognition result 120b once all audio data 110 has been received, after user 104 has finished speaking utterance 106. Therefore, the output of the final speech recognition result 120b for the input utterance 106 can be delayed from the partial speech recognition result 120a.

[0022] User device 10 and / or remote computing device 60 also execute user interface generator 107, which is configured to present a representation of the transcription 120 of utterance 106 to user 104 of user device 10. As described in more detail below, user interface generator 107 may stream partial speech recognition results 120a during time 1 and subsequently display the final speech recognition result 120b during time 2. In some configurations, the transcription 120 output from ASR system 109 may be processed, for example, by a natural language understanding (NLU) module executing on user device 10 or remote computing device 60 to execute a user command / query specified by utterance 106. Alternatively or additionally, a text-to-speech system (not shown) (e.g., executing on any combination of user device 10 or remote computing device 60) may convert the transcription 120 into synthesized speech for audible output by user device 10 and / or another device.

[0023] exist Figure 1A In the example, user 104 in voice environment 100a interacts with a program or application 50 (e.g., digital assistant application 50a) using user device 10 of ASR system 109. For example, Figure 1A The text depicts a conversation between user 104 and digital assistant application 50a, with digital assistant application 50a displaying digital assistant interface 18 on the screen of user device 10. In this example, user 104 asks digital assistant application 50a, “What song is playing right now?” This question from user 104 is a spoken word 106 captured by audio capture device 16a and processed by audio system 16 of user device 10. In this example, audio system 16 receives spoken word 106 and converts it into an audio frame 110 for input to ASR system 109.

[0024] Continuing the example, when user 104 speaks, model 200 uses the first encoder 210 (i.e., ...) upon receiving the sound frame 110 corresponding to utterance 106. Figure 2A The sound frame 110 is encoded, and then decoder 204 is used. Figure 2A The encoded representation of sound frame 110 is decoded into a partial speech recognition result 120a. During time 1, the user interface generator 107 presents a representation of the partial speech recognition result 120a of speech 106 to the user 104 of the user device 10 via the digital assistant interface 18 in a streaming manner, so that words, word fragments and / or single characters appear on the screen as soon as they are spoken.

[0025] After receiving all (or a certain number) of the sound frames 110 corresponding to the utterance 106 and after the first encoder 210 has encoded these sound frames 110, the second encoder 220 (i.e., Figure 2A The decoder 204 encodes the output from the first encoder 210 to generate an encoding of a set of acoustic frames 110 corresponding to the utterance 106 already encoded by the first encoder 210. The decoder 204 then decodes the acoustic frames 110 already encoded by the second encoder 220 into the final speech recognition result 120b. For example, while the first encoder 210 encodes all acoustic frames 110 corresponding to the utterance 106 (e.g., as acoustic frames 110 are received), the second encoder 220 encodes all acoustic frames 110 already encoded by the first encoder 210. In this respect, by encoding on multiple encoded acoustic frames 110, the second encoder 210 is able to provide greater context awareness in a non-streaming manner (e.g., by receiving a representation of all acoustic frames 110 of the utterance 106), which potentially reconciles or corrects aspects of the utterance 106 that may have been missed or misunderstood due to the streaming nature of the first encoder 210. In some examples, an indication that user 104 has completed speaking utterance 106 (e.g., an endpoint) triggers the second encoder 220 of model 200 to encode all audio frames 110. During time 2, user interface generator 107 presents a representation of the final speech recognition result 120b of utterance 106 to user 104 of user device 10 via digital assistant interface 18. In some implementations, user interface generator 107 replaces (or modifies) the representation of partial speech recognition result 120a with the representation of final speech recognition result 120b. For example, final speech recognition result 120b is considered more accurate than partial speech recognition result 120a, and final speech recognition result 120b is ultimately displayed as transcription 120 to correct any terms that may have been misrecognized in partial speech recognition result 120a. In this example, the streaming partial speech recognition result 120a output by model 200 and displayed on the screen of user device 10 at time 1 is associated with low latency and provides responsiveness to user 104, indicating that his / her query is being processed. Therefore, the partial speech recognition result 120a may incorrectly predict that user 104's utterance 106 is "What song is play right now?". The final speech recognition result 120, output by model 200 and displayed on the screen at time 2 with an increased delay, improves speech recognition quality in terms of accuracy by recognizing that user 104 said "playing". However, since the user interface generator 107 displays the partial speech recognition result when the user speaks utterance 106, the higher delay associated with generating and ultimately displaying the final recognition result 120b is less noticeable to user 104.

[0026] exist Figure 1A In the example shown, digital assistant application 50a can respond to a question posed by user 104 using natural language processing. Natural language processing generally refers to the process of interpreting written language (e.g., partial speech recognition results 120a and / or final speech recognition results 120b) and determining whether the written language prompts any action. In this example, digital assistant application 50a uses natural language processing to recognize that the question from user 10 relates to the user's environment, and more specifically, to a song playing near the user. By recognizing these details using natural language processing, the automated assistant returns a response 19 to the user's query, where response 19 states "Tweezer is playing right now." In some configurations, natural language processing occurs on a remote computing device 60 that communicates with the data processing hardware 12 of user device 10.

[0027] Figure 1B This is another example of speech recognition using the ASR system 109 with a voice environment 100b. As shown in the example, user 104 interacts with voicemail applications 50, 50b, which display voicemail application interfaces 18, 18b on the screen of user device 10, to transcribe a voicemail message left by Jane Doe for user 104. In this example, latency is not important. Without considering latency, model 200 of ASR system 109 is able to utilize the full context of the audio by waiting until all the sound frames 110 corresponding to the voicemail are generated. This voicemail scenario also illustrates how model 200 can handle long-format speech, as voicemails are often multiple sentences or even paragraphs. The ability to handle long-format speech is particularly advantageous compared to other ASR models (e.g., two-pass models with LAS decoders), because these two-pass models typically lead to performance degradation (e.g., higher word deletion rates on long-format speech) when recognizing long-format speech. For example, by combining a cascaded encoder 202 (e.g., a first encoder 210 and a second encoder 220) with an RNN-T decoder as decoder 204, model 200 is able to consume the full context of the input audio when performing speech recognition on long-format speech without causing performance degradation as in a two-pass ASR model.

[0028] Continue to refer to Figure 1B , such as regarding Figure 1AThe model 200 encodes the sound frame 110 using the first encoder 210 upon receiving it. After receiving all sound frames 110 and encoding them using the first encoder 210, the model 200 provides the first encoder output as input to the second encoder 220. The second encoder 220 encodes the first encoder output before the decoder 204 generates the final speech recognition result 120b. Subsequently, the user interface generator 107 presents a representation of the final speech recognition result 120b via the digital assistant interface 18b, without first displaying a portion of the speech recognition result 120a. For example, the final speech recognition result 120b is a transcription of a long-format voicemail from Jane Doe: “Give me a call back when you get this. Just trying to figure out plans for the New Year.”

[0029] Figures 2A to 2C Example models 200a-c are included, operating in various combinations of streaming and non-streaming modes. Specifically, each of models 200a-c includes a cascaded encoder 202 and a decoder 204. A cascaded encoder 202 refers to a model structure in which the encoding path includes two cascaded encoders 210, 220, such that the output of one encoder 210 is fed into the input of another encoder 220 before decoding. Here, encoders 210, 220 can be cascaded regardless of the underlying architecture of each encoder. In some examples, encoders 210, 220 include a stack of 512-dimensional convolutional enhancement transformer layers. Causal convolutions and left contextual attention layers can be used for each convolutional enhancement transformer layer to strictly restrict the model from using future inputs. Multi-head (e.g., 8-head) attention mechanisms can be used for self-attention layers. Cascaded encoders 210, 220 can include 17 convolutional enhancement transformer layers. Here, the first encoder 210 may include 15 convolutional enhancement transformer layers, while the second encoder 220 may include two convolutional enhancement transformer layers employing an additional right context (e.g., 5.04 seconds). Alternatively, other types of layers (e.g., transformer layers) incorporating self-attention mechanisms may be used instead of the convolutional enhancement transformer layers. The first encoder 210 may be referred to as a causal encoder, and the second encoder 220 may be referred to as a non-causal encoder.

[0030] In other implementations, one encoder is built using an LSTM structure, while the other is built using bidirectional LSTM layers or convolutional enhancement transformer layers (e.g., convolutional enhancement transformer-permutator). In other words, encoders 210 and 220 can have different or similar architectures. For example, cascaded encoder 202 can be roughly similar to the acoustic model (AM) in a conventional ASR system and can include a recurrent network of stacked long short-term memory (LSTM) layers. Here, the first encoder 210 is a streaming encoder including unidirectional long short-term memory (LSTM) layers, while the second encoder 220 is a non-streaming encoder including bidirectional LSTM layers or convolutional enhancement transformer layers. In cascaded encoder 202, where both encoders 210 and 220 include LSTM layers, the second encoder 220, which receives the output of the first encoder 210, can utilize the LSTM layers of the first encoder 210, such that the second encoder 220 includes fewer LSTM layers than the first encoder 210 (and fewer LSTM layers than a completely non-streaming model). By having fewer LSTM layers, the cascade encoder 202 can reduce the number of computationally expensive bidirectional layers, making the model 200 more streamlined than simply combining a traditional streaming model with a traditional non-streaming model.

[0031] refer to Figure 2A The first encoder 210 reads a series of d-dimensional feature vectors (e.g., Figure 1A and 1B The sound frame shown is 110)x=(x1,x2,···,x T ), where x t ∈R d And at each time step, a first higher-order feature representation 203 is generated. This first higher-order feature representation is represented as e. s Similarly, the second encoder 220 is cascaded to the first encoder 210 and trained to receive the first high-order feature e. s As input, and outputting a second higher-order feature representation of 205. This second higher-order feature representation is represented as e. a The first encoder 210 and the second encoder 220 are both directly connected to and shared by the decoder 204. Therefore, the decoder 204 receives the first higher-order feature representation e. s Second higher-order feature representation e a Both are used as input.

[0032] Decoder 204 may include a recurrent neural network-permuter (RNN-T) architecture with joint layer 230 and prediction network 240. Decoder 204 uses joint layer 230 to combine (i.e., when model 200 operates in non-streaming mode) the first high-order feature representation e output by cascade encoder 202. sSecond higher-order feature representation e a , and from the previously predicted y r–1 The embedding output of the prediction network 240 is used to generate the decoder output. When model 200 operates in streaming mode, the joint layer 240 receives the output of the prediction network 240, as well as the first high-order feature representation e output only from the first encoder 210. s Given N previous non-blank symbols and previous units {y} i-1 ,...,y i–N Given input x, the decoder output can be the current sub-word unit y. i The probability distribution P(y) on i |y i-1 (,...,y0,x). Although not explicitly stated, model 200 may include a Softmax layer that receives the output of decoder 204. In some implementations, the Softmax layer is separate from decoder 204 and processes the output y from decoder 204. r Then, the output of the Softmax layer is used to select orthogonal elements during the directional search process. In some implementations, the Softmax layer is integrated with decoder 204, so that the output y of decoder 204... r This represents the output of the Softmax layer.

[0033] Decoder 204 is configured to generate a probability distribution over possible speech recognition hypotheses at each output step. In other words, federated network 230 generates a probability distribution over possible speech recognition hypotheses at each output step (e.g., time step). Here, a “possible speech recognition hypothesis” corresponds to a set of output labels / symbols (also referred to as “speech units”), each output label / symbol representing a morpheme (e.g., symbol / character) or word fragment in a particular natural language. For example, when the natural language is English, the set of output labels may include twenty-seven (27) symbols, for example, one label for each of the 26 letters of the English alphabet, and one label for a space. Thus, federated network 230 may output a set of values ​​indicating the probability of occurrence of each of the predetermined set of output labels. This set of values ​​may be a vector (e.g., a one-hot vector) and may indicate a probability distribution over the set of output labels. In some cases, the output labels are morphemes (e.g., a single character, and possibly punctuation marks and other symbols), but the set of output labels is not limited to this. For example, in addition to or instead of morphemes, the set of output labels may include word fragments and / or entire words. The output labels can also be other types of speech units, such as phonemes or subphonemes. The output distribution of the joint network 230 can include the posterior probability value of each of the different output labels. Thus, if there are 100 different output labels representing different characters or other symbols, the output of the joint network 230 can include 100 different probability values, one probability value for each output label. The probability distribution can then be used to select candidate orthogonal elements (e.g., characters, word fragments, and / or words) during the directional search process (e.g., via a Softmax layer) and assign scores to the candidate orthogonal elements to determine the transcription 120. In some examples, the first encoder 210 of the cascade encoder model 200 consists of eight 2048-dimensional LSTM layers, each followed by a 640-dimensional projection layer. In these examples, the second encoder 220 of the model 200 can consist of two bidirectional LSTM layers with approximately 10 million parameters.

[0034] Within decoder 204, prediction network 240 may have two 2048-dimensional LSTM layers, each followed by a 640-dimensional projection layer, allowing the LSTM-based prediction network to have approximately 23.4 million parameters. In other configurations, instead of LSTM layers, prediction network 240 may alternatively include convolutional enhancement transformer layers or transformer layers. In other configurations, prediction network 240 includes a V2 embedding lookup table that includes an embedding prediction network. At each time step, the V2 embedding lookup table may receive the previous two predictions (e.g., one-hot vectors) output by joint network 230 as input, compute the corresponding embeddings d1, d2 for each of the previous two predictions, and provide a cascaded output [d1, d2] to joint layer 230. In contrast, V2 embedding lookup table may have only about two (2) million parameters, while the LSTM-based prediction network may include approximately 23.4 million parameters. Finally, joint network 230 may also be a single-layer neural network with 640 hidden units. The Softmax layer can be constructed using multiple training datasets 132, 132a–n ( Figure 3 The first encoder 210 consists of a unified set of word fragments or glyphs generated from all unique word fragments or glyphs in the cascade encoder model 200. In some implementations, to limit the amount of future context seen by the cascade encoder model 200, the second encoder 220 uses a certain number of convolutional booster transformer layers (e.g., two layers) along with a specific amount of right context (e.g., five seconds of right context), while the first encoder 210 continues to use LSTM layers. For these implementations, each convolutional booster transformer layer in the second encoder 220 may have 640 units to match the LSTM layers and add approximately 10 million additional parameters.

[0035] continue Figure 2A In the example provided, in some implementations, model 200a operates in both streaming and non-streaming modes in parallel. When operating in both streaming and non-streaming modes simultaneously, model 200a first performs streaming speech recognition on the audio data 110 using the first encoder 210 to generate a first high-order representation e for both the second encoder 220 and the decoder 204. s Decoder 204 then bases the first higher-order representation e on... s The model generates partial speech recognition results 120 and 120a. Model 200b also performs non-streaming speech recognition on the encoded audio data 110, wherein the second encoder 220 uses a first high-order representation e received from the first encoder 210. s To generate the second higher-order representation e a Decoder 204 then bases the second higher-order representation e on... aThe final speech recognition results 120 and 120b are generated. As shown in the timeline, decoder 204 uses the output from first encoder 210 to stream the partial speech recognition result 120a, and then waits until the utterance is complete before operating in a non-streaming mode, which uses the output from second encoder 220 to generate the final speech recognition result 120b. Therefore, the final speech recognition result 120b for input utterance 106 can be delayed from the partial speech recognition result 120a.

[0036] refer to Figure 2B In some implementations, model 200b operates only in streaming mode. This might happen, for example, when user 104 is using an application such as voice search or dictation on the device, which requires minimal latency. Here, model 200b performs streaming speech recognition on audio data 110 using only the first encoder 210 to generate a first higher-order representation 203e for decoder 204. s Decoder 204 then produces partial speech recognition results 120, 120a. Because the streaming mode of model 200b quickly produces partial speech recognition results 120, 120a, the inaccuracy of the term "play" is generally acceptable to user 104.

[0037] refer to Figure 2C In some implementations, model 200c operates only in non-streaming mode. Non-streaming mode can occur, for example, in non-latency-intensive applications, such as when user 104 is viewing a transcription of a voicemail left on his / her phone (e.g., Figure 1B As mentioned above, this type of application benefits from using future context to improve performance in exchange for increased processing time. Here, model 200c first uses the first encoder 210 to generate a first higher-order representation 203e at each time step. s The input is fed into the second encoder 220, but the decoder 204 does not decode any of the first higher-order representations e. s Then, model 200c performs non-streaming speech recognition on all audio data 110, wherein the second encoder 220 uses the first high-order representation e received from the first encoder 210. s To generate the second higher-order representation 205e a Decoder 204 then produces the final speech recognition results 120, 120b. Since real-time generation of streaming speech recognition is of little value to the user and latency is not a factor, model 200c can simply operate only in non-streaming mode to produce the final speech recognition results 120, 120b.

[0038] Figure 3An example of a training process 300 is shown for training model 200 to be operable for both streaming and / or non-streaming modes. In some configurations, training process 300 in... Figure 1A and 1B The training process 300 executes on the remote computing device 60. The training process 300 obtains multiple training utterances 132, 132a–n stored in the sample database 130, and trains the model 200 on the training utterances 132. The sample database 130 may reside on the memory hardware of the remote computing device 60. (As mentioned above regarding...) Figure 2A As discussed, the first encoder 210 and the second encoder 220 share the same decoder 204 and can be trained in a single stage, thus simplifying the training process 300. This means that the non-streaming encoder 220 can directly train the output of the streaming encoder 210 (e.g., the first higher-order representation e). s It is trained on the input acoustic features, rather than on the input acoustic frame 110.

[0039] like Figure 3 As shown, model 200 has two processing paths, one for the streaming mode of model 200b. Figure 2B As shown), a non-streaming mode for model 200c (as shown) Figure 2C (As shown in the diagram). Because there are two input processing paths in the training process 300, the model loss includes two loss functions. Specifically, given the input training utterance 132, the loss of the streaming mode of model 200b is roughly defined as the sum of the negative log probabilities corresponding to the probability distributions on the possible speech recognition assumptions. That is, the model loss from the first encoder 210 connected to the decoder 204 is defined as... Given the input training utterance 132, the model loss for the non-streaming mode is also roughly defined as the sum of negative log probabilities corresponding to the probability distributions over possible speech recognition assumptions. Therefore, the model loss from the second encoder 220 connected to the decoder 204 is defined as... Based on these representations, the total loss between the two input paths is calculated as a weighted sum of the losses for each input path. Where λ is a weighting term. In the training process 300, jointly training the cascade encoders 210 and 220 involves minimizing the weighted sum of the losses between the two input processing paths.

[0040] At each time step, during training process 300, for each training utterance 132, training can occur in either streaming or non-streaming mode. In other words, the input processing path is randomly selected to train either model 200b or model 200c. By sampling the training utterances 132, the training process only needs to compute the loss once for each training utterance 132 at each training step, which significantly speeds up training process 300. In some implementations, where longer training times are permissible, an alternative training process is employed to train each input processing path with each training utterance, and the loss of both model 200b and model 200c is computed at each training step for each training utterance 132.

[0041] In the example shown, training utterances 132b and 132c are selected to train a first processing path represented by a cascade encoder model 200b. The cascade encoder model 200b receives training utterances 132b and 132c, and a first encoder 210 converts the training utterances 132b and 132c into a first higher-order feature representation (e.g., an audio embedding) as output. The decoder 204 then receives the first higher-order feature representation of the training utterances 132b and 132c as input and generates an output, testing the accuracy of that output. Similarly, training utterances 132a and 132d are selected to train a second processing path represented by a cascade encoder model 200c. The cascade encoder model 200c receives training utterances 132a and 132d, and a first encoder converts the training utterances 132a and 132d into a first higher-order feature representation (e.g., an audio embedding) as output. The second encoder 220 receives the first higher-order feature representations of training utterances 132a and 132d as input and generates the second higher-order feature representations of training utterances 132a and 132d as output. The decoder 204 then receives the second higher-order feature representations of training utterances 132a and 132d as input and generates an output, testing the accuracy of the output. This ensures that the model 200 learns to operate in either streaming or non-streaming mode during inference.

[0042] Figure 4 The flowchart includes an example arrangement of operations for a method 400 that performs streaming and non-streaming speech recognition using a cascade encoder model 200. In operation 402, method 400 includes receiving a series of acoustic frames 110 as input to the cascade encoder model 200. In operation 404, method 400 further includes performing streaming and non-streaming speech recognition on the series of acoustic frames 110 using the cascade encoder model.

[0043] In operation 406, method 400 includes generating a first high-order feature representation of a corresponding sound frame 110 in the series of sound frames 110 by the first encoder 210 at each of the multiple output steps. Method 400 further includes, in operation 408, receiving the first high-order feature representation generated by the first encoder 210 at each of the multiple output steps as input to the second encoder 220. In operation 410, method 400 further includes generating a second high-order feature representation of the corresponding first high-order feature frame by the second encoder 220 at each of the multiple output steps. Method 400 further includes, in operation 412, receiving the second high-order feature representation generated by the second encoder 220 at each of the multiple output steps as input to the decoder 204. In operation 414, method 400 further includes generating a first probability distribution over possible speech recognition hypotheses at each of the multiple time steps.

[0044] Figure 5 This is a schematic diagram of an example computing device 500, which can be used to implement the systems (e.g., audio subsystem 108, ASR system 109, user interface generator 107, and / or model 200) and methods (e.g., method 400) described in this document. The computing device 500 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components, connections, and relationships between components shown herein, as well as the functionality of the components, are intended to be exemplary only and are not intended to limit the implementation of the inventions described and / or claimed in this document.

[0045] Computing device 500 includes a processor 510 (e.g., data processing hardware), a memory 520 (e.g., memory hardware), a storage device 530, a high-speed interface / controller 540 connected to the memory 520 and a high-speed expansion port 550, and a low-speed interface / controller 560 connected to a low-speed bus 570 and the storage device 530. Each of components 510, 520, 530, 540, 550, and 560 is interconnected using various buses, and the components may be mounted on a general-purpose motherboard or otherwise. The processor 510 can process instructions for execution within computing device 500, including instructions stored in the memory 520 or on the storage device 530 for displaying graphical information of a graphical user interface (GUI) on an external input / output device, such as a display 580 coupled to the high-speed interface 540. In other implementations, multiple processors and / or multiple buses may be used with multiple memories and various types of memory, depending on the situation. Furthermore, multiple computing devices 500 can be connected, each of which provides a portion of the necessary operation (e.g., as a server group, blade server cluster, or multiprocessor system).

[0046] Memory 520 stores information non-temporarily within computing device 500. Memory 520 may be a computer-readable medium, a volatile memory cell, or a non-volatile memory cell. Non-temporarily stored memory 520 may be a physical means for temporarily or permanently storing programs (e.g., instruction sequences) or data (e.g., program state information) for use by computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., commonly used in firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.

[0047] Storage device 530 provides large-capacity storage for computing device 500. In some implementations, storage device 530 is a computer-readable medium. In various implementations, storage device 530 may be a floppy disk device, hard disk device, optical disk device, magnetic tape device, flash memory or other similar solid-state storage device, or an array of devices, including devices in a storage area network or other configuration. In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as memory 520, storage device 530, or memory on processor 510.

[0048] High-speed controller 540 manages bandwidth-intensive operations of computing device 500, while low-speed controller 560 manages lower bandwidth-intensive operations. This allocation of responsibilities is merely exemplary. In some implementations, high-speed controller 540 is coupled to memory 520, display 580 (e.g., via a graphics processor or accelerometer), and high-speed expansion port 550 which can accept various expansion cards (not shown). In some implementations, low-speed controller 560 is coupled to storage device 530 and low-speed expansion port 590. Low-speed expansion port 590, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), may be coupled to one or more input / output devices, such as a keyboard, pointing device, scanner, or networking device such as a switch or router, for example, via a network adapter.

[0049] As shown in the figure, the computing device 500 can be implemented in a variety of different forms. For example, the computing device can be implemented as a standard server 500a, or multiple times in a group of such servers 500a, as a laptop computer 500b, or as part of a rack server system 500c.

[0050] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be used for special or general purposes and coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transfer data and instructions to the storage system, at least one input device, and at least one output device.

[0051] A software application (i.e., a software resource) can refer to computer software that instructs a computing device to perform a task. In some examples, a software application may be referred to as an "application," "application program," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and game applications.

[0052] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages, and / or in assembly / machine language. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transitory computer-readable medium, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0053] The processes and logic flows described in this specification can be executed by one or more programmable processors, also known as data processing hardware, which execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). For example, processors suitable for executing computer programs include both general-purpose microprocessors and special-purpose microprocessors, as well as any one or more processors of any kind of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or operatively coupled to receive data from or transfer data to said one or more mass storage devices, or both. However, a computer does not need to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0054] To provide interaction with a user, one or more aspects of this disclosure can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor or touchscreen) to display information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending web pages to a web browser on the user's client device in response to a request received from a web browser.

[0055] Several implementations have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other implementations are within the scope of the appended claims.

Claims

1. An automatic speech recognition (ASR) model, comprising: A first encoder, configured to: Receives a series of sound frames as input; as well as In each of the multiple output steps, a first high-order feature representation of the corresponding acoustic frame in the series of acoustic frames is generated; The second encoder is configured to: The first high-order feature representation generated by the first encoder in each of the plurality of output steps is received as input; as well as Each of the plurality of output steps generates a second high-order feature representation of the corresponding first high-order feature frame; as well as Decoder, the decoder is configured to: The second higher-order feature representation generated by the second encoder in each of the plurality of output steps is received as input; as well as At each of the multiple time steps, a first probability distribution is generated over the possible speech recognition hypotheses. The first high-order feature representation generated by the first encoder in each of the plurality of output steps is received as input; as well as At each of the plurality of time steps, a second probability distribution is generated over possible speech recognition hypotheses. The decoder includes: A prediction network, which is configured to: It receives a series of non-whitespace symbols as input from the output of the final Softmax layer; and Each of the plurality of output steps generates a dense representation; and A federated network, which is configured to: Receive the dense representation generated by the prediction network at each of the plurality of output steps, and one of the following as input: When the ASR model operates in streaming mode, the first high-order feature representation generated by the first encoder at each of the plurality of output steps; and When the ASR model operates in non-streaming mode, the second high-order feature representation generated by the second encoder at each of the plurality of output steps; and Each of the plurality of output steps generates one of the following: When the ASR model operates in the streaming mode, the second probability distribution on possible speech recognition assumptions; and The first probability distribution on possible speech recognition assumptions when the ASR model operates in the non-streaming mode.

2. The ASR model according to claim 1, wherein, The second encoder generates the second high-order feature representation without receiving any of the said acoustic frames as input.

3. The ASR model according to claim 1, wherein, The decoder is further configured to generate partial speech recognition results based on the second probability distribution on possible speech recognition assumptions.

4. The ASR model according to claim 1, wherein, The prediction network includes a prediction network based on Long Short-Term Memory (LSTM).

5. The ASR model according to claim 1, wherein, The prediction network includes a V2 embedding lookup table.

6. The ASR model according to claim 1, wherein, The first encoder includes a causal encoder, which includes one of the following: Multiple unidirectional Long Short-Term Memory (LSTM) layers; Multiple convolutional enhancement transformer layers; or Multiple converter layers.

7. The ASR model according to claim 1, wherein, The second encoder includes a noncausal encoder, which includes one of the following: One or more bidirectional Long Short-Term Memory (LSTM) layers; Multiple convolutional enhancement transformer layers; or Multiple converter layers.

8. The ASR model according to any one of claims 1 to 7, further comprising: Receive a set of training utterances; as well as The first encoder and the second encoder are jointly trained using the set of training utterances through the following operations: Generate the first loss of the first encoder; Generate the second loss of the second encoder; as well as Minimize the weighted sum of the first loss and the second loss.

9. A computer-implemented method, which, when executed on data processing hardware (12), causes the data processing hardware (12) to perform operations, the operations comprising: It receives a series of sound frames as input to the Automatic Speech Recognition (ASR) model; as well as The ASR model is used to perform streaming speech recognition and non-streaming speech recognition on the series of audio frames through the following operations: The first high-order feature representation of the corresponding acoustic frame in the series of acoustic frames is generated by the first encoder at each of the multiple output steps; The first high-order feature representation generated by the first encoder in each of the plurality of output steps is received as input to the second encoder. The second encoder generates a second high-order feature representation of the corresponding first high-order feature frame in each of the plurality of output steps; Receives a second higher-order feature representation generated by the second encoder at each of the plurality of output steps as input to the decoder; and At each of the multiple time steps, a first probability distribution is generated over the possible speech recognition hypotheses. Wherein, when the ASR model operates in streaming mode, the operation further includes: The first higher-order feature representation generated by the first encoder in each of the plurality of output steps is received as input to the decoder; The decoder generates a second probability distribution over possible speech recognition hypotheses at each of the plurality of time steps; It receives a series of non-whitespace symbols output from the final Softmax layer as input to the prediction network; The prediction network generates a dense representation at each of the plurality of output steps; The network receives a dense representation generated by the prediction network at each of the plurality of steps and a first high-order feature representation generated by the first encoder at each of the plurality of output steps as input to the joint network; and The joint network generates a second probability distribution over possible speech recognition hypotheses at each of the plurality of output steps; and Wherein, when the ASR model operates in non-streaming mode, the operation further includes: The network receives a series of non-blank symbols output from the final Softmax layer as input to the prediction network. The prediction network generates a dense representation at each of the plurality of output steps; The network receives a dense representation generated by the prediction network at each of the plurality of output steps and a second high-order feature representation generated by the second encoder at each of the plurality of output steps as input to the joint network; and The joint network generates a first probability distribution over possible speech recognition hypotheses at each of the plurality of output steps.

10. The method according to claim 9, wherein, The second encoder generates the second high-order feature representation without receiving any of the said acoustic frames as input.

11. The method according to claim 9, wherein, The operation further includes generating partial speech recognition results based on the second probability distribution on possible speech recognition assumptions.

12. The method according to claim 9, wherein, The decoder includes a prediction network based on Long Short-Term Memory (LSTM).

13. The method according to claim 9, wherein, The decoder includes a prediction network based on a V2 embedding lookup table.

14. The method according to claim 9, wherein, The first encoder includes a causal encoder, which includes one of the following: Multiple unidirectional Long Short-Term Memory (LSTM) layers; Multiple convolutional enhancement transformer layers; or Multiple converter layers.

15. The method according to claim 9, wherein, The second encoder includes a noncausal encoder, the noncausal encoder comprising: One or more bidirectional Long Short-Term Memory (LSTM) layers; Multiple convolutional enhancement transformer layers; or Multiple converter layers.

16. The method according to any one of claims 9 to 15, wherein, The operation further includes: Receive a set of training utterances; and The first encoder and the second encoder are jointly trained using the set of training utterances through the following operations: Generate the first loss of the first encoder; Generate the second loss of the second encoder; and Minimize the weighted sum of the first loss and the second loss.

Images