Hybrid model attention for flexible streaming and non-streaming automatic speech recognition

Abstract

A method (500) for an automated speech recognition (ASR) model (200) that unifies streaming and non-streaming speech recognition includes receiving a sequence of acoustic frames (110). The method includes generating, using an audio encoder (300) of the automated speech recognition (ASR) model, a higher-order feature representation of a corresponding acoustic frame in the sequence of acoustic frames. The method further includes generating, using a joint network (230) of the ASR model, a probability distribution over possible speech recognition hypotheses for a corresponding time step based on the higher-order feature representation generated by the audio encoder at the corresponding time step. The audio encoder includes a neural network that applies a mixture model (MiMo) attention to compute an attention probability distribution function (PDF) using a set of softmax mixing components over a context window.

Description

Technical Field

[0001] This disclosure relates to hybrid model attention for flexible streaming and non-streaming automatic speech recognition. Background Technology

[0002] Automatic speech recognition (ASR) systems have evolved from multiple models, each with a specific purpose, to ensemble models where a single neural network directly maps audio waveforms (i.e., input sequences) to output sentences (i.e., output sequences). This ensemble has produced sequence-to-sequence means that, given a sequence of audio features, generate sequences of words (or epigraphs). Through this ensemble structure, all components of the model can be jointly trained as a single end-to-end (E2E) neural network. Here, an end-to-end model refers to a model whose architecture is entirely built from neural networks. The complete neural network operates without external and / or manually designed components (e.g., finite-state transducers, dictionaries, or text normalization modules). Furthermore, when training E2E models, these models typically do not require bootstrapping from decision trees or temporal alignment from separate systems. These end-to-end automatic speech recognition (ASR) systems have achieved significant progress, outperforming conventional ASR systems on several common benchmarks, including word error rate (WER). The architecture of an E2E ASR model is highly application-dependent. For example, many applications involving user interaction (such as voice search or on-device dictation) require the model to perform recognition in a streaming manner. Other applications (such as offline video caption generation) do not require the model to be streaming and can leverage future context to improve performance. Summary of the Invention

[0003] One aspect of this disclosure provides an automated speech recognition (ASR) model for unifying streaming and non-streaming speech recognition. The ASR model includes an audio encoder configured to receive a sequence of acoustic frames as input. The audio encoder is also configured to generate high-order feature representations of corresponding acoustic frames in the sequence of acoustic frames at each of multiple time steps. The ASR model also includes a joint network configured to receive the high-order feature representations generated by the audio encoder at each of the multiple time steps as input. The joint network is also configured to generate probability distributions over possible speech recognition hypotheses at each of the multiple time steps. The audio encoder includes a neural network that applies Mixture Model Attention (MiMo) to compute an attention probability distribution function (PDF) using a set of mixed components with softmax applied over a context window.

[0004] Implementations of this disclosure may include one or more of the following optional features. In some implementations, the set of hybrid components for MiMo attention is designed to cover all possible contextual use cases during inference. In other implementations, each hybrid component in the set of hybrid components includes a fully normalized softmax.

[0005] Furthermore, the ASR model can switch between streaming and non-streaming modes by adjusting the hybrid weights of the MiMo attention. In some implementations, the neural network of the audio encoder includes multiple convolutional enhancement transformer layers. In other implementations, the neural network of the audio encoder includes multiple transformer layers.

[0006] In some implementations, the ASR model includes a label encoder configured to receive a sequence of non-blank symbols output by a final softmax layer as input and to generate a dense representation at each of multiple time steps. In these implementations, the joint network is also configured to receive the dense representation generated by the label encoder at each of the multiple time steps as input. In these implementations, the label encoder may include a neural network comprising a transformer layer, a convolutional augmentation transformer layer, or a long short-term memory (LSTM) layer.

[0007] Furthermore, the label encoder may include a lookup table embedding model configured to look up a dense representation for each of multiple time steps. In some examples, each possible speech recognition hypothesis generated at multiple time steps corresponds to a set of output labels, each output label representing a word or phrase in natural language.

[0008] Another aspect of this disclosure provides a computer-implemented method for an automated speech recognition (ASR) model that unifies streaming and non-streaming speech recognition. When executed on data processing hardware, the computer-implemented method causes the data processing hardware to perform operations including receiving a sequence of acoustic frames. Further operations performed at each of a plurality of time steps include generating high-order feature representations of the corresponding acoustic frames in the acoustic frame sequence using an audio encoder of the ASR model. These operations also include generating a probability distribution over possible speech recognition hypotheses at the corresponding time step using a joint network of the ASR model, based on the high-order feature representations generated by the audio encoder at the corresponding time step. The audio encoder includes a neural network that applies Mixture Model Attention (MiMo) to compute an attention probability distribution function (PDF) using a set of mixed components with softmax applied over a context window.

[0009] This aspect may include one or more of the following optional features. In some implementations, the set of hybrid components of MiMo attention is designed to cover all possible contextual use cases during inference. In other implementations, each hybrid component in the set of hybrid components includes a fully normalized softmax.

[0010] Furthermore, the ASR model can switch between streaming and non-streaming modes by adjusting the hybrid weights of the MiMo attention. In some implementations, the neural network of the audio encoder includes multiple convolutional enhancement transformer layers. In other implementations, the neural network of the audio encoder includes multiple transformer layers.

[0011] In some implementations, the operation further includes generating a dense representation using a label encoder of the ASR model at each of multiple time steps, the label encoder being configured to receive a sequence of non-whitespace symbols output by the final softmax layer of the ASR model. In these implementations, the probability distribution generated at the corresponding time step based on possible speech recognition hypotheses is also based on a sequence of non-whitespace output symbols output by the final softmax layer at the corresponding time step. In these implementations, the label encoder may comprise a neural network with transformer layers, convolutional augmentation transformer layers, or long short-term memory (LSTM) layers.

[0012] Furthermore, the label encoder may include a lookup table embedding model configured to look up a dense representation for each of multiple time steps. In some examples, each possible speech recognition hypothesis generated at multiple time steps corresponds to a set of output labels, each output label representing a word or phrase in natural language.

[0013] Details of one or more embodiments of this disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings and from the claims. Attached Figure Description

[0014] Figure 1A and Figure 1B This is a schematic diagram of an example speech environment in which a speech recognition model trained using hybrid model attention switches between streaming and non-streaming speech recognition modes.

[0015] Figure 2 This is a schematic diagram of an example speech recognition model with a transducer model architecture.

[0016] Figures 3A-3C The example attention probability matrix is ​​illustrated.

[0017] Figure 4 This is an example table illustrating the word error rates of the baseline and hybrid model streaming / non-streaming convolutional enhancement transformers.

[0018] Figure 5 This is a flowchart illustrating an example layout of operations for a method that unifies streaming and non-streaming speech recognition.

[0019] Figure 6 This is a schematic diagram of an example computing device that can be used to implement the systems and methods described in this paper.

[0020] The same reference numerals in the various figures indicate the same elements. Detailed Implementation

[0021] Automated speech recognition (ASR) systems focus not only on providing quality / accuracy (e.g., low word error rate (WER)) but also on providing low latency (e.g., short delay between the user speaking and the occurrence of the transcription). Recently, end-to-end (E2E) ASR models have gained popularity for achieving state-of-the-art performance in both accuracy and latency. Compared to conventional hybrid ASR systems that include separate acoustic, articulation, and language models, E2E models apply sequence-to-sequence approaches to jointly learn acoustic and language modeling in a single neural network, which is trained end-to-end based on training data (e.g., utterance-transcription pairs). Here, an E2E model refers to a model whose architecture is entirely built from neural networks. The complete neural network operates without the need for external and / or manually designed components (e.g., finite-state transducers, dictionaries, or text normalization modules). Furthermore, when training E2E models, these models typically do not require sampling with replacement from decision trees or temporal alignment from separate systems.

[0022] When using ASR systems today, it may be necessary for the ASR system to decode speech in a streaming manner, which corresponds to displaying a description of the speech in real time or even faster than real time as the user speaks. To illustrate, when an ASR system is displayed on a user computing device (e.g., a mobile phone) experiencing direct user interaction, applications running on the user device and using the ASR system (e.g., digital assistant applications) may need to make speech recognition streaming so that words, phrases, and / or individual characters appear on the screen as soon as they are spoken. Additionally, users of the user device may have a low tolerance for latency. For example, when a user utters a query requesting a digital assistant to retrieve details of an upcoming appointment from a calendar application, the user expects the digital assistant to provide a response conveying the retrieved details as quickly as possible. Due to this low tolerance, ASR systems strive to operate on user devices in a way that minimizes the impact of latency and inaccuracies that could negatively affect the user experience. However, attention-based sequence-to-sequence models (such as the listen-attention-spelling (LAS) model, which operates by examining the entire audio input sequence before generating the output text) do not allow for streaming output as input is received. Due to this limitation, deploying attention-based sequence-to-sequence models for latency-sensitive and / or real-time speech transcription applications can be problematic. This makes the LAS model itself not an ideal model for latency-sensitive applications and / or applications that provide real-time streaming transcription capabilities while the user is speaking.

[0023] Another form of sequence-to-sequence model, known as a recurrent neural network transducer (RNN-T), does not employ an attention mechanism. Unlike other sequence-to-sequence models that typically require processing the entire sequence (e.g., an audio waveform) to produce an output (e.g., a sentence), RNN-T continuously processes input samples and streams output symbols, a feature particularly attractive for real-time communication. For example, speech recognition using RNN-T can output characters one by one as spoken. Here, RNN-T uses a feedback loop, which feeds back the symbols predicted by the model to itself to predict the next symbol. Because decoding an RNN-T involves beam search through a single neural network rather than a large decoder graph, RNN-T can scale to a fraction of the size of server-based speech recognition models. With this reduced size, RNN-T can be deployed entirely on a device and can operate offline (i.e., without a network connection); thus avoiding the unreliability issues of communication networks.

[0024] Because they cannot apply look-ahead audio context (e.g., right context) when predicting recognition results, RNN-T models still lag behind large, state-of-the-art conventional models (e.g., server-based models with independent AM, PM, and LM) and attention-based sequence-to-sequence models (e.g., LAS models) in terms of quality (e.g., speech recognition accuracy, typically measured by word error rate (WER)). Recently, transformer-transducer (TT) and convolutional augmented transformer-transducer (CT) models have become more popular than RNN-T models because they can process the entire audio sequence to compute the attention probability density function at each time step. Therefore, TT and CT models can be used for both streaming and non-streaming speech recognition due to their ability to utilize look-ahead context. However, the two models are trained separately to perform the corresponding speech recognition in streaming and non-streaming speech recognition. For example, a user might use a streaming speech recognition model such as an RNN-T, CT, or TT model to recognize conversational queries and a separate non-streaming speech recognition model to recognize non-conversational queries. Typically, the direction a user directs their voice towards an application can be used to identify which of the streaming or non-streaming speech recognition models is being used for speech recognition. Requires different and separate speech recognition models to perform speech recognition based on the application and / or query type, which is computationally expensive and requires sufficient memory capacity to store the corresponding models on the user's device. Even if one of the models can be executed on a remote server, the additional cost of connecting to the remote server and bandwidth limitations can impact speech recognition performance and ultimately affect the user experience.

[0025] The implementation described in this paper aims to unify streaming and non-streaming acoustic encoders by training speech recognition models (e.g., TT or CT models) using hybrid model attention, allowing the trained speech recognition model to switch between streaming and non-streaming speech recognition modes during inference. A fundamental limitation of acoustic encoders based on transformers and convolutional enhancement transformers lies in the computation of the attention probability density function (PDF) at each time step. Specifically, a single softmax constraint on the entire context window restricts the model to use the same context window size during training and inference. The proposed hybrid model (MiMo) attention provides a flexible alternative that alleviates the requirement to constrain the context window size during inference to the same size used during training. It will be apparent that MiMo attention computes the attention PDF using a hybrid model with softmax over the context window. Support for hybrid components is designed to cover all possible use cases during inference, such as full / global sequence context, finite left + right context, and left-only context. Since each hybrid component is a fully normalized softmax, it is possible to apply any subset of the hybrid components during inference without producing any mismatches. Therefore, based on current applications, CT and TT models trained with MiMO attention can be used for various streaming and non-streaming modes during inference. It is worth noting that MiMO attention does not add any additional complexity, parameters, or training loss to the model.

[0026] Figure 1A and 1B Examples of voice environments 100, 100a-b are provided. In voice environment 100, user 104 interacts with a computing device such as user device 10 via voice input. User device 10 (also commonly referred to as device 10) is configured to capture sound (e.g., streaming audio data) from one or more users 104 within voice environment 100. Here, streaming audio data may refer to spoken utterances 106 produced by user 104, which serve as audible communication, audible queries, or commands to device 10 captured by device 10. The voice-enabled system of device 10 responds to queries or commands by answering them and / or causing the command to be executed / fulfilled by one or more downstream applications.

[0027] User equipment 10 can correspond to any computing device associated with user 104 and capable of receiving audio data. Some examples of user equipment 10 include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smartwatches), smart appliances, Internet of Things (IoT) devices, vehicle entertainment systems, smart displays, smart speakers, etc. User equipment 10 includes data processing hardware 12 and memory hardware 14 that communicates with and stores instructions to the data processing hardware 12, which, when executed by the data processing hardware 12, cause the data processing hardware 12 to perform one or more operations. User equipment 10 also includes an audio system 16 having: audio capture devices (e.g., microphones) 16, 16a for capturing spoken speech 106 within the speech environment 100 and converting it into electrical signals; and voice output devices (e.g., speakers) 16b, 16b for transmitting audible audio signals (e.g., as output audio data from device 10). Although user equipment 10 implements a single audio capture device 16a in the illustrated example, user equipment 10 may implement an array of audio capture devices 16a without departing from the scope of this disclosure, wherein one or more capture devices 16a in the array may not physically reside on user equipment 10, but may communicate with audio system 16.

[0028] In the speech environment 100, an automated speech recognition (ASR) system 109, implementing an ASR model 200 trained using hybrid model (MiMo) attention, resides on a user device 10 of user 104 and / or a remote computing device 60 (e.g., one or more remote servers of a distributed system executing in a cloud computing environment) communicating with user device 10 via network 40. User device 10 and / or remote computing device 60 also include an audio subsystem 108 configured to receive utterance 106 spoken by user 104 and captured by audio capture device 16a, and to convert utterance 106 into a corresponding digital format associated with an input acoustic frame 110 that can be processed by the ASR system 109. Figure 1A In the example shown, user 104 utters a corresponding utterance 106, and audio subsystem 108 converts utterance 106 into corresponding audio data (e.g., acoustic frames) 110 for input to ASR system 109. Subsequently, model 200 receives the audio data 110 corresponding to utterance 106 as input and generates / predicts a corresponding transcription 120 (also referred to as recognition result / hypothesis 120) of utterance 106 as output.

[0029] Model 200 includes an acoustic encoder 300 trained using MiMo attention, which employs a hybrid model with softmax over a context window to compute the attention PDF at each time step. Support for the hybrid components is designed to cover all possible use cases during inference, such as full / global sequence context, finite left + right context, and left-only context. Because each hybrid component is a fully normalized softmax, it is possible to apply any subset of the hybrid components during inference without producing any mismatches.

[0030] Model 200 also includes decoders 220 and 230, which enable model 200 to be a single model capable of operating in both streaming and non-streaming modes (e.g., as opposed to two separate models where each model is dedicated to either streaming or non-streaming mode). For example, as... Figure 1A As shown, the digital assistant application 50 running on user device 10 may require speech recognition to be streaming, so that words, phrases, and / or individual characters appear on the screen as soon as they are spoken. Additionally, the user 104 of user device 10 may have a low tolerance for latency when issuing queries for execution by the digital assistant application 50. In these scenarios where applications require minimal latency, model 200 operates in streaming mode, where model 200 can provide streaming transcription capabilities in real time as user 104 speaks utterance 106.

[0031] User device 10 and / or remote computing device 60 also execute user interface generator 107, which is configured to present a representation of the transcription 120 of utterance 106 to user 104 of user device 10. As described in more detail below, user interface generator 107 may stream speech recognition results 120a, which are subsequently displayed as final speech recognition result 120. In some configurations, the transcription 120 output from ASR system 109 is processed, for example, by a natural language understanding (NLU) module executed on user device 10 or remote computing device 60 to execute the user command / query specified by utterance 106. Additionally or alternatively, a text-to-speech system (not shown) (e.g., executed on any combination of user device 10 or remote computing device 60) may convert the transcription 120 into synthesized speech for audible output by user device 10 and / or another device.

[0032] exist Figure 1A In the example, user 104 in voice environment 100a interacts with a program or application 50 (e.g., digital assistant application 50a) using user device 10 of ASR system 109. For example, Figure 1AThe text depicts a user 104 communicating with a digital assistant application 50a, where the application displays a digital assistant interface 18 on the screen of the user device 10 to illustrate a dialogue between the user 10 and the digital assistant of the application 50a. In this example, the user 104 asks the application 50a, “What time is the concert tonight?” This question from the user 104 is a spoken word 106 captured by an audio capture device 16a and processed by the audio system 16 of the user device 10. In this example, the audio system 16 receives the spoken word 106 and converts it into an acoustic frame 110 for input to the ASR system 109.

[0033] Continuing this example, model 200, while receiving acoustic frames 110 corresponding to utterances 106 spoken by user 104, uses audio encoder 300 (i.e., Figure 2 The acoustic frame 110 is encoded, and then decoders 220 and 230 are used. Figure 2 The encoded representation of acoustic frame 110 is decoded into a streaming speech recognition result 120a. The user interface generator 107 presents the streaming speech recognition result 120a of utterance 106 to the user 104 of user device 10 via digital assistant interface 18, so that words, phrases, and / or individual characters appear on the screen as they are spoken. The user interface generator 107 also presents the final speech recognition result 120 of utterance 106 to the user 104 of user device 10 via digital assistant interface 18.

[0034] exist Figure 1A In the example shown, digital assistant application 50a can use natural language processing (NLP) to respond to a question posed by user 104. NLP generally refers to the process of interpreting written language (e.g., partial speech recognition results 120a and / or final speech recognition results 120b) and determining whether the written language prompts any action. In this example, digital assistant application 50a uses NLP to identify a question from user 104 about the user's environment and, more specifically, about a song playing near the user. By identifying these details using NLP, the automated assistant returns a response 19 to the user's query, where response 19 states "The doors open at 6:30 pm". In some configurations, NLP occurs on a remote computing device 60 that communicates with the data processing hardware 12 of user device 10.

[0035] Figure 1BThis is another example of speech recognition using the ASR system 109 with a voice environment 100b. As shown in the example, user 104 interacts with voicemail applications 50, 50b, which display voicemail application interfaces 18, 18b on the screen of user device 10, to transcribe a voicemail left by Jane Doe for user 104. In this example, latency is not important, but the accuracy of transcription is important when dealing with long-tailed proper nouns or rare words. Model 200 of ASR system 109 is able to utilize the full context of the audio by waiting until all acoustic frames 110 corresponding to the voicemail are generated. This voicemail scenario also illustrates how model 200 can handle long-form speech, as voicemails are often multiple sentences or even several paragraphs. The ability to handle long-form speech is particularly advantageous over other ASR models, such as two-pass models with LAS decoders, because these two-pass models often encounter long-form problems (e.g., higher word deletion rates on long-form speech) when applied to long-form conditions.

[0036] Continue to refer to Figure 1B , such as regarding Figure 1A In the discussed model 200, the acoustic frame 110 is encoded using an audio encoder 300 that receives the acoustic frame 110. Figure 1A Unlike the example (where the audio encoder 300 operates in streaming mode using only the left context), the audio encoder 300 is switched to operate in non-streaming mode using both the left and right contexts. Here, the hybrid weights of the audio encoder 300 trained using MiMo attention can be set to appropriate values ​​associated with the non-streaming mode. After model 200 receives all acoustic frames 110 and encodes them using the audio encoder 300, model 200 provides the audio encoder output as input to decoders 220 and 230 to generate the final speech recognition result 120b.

[0037] refer to Figure 2Model 200 can provide end-to-end (E2E) speech recognition by integrating acoustic, pronunciation, and language models into a single neural network, without requiring a dictionary or separate text normalization component. Various architectures and optimization mechanisms enable increased accuracy and reduced model training time. Model 200 can include a Transformer-Transducer (TT) or Convolution-Augmented Transformer-Transducer (CT) model architecture that adheres to latency constraints associated with interactive applications. An example TT model is described in U.S. Application No. 17 / 210,465, filed March 23, 2021, the contents of which are incorporated herein by reference in their entirety. An example CT model is described in “Conformer: Convolution-augmented Transformer for Speech Recognition,” arxiv.org / pdf / 2005.08100, the contents of which are incorporated herein by reference in their entirety. Model 200 offers a small computational footprint and utilizes less memory requirements than conventional ASR architectures, making the model architecture suitable for performing speech recognition entirely on user device 10 (e.g., without communication with remote server 60). Model 200 includes an audio encoder 300, a tag encoder (e.g., a prediction network) 220, and a joint network 230. The tag encoder 220 and the joint network 230 together form a decoder. The audio encoder 300, which is broadly similar to the acoustic model (AM) in a conventional ASR system, includes a neural network with multiple convolutional enhancement transformer layers or transformer layers. For example, the audio encoder 300 reads a sequence of d-dimensional feature vectors (e.g., acoustic frames 110 (Figure 1)), x = (x1, x2, ..., x...). T ),in, Furthermore, a higher-order feature representation is generated at each time step. This higher-order feature representation is denoted as ah1,...,ah T .

[0038] Similarly, the tag encoder 220 may also include a transformer layer, a convolutional enhancement transformer layer, a neural network with a long short-term memory (LSTM) layer, or a lookup table embedding model, which, like a language model (LM), will process the non-whitespace symbol sequence y0,...,y0 so far, output by the final Softmax layer 240. ui-1 Processing is done by encoding a dense representation Ih that encodes the history of predicted labels. u .

[0039] Finally, utilizing the TT or CT model architecture, the representations generated by the audio and tag encoders 300 and 220 are processed by the joint network 230 using a dense layer J. u,t Combining. Then, the joint network 230 predicts P(z) u，t|x,t,y1,...,y u-1 The distribution over the next output symbol is the probability distribution. In other words, the joint network 230 generates a probability distribution over possible speech recognition hypotheses at each output step (e.g., time step). Here, a “possible speech recognition hypothesis” corresponds to a set of output labels (also called “speech units”), each output label specifying a natural language representation of a morpheme (e.g., symbol / character) or a word fragment. For example, when the natural language is English, the set of output labels may include twenty-seven (27) symbols, for example, one label for each of the 26 letters of the English alphabet and one label for a space. Thus, the joint network 230 can output a set of values ​​indicating the probability of occurrence of each of the predetermined set of output labels. This set of values ​​can be a vector and can indicate a probability distribution over the set of output labels. In some cases, the output labels are morphemes (e.g., a single character, and possibly punctuation marks and other symbols), but the set of output labels is not limited to this. For example, in addition to or in place of morphemes, the set of output labels may include word fragments and / or entire words. The output distribution of the joint network 230 can include the posterior probability value for each different output label. Therefore, if there are 100 different output labels representing different characters or other symbols, then the output z of the joint network 230... u,t It can include 100 different probability values, one for each output label. The probability distribution can then be used to select candidate orthographic elements (e.g., graphemes, word segments, and / or words) and assign them scores for determining transcription 120 during a bundle search process (e.g., via a Softmax layer 240).

[0040] The Softmax layer 240 can employ any technique to select the output label / symbol with the highest probability in the distribution as the next output symbol predicted by model 200 at the corresponding output step. In this way, model 200 does not make a conditional independence assumption, but rather the prediction of each symbol is conditional not only on acoustics but also on the sequence of labels output so far.

[0041] The implementation described in this paper aims to replace the multi-head attention mechanism in the transformer or convolutional enhancement transformer layer of the audio encoder 300 with MiMo attention to provide greater flexibility in switching between streaming and non-streaming modes during inference. MiMo attention is used to generate PDFs from the input sequence of feature vectors (x0,...,x) over T time steps. T-1 ) Calculate the output sequence (y0,...,y) of the eigenvectors T-1 For T time steps, the nonnormalized attention score is indicated. To be used to generate y at the current time step kThe formula does not assume how these scores are generated. For example, they could be computed by taking the dot product between the query vector at time step k and all T key vectors. While conventional attention uses a single softmax to convert the attention score into a PDF over T time steps, MiMo attention applies a mixture of M softmaxes to compute the attention PDF as follows.

[0042]

[0043] Among them, w0,...,w M–1 It uses hybrid weights. Advantageously, hybrid models allow for the determination / setting of support for softmax hybrid components during training to cater to all possible contextual use cases during inference. This flexibility is not available in standard / regular attention.

[0044] In the non-restrictive example, the M softmaxes are configured as two hybrid components, comprising a first component spanning kL, k time steps and a second component spanning [k+1, k+R]. In other words, the first softmax operates on the left + center context for L+1 time steps, while the second softmax operates on the right context for R time steps relative to the center frame of time step k. Figures 3A-3C Schematic diagram 301 illustrates the attention probability matrices 301a, 301b, and 301c of the simple case of MiMo attention with size T×T and M=2 hybrid components. Figure 3A The T×T attention probability matrix 301a is written as a hybrid model consisting of left-only (causal, represented by horizontal dashed lines) and right-only (non-causal, represented by vertical dashed lines) attention matrices. MiMo attention splits this diagonal matrix into a weighted sum of the lower-band diagonal matrix of the left + center context and the upper-band diagonal matrix of the right context. All matrices are row-randomized (i.e., each row sums to 1) and thus represent a correctly normalized attention PDF.

[0045] During inference, it is possible to use both left and right contexts (non-streaming mode), use only the left context by setting the mixed weight w0=1 (streaming mode), or use only the right context by setting w1=1. It is easy to see that MiMo attention is a general framework that can adapt to all types of contexts and use cases during inference.

[0046] Training an ASR model using MiMo attention does not change the training loss or add any additional complexity. In fact, it can be useful to add noise to the mixed weights during training. For the case of M=2 above, u ~ uniform(0, w1) can be sampled in each training batch and used to perturb the mixed weights to w0+u and w1–u.

[0047] Figure 4 Table 400 shows the WERs for the test-clean and test-other example datasets. Table 400 illustrates the WERs for the baseline and MiMo streaming / non-streaming convolutional enhancement transforms for Librispeech-960. In Table 400, (L, R) represents the dimensions of the left + center and right context. All checkpoints are selected based on the dev-clean WER. Two baseline ASR models were trained—a left-context-only model with a context size of 65; and a non-streaming model with a left context size of 65 and a right context size of 64. The baseline non-streaming (65, 64) model gives a lower WER than the corresponding streaming (65, 0) model when using the matched non-streaming (65, 64) inference graph.

[0048] Model 200 can be trained on a remote server. The trained model 200 can then be pushed to the user's device to perform on-device speech recognition.

[0049] Figure 5 This is a flowchart illustrating an example layout of the operation of the unified streaming and non-streaming speech recognition method 500. Data processing hardware 12 ( Figure 1A ) can execute the memory stored in hardware 14 ( Figure 1A The instructions on the device are used to execute the example operational arrangement of method 500. In operation 502, method 500 includes receiving a sequence of acoustic frames 110. Method 500 can then continue to operations 504 and 506, which are executed in each of the plurality of time steps.

[0050] In operation 504, method 500 includes generating high-order feature representations of corresponding acoustic frames 110 in a sequence of acoustic frames 110 using an audio encoder 300 of an Automatic Speech Recognition (ASR) model 200. In operation 506, method 500 includes generating a probability distribution over possible speech recognition hypotheses at corresponding time steps using a joint network (230) of the ASR model based on the high-order feature representations generated by the audio encoder 300 at corresponding time steps. Here, the audio encoder 300 includes a neural network that applies Mixed Model Attention (MiMo) to compute an attention probability distribution function (PDF) using a set of mixed components with softmax applied over a context window.

[0051] A software application (i.e., a software resource) can refer to computer software that instructs a computing device to perform a task. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and game applications.

[0052] Non-transitory memory can be a physical device used for temporary or permanent storage of programs (e.g., instruction sequences) or data (e.g., program state information) for use by a computing device. Non-transitory memory can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.

[0053] Figure 6 This is a schematic diagram of an example computing device 600 that can be used to implement the systems and methods described in this document. The computing device 600 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components illustrated herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the embodiments of the invention described and / or claimed in this document.

[0054] Computing device 600 includes a processor 610, memory 620, storage device 630, a high-speed interface / controller 640 connected to memory 620 and high-speed expansion port 650, and a low-speed interface / controller 660 connected to low-speed bus 670 and storage device 630. Each of components 610, 620, 630, 640, 650, and 660 is interconnected using various buses and may be mounted on a common motherboard or otherwise, as appropriate. Processor 610 is capable of processing instructions for execution within computing device 600, including instructions stored in memory 620 or storage device 630, to display graphical information for a graphical user interface (GUI) on an external input / output device (e.g., a display 680 coupled to high-speed interface 640). In other embodiments, multiple processors and / or multiple buses, as well as multiple memories and memory types, may be used as appropriate. Furthermore, multiple computing devices 600 may be connected, each providing a portion of the necessary operation (e.g., as a server group, a set of blade servers, or a multiprocessor system).

[0055] Memory 620 stores information non-temporarily within computing device 600. Memory 620 may be a computer-readable medium, a volatile memory cell, or a non-volatile memory cell. Non-temporary memory 620 may be a physical device for temporarily or permanently storing programs (e.g., instruction sequences) or data (e.g., program state information) for use by computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.

[0056] Storage device 630 provides mass storage for computing device 600. In some embodiments, storage device 630 is a computer-readable medium. In various embodiments, storage device 630 may be a floppy disk device, hard disk device, optical disk device, magnetic tape device, flash memory or other similar solid-state storage device, or device array, including devices in a storage area network or other configuration. In additional embodiments, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as memory 620, storage device 630, or memory on processor 610.

[0057] High-speed controller 640 manages bandwidth-intensive operations of computing device 600, while low-speed controller 660 manages lower bandwidth-intensive operations. This allocation of responsibilities is merely exemplary. In some embodiments, high-speed controller 640 is coupled to memory 620, display 680 (e.g., via a graphics processor or accelerator), and high-speed expansion port 650, which can accept various expansion cards (not shown). In some embodiments, low-speed controller 660 is coupled to storage device 630 and low-speed expansion port 690. Low-speed expansion port 690, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), may be coupled to one or more input / output devices, such as keyboards, pointing devices, scanners, or networking devices such as switches or routers, for example, via a network adapter.

[0058] The computing device 600 can be implemented in a variety of different forms, as shown in the figure. For example, it can be implemented as a standard server 600a or multiple times as a laptop computer 600b in a set of such servers 600a, or as part of a rack server system 600c.

[0059] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can be included in one or more computer programs executable and / or interpretable on a programmable system comprising at least one programmable processor, which may be dedicated or general-purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transmit data and instructions to the storage system, at least one input device, and at least one output device.

[0060] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” mean any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, which includes a machine-readable medium that receives machine instructions as machine-readable signals. The term “machine-readable signal” means any signal used to provide machine instructions and / or data to a programmable processor.

[0061] The processes and logical flows described in this specification can be executed by one or more programmable processors (also known as data processing hardware) that execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logical flows can also be executed by special-purpose logic circuitry (e.g., FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits)). For example, processors suitable for executing computer programs include both general-purpose and special-purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are the processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or operatively coupled to receive data from or transfer data to one or more mass storage devices for storing data, or both. However, a computer does not need to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example: semiconductor storage devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Processors and memory can be supplemented by or incorporated into dedicated logic circuitry.

[0062] To provide interaction with a user, one or more aspects of this disclosure can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touchscreen) for displaying information to the user; and optionally, a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending web pages to a web browser on the user's client device in response to a request received from a web browser.

[0063] Various embodiments have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other embodiments are also within the scope of the appended claims.

Claims

1. An automated speech recognition (ASR) model for unifying streaming and non-streaming speech recognition, the ASR model comprising: An audio encoder, the audio encoder being configured to: The sequence of received acoustic frames is used as input; as well as At each of the multiple time steps, the attention probability distribution function PDF is used to generate a high-order feature representation of the corresponding acoustic frame in the sequence of acoustic frames; Tag encoder, the tag encoder being configured to: It receives a sequence of non-whitespace symbols output from the final softmax layer as input; as well as A dense representation is generated at each of the plurality of time steps; as well as A federated network, which is configured as follows: The system receives the high-order feature representation generated by the audio encoder at each of the plurality of time steps and the dense representation generated by the tag encoder at each of the plurality of time steps as input; as well as At each of the plurality of time steps, a probability distribution is generated over the possible speech recognition hypotheses for that time step. The audio encoder includes a neural network that applies a hybrid model, MiMo attention, to compute an attention probability distribution function (PDF) using a set of hybrid components with softmax applied over a context window. The ASR model switches between streaming and non-streaming modes by adjusting the hybrid weights of the MiMo attention.

2. The ASR model according to claim 1, wherein, The set of hybrid components of the MiMo attention is designed to cover all possible contextual use cases during inference.

3. The ASR model according to claim 1, wherein, Each of the hybrid components in the set of hybrid components includes a fully normalized softmax.

4. The ASR model according to claim 1, wherein, The neural network of the audio encoder includes multiple convolutional enhancement transformer layers.

5. The ASR model according to claim 1, wherein, The neural network of the audio encoder includes multiple transformer layers.

6. The ASR model according to claim 1, wherein, The tag encoder includes a neural network comprising a transformer layer, a convolutional augmentation transformer layer, or a long short-term memory (LSTM) layer.

7. The ASR model according to claim 1, wherein, The tag encoder includes a lookup table embedding model configured to look up the dense representation for each of the plurality of time steps.

8. The ASR model according to any one of claims 1-7, wherein, The possible speech recognition hypotheses generated at each of the plurality of time steps correspond to a set of output labels, each output label representing a word or phrase in natural language.

9. A computer-implemented method, which, when executed on data processing hardware, causes the data processing hardware to perform operations, the operations including: Receive the sequence of acoustic frames; At each of the multiple time steps: An audio encoder using an Automatic Speech Recognition (ASR) model generates high-order feature representations of the corresponding acoustic frames in the sequence of acoustic frames using the attention probability distribution function (PDF). At each of the plurality of time steps, a dense representation is generated using the label encoder of the ASR model, the label encoder being configured to receive a sequence of non-whitespace symbols output by the final softmax layer of the ASR model; as well as Using the joint network of the ASR model, based on the high-order feature representation generated by the audio encoder at the corresponding time step and the dense representation generated by the label encoder at the corresponding time step, a probability distribution on the possible speech recognition hypotheses at the corresponding time step is generated. The audio encoder includes a neural network that applies a hybrid model, MiMo attention, to compute the attention probability distribution function PDF using a set of hybrid components with softmax applied over a context window. The ASR model switches between streaming and non-streaming modes by adjusting the hybrid weights of the MiMo attention.

10. The computer-implemented method according to claim 9, wherein, The set of hybrid components of the MiMo attention is designed to cover all possible contextual use cases during inference.

11. The computer-implemented method according to claim 9, wherein, Each of the hybrid components in the set of hybrid components includes a fully normalized softmax.

12. The computer-implemented method according to claim 9, wherein, The neural network of the audio encoder includes multiple convolutional enhancement transformer layers.

13. The computer-implemented method according to claim 9, wherein, The neural network of the audio encoder includes multiple transformer layers.

14. The computer-implemented method according to claim 9, wherein, The tag encoder includes a neural network comprising a transformer layer, a convolutional augmentation transformer layer, or a long short-term memory (LSTM) layer.

15. The computer-implemented method according to claim 9, wherein, The tag encoder includes a lookup table embedding model configured to look up the dense representation for each of the plurality of time steps.

16. The computer-implemented method according to any one of claims 9-15, wherein, The possible speech recognition hypotheses generated at each of the plurality of time steps correspond to a set of output labels, each output label representing a word or phrase in natural language.

Images