Quantization-aware training for universal speech models with recurrent neural network-transducer decoder

Abstract

A method (700) includes obtaining a plurality of training samples (402), each training sample including audio data (404) characterizing a corresponding speech utterance (304, 306) and a transcription (420) of the corresponding speech utterance. The method also includes training an automatic speech recognition (ASR) model (200) on the plurality of training samples, the ASR model having a recurrent neural network-transducer (RNN-T) architecture. The method also includes quantizing the trained ASR model to an integer target fixed bit width, and providing the quantized trained ASR model to a user device (10).

Description

Technical Field

[0001] This disclosure relates to quantization-aware training for a general speech model with a recurrent neural network-transducer decoder. Background Technology

[0002] With recent advancements in large-scale general speech models (USM), the quality of end-to-end automatic speech recognition (ASR) models has been revolutionized. However, these models are enormous (e.g., billions of parameters), requiring substantial amounts of memory and computing power, resulting in high deployment costs. Therefore, efficient training and model compression algorithms have become an unprecedentedly important research topic. Summary of the Invention

[0003] One aspect of this disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the hardware to perform operations including obtaining a plurality of training samples, each training sample including audio data representing a corresponding speech utterance and a transcription of the corresponding speech utterance. These operations also include training an automatic speech recognition (ASR) model on the plurality of training samples. The ASR model includes a recurrent neural network-transducer (RNN-T) architecture. These operations further include quantizing the trained ASR model to an integer target fixed bit width and providing the quantized trained ASR model to a user device.

[0004] Implementations of this disclosure may include one or more of the following optional features. In some implementations, quantizing the trained ASR model includes: quantizing the trained ASR model to a target fixed-bit width equal to two using per-channel asymmetric quantization with scale backpropagation. In another implementation, quantizing the trained ASR model includes: quantizing the trained ASR model to a target fixed-bit width equal to two using per-channel asymmetric quantization with scale backpropagation, clipping, and sub-channel splitting. In yet another implementation, quantizing the trained ASR model includes: quantizing the trained ASR model to a target fixed-bit width equal to one using absolute maximum binarization. In still another implementation, quantizing the trained ASR model includes: subtracting the per-channel average from multiple weights of the trained ASR model to provide multiple scaled weights; and binarizing the multiple scaled weights to a target fixed-bit width equal to one. In another implementation, quantizing the trained ASR model includes: increasing the number of channels in the multiple input weights of the trained ASR model by splitting multiple input weights into sub-channels; subtracting the average value of each sub-channel from the multiple input weights of the trained ASR model to provide multiple scaled weights; binarizing the multiple scaled weights; using pseudo-quantization-aware training, dequantizing the binarized scaled weights by multiplying the binarized scaled weights by a scaling factor to provide dequantized weights; reshaping the dequantized weights to the same shape as the multiple input weights; and calculating einsum on the input activations applied to the reshaped dequantized weights to quantize the trained ASR model to a target fixed-bit width equal to one.

[0005] In some examples, obtaining multiple training samples involves receiving training data that includes: a corpus of transcribed non-synthetic speech utterances, where each transcribed non-synthetic speech utterance is paired with a corresponding transcription; and a corpus of untranscribed non-synthetic speech utterances. Each untranscribed non-synthetic speech utterance is not paired with a corresponding transcription. In these examples, a teacher ASR model is trained on the corpus of transcribed non-synthetic speech utterances to teach the teacher ASR model how to predict the corresponding transcription from the non-synthetic speech utterances, and the trained teacher model is used to process both the corpus of transcribed non-synthetic speech utterances and the corpus of untranscribed non-synthetic speech utterances to predict the corresponding pseudo-real value labels.

[0006] In some implementations, the ASR model includes an audio encoder and a decoder. The decoder includes a prediction network and a joint network. In these implementations, quantizing the ASR model may include quantizing the audio encoder but not the decoder. Additionally, the audio encoder may include multiple self-attention layers, each including a multi-head attention mechanism. The self-attention layers may include conformer layers or transformer layers. Notably, the speech utterances and transcriptions from multiple training samples can span multiple different languages, enabling the trained ASR model to include a multilingual ASR model.

[0007] Another aspect of this disclosure provides a system including data processing hardware and memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations including acquiring a plurality of training samples, each training sample including audio data representing a corresponding speech utterance and a transcription of the corresponding speech utterance. These operations also include training an automatic speech recognition (ASR) model on the plurality of training samples. The ASR model includes a recurrent neural network-transducer (RNN-T) architecture. These operations further include quantizing the trained ASR model to an integer target fixed bit width and providing the quantized trained ASR model to a user device.

[0008] This aspect of the disclosure may include one or more optional features from the following optional features. In some implementations, quantizing the trained ASR model includes: quantizing the trained ASR model to a target fixed-bit width equal to two using per-channel asymmetric quantization with scale backpropagation. In another implementation, quantizing the trained ASR model includes: quantizing the trained ASR model to a target fixed-bit width equal to two using per-channel asymmetric quantization with scale backpropagation, clipping, and sub-channel splitting. In yet another implementation, quantizing the trained ASR model includes: quantizing the trained ASR model to a target fixed-bit width equal to one using absolute maximum binarization. In yet another implementation, quantizing the trained ASR model includes: subtracting the per-channel average from a plurality of weights of the trained ASR model to provide a plurality of scaled weights; and binarizing the plurality of scaled weights to a target fixed-bit width equal to one. In another implementation, quantizing the trained ASR model includes: increasing the number of channels in the multiple input weights of the trained ASR model by splitting multiple input weights into sub-channels; subtracting the average value of each sub-channel from the multiple input weights of the trained ASR model to provide multiple scaled weights; binarizing the multiple scaled weights; using pseudo-quantization-aware training, dequantizing the binarized scaled weights by multiplying the binarized scaled weights by a scaling factor to provide dequantized weights; reshaping the dequantized weights to the same shape as the multiple input weights; and calculating einsum on the input activations applied to the reshaped dequantized weights to quantize the trained ASR model to a target fixed-bit width equal to one.

[0009] In some examples, obtaining multiple training samples involves receiving training data that includes: a corpus of transcribed non-synthetic speech utterances, where each transcribed non-synthetic speech utterance is paired with a corresponding transcription; and a corpus of untranscribed non-synthetic speech utterances. Each untranscribed non-synthetic speech utterance is not paired with a corresponding transcription. In these examples, a teacher ASR model is trained on the corpus of transcribed non-synthetic speech utterances to teach the teacher ASR model how to predict the corresponding transcription from the non-synthetic speech utterances, and the trained teacher model is used to process both the corpus of transcribed non-synthetic speech utterances and the corpus of untranscribed non-synthetic speech utterances to predict the corresponding pseudo-real value labels.

[0010] In some implementations, the ASR model includes an audio encoder and a decoder. The decoder includes a prediction network and a joint network. In these implementations, quantizing the ASR model may include quantizing the audio encoder but not the decoder. Additionally, the audio encoder may include multiple self-attention layers, each including a multi-head attention mechanism. The self-attention layers may include conformer layers or transformer layers. Notably, the speech utterances and transcriptions from multiple training samples can span multiple different languages, enabling the trained ASR model to include a multilingual ASR model.

[0011] Details of one or more implementations of this disclosure are set forth in the accompanying drawings and the following description. Other aspects, features, and advantages will become apparent from the specification, drawings, and claims. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of an example system for performing speech recognition.

[0013] Figure 2 This is a schematic diagram of the Recurrent Neural Network-Transducer (RNN-T) model architecture.

[0014] Figure 3A and Figure 3B This is a schematic diagram of an example training initialization process used to train a teacher model to learn how to predict real-value pseudo-labels from speech utterances.

[0015] Figure 4 This is a schematic diagram of an example training process used to train and quantize a speech recognition model.

[0016] Figure 5 This is an example algorithm for absolute average binarization technique.

[0017] Figure 6 It is a combination of sub-channel splitting and absolute average binarization. Figure 4 A schematic diagram illustrating an example of the training process.

[0018] Figure 6 yes Figure 4 Example algorithm for the training process.

[0019] Figure 7 This is a flowchart illustrating an example of the operational steps involved in training and quantifying a speech recognition model.

[0020] Figure 8 This is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0021] In the various figures, the same reference numerals indicate the same elements. Detailed Implementation

[0022] With recent advancements in large-scale general speech models (USM), the quality of end-to-end automatic speech recognition (ASR) models has been revolutionized. However, these models are enormous (e.g., billions of parameters), requiring substantial amounts of memory and computing power, resulting in high deployment costs. Therefore, efficient training and model compression algorithms have become an unprecedentedly important research topic.

[0023] Recently, with the rapid emergence of large-scale datasets and high-capacity data processing hardware (such as graphics processing units (GPUs) and tensor processing units (TPUs)), self-supervised learning (SSL) ASR models have become increasingly larger in size by scaling up regular SSL ASR models to capture multi-domain and multilingual distributions. Therefore, these SSL ASR models can be used as general-purpose foundational models for most speech processing tasks. However, due to the high deployment costs of these SSL ASR models because of their large size, efficient training and model compression algorithms have become an unprecedentedly important research topic.

[0024] Quantization is a technique that reduces the computational and memory costs of ASR models by using lower-precision data types (e.g., 8-bit integers) instead of the usual 32-bit floating-point values ​​to represent weights and / or activations. However, one drawback of such techniques is that performance degrades as the amount of training data used to train the model increases (e.g., millions of hours of speech). Sparsity is another technique for reducing the computational and memory costs of ASR models. When using sparsity, nodes are pruned based on the entropy of the weights and node activity. Combining quantization with sparsity is another technique that reduces the computational and memory costs of USM by using a sparse mask to prune the weights of the USM model and then quantizing the weights of the USM model with 4 or 2 bits. However, USM with 2-bit weight quantization, obtained from the combined quantization and sparsity techniques, suffers a drop in accuracy to 1 / 3 in terms of word error rate (WER). For models with 1-bit weight quantization (binarization), there are significant challenges regarding WER degradation.

[0025] The implementation described in this paper involves training a USM model with a recurrent neural network-transducer (RNN-T) architecture using a weight-only binarization technique to achieve a reduction in model size without a significant degradation in accuracy. While 4-bit quantization typically does not cause a quality change for most ASR models, the effectiveness of ASR models with 2-bit weight quantization depends on both model size and data size. That is, since USMs can include over a billion parameters and are trained on millions of hours of speech data, performing quantization-aware training to achieve 2-bit or fewer weight quantizations can lead to undesirable WER degradation in these models. For example, a USM model with a Connectionized Temporal Classification (CTC) decoder trained using native quantization-aware and sparse-aware training to achieve 2-bit weight quantization has been shown to suffer a 1 / 3 drop in accuracy in terms of WER. In contrast, the specific implementation disclosed in this paper applies weight-only binarization to train the USM RNN-T model by combining absolute average binarization with subchannel quantization, reducing the size of the USM RNN-T model by 6.2% from its original size, while increasing the average WER by only 1.9% compared to the floating-point baseline model. It is noteworthy that the technique of using absolute average binarization disclosed in this paper does so without weight centering, thus simplifying model quantization without affecting model accuracy compared to quantization techniques with weight centering.

[0026] Traditional CTC-based USM models are large models capable of performing ASR on multiple languages ​​(e.g., 300 languages). They use an audio encoder with multiple multi-head attention layers and employ CTC loss during training. Trainable language embeddings are injected as first terms into the encoder, enabling the model to handle approximately 300 languages. The audio encoding output from the audio encoder is then fed into the CTC decoder for predicting speech recognition results. The training process of the USM CTC model involves a complex four-step process. In the first step, the 600 million-parameter USM CTC is initially trained on a small supervised dataset to provide a teacher model, which generates pseudo-labels on both the supervised and unsupervised datasets. In the second step, the audio encoder is pre-trained using BERT-based speech pre-training (BEST-RQ) with a random projection quantizer for self-supervised learning. In the third step, the complete USM CTC model is trained using the pseudo-labels generated by the teacher model from the supervised and unsupervised datasets in the first step. Finally, the fourth step involves fine-tuning the USM CTC model with task-specific data to improve performance.

[0027] In contrast, our implementation simplifies the training of CTC-based USM models by replacing the CTC decoder with an RNN-T decoder and training the USM RNN-T model using RNN-T loss. Since RNN-T loss is known to have higher performance than CTC loss, the training process for the USM RNN-T model can eliminate the BEST-RQ pre-training and fine-tuning steps (e.g., steps 2 and 4) used in the training of CTC-based USM models. Therefore, our implementation involves a training process for the USM RNN-T model that applies only two steps, providing 2-bit weight quantization with only a 1.4% WER degradation, in contrast to the 3x increase in WER degradation shown by a USM CTC-based model trained using a four-step process.

[0028] Figure 1 This is an example of a system 100 operating in a voice environment 101. In the voice environment 101, a user 104 interacts with a computing device such as user device 10 via voice input. User device 10 (also commonly referred to as device 10) is configured to capture sound (e.g., streaming audio data) from one or more users 104 within the voice environment 100. Here, streaming audio data may refer to spoken utterances 106 of user 104, which serve as an audible query, a command to device 10, or audible communication captured by device 10. The voice-enabled system of device 10 can process queries or commands by answering queries and / or causing one or more downstream applications to execute / fulfill commands.

[0029] User device 10 may correspond to any computing device associated with user 104 and capable of receiving audio data. Some examples of user device 10 include, but are not limited to, mobile devices (e.g., mobile phones, tablet computers, laptop computers, etc.), computers, wearable devices (e.g., smartwatches), smart appliances, Internet of Things (IoT) devices, vehicle infotainment systems, smart displays, smart speakers, etc. User device 10 includes data processing hardware 12 and memory hardware 14 that communicates with and stores instructions to the data processing hardware 12, which, when executed by the data processing hardware 12, cause the data processing hardware 12 to perform one or more operations. User device 10 further includes an audio system 16 having: audio capture devices (e.g., microphones) 16, 16a for capturing spoken words 106 within a speech environment 100 and converting them into electrical signals; and voice output devices (e.g., speakers) 16, 16b for transmitting audible audio signals (e.g., as output audio data from device 10). Although the user device 10 implements a single audio capture device 16a in the example shown, the user device 10 may implement an array of audio capture devices 16a without departing from the scope of this disclosure, wherein one or more capture devices 16a in the array may not physically reside on the user device 10, but may communicate with the audio system 16.

[0030] In the speech environment 100, the Automatic Speech Recognition (ASR) system 118 includes an ASR model 200 (such as an ASR model with a recurrent neural network-transducer (RNN-T) model architecture or other transducer / multipath model) residing on the user device 10 of the user 104 and / or on a remote computing device 60 (e.g., one or more remote servers of a distributed system running in a cloud computing environment) communicating with the user device 10 via a network 40. The remote computing device 60 is equipped with data processing hardware 62 and memory hardware 64. The user device 10 and / or the remote computing device 60 also includes an audio subsystem 108 configured to receive utterances 106 spoken by the user 104 and captured by an audio capture device 16a, and to convert the utterances 106 into a corresponding digital format associated with an input acoustic frame 110 that can be processed by the ASR system 118. In the example shown, the user utters a corresponding utterance 106, and the audio subsystem 108 converts the utterance 106 into corresponding audio data (e.g., acoustic frames) 110 for input to the ASR system 118. Subsequently, the model 200 receives the audio frame 110 (i.e., audio data) corresponding to the utterance 106 as input and generates / predicts a corresponding transcription 120 (e.g., speech recognition result / hypothesis) of the utterance 106 as output.

[0031] User device 10 and / or remote computing device 60 also execute user interface generator 107, which is configured to present a representation of the transcription 120 of utterance 106 to user 104 of user device 10. As described in more detail below, user interface generator 107 can display the speech recognition result 120 in a streaming manner. In some configurations, the transcription 120 output from ASR system 118 is processed, for example, by a natural language understanding (NLU) module executing on user device 10 or remote computing device 60 to execute the user command / query specified by utterance 106. Alternatively or additionally, a text-to-speech system (not shown) (e.g., executing on any combination of user device 10 or remote computing device 60) can convert the transcription into synthesized speech for audible output by user device 10 and / or another device.

[0032] In the example shown, user 104 interacts with a program or application 50 (e.g., digital assistant application 50) on user device 10 using ASR system 118. For example, Figure 1 The text depicts a user 104 communicating with a digital assistant application 50, and the digital assistant application 50 displaying a digital assistant interface 18 on the screen of the user device 10 to illustrate the dialogue between the user 104 and the digital assistant application 50. In this example, the user 104 asks the digital assistant application 50, “What time is the concert tonight?” This question from the user 104 is spoken utterance 106, which is captured by the audio capture device 16a and processed by the audio system 16 of the user device 10. In this example, the audio subsystem 108 receives the spoken utterance 106 and converts it into an acoustic frame 110 for input to the ASR system 118.

[0033] refer to Figure 2Example frame-aligned transducer model 200 includes a recurrent neural network-transducer (RNN-T) model architecture that adheres to latency constraints associated with interactive applications. The use of the RNN-T model architecture is exemplary, and the frame-aligned transducer model 200 may include other architectures such as transformer-transducer and conformer-transducer model architectures. The RNN-T model 200 offers a small computational footprint and uses less memory than conventional ASR architectures, making it suitable for performing speech recognition entirely on user device 102 (e.g., without needing to communicate with a remote server). The RNN-T model 200 includes an encoder network 210 and a decoder 250, the decoder including a prediction network 220 and a joint network 230. The encoder network 210, broadly similar to the acoustic model (AM) in a conventional ASR system, includes a recurrent network of stacked or stacked long short-term memory (LSTM) layers of multi-head attention layers (e.g., conformer or transformer layers). For example, the encoder reads… d 110-dimensional feature vectors (e.g., acoustic frames 110) Figure 1 The sequence x = (x) 1 , x 2 , ··· , x T ),in x t ∈ d Furthermore, at each output step, a higher-order feature representation is generated. This higher-order feature representation is represented as... .

[0034] Similarly, prediction network 220 is also an LSTM network, which, like the language model (LM), will generate a sequence of non-whitespace symbols output so far by the final Softmax layer 240. y 0 , . . . , y ui-1 Processing as dense representation Finally, utilizing the RNN-T model architecture, the representations generated by the encoder network 210 and the predictor / decoder network 220 are combined by a joint network 230. The predictor network 220 can be replaced by an embedding lookup table to improve latency by outputting the looked-up sparse embeddings instead of processing dense representations. Then, the joint network predicts... This is a distribution of the next output symbol. In other words, the joint network 230 generates a probability distribution over possible speech recognition hypotheses at each output step (e.g., time step). Here, a "possible speech recognition hypothesis" corresponds to a set of output labels, each representing a symbol / character of a specified natural language. For example, when the natural language is English, this set of output labels may include twenty-seven (27) symbols, for example, one label for each of the 26 letters of the English alphabet, and one label for a space. Thus, the joint network 230 may output a set of values ​​indicating the probability of each output label appearing in a predetermined set of output labels. This set of values ​​may be a vector and may indicate a probability distribution over the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, and possibly punctuation marks and other symbols), but the set of output labels is not limited to this. For example, in addition to graphemes or alternative graphemes, the set of output labels may also include word pieces, phonemes, and / or entire words. The output distribution of the joint network 230 may include posterior probability values ​​for each label in the different output labels. Therefore, if there are 100 different output labels representing different characters or other symbols, then the output y of the joint network 230... i It can include 100 different probability values, with one probability value corresponding to each output label. The probability distribution can then be used (e.g., by a Softmax layer 240) to select candidate orthographic elements (e.g., graphemes, word segments, and / or words) during the beam search process and assign them scores to determine the transcription 120.

[0035] The Softmax layer 240 can employ any technique to select the output label / symbol with the highest probability in the distribution as the next output symbol predicted by the RNN-T model 200 at the corresponding output step. In this way, the RNN-T model 200 does not make a conditional independence assumption, but rather conditions the prediction of each symbol not only acoustically but also with the sequence of labels output so far. The RNN-T model 200 does assume that the output symbol is independent of future acoustic frames 110, which allows the RNN-T model to be used in a streaming manner.

[0036] In some examples, the encoder network (i.e., the audio encoder) 210 of the RNN-T model 200 includes a stack of multi-head attention layers / blocks with self-attention mechanisms, such as conformer layers / blocks. In some configurations, the audio encoder 210 includes 16 conformer layers / blocks. In some examples, the relative attention in each self-attention layer is equal to 12 attention heads. Here, each conformer block includes a series of multi-head self-attention layers, depthwise convolutional layers, and feedforward layers. In other examples, the stack of self-attention layers / blocks may include transformer layers / blocks. The prediction network 220 may have three 2,048-dimensional LSTM layers, each followed by a 640-dimensional projection layer. Alternatively, the prediction network 220 may include a stack of transformer blocks or conformer blocks, or an embedding lookup table instead of LSTM layers. Finally, the joint network 230 may also have 640 hidden units. The Softmax layer 240 can consist of a uniform set of word pieces or morphemes, which is generated using all unique word pieces or morphemes from multiple training datasets. The vocabulary size of the decoder 250 can be 16,384 word pieces.

[0037] Figure 3A and Figure 3B The initialization process 300 and training process 400 for training the teacher USM CTC model 301 to learn how to predict the true value labels 420 of supervised and unsupervised training sets are shown. Figure 4 These ground truth labels are used to apply quantized perceptual training to the ASR model 200, which includes a USM RNN-T model architecture. The teacher USM CTC model 301 includes an audio encoder 210 and a CTC decoder 350. The teacher USM CTC model 301 may include approximately 600 million parameters. For simplicity, the training initialization process 300 includes a supervised training section 300a for training the teacher USM CTC model 350. Figure 3A ) and the label prediction part 300b for both untranscribed speech 306 belonging to the unsupervised training dataset and transcribed speech 304 belonging to the supervised training dataset to predict pseudo-true labels 420. Figure 3BTranscribed speech utterances 306 belong to a corpus of transcribed non-synthetic speech utterances, with each transcribed non-synthetic speech utterance paired with a corresponding transcript 320. The corresponding transcripts 320 may be tagged by a human transcriber. Untranscribed speech utterances 304 belong to a corpus of untranscribed non-synthetic speech utterances not paired with corresponding transcripts. The total length of the corpus of transcribed non-synthetic speech utterances 304 can be significantly shorter than the total length of the corpus of untranscribed non-synthetic speech utterances 304. For example, the corpus of transcribed non-synthetic speech utterances 304 may include approximately 680,000 hours of speech across 56 different languages, while the corpus of untranscribed non-synthetic speech utterances 306 may include approximately 4.2 million hours of speech across 75 different languages.

[0038] refer to Figure 3A In some implementations, the audio encoder 210 includes a Conformer encoder comprising a stack of Conformer blocks, each of which includes a series of multi-head self-attention layers, depthwise convolutional layers, and feedforward layers. Alternatively, the audio encoder 210 may include another type of encoder with a stack of self-attention layers / blocks, such as a transformer encoder. The Conformer encoder 210 can be naturally split into a feature encoder (comprising convolutional subsampling blocks) and a context network (comprising linear layers and a stack of Conformer blocks). In some implementations, the convolutional subsampling blocks have two two-dimensional convolutional layers, each with a stride of (2, 2), resulting in a feature sequence length reduced to 1 / 4. The convolutional subsampling blocks receive input features / vectors (e.g., Mel spectrograms, etc.) associated with each transcribed non-synthetic speech utterance 304. Figure 1 The sequence of acoustic frames 110 is taken as input, and an encoded audio representation is generated for each of the multiple output steps. sup 314 is output.

[0039] The CTC decoder 350 may include a phoneme decoder configured to decode sequences of phonemes, a word segment decoder configured to decode sequences of word segments, or a character segment decoder configured to decode sequences of characters. The CTC decoder 350 receives each encoded audio representation 324 output from the encoder 210 as input and generates a probability distribution 394 as output regarding possible non-synthetic speech recognition hypotheses for the corresponding transcribed non-synthetic speech utterance 304 at the corresponding time step. In some examples, the probability distribution 394 regarding possible non-synthetic speech recognition hypotheses includes one of possible phoneme labels, possible word segment labels, or possible character segment labels. Subsequently, the supervised loss module 340 may determine a non-synthetic speech loss term 344 based on the probability distribution 394 regarding possible non-synthetic speech recognition hypotheses and the corresponding transcript 320 paired with the transcribed non-synthetic speech utterance 304. Here, the corresponding transcript 320 is used as a ground truth transcript and may include sequences of target phonemes, target word segments, and / or target characters. The supervised training part 300a can train the USM CTC model on the non-synthetic speech loss term 344 by updating the parameters of the audio encoder 210 (and optionally the CTC decoder 350) using the non-synthetic speech loss term 344.

[0040] refer to Figure 3B The label prediction part 300b of the training initialization process 300 uses the trained USM CTC model 301 from the supervised training part 301 to predict the pseudo-real value labels 420 for all speech utterances in the corpus of transcribed non-synthetic speech utterances 304 and the corpus of untranscribed non-synthetic speech utterances 306. It is worth noting that although each of the transcribed non-synthetic speech utterances includes a corresponding transcription 320, the pseudo-real value labels 420 predicted by the trained USM CTC model 301 for the transcribed non-synthetic speech utterances 304 will be used in the training process 400 (for training the USM RNN-T model 200) to predict the pseudo-real value labels 420. Figure 4 During this period, the corresponding transcription 320 is replaced.

[0041] For each corresponding transcribed non-synthetic speech utterance 304 and each corresponding untranscribed non-synthetic speech utterance 306, the audio encoder 210 receives input features / vectors (e.g., Mel spectrograms, etc.) associated with the corresponding transcribed non-synthetic speech utterance 304 or the corresponding untranscribed non-synthetic speech utterance 304. Figure 1 The sequence of acoustic frames 110 is taken as input, and an encoded audio representation is generated for each of the multiple output steps. sup) 314 is the output. The CTC decoder 350, including a phoneme decoder, word slice decoder, or character decoder, receives the encoded audio representation 324 output from the encoder 210 as input and generates corresponding pseudo-real value labels 420 for the corresponding transcribed non-synthetic speech 304 or the corresponding untranscribed non-synthetic speech 306 as output. Each transcribed non-synthetic speech 304 can be paired with a corresponding pseudo-real value label 420, and each untranscribed non-synthetic speech 306 can be paired with a corresponding pseudo-real value label 420 to form multiple multilingual supervised training samples 402. Figure 4 ).

[0042] After the label prediction part 300b of the training initialization process 300 predicts the pseudo-real value labels 420 of transcribed and untranscribed non-synthetic speech utterances 304 and 306 to form a supervised multilingual training utterance set 402, Figure 4 A training process 400 is illustrated for training a USM RNN-T model 200 to learn how to transcribe spoken utterance on multiple supervised training samples 402, 402a-n using quantized perceptual training. The number of training samples 402 in the multiple supervised training samples 402 may be equal to the sum of the number of transcribed non-synthetic speech utterances 304 and the number of untranscribed non-synthetic speech utterances 304. Here, each training sample 402 includes audio data 404 representing the corresponding speech utterance in either the transcribed non-synthetic speech utterance 304 or the untranscribed non-synthetic speech utterance 306, and a corresponding pseudo-ground value label 420 predicting the corresponding speech utterance in either the transcribed non-synthetic speech utterance 304 or the untranscribed non-synthetic speech utterance 306. The pseudo-ground value label 420 corresponds to the transcription of the corresponding speech utterance 304, 306.

[0043] A model trainer 150 running on a remote computing system 60 can perform a training process 400 to train an ASR model 200 (e.g., a USM RNN-T model 200) on multiple multilingual supervised training samples 402, while using quantization-aware training and optionally weight-only binarization to compress the size of the resulting ASR model 200. For each training sample 402, the ASR model 200 processes the corresponding audio data 404 to predict the corresponding speech recognition result 252, and a loss module 260 generates a corresponding training loss 270 based on the speech recognition result 252 and the corresponding pseudo-real value label 420. Since the ASR model 200 includes an RNN-T decoder, the training loss 270 includes an RNN-T loss. During the training process 400 of training the ASR model 200, the model trainer 150 applies various quantization-aware training (QAT) 160 techniques to reduce the model size while maintaining acceptable WER degradation.

[0044] The audio encoder 210 of the ASR model 200 includes multiple weights, and the training process 400 performs QAT 160 by quantizing each of the multiple weights of the encoder based on an integer with a fixed bit weight width. QAT 160 can quantize the weights of the linear and projective attention layers in the audio encoder 210. QAT 160 may not quantize the convolutional layers and the layers of the RNN-T decoder 250 in the audio encoder 210 because their size is negligible compared to the size of the audio encoder. Therefore, the convolutional layers in the audio encoder 210 and the layers of the RNN-T decoder 250 retain their original floating-point values.

[0045] In some examples, per-channel asymmetric quantization with scale backpropagation is used to quantize the trained ASR model 200 to provide 2-bit weighted quantization to the ASR model 200. The training process can include 270,000 training iterations over 1.7 days. Here, the resulting ASR model 200 with 2-bit weighted quantization has a 1.4% WER degradation compared to the USM CTC model trained through combined quantization and sparsity, which suffers a WER degradation to 1 / 3, and the model size is reduced from its floating-point baseline to 8.8%. It can be assumed that the lower accuracy of the USM CTC model can be attributed to its four-stage training process.

[0046] In some additional examples, the trained ASR model 200 is quantized using per-channel asymmetric quantization with scale backpropagation, clipping, and sub-channel splitting to provide 2-bit weighted quantization to the ASR model 200. The training process can include 270,000 training iterations over 1.7 days. Here, the resulting ASR model 200 with 2-bit weighted quantization has a model size reduced to 12.6% from its floating-point baseline due to additional metadata attributable to its block size of 64.

[0047] By adding binarization, QAT 160 can provide 1-bit weight quantization to the trained ASR model 200 (i.e., the USM RNN-T model 200). The training process, applying different binarization techniques, requires 446,000 training iterations over 3.4 days. For example, QAT 160 can use absolute maximum binarization to quantize the trained ASR model to provide 1-bit weight quantization to ASR model 200. However, applying absolute maximum binarization as the binarization technique used to quantize the trained ASR model 200 causes the model to diverge and results in a 100% WER. It can be assumed that absolute maximum binarization causes the model to diverge because the scaling factor is defined by the maximum value of the absolute weights.

[0048] In some implementations, absolute average binarization is applied for quantization to provide 1-bit weight quantization without causing the ASR model 200 to diverge. One technique for applying absolute average binarization may include subtracting the per-channel average from multiple weights of a trained ASR model (e.g., of the audio encoder 210) to provide multiple scaled weights, and then binarizing the multiple scaled weights to the target 1-bit weight quantization. Figure 5 An example algorithm 500 is shown that applies this technique of absolute average binarization. This algorithm works by subtracting the average value per channel (row 7) and then adjusting the weights. x Binarize (line 14) to center the weights x It is worth noting that all zeros are set to equal to a small positive number (line 13, such that...). symbol The function only returns 1 and -1. Weight centering is not applied.

[0049] In another technique, QAT 160 combines absolute average binarization with subchannel quantization, which includes a predefined block size with a value equal to 64. Figure 6 A schematic diagram 600 illustrates an example of a training process 400 that applies a combination of sub-channel splitting and absolute average binarization. Here, the multiple input weights of the trained ASR model 200 have a [2x6] shape, and the training process 400 increases the number of channels in the multiple input weights by splitting the multiple weights into sub-channels. The sub-channels can have a block size equal to a value of 3, resulting in a change in the weight shape to [4x3]. Without departing from the scope of this disclosure, the block size of the sub-channels can be set to a value other than 3. Subsequently, the training process subtracts the average value per sub-channel from the multiple input weights to provide multiple scaled weights and binarizes the multiple scaled weights. Using pseudo-QAT, the training process 400 then dequantizes the binarized scaled weights by multiplying the binarized scaled weights by a scaling factor to provide dequantized weights. Subsequently, the dequantized weights are reshaped back to their original [2x6] shape. Then, during training 400, the input activations applied to the reshaped dequantized weights are computed to calculate einsum, quantizing the trained ASR model 200 to 1-bit weight quantization. Notably, the dequantization step of multiplying the binarized, scaled weights by a scaling factor is performed on the einsum output, causing the input activations to be reshaped to match the shape of the binarized weights, for example, a [4x3] shape. The trained ASR model 200 with 1-bit weight quantization, obtained from the combined absolute average binarization and subchannel quantization techniques, achieves a reduction in model size to 6.2% and an average WER reduction of only 1.9% compared to its floating-point baseline.

[0050] Figure 7 This is a flowchart illustrating an exemplary arrangement of the operation of a method 700 for training and quantizing an Automatic Speech Recognition (ASR) model 200 with a recurrent neural network-transducer (RNN-T) architecture. Method 700 may be based on memory hardware 820 (…). Figure 8 Instructions on the data processing hardware 810 ( Figure 8 The data processing hardware 810 may include the data processing hardware 62 of the remote computing system 60, and the memory hardware 820 may include the memory hardware 64 of the remote computing system 60.

[0051] At operation 702, method 700 includes obtaining multiple training samples 402, each training sample including audio data representing the corresponding speech utterances 304, 306 and a transcription 420 of the corresponding speech utterances 304, 306. The transcription 420 corresponds to pseudo-real value labels. At operation 704, method 700 includes training an ASR model 200 on the multiple training samples 402. Here, supervised training is applied, wherein the ASR model 200 is trained to learn how to predict the pseudo-real value labels of the corresponding speech utterances 304, 306 represented by the audio data 404.

[0052] At operation 706, method 700 includes quantizing the trained ASR model to an integer target fixed bit width. In some examples, the target fixed bit width is equal to 2 bits. In other examples, the target fixed bit width is equal to 1 bit. At operation 708, method 700 includes providing the quantized trained ASR model to user device 10. User device 10 can execute a trained ASR model 200 with a compressed size from quantized perceptual training, such that ASR model 200 can perform speech recognition on user device 10. Alternatively or additionally, trained ASR model 200 can be executed on a remote computing device communicating with user device 10 for performing speech recognition on spoken utterance captured by user device 10 and transmitted to the remote computing device.

[0053] Figure 8 This is a schematic diagram of an example computing device 800 that can be used to implement the systems and methods described in this document. The computing device 800 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit the ways in which the inventions described and / or claimed in this document can be implemented.

[0054] The computing device 800 includes a processor 810, a memory 820, a storage device 830, a high-speed interface / controller 840 connected to the memory 820 and a high-speed expansion port 850, and a low-speed interface / controller 860 connected to a low-speed bus 870 and the storage device 830. Each of the components 810, 820, 830, 840, 850, and 860 is interconnected using various buses and can be mounted on a common motherboard or otherwise. The processor 810 can process instructions for execution within the computing device 800, including instructions stored in the memory 820 or the storage device 830, to display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 880 coupled to the high-speed interface 840. In other implementations, multiple processors and / or multiple buses, as well as multiple memories and various types of memory, can be used as appropriate. Moreover, multiple computing devices 800 can be connected, with each device providing a portion of the necessary operation (e.g., as a server group, blade server group, or multiprocessor system).

[0055] Memory 820 stores information non-temporarily within computing device 800. Memory 820 may be a computer-readable medium, a volatile memory cell, or a non-volatile memory cell. Non-temporary memory 820 may be a physical means for temporarily or permanently storing programs (e.g., instruction sequences) or data (e.g., program state information) for use by computing device 800. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.

[0056] Storage device 830 provides mass storage for computing device 800. In some implementations, storage device 830 is a computer-readable medium. In various implementations, storage device 830 may be a floppy disk device, hard disk device, optical disk device, magnetic tape device, flash memory or other similar solid-state storage device, or an array of devices (including devices in a storage area network or other configuration). In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as memory 820, storage device 830, or memory on processor 810.

[0057] High-speed controller 840 manages bandwidth-intensive operations of computing device 800, while low-speed controller 860 manages lower bandwidth-intensive operations. This allocation of responsibilities is merely exemplary. In some implementations, high-speed controller 840 is coupled to memory 820, display 880 (e.g., via a graphics processor or accelerator), and high-speed expansion port 850 which accepts various expansion cards (not shown). In some implementations, low-speed controller 860 is coupled to storage device 830 and low-speed expansion port 890. Low-speed expansion port 890, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), may be coupled to one or more input / output devices, such as keyboards, pointing devices, scanners, or networking devices, such as switches or routers, for example, via a network adapter.

[0058] As shown in the figure, the computing device 800 can be implemented in many different forms. For example, the computing device can be implemented as a standard server 800a or multiple times in a set of such servers 800a, as a laptop computer 800b, or as part of a rack server system 800c.

[0059] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuit systems, integrated circuit systems, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system, which includes at least one programmable processor, which may be dedicated or general-purpose and is coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transfer data and instructions to the storage system, at least one input device, and at least one output device.

[0060] A software application (i.e., a software resource) can refer to computer software that causes a computing device to perform a task. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and game applications.

[0061] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0062] The processes and logic flows described in this specification can be executed by one or more programmable processors, also known as data processing hardware, which execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by special-purpose logic circuit systems, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). For example, processors suitable for executing computer programs include both general-purpose microprocessors and special-purpose microprocessors, as well as any one or more processors of any type of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or operatively coupled to receive data from or transfer data to said mass storage device, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. The processor and memory may be supplemented by or incorporated into a dedicated logic circuit system.

[0063] To provide interaction with the user, one or more aspects of this disclosure can be implemented on a computer having a display device for displaying information to the user (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touchscreen) and possibly a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's client device in response to a request received from a web browser.

[0064] Various implementations have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other implementations are within the scope of the appended claims.

Claims

1. A computer-implemented method (700), said computer-implemented method being executed on data processing hardware (810), said computer-implemented method causing said data processing hardware (810) to perform operations, said operations including: Obtain a plurality of training samples (402), each of the plurality of training samples (402) comprising: Audio data (404) representing the corresponding speech utterances (304, 306); and Transcription (420) of the corresponding speech utterances (304, 306); An automatic speech recognition (ASR) model (200) is trained on the plurality of training samples (402), the ASR model (200) comprising a recurrent neural network-transducer RNN-T architecture; The trained ASR model (200) is quantized into an integer target fixed bit width; and The quantized trained ASR model (200) is provided to the user device (10).

2. The method (700) as described in claim 1, wherein, Quantizing the trained ASR model (200) includes: using per-channel asymmetric quantization with scale backpropagation to quantize the trained ASR model (200) to the target fixed bit width, the target fixed bit width being equal to two.

3. The method (700) as described in claim 1, wherein, Quantizing the trained ASR model (200) includes: using per-channel asymmetric quantization with scale backpropagation, amplitude limiting and sub-channel splitting to quantize the trained ASR model (200) to the target fixed bit width, the target fixed bit width being equal to two.

4. The method (700) as described in claim 1, wherein, Quantizing the trained ASR model (200) includes: using absolute maximum binarization to quantize the trained ASR model (200) to the target fixed bit width, the target fixed bit width being equal to one.

5. The method (700) as claimed in claim 1, wherein, Quantization of the trained ASR model (200) includes: Subtract the per-channel average from the multiple weights of the trained ASR model (200) to provide multiple scaled weights; and The multiple expanded and shrunk weights are binarized into the target fixed bit width, which is equal to one.

6. The method (700) as claimed in claim 1, wherein, Quantization of the trained ASR model (200) includes: The number of channels in the multiple input weights of the trained ASR model (200) is increased by splitting the multiple input weights into sub-channels; The average value of each sub-channel is subtracted from the multiple input weights of the trained ASR model (200) to provide multiple scaled weights; Binarize the multiple expanded and shrunk weights; Using pseudo-quantization-aware training, the binarized expanded weights are dequantized by multiplying the binarized expanded weights by a scaling factor to provide dequantized weights. The dequantized weights are reshaped to the same shape as the plurality of input weights; and The input activations applied to the reshaped dequantized weights are used to compute einsum to quantize the trained ASR model (200) to the target fixed bit width, which is equal to one.

7. The method (700) according to any one of claims 1 to 6, wherein, Obtaining the plurality of training samples (402) includes: Receive training data, the training data including: A corpus of transcribed non-synthetic speech utterances (304), each transcribed non-synthetic speech utterance (304) paired with a corresponding transcription (320); and A corpus of untranscribed non-synthetic speech utterances (306), where each untranscribed non-synthetic speech utterance (306) is not paired with a corresponding transcription; A teacher model (301) is trained on the corpus of transcribed non-synthetic speech utterances (304) to teach the teacher model (301) how to predict the corresponding transcription (320) from the non-synthetic speech utterances (304); and The trained teacher model (301) is used to process the corpus of transcribed non-synthetic speech utterances (304) and the corpus of untranscribed non-synthetic speech utterances (306) to predict the corresponding pseudo-real value labels (420).

8. The method (700) according to any one of claims 1 to 7, wherein, The ASR model (200) includes an audio encoder (210) and a decoder (250), the decoder (250) including a prediction network (220) and a joint network (230).

9. The method (700) as claimed in claim 8, wherein, Quantizing the ASR model (200) includes quantizing the audio encoder (210) but not quantizing the decoder (250).

10. The method (700) as claimed in claim 8 or 9, wherein, The audio encoder (210) includes multiple multi-head self-attention layers, each of which includes a multi-head attention mechanism.

11. The method (700) of claim 10, wherein, The multiple multi-head self-attention layers include a conformer layer or a transformer layer.

12. The method (700) according to any one of claims 1 to 11, wherein: The spoken utterances (304, 306) and the transcriptions (420) of the multiple training samples (402) span multiple different languages; and The trained ASR models (200) include multilingual ASR models (200).

13. A system (100) comprising: Data processing hardware (810); as well as A memory hardware (820) communicating with the data processing hardware (810), the memory hardware (820) storing instructions that, when executed on the data processing hardware (810), cause the data processing hardware (810) to perform operations, the operations including: Obtain a plurality of training samples (402), each of the plurality of training samples (402) comprising: Audio data (404) representing the corresponding speech utterances (304, 306); and Transcription (420) of the corresponding speech utterances (304, 306); An automatic speech recognition (ASR) model (200) is trained on the plurality of training samples (402), the ASR model (200) comprising a recurrent neural network-transducer RNN-T architecture; The trained ASR model (200) is quantized into an integer target fixed bit width; and The quantized trained ASR model (200) is provided to the user device (10).

14. The system (100) as claimed in claim 13, wherein, Quantizing the trained ASR model (200) includes: using per-channel asymmetric quantization with scale backpropagation to quantize the trained ASR model (200) to the target fixed bit width, the target fixed bit width being equal to two.

15. The system (100) as claimed in claim 13, wherein, Quantizing the trained ASR model (200) includes: using per-channel asymmetric quantization with scale backpropagation, amplitude limiting and sub-channel splitting to quantize the trained ASR model (200) to the target fixed bit width, the target fixed bit width being equal to two.

16. The system (100) as claimed in claim 13, wherein, Quantizing the trained ASR model (200) includes: using absolute maximum binarization to quantize the trained ASR model (200) to the target fixed bit width, the target fixed bit width being equal to one.

17. The system (100) as claimed in claim 13, wherein, Quantization of the trained ASR model (200) includes: Subtract the per-channel average from the multiple weights of the trained ASR model (200) to provide multiple scaled weights; and The multiple expanded and shrunk weights are binarized into the target fixed bit width, which is equal to one.

18. The system (100) as claimed in claim 13, wherein, Quantization of the trained ASR model (200) includes: The number of channels in the multiple input weights of the trained ASR model (200) is increased by splitting the multiple input weights into sub-channels; The average value of each sub-channel is subtracted from the multiple input weights of the trained ASR model (200) to provide multiple scaled weights; Binarize the multiple expanded and shrunk weights; Using pseudo-quantization-aware training, the binarized expanded weights are dequantized by multiplying the binarized expanded weights by a scaling factor to provide dequantized weights. The dequantized weights are reshaped to the same shape as the plurality of input weights; and The input activations applied to the reshaped dequantized weights are used to compute einsum to quantize the trained ASR model (200) to the target fixed bit width, which is equal to one.

19. The system (100) as claimed in any one of claims 13 to 18, wherein, Obtaining the plurality of training samples (402) includes: Receive training data, the training data including: A corpus of transcribed non-synthetic speech utterances (304), each transcribed non-synthetic speech utterance (304) paired with a corresponding transcription (320); and A corpus of untranscribed non-synthetic speech utterances (306), where each untranscribed non-synthetic speech utterance (306) is not paired with a corresponding transcription; A teacher model (301) is trained on the corpus of transcribed non-synthetic speech utterances to teach the teacher model (301) how to predict the corresponding transcription (320) from the non-synthetic speech utterances; and The trained teacher model (301) is used to process the corpus of transcribed non-synthetic speech utterances (304) and the corpus of untranscribed non-synthetic speech utterances (306) to predict the corresponding pseudo-real value labels (420).

20. The system (100) as claimed in any one of claims 13 to 19, wherein, The ASR model (200) includes an audio encoder (210) and a decoder (250), the decoder (250) including a prediction network (220) and a joint network (230).

21. The system (100) as claimed in claim 20, wherein, Quantizing the ASR model (200) includes quantizing the audio encoder (210) but not quantizing the decoder (250).

22. The system (100) as claimed in claim 20 or 21, wherein, The audio encoder (210) includes multiple multi-head self-attention layers, each of which includes a multi-head attention mechanism.

23. The system (100) as claimed in claim 22, wherein, The multiple multi-head self-attention layers include a conformer layer or a transformer layer.

24. The system (100) as claimed in any one of claims 13 to 23, wherein: The spoken utterances (304, 306) and the transcriptions (420) of the multiple training samples (402) span multiple different languages; and The trained ASR models (200) include multilingual ASR models (200).

Images