4-bit Conformer with Accurate Quantization Training for Speech Recognition

Abstract

A method (500) for training a model includes obtaining a plurality of training samples (152), each of which comprises a respective speech utterance (152) and a respective text utterance (154) representing a transcription of the respective speech utterance. An automatic speech recognition (ASR) model (200) is trained on the plurality of training samples using quantization-aware training with native integer arithmetic. The trained automatic speech recognition ASR model is quantized to an integer target fixed bit-width (162). The quantized trained automatic speech recognition ASR model comprises a plurality of weights (202). Each weight of the plurality of weights comprises an integer having a target fixed bit-width. The quantized trained automatic speech recognition ASR model is provided to a user device (10).

Description

[Technical field]

[0001] This disclosure relates to accurate quantization training for speech recognition. [Background technology]

[0002] Modern automatic speech recognition (ASR) systems focus on providing not only good quality (e.g., low word error rate (WER)) but also low latency (e.g., short delay between when a user speaks and when a transcription appears). Furthermore, when using automatic speech recognition ASR systems today, there is a demand for the automatic speech recognition ASR system to decode utterances in real time or in a streaming format that supports faster than real time. To explain, when an automatic speech recognition ASR system is deployed on a mobile phone that experiences direct user interactivity, a mobile phone application that uses the automatic speech recognition ASR system may require speech recognition to be streamed so that words are displayed on the screen as soon as they are spoken. Here, mobile phone users may have a low tolerance for latency. Due to this low tolerance, speech recognition is intended to be performed on mobile devices in a manner that minimizes the impact of latency and inaccuracies that may adversely affect the user's experience. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2021 / 258752 Summary of the Invention [Problem to be solved by the invention]

[0004] However, mobile phones often have limited resources, which in turn limits the size of the automatic speech recognition (ASR) models. [Means for solving the problem]

[0005] One aspect of the present disclosure provides a method for training an automatic speech recognition (ASR) model. The computer-implemented method, when executed on data processing hardware, causes the data processing hardware to perform operations. The operations include obtaining a plurality of training samples. Each training sample of the plurality of training samples comprises a respective speech utterance and a respective text utterance representing a transcription of the respective speech utterance. The method includes training an automatic speech recognition ASR model on the plurality of training samples using quantization-aware training with native integer arithmetic. The method also includes quantizing the trained automatic speech recognition ASR model to a target fixed bit-width of integers. The quantized trained automatic speech recognition ASR model comprises a plurality of weights. Each weight of the plurality of weights comprises an integer having a target fixed bit-width. The method includes providing the quantized trained automatic speech recognition ASR model to a user device.

[0006] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, the target fixed bit-width is "4". In some examples, the automatic speech recognition ASR model further comprises a plurality of activation functions. Each activation function of the plurality of activation functions may comprise an integer having a target fixed bit-width. In other examples, the automatic speech recognition ASR model further comprises a plurality of activation functions, each activation function of the plurality of activation functions comprises an integer having a fixed bit-width greater than the target fixed bit-width. In yet other examples, the automatic speech recognition ASR model further comprises a plurality of activation functions, each activation function of the plurality of activation functions comprises a floating point value.

[0007] Optionally, quantizing the trained automatic speech recognition ASR model comprises determining a scale factor based on an estimated maximum value of the axis to be quantized and a target fixed bit width. In some embodiments, the automatic speech recognition ASR model comprises one or more multi-head attention layers. In some of these embodiments, the one or more multi-head attention layers comprise one or more conformer layers or one or more transformer layers. The automatic speech recognition ASR model may include multiple encoders and multiple decoders, and quantizing the automatic speech recognition ASR model may include quantizing the multiple encoders and not quantizing the multiple decoders. In some examples, the automatic speech recognition ASR model comprises an audio encoder, and the audio encoder comprises a cascaded encoder having a first causal encoder and a second non-causal encoder.

[0008] Another aspect of the present disclosure provides a system for training an automatic speech recognition ASR model. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed by the data processing hardware, cause the data processing hardware to perform operations. The operations include obtaining a plurality of training samples. Each training sample of the plurality of training samples includes a respective speech utterance and a respective text utterance representing a transcription of the respective speech utterance. The method includes training an automatic speech recognition ASR model on the plurality of training samples using quantization-aware training with native integer arithmetic. The method also includes quantizing the trained automatic speech recognition ASR model to a target fixed bit-width of an integer. The quantized trained automatic speech recognition ASR model includes a plurality of weights. Each weight of the plurality of weights includes an integer having a target fixed bit-width. The method includes providing a quantized, trained automatic speech recognition (ASR) model to a user device.

[0009] This aspect may include one or more of the following optional features: In some implementations, the target fixed bit-width is "4". In some examples, the automatic speech recognition ASR model may further comprise a plurality of activation functions, where each activation function of the plurality of activation functions comprises an integer having a target fixed bit-width. In other examples, the automatic speech recognition ASR model further comprises a plurality of activation functions, where each activation function of the plurality of activation functions comprises an integer having a fixed bit-width greater than the target fixed bit-width. In yet other examples, the automatic speech recognition ASR model further comprises a plurality of activation functions, where each activation function of the plurality of activation functions comprises a floating-point value.

[0010] Optionally, quantizing the trained automatic speech recognition ASR model comprises determining a scale factor based on an estimated maximum value of the axis to be quantized and a target fixed bit width. In some embodiments, the automatic speech recognition ASR model comprises one or more multi-head attention layers. In some of these embodiments, the one or more multi-head attention layers comprise one or more conformer layers or one or more transformer layers. The automatic speech recognition ASR model may include multiple encoders and multiple decoders, and quantizing the automatic speech recognition ASR model may include quantizing the multiple encoders and not quantizing the multiple decoders. In some examples, the automatic speech recognition ASR model comprises an audio encoder, and the audio encoder comprises a cascaded encoder having a first causal encoder and a second non-causal encoder.

[0011] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0012] [Figure 1A] 1 is a schematic diagram of an exemplary system for performing speech recognition. [Figure 1B] FIG. 1B is a schematic diagram of components for training an automatic speech recognition (ASR) model of the system of FIG. 1A. [Diagram 2] FIG. 1B is a schematic diagram of an exemplary recurrent neural network-transducer (RNN-T) model of the system of FIG. 1A. [Diagram 3] FIG. 3 is a schematic diagram of an exemplary prediction network of the RNN-T model of FIG. [Figure 4] FIG. 1 is a schematic diagram of a quantization algorithm for quantization aware training with native integer arithmetic. [Diagram 5]1 is a flowchart of an exemplary arrangement of operations for a method of training an automatic speech recognition (ASR) model. [Figure 6] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Like reference symbols in the various drawings indicate like elements. With the rapid growth of voice search and speech interactive features, automatic speech recognition (ASR) has become an essential component for user interactive services and devices (e.g., search by voice in search engines and smartphones). Modern automatic speech recognition (ASR) applications are often developed based on end-to-end models, which have been shown to achieve significant improvements in recognition performance with much smaller model sizes compared to traditional hybrid systems. Improving latency and model size without compromising recognition quality has been actively pursued to benefit live automatic speech recognition (ASR) applications in both server-side and on-device models.

[0014] Quantization is a technique to reduce the computational and memory costs of automatic speech recognition (ASR) models by representing weights and / or activation functions in lower precision data types (e.g., 1 and 8-bit integers) instead of traditional 32-bit floating-point values. Among the latest model quantization methods, "post-training quantization" (PTQ) using 8-bit integers (int8) is a popular and easy-to-use technique that has been successfully applied in many applications. However, one drawback of such techniques is the risk of performance degradation due to reduced precision. Another limitation of post-training quantization PTQ is the lack of control over the quantization of the model. For example, post-training quantization PTQ may not support 4-bit integer (int4) quantization or customized quantization of a selected set of layers.

[0015] The embodiments herein include a model trainer that trains an automatic speech recognition (ASR) model by using native "quantization aware training" (QAT) with native integer arithmetic. In contrast to some methods that use "fake" quantization aware training QAT (i.e., using floating point arithmetic and then converting floating point to integer using conversion), the native quantization aware training QAT uses native integer arithmetic to perform quantization operations (e.g., matrix multiplication) and produces a model that does not have any difference in accuracy during training and inference. That is, "fake quantization" may have numerical differences between training mode (i.e., with floating point arithmetic) and inference mode (i.e., with integer arithmetic) when floating point operations do not fit into the mantissa bits during training.

[0016] The embodiments herein include a model trainer that trains an automatic speech recognition ASR model by using a native quantization-aware training QAT. This approach ensures that "what you train is what you get"; that is, with native integer arithmetic, there is no numerical difference between training forward propagation and inference. Thus, the trained model may be run on multiple applications, such as the cloud (e.g., on a tensor processing unit (TPU)) or both models applications with the same performance. The model trainer minimizes the number of operations used for quantization, thus reducing training time compared to conventional techniques. The automatic speech recognition ASR model may include one or more multi-head attention layers, such as one or more conformer layers and / or one or more transformer layers.

[0017] 1A is an example of a system 100 operating in a speech environment 101. In the speech environment 101, a way for a user 104 to interact with a computing device, such as a user device 10, may be through voice input. The user device 10 (also commonly referred to as device 10) is configured to capture sounds (e.g., streaming audio data) from one or more users 104 in the speech environment 100. Here, the streaming audio data may refer to utterances 106 spoken by the user 104 that function as audible queries, commands to the device 10, or audible communications captured by the device 10. The speech-enabled system of the device 10 may address the queries or commands by answering the queries and / or executing the commands for one or more downstream applications.

[0018] The user device 10 may correspond to any computing device associated with a user 104 and enabled to receive audio data. Some examples of the user device 10 include, but are not limited to, a mobile device (e.g., a mobile phone, a tablet, a laptop, etc.), a computer, a wearable device (e.g., a smart watch), a smart appliance, an Internet of Things (IoT) device, an in-vehicle infotainment system, a smart display, a smart speaker, etc. The user device 10 includes data processing hardware 12 and memory hardware 14 in communication with the data processing hardware 12, storing instructions that, when executed by the data processing hardware 12, cause the data processing hardware 12 to perform one or more operations. The user device 10 further includes an audio system 16 including audio capture devices (e.g., microphones) 16, 16a for capturing and converting speech 106 spoken within the audio environment 100 into electrical signals, and speech output devices (e.g., speakers) 16, 16b for communicating audible audio signals (e.g., as output audio data from the device 10). In the illustrated example, the user device 10 implements a single audio capture device 16a, however, the user device 10 may implement an array of audio capture devices 16a without departing from the scope of this disclosure, such that one or more capture devices 16a in the array may communicate to the audio system 16 without being physically present at the user device 10.

[0019] In the speech environment 100, an automatic speech recognition (ASR) system 118 comprises a model 200 (such as a recurrent neural network-transducer (RNN-T) model or other conformal transducer model / multi-path model) residing on a user device 10 of a user 104 and / or on a remote computing device 60 (e.g., one or more remote servers of a distributed system executing within a cloud computing environment) that communicates to the user device 10 via a network 40. The remote computing device comprises data processing hardware 62 and memory hardware 64. The user device 10 and / or the remote computing device 60 also comprises an audio subsystem 108 configured to receive an utterance 106 spoken by the user 104 and captured by the audio capture device 16a as well as to convert the utterance 106 into a corresponding digital format associated with input acoustic frames 110 that can be processed by the automatic speech recognition ASR system 118. In the illustrated example, a user speaks each utterance 106, and an audio subsystem 108 converts the utterance 106 into corresponding audio data (e.g., acoustic frames) 110 for input to an automatic speech recognition ASR system 118. A model 200 then receives as input the audio frames 110 (i.e., audio data) corresponding to the utterance 106 and generates / predicts as output a corresponding transcription 120 (e.g., speech recognition result / hypothesis) of the utterance 106.

[0020] The user device 10 and / or the remote computing device 60 also execute a user interface generator 107 configured to present a representation of a transcription 120 of the utterance 106 to the user 104 of the user device 10. As described in further detail below, the user interface generator 107 may display the speech recognition results 120 in a streaming format. In some configurations, the transcription 120 output from the automatic speech recognition ASR system 118 is processed, for example by a natural language understanding (NLU) module executing on the user device 10 or the remote computing device 60, to execute the user command / query specified by the utterance 106. Additionally or alternatively, a text-to-speech system (not shown) (e.g., executing on any combination of the user device 10 or the remote computing device 60) is enabled to convert the transcription into synthetic speech for audible output by the user device 10 and / or other devices.

[0021] In the illustrated example, a user 104 interacts with a program or application 50 (e.g., a digital assistant application 50) of a user device 10 that uses an automatic speech recognition ASR system 118. For example, FIG. 1A depicts a user 104 communicating with a digital assistant application 50 and the digital assistant application 50 displaying a digital assistant interface 18 on the screen of the user device 10 to show a conversation between the user 104 and the digital assistant application 50. In this example, the user 104 asks the digital assistant application 50, "What time is the concert tonight?" This question from the user 104 is a spoken utterance 106 that is captured by an audio capture device 16a and processed by an audio system 16 of the user device 10. In this example, the audio system 16 receives the spoken utterance 106 and converts it into acoustic frames 110 for input to the automatic speech recognition ASR system 118.

[0022] 1B, the remote computing device 60 trains the model 200 of FIG. 1A by executing the model trainer 150. The model trainer 150 obtains (e.g., from the memory hardware 64) a plurality of training samples 152, 152a-152n. Each training sample 152 comprises a spoken training utterance 154 (i.e., a sequence of input audio features) and a corresponding text utterance 156 representing a transcription 156 of the utterance 154. The model trainer 150 trains the model 200 on the plurality of training samples 152 by using quantization aware training with native integer arithmetic (QAT). As discussed in more detail below, the model trainer 150 uses quantization aware training by determining a scale factor 160 during training. The model trainer 150 quantizes the model 200 to an integer fixed bit width 162 during or after training. The integer fixed bit width 162 represents the number of bits allocated to each native integer operation. For example, if the integer fixed bit width 162 is "8", the model 200 is quantized to an 8-bit integer (i.e., int8). In another example, if the integer fixed bit width 162 is "4", the model 200 is quantized to a 4-bit integer (i.e., int4). Other examples, such as 6-bit integers, are possible. The integer fixed bit width 162 may be configurable by the user and may depend on the use case of the model.

[0023] Traditional quantization-aware techniques rely on training a "fake" quantization-aware QAT. For example, many popular systems use tf.quantization.fake_quant_ *The model is quantized using tf operations. These operations are used during server-side inference, but for on-device models (e.g., user devices such as smartphones), the fake quantization operations must be converted to integer operations using a conversion operation (e.g., TFlite). Therefore, because these conventional techniques require this additional conversion step to convert the fake quantization operations to real integer operations, existing application programming interfaces (APIs) only support estimation of minimum and maximum values ​​per channel for the last dimension. This is not ideal, such as in some use cases where the channel dimension is not the last dimension. In these cases, conventional techniques permutate the dimensions of the input tensor to make it the last dimension, and then the tensor must be "fake" quantized using the application programming interface API. Finally, the dimensions must be permuted back to the original order of the input tensor. These additional permutation operations increase training time. In contrast, the model trainer 150 uses native integer operations (e.g., tf operations). As a result, model 200 can be used for training and inference in both mobile and tensor processing unit (TPU) applications, and training can be further accelerated by using hardware-supported integer arithmetic (e.g., matrix multiplication).

[0024] During quantization, the model trainer 150 reduces the size of the model 200 by adjusting the size of one or more weights 202 and / or activation functions 204 of the model 200. Traditionally, both the weights 202 and activation functions 204 of an automatic speech recognition ASR model are represented by float32 values ​​that occupy 32-bit space and often require complex calculations to process. In "post-training quantization" PTQ, these float32 values ​​may be "clipped" or rounded to reduce precision and memory requirements. In contrast, the model trainer 150 uses native integer arithmetic to represent the weights 202 of the model 200 by using integers whose size is determined by a fixed bit width 162. In some examples, the model trainer 150 quantizes the weights 202 and activation functions 204 by the same fixed bit width 162 (e.g., 4 bits or 8 bits). In other examples, the model trainer 150 quantizes the weights 202 and activation functions by using different fixed bit widths 162. For example, the model trainer 150 quantizes the activation functions 204 using a fixed bit-width 162 that is larger than the fixed bit-width 162 of the weights 202 (e.g., 4 bits for the weights 202 and 8 bits for the activation functions 204). In yet another example, the model trainer 150 quantizes the weights 202 but does not quantize the activation functions 204 (e.g., the activation functions 204 are represented by floating-point values ​​such as float32).

[0025] The model trainer 150 may quantize only a portion of the model 200. In some examples, when the model 200 includes multiple encoders and multiple decoders, the model trainer 150 quantizes only the encoders while not quantizing the decoders, since the memory requirements of the decoders are minimal in some scenarios. After training and quantization, the model trainer 150 may provide the quantized trained model 200 to the user device 10.

[0026] Referring now to FIG. 2, the exemplary model 200, 200a comprises a recurrent neural network-transducer (RNN-T) model architecture that adheres to the latency constraints associated with interactive applications. The use of the RNN-T model architecture is exemplary, and the model 200 may include other architectures, such as transformer-transducer and conformer-transducer model architectures, among others. The RNN-T model 200a provides a small computational footprint and utilizes less memory requirements than conventional automatic speech recognition ASR architectures, making the RNN-T model architecture suitable for performing speech recognition entirely on the user device 10 (e.g., no communication with a remote server is required). In this example, the RNN-T model 200a comprises an encoder network 210, a prediction network 300, and a joint network 230. The encoder network 210, which is roughly similar to an acoustic model (AM) in a conventional automatic speech recognition (ASR) system, comprises a stack of self-attention layers (e.g., conformer or transformer layers) or a recurrent network of stacked long short-term memory (LSTM) layers. For example, the encoder may receive a sequence of d-dimensional feature vectors (e.g., acoustic frames) x = (x 1 ,x 2 ,···,x T ), in the formula

[0027]

number

[0028] At each output step, a high-level feature representation is generated.

[0029]

number

[0030] It is expressed as: Similarly, the prediction network 300 may also be an LSTM network, which, like a language model (LM), computes the sequence of non-empty symbols y output by the final softmax layer 240 so far. 0 , , y ui-1 , a dense representation

[0031]

number

[0032] Finally, using the RNN-T model architecture, the representations generated by the encoder and predictor / decoder networks 210, 300 are combined by the joint network 230. The predictor network 300 can be replaced by an embedding lookup table to improve latency by outputting a sparse embedding lookup instead of processing a dense representation. The joint network then calculates the distribution of the next output symbol,

[0033]

number

[0034] In other words, the joint network 230 generates a probability distribution of possible speech recognition hypotheses at each output step (e.g., time step), where a "possible speech recognition hypothesis" corresponds to a set of output labels, each of which represents a symbol / character of a specified natural language. For example, when the natural language is English, the set of output labels may include twenty-seven (27) symbols. For example, there is one label for each of the 26 letters of the English alphabet and one label representing a space. Thus, the joint network 230 may output a set of values ​​indicating the likelihood of occurrence of each of the given set of output labels. This set of values ​​may be a vector, and thus may indicate a probability distribution over the set of output labels. In some cases, the output labels are graphemes (individual characters, potential punctuation or other symbols, etc.), but the set of output labels is not limited to such. For example, the set of output labels may comprise parts of words and / or entire words in addition to or instead of graphemes. The output distribution of the joint network 230 may include a posterior probability value for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, the output y of the joint network 230 may be i may have 100 different probability values, one for each output label. The probability distributions may then be used to select candidate orthographic elements (e.g., graphemes, word fragments, and / or words) in a beam search process (e.g., by a softmax layer 240) and assign scores to determine the transcription 120.

[0035] The softmax layer 240 may use any technique to select the output label / symbol that has the highest probability in the distribution as the next output symbol predicted by the RNN-T model 200 at the corresponding output step. In this way, the RNN-T model 200 does not make a conditional independence assumption, but rather the prediction of each symbol is conditioned not only on the acoustics but also on the sequence of labels output so far. The RNN-T model 200 assumes that the output symbols are independent of future acoustic frames 110, allowing the RNN-T model to be used in a streaming manner.

[0036] In some examples, the encoder network (i.e., audio encoder) 210 of the RNN-T model 200 includes a stack of multi-head attention or self-attention layers / blocks, such as one or more conformer blocks / layers and / or one or more transformer blocks / layers. Optionally, the encoder 210 (i.e., audio encoder) includes a first-pass causal encoder and a second-pass acausal encoder for a multi-pass architecture. This multi-pass model integrates streaming and non-streaming automatic speech recognition (ASR). The causal encoder uses only the left context to generate partial results with minimal latency. The acausal encoder uses both the left and right contexts to provide more accurate hypotheses. In this example, each conformer block includes a series of multi-head self-attention, deep unit convolution, and feedforward layers. The prediction network 300 may have two 2048-dimensional LSTM layers, each followed by a 640-dimensional projection layer. Alternatively, the prediction network 300 may include a stack of transformer or conformer blocks, or an embedded lookup table, instead of the LSTM layers. Finally, the joint network 230 may also have 640 hidden units. The softmax layer 240 may be configured with a unified word chunk or grapheme set that is generated by using all unique word chunks or graphemes in the multiple training datasets.

[0037] FIG. 3 illustrates a sequence of non-empty symbols y , restricted to the N previous non-empty symbols 301a-301n output by the final softmax layer 240. ui-n , , y ui-11 illustrates an example prediction network 300 of the RNN-T model 200a receiving as input N. In some examples, N is equal to 2. In other examples, N is equal to 5, but the disclosure is non-limiting and N may be equal to any integer. The sequence of non-blank symbols 301a-301n represents the initial speech recognition result 120a (FIG. 1). In some implementations, the prediction network 300 includes a multi-head attention mechanism 302 that shares a shared embedding matrix 304 across each head 302A-302H of the multi-head attention mechanism. In one example, the multi-head attention mechanism 302 includes four heads. However, any number of heads can be used in the multi-head attention mechanism 302. Notably, the multi-head attention mechanism significantly improves performance while minimizing the increase in model size. As explained in more detail below, each head 302A-302H has its own row of position vectors 308, and rather than increasing the model size by concatenating the outputs 318A-318H from all heads, the outputs 318A-318H are instead averaged by a head averaging module 322.

[0038] Referring to the first head 302A of the multi-head attention mechanism 302, the first head 302A uses a shared embedding matrix 304 to estimate the sequence of non-empty symbols y ui-n , , y ui-1 For each non-whitespace symbol 301 in

[0039]

number

[0040] In particular, the shared embedding matrix 304 is shared among all heads of the multi-head attention mechanism 302, so that all other heads 302B-302H generate the same corresponding embedding 306 for each non-whitespace symbol. The first head 302A also generates a sequence of non-whitespace symbols y ui-n , , y ui-1 For each corresponding non-blank symbol in Aa~An 308, 308Aa-308An (e.g.

[0041]

number

[0042] Each position vector PV 308 assigned to each non-blank symbol indicates a position in the history of the sequence of non-blank symbols (e.g., the N previous non-blank symbols output by the final softmax layer 240). For example, the first position vector PV Aa is assigned to the most recent position in the history, while the last position vector PV An is assigned to the last position in the history of the N previous non-empty symbols output by the final softmax layer 240. In particular, each of the embeddings 306 may include the same dimensionality (i.e., dimension size) as each of the position vectors PV 308.

[0043] A sequence of non-blank symbols 301a through 301n ui-n , , y ui-1 The corresponding embedding generated by the shared embedding matrix 304 for each non-blank symbol 301 in is the same for all heads 302A-302H of the multi-head attention mechanism 302, but each head 302A-302H defines a different set / row of position vectors 308. For example, the first head 302A defines a position vector PV Aa~An The second head 302B defines a position vector PV Ba~Bn , and the H-th head 302H defines a position vector PV Ha~HnThe other different rows 308Ha to 308Hn are defined.

[0044] For each non-blank symbol in the sequence of received non-blank symbols 301a-301n, the first head 302A, via the weight layer 310, weights the corresponding embedding 306 in proportion to the similarity between the corresponding embedding and the respective position vector PV 308 assigned thereto. In some examples, the similarity may include a cosine similarity (e.g., cosine distance). In the example shown, the weight layer 310 outputs a sequence of weighted embeddings 312, 312Aa-312An, each weighted embedding associated with a corresponding embedding 306 that is weighted in proportion to the respective position vector PV 308 assigned thereto. Stated another way, the weighted embedding 312 output by the weight layer 310 for each embedding 306 may correspond to a dot product between the embedding 306 and the respective position vector PV 308. The weighted embedding 312 may be interpreted as being added to the embedding in proportion to the extent to which the embedding resembles the positioning associated with each position vector PV 308. To speed up computations, the prediction network 300 includes a non-recurrent layer, so that the sequence of weighted embeddings 312Aa-312An are not concatenated, but are instead averaged by a weighted average module 316. Thus, as output from the first head 302A, a weighted average 318A of the weighted embeddings 312Aa-312An is generated, which is given by:

[0045]

number

[0046] In Equation 1, h represents the index of the head 302, n represents the position in the context, and e represents the embedding dimension. e, with the size of the corresponding dimension. The position vector PV 308 does not need to be trainable and may contain random values. In particular, even if the weighted embedding 312 is averaged, the position vector PV 308 can potentially preserve position history information, alleviating the need to provide recurrent connections at each layer of the prediction network 300.

[0047] The operations described above with respect to the first head 302A are similarly performed for each of the other heads 302B-302H of the multi-head attention mechanism 302. Due to the different set of positioned vectors PV 308 defined by each head 302, the weight layer 310 outputs a sequence of weighted embeddings 312Ba-312Bn, 312Ha-312Hn in each of the other heads 302B-302H that is different from the sequence of weighted embeddings 312Aa-312Aa in the first head 302A. The weighted average module 316 then generates a weighted average 318B-318H of each of the corresponding weighted embeddings 312 of the sequence of non-blank symbols as output from each of the other corresponding heads 302B-302H.

[0048] In the illustrated example, the prediction network 300 includes a head averaging module 322 that averages the weighted averages 318A-H output from corresponding heads 302A-H. A projection layer 326 with SWISH may receive as input an output 324 from the head averaging module 322 corresponding to the average of the weighted averages 318A-H and generate as output a projection output 328. A final layer normalization 330 normalizes the projection output 328 to reduce the number of time steps to a single embedding vector P at the corresponding time step. ui The prediction network 300 is enabled to provide a single embedding vector P ui Generates only 350.

[0049] In some configurations, the prediction network 300 does not implement a multi-head attention mechanism 302, and performs the operations described above only for the first head 302A. In these configurations, the weighted average 318A of the weighted embeddings 312Aa-312An is simply passed through the projection layer 326 and layer normalization 330 to produce a single embedding vector P ui 350 is offered.

[0050] In some implementations, to further reduce the size of the RNN-T decoder, i.e., the prediction network 300 and the joint network 230, parameter tying between the prediction network 300 and the joint network 230 is applied. Specifically, the vocabulary size |V| and the embedding dimension d e In this case, the shared embedding matrix 304 in the prediction network is

[0051]

number

[0052] Meanwhile, the last hidden layer has a dimension size of d h Therefore, the weights of the feedforward projection from the hidden layer to the output logic are

[0053]

number

[0054] Therefore, the feedforward layer corresponding to the last layer of the joint network 230 has a weight matrix [d h , |V]|. The prediction network 300 has an embedding dimension d e The size of the last hidden layer of the joint network 230 is the dimension d hBy combining the embedding dimension d, the feedforward projection weights of the joint network 230 and the shared embedding matrix 304 of the prediction network 300 can share their weights for all non-blank symbols with a simple transpose transformation. Because the two matrices share all their values, the RNN-T decoder only needs to store their values ​​once in memory instead of storing two individual matrices. e The size of the hidden layer is d h By setting the embedding dimension d equal to the size of e We reduce the number of parameters equal to the product of |V| and the vocabulary size |V|. This weight tying corresponds to a regularization technique.

[0055] Referring now to FIG. 4, algorithm 400 illustrates native quantization of 8-bit integers (int8) in TensorFlow. Using algorithm 400, model trainer 150 determines scale factor 160 during training by first estimating the maximum value for the axis being quantized (because algorithm 400 supports per-channel quantization). Model trainer 150 then determines scale factor 160 by dividing the maximum value by integer representation value 410 (i.e., 127.0 in this example). Integer representation value 410 is based on desired integer fixed bit-width 162. That is, scale factor 160 is based on the estimated maximum value for the axis being quantized and target fixed bit-width 162. For example, for 8-bit quantization (i.e., integer fixed bit-width of 8), integer representation value 410 is 127.0. As another example, for 4-bit quantization (i.e., integer fixed bit-width of 4), integer representation value is 4.0. After determining the scale factor 160, the model trainer 150 quantizes the input tensor by dividing it by the scale factor 160 and casting it to an integer. Inverse quantization (dequantization) can be used by multiplying the tensor by the scale factor 160.

[0056] FIG. 5 is a flow chart of an exemplary arrangement of operations of a method 500 for training an automatic speech recognition ASR model 200. The method 500 includes, at operation 502, obtaining a plurality of training samples 152. Each of the plurality of training samples 152 includes a respective speech utterance 154 and a respective text utterance 156 representing a transcription of the respective speech utterance 154. The method 500 includes, at operation 504, training an automatic speech recognition (ASR) model 200 on the plurality of training samples 152 by using "quantization aware training" (QAT) with native integer arithmetic. At operation 506, the method 500 includes quantizing the trained automatic speech recognition ASR model 200 to a target fixed bit width 162 of integers. The quantized trained automatic speech recognition ASR model 200 includes a plurality of weights 202. Each weight 202 of the plurality of weights 202 comprises an integer having a target fixed bit-width 162. In operation 508, the method 500 comprises providing the quantized trained automatic speech recognition ASR model 200 to the user device 10.

[0057] 6 is a schematic diagram of an exemplary computing device 600 that can be used to implement the systems and methods described in this document. The computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown, their connections and relationships, and their functions are for illustrative purposes only and are not intended to limit the implementation of the invention described and / or claimed in this document.

[0058] Computing device 600 includes a processor 610, a memory 620, a storage device 630, a high-speed interface / controller 640 that connects to memory 620 and a high-speed expansion port 650, and a low-speed interface / controller 660 that connects to a low-speed bus 670 and storage device 630. Each of the components (610, 620, 630, 640, 650, and 660) are interconnected using various buses and may be mounted on a common motherboard or may exist otherwise as desired. Processor 610 is enabled to process instructions for execution within computing device 600, including instructions stored in memory 620 or storage device 630, and to display graphical information of a graphical user interface (GUI) on an external input / output device, such as a display 680 coupled to high-speed interface 640. In other implementations, multiple processors and / or multiple buses may be used as desired, along with multiple memories and types of memories. Also, multiple computing devices 600 may be connected, each performing some of the required operations (eg, as a bank of servers, a group of blade servers, or a multi-processor system).

[0059] The memory 620 stores non-transient information within the computing device 600. The memory 620 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-transient memory 620 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0060] The storage device 630 can provide mass storage for the computing device 600. In some embodiments, the storage device 630 is a computer-readable medium. In various different implementations, the storage device 630 can be a device array, including a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or a storage area network or other configuration of devices. In additional embodiments, the computer program product is tangibly embodied in an information carrier. The computer program product comprises instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable medium or a machine-readable medium, such as the memory 620, the storage device 630, or a memory on the processor 610.

[0061] The high-speed controller 640 manages more of the bandwidth-intensive operations of the computing device 600, and the low-speed controller 660 manages less of the bandwidth-intensive operations. Such an allocation of roles is merely an example. In some implementations, the high-speed controller 640 is coupled to the memory 620, the display 680 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 650 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 660 is coupled to the storage device 630 and the low-speed expansion port 690. The low-speed expansion port 690 may include various communication ports (USB, Bluetooth, Ethernet, wireless Ethernet, etc.), and may be coupled, via a network adapter or the like, to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a network device such as a switch or router.

[0062] The computing device 600 can be implemented in many different forms, as shown in the figure: for example, it can be implemented as a standard server 600a, or multiple times in a group of such servers 600a, as a laptop computer 600b, or as part of a rack server system 600c.

[0063] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may be specialized or general purpose and may include implementations in one or more computer programs executable and / or interpretable by a programmable system having at least one programmable processor, at least one input device, and at least one output device coupled to receive data and instructions from and transmit data and instructions to the storage system.

[0064] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform a task. In some examples, a software application may be referred to as an "application," an "app," or a "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0065] These computer programs (also known as programs, software, software applications, or codes) comprise machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic circuit (PLD)) used to provide machine instructions and / or data to a programmable processor comprising a machine-readable medium that receives the machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0066] The processes and logic flows described herein may be performed by one or more programmable processors, also referred to as data processing hardware, executing one or more computer programs to operate on input data and generate output to perform functions. The processes and logic flows may also be performed by special purpose logic circuitry, such as FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits). Processors suitable for executing computer programs include, for example, both general purpose and special purpose processors, as well as any one or more processors of any type of digital computer. Generally, a processor receives instructions and data from a read-only memory, a random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively coupled to receive data from or transmit data to them, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices, magnetic disks, such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0067] To interact with a user, one or more aspects of the present disclosure can be implemented in a computer having a display device, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, for displaying information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user. For example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, verbal, or tactile input. Additionally, the computer can interact with the user by sending and receiving documents to a device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0068] Although several embodiments have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A method as a computer-implemented method (500), which when executed on data processing hardware (62) causes the data processing hardware (62) to perform operations, the operations including: Obtaining a plurality of training samples (152), each of the training samples (152) comprising: each speech utterance (154); and a respective text utterance (156) representing a transcription of each of said speech utterances (154); obtaining a plurality of said training samples (152); training an automatic speech recognition (ASR) model (200) on the plurality of training samples (152) using quantization-aware training with native integer arithmetic; quantizing a trained automatic speech recognition ASR model (200) to a target fixed bit-width (162) of integers, the quantized trained automatic speech recognition ASR model (200) comprising a plurality of weights (202), each weight (202) of the plurality of weights (202) comprising an integer having the target fixed bit-width (162); providing the quantized trained automatic speech recognition ASR model (200) to a user device (10); The method (500).

2. The target fixed bit width (162) is "4".

2. The method of claim 1 (500).

3. The automatic speech recognition ASR model (200) further comprises a plurality of activation functions (204); each said activation function (204) of said plurality of activation functions (204) comprises an integer having said target fixed bit-width (162); The method (500) of claim 1 or claim 2.

4. The automatic speech recognition ASR model (200) further comprises a plurality of activation functions (204); each said activation function (204) of said plurality of activation functions (204) comprises an integer having a fixed bit-width greater than said target fixed bit-width (162); The method (500) of claim 1 or claim 2.

5. The automatic speech recognition ASR model (200) further comprises a plurality of activation functions (204); each said activation function (204) of the plurality of activation functions (204) comprises a floating point value; The method (500) of claim 1 or claim 2.

6. quantizing the trained automatic speech recognition ASR model (200) comprises determining a scale factor (160) based on an estimated maximum value of an axis to be quantized and the target fixed bit width (162). The method (500) of any one of claims 1 to 5.

7. The automatic speech recognition (ASR) model (200) includes one or more multi-head attention layers (302). The method (500) of any one of claims 1 to 6.

8. The one or more multi-head attention layers (302) comprise one or more conformer layers or one or more transformer layers.

8. The method of claim 7 (500).

9. The automatic speech recognition (ASR) model (200) comprises a plurality of encoders and a plurality of decoders; quantizing the automatic speech recognition ASR model (200) comprises quantizing a number of the encoders and not quantizing a number of the decoders. The method (500) of any one of claims 1 to 8.

10. The automatic speech recognition (ASR) model (200) comprises an audio encoder (210); the audio encoder (210) comprises a cascaded encoder comprising a first causal encoder and a second non-causal encoder; The method (500) of any one of claims 1 to 9.

11. A system (100), comprising: Data processing hardware (62); and memory hardware (64) in communication with said data processing hardware (62); It is equipped with The memory hardware (64) stores instructions that, when executed on the data processing hardware (62), cause the data processing hardware (62) to perform operations, the operations being: Obtaining a plurality of training samples (152), each of the training samples (152) comprising: each speech utterance (154); and a respective text utterance (156) representing a transcription of each of said speech utterances (154); obtaining a plurality of said training samples (152); training an automatic speech recognition (ASR) model (200) on the plurality of training samples (152) using quantization-aware training with native integer arithmetic; quantizing a trained automatic speech recognition ASR model (200) to a target fixed bit-width (162) of integers, the quantized trained automatic speech recognition ASR model (200) comprising a plurality of weights (202), each weight (202) of the plurality of weights (202) comprising an integer having the target fixed bit-width (162); providing the quantized trained automatic speech recognition ASR model (200) to a user device (10); The system (100) comprises:

12. The target fixed bit width (162) is "4". The system (100) of claim 11.

13. The automatic speech recognition ASR model (200) further comprises a plurality of activation functions (204); each said activation function (204) of said plurality of activation functions (204) comprises an integer having said target fixed bit-width (162); The system (100) according to claim 11 or claim 12.

14. The automatic speech recognition ASR model (200) further comprises a plurality of activation functions (204); each said activation function (204) of said plurality of activation functions (204) comprises an integer having a fixed bit-width greater than said target fixed bit-width (162); The system (100) according to claim 11 or claim 12.

15. The automatic speech recognition ASR model (200) further comprises a plurality of activation functions (204); each said activation function (204) of the plurality of activation functions (204) comprises a floating point value; The system (100) according to claim 11 or claim 12.

16. quantizing the trained automatic speech recognition ASR model (200) comprises determining a scale factor (160) based on an estimated maximum value of an axis to be quantized and the target fixed bit width (162). A system (100) according to any one of claims 11 to 15.

17. The automatic speech recognition (ASR) model (200) includes one or more multi-head attention layers (302). A system (100) according to any one of claims 11 to 16.

18. The one or more multi-head attention layers (302) comprise one or more conformer layers or one or more transformer layers.

20. The system (100) of claim 17.

19. The automatic speech recognition (ASR) model (200) comprises a plurality of encoders and a plurality of decoders; quantizing the automatic speech recognition ASR model (200) comprises quantizing a number of the encoders and not quantizing a number of the decoders. A system (100) according to any one of claims 11 to 18.

20. The automatic speech recognition (ASR) model (200) comprises an audio encoder (210); the audio encoder (210) comprises a cascaded encoder comprising a first causal encoder and a second non-causal encoder; A system (100) according to any one of claims 11 to 19.

Images