4-bit Conformer with accurate quantitative training for speech recognition

Abstract

A method (500) for training a model includes obtaining a plurality of training samples (152). Each respective training sample of the plurality of training samples includes a respective speech utterance (152) and a respective textual utterance (154) representing a transcription of the respective speech utterance. The method includes training an automatic speech recognition (ASR) model on a plurality of training samples using quantitative perceptual training with native integer operations (200). The method further includes quantizing the trained ASR model to an integer target fixed bit width (162). The quantized trained ASR model includes a plurality of weights (202). Each of the plurality of weights includes an integer having a target fixed bit width. The method comprises providing the quantized trained ASR model to the user device (10).

Description

Technical Field

[0001] The present disclosure relates to accurate quantization training for speech recognition. Background Art

[0002] Modern automatic speech recognition (ASR) systems focus not only on providing high quality (e.g., low word error rate (WER)), but also on providing low latency (e.g., a short delay between a user speaking and the transcription appearing). In addition, when ASR systems are used today, they are required to decode speech in a streaming manner that corresponds to or is even faster than real time. For example, when an ASR system is deployed on a mobile phone that experiences direct user interaction, an application on the mobile phone using the ASR system may require that speech recognition is streamed so that the words appear on the screen as soon as they are spoken. Here, users of mobile phones may also have a low tolerance for latency. Due to this low tolerance, speech recognition strives to run on mobile devices in a manner that minimizes the impact of latency and inaccuracy that may adversely affect the user's experience. However, mobile phones often have limited resources that limit the size of the ASR model. Summary of the invention

[0003] One aspect of the present disclosure provides a method for training an automatic speech recognition (ASR) model. The computer-implemented method causes the data processing hardware to perform operations when executed on the data processing hardware. The operations include: obtaining a plurality of training samples. Each corresponding training sample in the plurality of training samples includes a corresponding speech utterance and a corresponding text utterance representing a transcription of the corresponding speech utterance. The method includes: training the ASR model on the plurality of training samples using quantization-aware training using native integer operations. The method also includes: quantizing the trained ASR model to an integer target fixed bit width. The quantized trained ASR model includes a plurality of weights. Each weight in the plurality of weights includes an integer having a target fixed bit width. The method includes: providing the quantized trained ASR model to a user device.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the target fixed bit width is four. In some examples, the ASR model further includes a plurality of activations, and each of the plurality of activations may include an integer having a target fixed bit width. In some examples, the ASR model further includes a plurality of activations, and each of the plurality of activations includes an integer having a fixed bit width greater than the target fixed bit width. In yet other examples, the ASR model further includes a plurality of activations, and each of the plurality of activations includes a floating point value.

[0005] Optionally, the quantized trained ASR model includes determining a scaling factor based on an estimated maximum value of the axis to be quantized and a target fixed bit width. In some implementations, the ASR model includes one or more multi-headed attention layers. In some of these implementations, the one or more multi-headed attention layers include one or more conformer layers or one or more transformer layers. The ASR model may include multiple encoders and multiple decoders, and the quantized ASR model may include quantizing multiple encoders without quantizing multiple decoders. In some examples, the ASR model includes an audio encoder, and the audio encoder includes a cascaded encoder, the cascaded encoder including a first causal encoder and a second non-causal encoder.

[0006] Another aspect of the present disclosure provides a system for training an ASR model. The system includes: data processing hardware; and memory hardware that communicates with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include: obtaining multiple training samples. Each corresponding training sample in the multiple training samples includes a corresponding speech utterance and a corresponding text utterance representing a transcription of the corresponding speech utterance. The method includes: training the ASR model on the multiple training samples using quantization-aware training using native integer operations. The method also includes: quantizing the trained ASR model to an integer target fixed bit width. The quantized trained ASR model includes multiple weights. Each weight in the multiple weights includes an integer with a target fixed bit width. The method includes: providing the quantized trained ASR model to a user device.

[0007] This aspect may include one or more of the following optional features. In some implementations, the target fixed bit width is four. In some examples, the ASR model further includes a plurality of activations, and each of the plurality of activations may include an integer having the target fixed bit width. In some examples, the ASR model further includes a plurality of activations, and each of the plurality of activations includes an integer having a fixed bit width greater than the target fixed bit width. In yet other examples, the ASR model further includes a plurality of activations, and each of the plurality of activations includes a floating point value.

[0008] Optionally, the quantized trained ASR model includes determining a scaling factor based on an estimated maximum value of the axis to be quantized and a target fixed bit width. In some implementations, the ASR model includes one or more multi-headed attention layers. In some of these implementations, the one or more multi-headed attention layers include one or more conformer layers or one or more transformer layers. The ASR model may include multiple encoders and multiple decoders, and the quantized ASR model may include quantizing multiple encoders without quantizing multiple decoders. In some examples, the ASR model includes an audio encoder, and the audio encoder includes a cascaded encoder, the cascaded encoder including a first causal encoder and a second non-causal encoder.

[0009] The details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the following description. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1A is a schematic diagram of an example system for performing speech recognition.

[0011] Figure 1B It is used for training Figure 1A A schematic diagram of the components of the automatic speech recognition (ASR) model of the system.

[0012] Figure 2 is used for Figure 1A Figure 1. Schematic diagram of an example recurrent neural network transducer (RNN-T) model of a system.

[0013] Figure 3 yes Figure 2 Schematic diagram of an example prediction network for the RNN-T model.

[0014] Figure 4 is a schematic diagram of the quantization algorithm used for quantization-aware training with native integer arithmetic.

[0015] Figure 5 is a flow diagram of an example arrangement of operations of a method for training an automatic speech recognition (ASR) model.

[0016] Figure 6 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0017] Like reference numbers in the various drawings indicate like elements. DETAILED DESCRIPTION

[0018] With the rapid development of voice search and speech interaction features, automatic speech recognition (ASR) has become an essential component for user interactive services and devices (e.g., search by voice function in search engines and smart phones). Modern ASR applications are often developed based on end-to-end models, which have been shown to achieve significant recognition performance improvements compared to conventional hybrid systems with much smaller model sizes. Improving latency and model size without compromising recognition quality has become an active pursuit for benefiting real-time ASR applications with both server-side models and on-device models.

[0019] Quantization is a technique to reduce the computational and memory cost of ASR models by representing weights and / or activations with lower precision data types (e.g., 8-bit integers) instead of conventional 32-bit floating point values. Among modern model quantization methods, post-training quantization (PTQ) with 8-bit integers (int8) is a popular and easy-to-use technique that has been successfully applied in many applications. However, one of the disadvantages of this technique is the potential performance degradation due to the loss of precision. Another limitation of PTQ is the lack of control over model quantization. For example, PTQ may not support 4-bit integer (int4) quantization or customized quantization of a selected set of layers.

[0020] Implementations herein include a model trainer that trains an ASR model using native quantization-aware training (QAT) that utilizes native integer operations. Compared to some approaches that use "pseudo" QAT (i.e., use floating point operations and later use conversion to convert floating point numbers to integers), native QAT performs quantized operations (e.g., matrix multiplication) using native integer operations, which generates a model that does not have any accuracy difference during training and inference. That is, when floating point operations cannot fit into the bits of the mantissa during training, "pseudo quantization" may have numerical differences between training (i.e., utilizing floating point operations) mode and inference (i.e., utilizing integer operations) mode.

[0021] The implementation of this article includes a model trainer that trains an ASR model using a native QAT. This approach ensures that "what you train is what you provide". That is, with native integer operations, there is no numerical difference between the forward propagation of training and inference. Therefore, the trained model can be run in multiple applications, such as in the cloud (e.g., on a tensor processing unit (TPU)) or in both model applications with the same performance. Compared with traditional techniques, the model trainer minimizes the number of operations used for quantization to reduce training time. The ASR model may include one or more multi-head attention layers, such as one or more conformer layers or one or more transformer layers.

[0022] Figure 1A 101 is an example of a system 100 operating in a speech environment 101. In the speech environment 101, the way in which a user 104 interacts with a computing device such as a user device 10 may be through voice input. The user device 10 (also generally referred to as a device 10) is configured to capture sound (e.g., streaming audio data) from one or more users 104 within the speech environment 100. Here, the streaming audio data may refer to an utterance 106 spoken by the user 104, which serves as an audible query, a command for the device 10, or an audible communication captured by the device 10. The voice-enabled system of the device 10 may process the query or command by answering the query and / or causing the command to be executed / fulfilled by one or more downstream applications.

[0023] The user device 10 may correspond to any computing device associated with the user 104 and capable of receiving audio data. Some examples of the user device 10 include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smart watches), smart home appliances, Internet of Things (IoT) devices, in-vehicle infotainment systems, smart displays, smart speakers, etc. The user device 10 includes data processing hardware 12 and memory hardware 14 that communicates with the data processing hardware 12 and stores instructions that, when executed by the data processing hardware 12, cause the data processing hardware 12 to perform one or more operations. The user device 10 further includes an audio system 16 having: an audio capture device (e.g., a microphone) 16, 16a for capturing spoken utterances 106 within the speech environment 100 and converting the spoken utterances 106 into electrical signals; and a speech output device (e.g., a speaker) 16, 16b for transmitting audible audio signals (e.g., as output audio data from the device 10). Although the user device 10 implements a single audio capture device 16 a in the example shown, the user device 10 may implement an array of audio capture devices 16 a without departing from the scope of the present disclosure, wherein one or more capture devices 16 a in the array may not physically reside on the user device 10 but may communicate with the audio system 16 .

[0024] In the speech environment 100, an automatic speech recognition (ASR) system 118 includes a model 200 (such as a recurrent neural network transducer (RNN-T) model or other conformer transducer model / multi-path model) that resides on a user device 10 of a user 104 and / or on a remote computing device 60 (e.g., one or more remote servers of a distributed system executed in a cloud computing environment) that communicates with the user device 10 via a network 40. The remote computing device is equipped with data processing hardware 62 and memory hardware 64. The user device 10 and / or the remote computing device 60 also includes an audio subsystem 108 that is configured to receive an utterance 106 spoken by the user 104 and captured by the audio capture device 16a, and convert the utterance 106 into a corresponding digital format associated with an input acoustic frame 110 that can be processed by the ASR system 118. In the example shown, a user speaks a respective utterance 106, and the audio subsystem 108 converts the utterance 106 into corresponding audio data (e.g., acoustic frames) 110 for input to an ASR system 118. Thereafter, the model 200 receives as input the audio frames 110 (i.e., audio data) corresponding to the utterance 106, and generates / predicts as output a corresponding transcription 120 (e.g., speech recognition results / hypotheses) of the utterance 106.

[0025] The user device 10 and / or the remote computing device 60 also executes a user interface generator 107 that is configured to present a representation of a transcription 120 of the utterance 106 to the user 104 of the user device 10. As described in more detail below, the user interface generator 107 can display the speech recognition results 120 in a streaming manner. In some configurations, the transcription 120 output from the ASR system 118 is processed, for example, by a natural language understanding (NLU) module executed on the user device 10 or the remote computing device 60 to execute the user command / query specified by the utterance 106. Additionally or alternatively, a text-to-speech system (not shown) (e.g., executed on any combination of the user device 10 or the remote computing device 60) can convert the transcription into synthesized speech for audible output by the user device 10 and / or another device.

[0026] In the example shown, the user 104 interacts with a program or application 50 (e.g., a digital assistant application 50) of the user device 10 using the ASR system 118. For example, Figure 1AA user 104 is depicted communicating with a digital assistant application 50 and the digital assistant application 50 displays a digital assistant interface 18 on a screen of a user device 10 depicting a conversation between the user 104 and the digital assistant application 50. In this example, the user 104 asks the digital assistant application 50, "What time is the concert tonight?" This question from the user 104 is a spoken utterance 106 that is captured by an audio capture device 16a and processed by an audio system 16 of the user device 10. In this example, the audio system 16 receives the spoken utterance 106 and converts the spoken utterance 106 into acoustic frames 110 for input to an ASR system 118.

[0027] Reference now Figure 1B , the remote computing device 60 executes the model trainer 150 to train Figure 1A A model 200 of the invention. The model trainer 150 obtains (e.g., from the memory hardware 64) a plurality of training samples 152, 152a-n. Each training sample 152 includes a spoken training utterance 154 (i.e., a sequence of input audio features) and a corresponding text utterance 156 representing a transcription 156 of the utterance 154. The model trainer 150 trains the model 200 on the plurality of training samples 152 using quantization-aware training (QAT) utilizing native integer operations. As discussed in more detail below, the model trainer 150 uses quantization-aware training by determining a scaling factor 160 during training. The model trainer 150 quantizes the model 200 to an integer fixed bit width 162 during training or after training. The integer fixed bit width 162 represents the number of bits allocated to each native integer operation. For example, when the integer fixed bit width 162 is eight, the model 200 is quantized to an 8-bit integer (i.e., int8). In another example, when the integer fixed bit width 162 is four, the model 200 is quantized to a 4-bit integer (ie, int4). Other examples are possible, such as a 6-bit integer. The integer fixed bit width 162 may be configurable by the user and depends on the use case of the model.

[0028] Conventional quantization-aware techniques rely on "pseudo" QATs. For example, many common systems use tf.quantization.fake_quant_* operations to quantize models. These operations are used during server-side inference, but for on-device models (e.g., user devices such as smartphones), conversion operations (e.g., TFlite) must be used to convert pseudo-quantization operations into integer operations. Therefore, these conventional techniques require this additional conversion step to convert pseudo-quantization operations into actual integer operations, and existing application programming interfaces (APIs) only support per-channel minimum and maximum estimates on the last dimension. This is not ideal because in some use cases, the channel dimension is not the last dimension. In these cases, conventional techniques must rearrange the dimensions of the input tensor so that the channel dimension becomes the last dimension, and then use the API to "pseudo" quantize the tensor. Finally, the dimensions must be rearranged back to the original order of the input tensor. These additional rearrangement operations increase training time. In contrast, the model trainer 150 uses native integer operations (e.g., tf operations). Therefore, the model 200 can be used for training and inference on both mobile applications and TPU applications. Furthermore, training can be additionally accelerated by using hardware-supported integer operations (e.g., matrix multiplication).

[0029] During quantization, the model trainer 150 reduces the size of the model 200 by adjusting the size of one or more weights 202 and / or activations 204 of the model 200. Conventionally, both the weights 202 and activations 204 of the ASR model are represented by float32 values, which occupy 32 bits of space and often require complex calculations to process. In PTQ, these float32 values ​​may be "clipped" or rounded to reduce precision and memory requirements. In contrast, the model trainer 150 uses native integer operations to represent the weights 202 of the model 200 using integers having a size determined by a fixed bit width 162. In some examples, the model trainer 150 quantizes the weights 202 and activations 204 according to the same fixed bit width 162 (e.g., 4 bits or 8 bits). In other examples, the model trainer 150 quantizes the weights 202 and activations using different fixed bit widths 162. For example, the model trainer 150 quantizes the activations 204 using a fixed bit width 162 that is larger than the fixed bit width 162 used for the weights 202 (e.g., 4 bits for the weights 202 and 8 bits for the activations 204). In yet other examples, the model trainer 150 quantizes the weights 202 without quantizing the activations 204 (e.g., the activations 204 are represented by floating point values ​​such as float32).

[0030] The model trainer 150 may quantize only a portion of the model 200. In some examples, when the model 200 includes multiple encoders and multiple decoders, the model trainer 150 quantizes only the encoders and does not quantize the decoders because in some scenarios, the memory requirements for the decoders are minimal. After training and quantization, the model trainer 150 may provide the quantized and trained model 200 to the user device 10.

[0031] Reference now Figure 2 , the example models 200, 200a include a recurrent neural network transducer (RNN-T) model architecture that complies with latency constraints associated with interactive applications. The use of the RNN-T model architecture is exemplary, and the model 200 may include other architectures such as transformer-transducer and conductor-transducer model architectures. The RNN-T model 200a provides a small computational footprint and utilizes fewer memory requirements than conventional ASR architectures, making the RNN-T model architecture suitable for performing speech recognition entirely on the user device 10 (e.g., without the need to communicate with a remote server). In this example, the RNN-T model 200a includes an encoder network 210, a prediction network 300, and a joint network 230. The encoder network 210, which is roughly similar to an acoustic model (AM) in a conventional ASR system, includes a stack of self-attention layers (e.g., Conformer layers or Transformer layers) or a recurrent network of stacked long short-term memory (LSTM) layers. For example, the encoder reads The order of the dimensional feature vector (e.g., acoustic frame) ,in , and a higher-order feature representation is produced at each output step. This higher-order feature representation is represented as .

[0032] Similarly, the prediction network 300 may also be an LSTM network that, like a language model (LM), converts the sequence of non-blank symbols outputted so far by the last Softmax layer 240 into Processing into dense representation Finally, using the RNN-T model architecture, the representations produced by the encoder and prediction / decoder networks 210, 300 are combined by the joint network 230. The prediction network 300 can be replaced by an embedding lookup table, thereby improving latency by outputting the sparse embeddings looked up instead of processing dense representations. The joint network then predicts , which is the distribution within the next output symbol. In other words, the joint network 230 generates a probability distribution within a possible speech recognition hypothesis at each output step (e.g., time step). Here, a "possible speech recognition hypothesis" corresponds to a set of output labels that each represent a symbol / character in a specified natural language. For example, when the natural language is English, the set of output labels may include twenty-seven (27) symbols, for example, one label for each of the 26 letters in the English alphabet, and one label specifies a space. Therefore, the joint network 230 can output a set of values ​​that indicates the probability of each of the predetermined set of output labels occurring. This set of values ​​can be a vector and can indicate a probability distribution within the set of output labels. In some cases, the output labels are graphemes (e.g., single characters, as well as potential punctuation marks and other symbols), but the set of output labels is not so limited. For example, in addition to or in place of graphemes, the set of output labels may include word pieces and / or entire words. The output distribution of the joint network 230 may include a posterior probability value for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, then the output of the joint network 230 100 different probability values ​​may be included, one for each output label. The probability distribution may then be used (e.g., by a Softmax layer 240) to select candidate orthographic elements (e.g., graphemes, word fragments, and / or words) and assign scores thereto in a beam search process for use in determining the transcription 120.

[0033] The Softmax layer 240 may employ any technique to select the output label / symbol with the highest probability in the distribution as the next output symbol predicted by the RNN-T model 200 at the corresponding output step. In this manner, the RNN-T model 200 does not make conditional independence assumptions, but instead the prediction of each symbol is conditioned not only on the acoustics, but also on the sequence of labels output so far. The RNN-T model 200 does assume that the output symbol is independent of future acoustic frames 110, which allows the RNN-T model to be employed in a streaming manner.

[0034] In some examples, the encoder network (i.e., audio encoder) 210 of the RNN-T model 200 includes a stack of multi-head attention layers or self-attention layers / blocks (such as one or more conformer blocks / layers and / or one or more transformer blocks / layers). Optionally, the encoder 210 (i.e., audio encoder) includes a first-path causal encoder and a second-path non-causal encoder for a multi-path architecture. This multi-path model unifies streaming and non-streaming ASR, where the causal encoder only uses the left context and produces partial results with minimal delay, while the non-causal encoder can provide more accurate assumptions by using both the left context and the right context. In this example, each conformer block includes a series of multi-head self-attention layers, depth-wise convolutional layers, and feedforward layers. The prediction network 300 can have two 2048-dimensional LSTM layers, each of which is also followed by a 640-dimensional projection layer. Alternatively, the prediction network 300 can include a stack of transformer or conformer blocks, or an embedding lookup table instead of an LSTM layer. Finally, the joint network 230 may also have 640 hidden units. The Softmax layer 240 may be composed of a unified word patch or grapheme set generated using all unique word patches or graphemes in the plurality of training data sets.

[0035] Figure 3 An exemplary prediction network 300 is illustrated for an RNN-T model 200a that receives a sequence of non-whitespace symbols. As input, the sequence of non-blank symbols is limited to the N previous non-blank symbols 301a-n output by the last Softmax layer 240. In some examples, N is equal to two. In other examples, N is equal to five, however, the present disclosure is non-restrictive and N can be equal to any integer. The sequence 301a-n of non-blank symbols indicates the initial speech recognition result 120a (Figure 1). In some implementations, the prediction network 300 includes a multi-headed attention mechanism 302, which shares a shared embedding matrix 304 across each head 302A-H of the multi-headed attention mechanism. In one example, the multi-headed attention mechanism 302 includes four heads. However, any number of heads can be used by the multi-headed attention mechanism 302. It is worth noting that the multi-headed attention mechanism significantly improves performance while minimizing the size of the model. As described in more detail below, each head 302A-H includes its own row 308 of position vectors, and rather than incurring an increase in model size by concatenating the outputs 318A-H from all heads, the outputs 318A-H are averaged by the head averaging module 322 .

[0036] Referring to the first head 302A of the multi-head attention mechanism 302, the head 302A is directed to a sequence of non-whitespace symbols received as input at corresponding time steps from a plurality of time steps. Each non-whitespace symbol 301 in generates a corresponding embedding 306, 306a-n (e.g., ). It is worth noting that because the shared embedding matrix 304 is shared across all heads of the multi-head attention mechanism 302, the other heads 302B-H all generate the same corresponding embedding 306 for each non-blank symbol. Head 302A also adds a sequence of non-blank symbols to Each corresponding non-whitespace symbol in is assigned a corresponding position vector 308, 308Aa-An (e.g., ). The corresponding position vector PV 308 assigned to each non-blank symbol indicates the position in the history of the sequence of non-blank symbols (e.g., the N previous non-blank symbols output by the last Softmax layer 240). For example, the first position vector PV is assigned to the most recent position in the history. , while assigning the last position vector to the last position in the history of N previous non-blank symbols output by the last Softmax layer 240 Notably, each of embeddings 306 may include the same number of dimensions (ie, dimensional size) as each of position vectors PV 308 .

[0037] Although the sequence 301a-n of non-whitespace symbols is targeted by the shared embedding matrix 304, , the corresponding embedding generated for each non-whitespace symbol 301 in the multi-head attention mechanism 302 is the same at all heads 302A-H, but each head 302A-H defines a different set / row of position vectors 308. For example, the first head 302A defines the row of position vectors 308Aa-An, the second header 302B defines different rows of position vectors , ..., the Hth header 302H defines another different row of position vectors .

[0038] For each non-blank symbol in the received sequence 301a-n of non-blank symbols, the first head 302A also weights the corresponding embedding 306 via the weight layer 310 in proportion to the similarity between the corresponding embedding and the corresponding position vector PV 308 assigned to the corresponding embedding. In some examples, the similarity may include a cosine similarity (e.g., a cosine distance). In the example shown, the weight layer 310 outputs a sequence 312 (i.e., 312Aa-An) of weighted embeddings each associated with the corresponding embedding 306, which is weighted in proportion to the corresponding position vector PV 308 assigned to the corresponding embedding 306. In other words, the weighted embedding 312 output by the weight layer 310 for each embedding 306 may correspond to the dot product between the embedding 306 and the corresponding position vector PV 308. The weighted embeddings 312 may be interpreted as focusing within the embedding in proportion to their similarity to the position associated with their corresponding position vector PV 308. To increase computational speed, the prediction network 300 includes a non-recurrent layer, and therefore the sequence of weighted embeddings 312Aa-An is not concatenated, but averaged by a weighted average module 316 to generate a weighted average 318A of the weighted embeddings 312Aa-An as output from the first head 302A, which is represented by: In equation 1, indicates the index of header 302, represents a position in the context, and represents the embedding dimension. Additionally, in Equation 1, and Including the size of the corresponding dimension. Position vector PV 308 does not have to be trainable and may include random values. It is worth noting that even if weighted embedding 312 is averaged, position vector PV 308 can potentially preserve position history information, thereby alleviating the need to provide recurrent connections at each layer of prediction network 300.

[0039] The operations described above with respect to the first head 302A are similarly performed by each other head 302B-H of the multi-head attention mechanism 302. Due to the different sets of positioning vectors PV 308 defined by each head 302, the weight layer 310 outputs a sequence of weighted embeddings 312Ba-Bn, 312Ha-Hn at each other head 302B-H that is different from the sequence of weighted embeddings 312Aa-Aa at the first head 302A. Thereafter, the weighted average module 316 generates a corresponding weighted average 318B-H of the corresponding weighted embeddings 312 in the sequence of non-blank symbols as output from each other corresponding head 302B-H.

[0040] In the example shown, the prediction network 300 includes a head average module 322 that averages the weighted averages 318A-H output from the corresponding heads 302A-H. A projection layer 326 with SWISH can receive as input an output 324 corresponding to the average of the weighted averages 318A-H from the head average module 322 and generate a projection output 328 as output. The final layer normalization 330 can normalize the projection output 328 to provide a single embedding vector Pu at corresponding time steps from multiple time steps. i 350. The prediction network 300 generates only a single embedding vector Pu at each of the multiple time steps after the initial time step i 350.

[0041] In some configurations, the prediction network 300 does not implement the multi-head attention mechanism 302, but only performs the operations described above with respect to the first head 302A. In these configurations, the weighted average 318A of the weighted embeddings 312Aa-An is simply passed through a projection layer 326 and a layer normalization 330 to provide a single embedding vector 350.

[0042] In some implementations, in order to further reduce the size of the RNN-T decoder (i.e., the prediction network 300 and the joint network 230), parameter binding between the prediction network 300 and the joint network 230 is applied. Specifically, for the vocabulary size and embedding dimension , the shared embedding matrix 304 at the prediction network is Meanwhile, the last hidden layer includes the dimension size of the joint network 230 , the feed-forward projection weights from the hidden layer to the output logit will be , where there are additional blank tokens in the vocabulary. Therefore, the feed-forward layer corresponding to the last layer of the joint network 230 includes the weight matrix By making the prediction network 300 embedding dimension The size of the last hidden layer of the joint network 230 The feed-forward projection weights of the combined network 230 and the shared embedding matrix 304 of the prediction network 300 can share their weights for all non-blank symbols via a simple transpose transformation. Because the two matrices share all their values, the RNN-T decoder only needs to store these values ​​once in memory, rather than storing two separate matrices. By increasing the size of the embedding dimension Set to be equal to the size of the hidden layer dimension , the RNN-T decoder reduces the embedding dimension to The size of the vocabulary This weight tying corresponds to the regularization technique.

[0043] Reference now Figure 4 , algorithm 400 illustrates native quantization for 8-bit integers (int8) in TensorFlow. Using algorithm 400, the model trainer 150 determines the scale factor 160 during training by first estimating the maximum value on the axis to be quantized (because algorithm 400 supports channel-by-channel quantization). Next, the model trainer 150 determines the scale factor 160 by dividing the maximum value by the integer representation value 410 (i.e., 127.0 in this example). The integer representation value 410 is based on the expected integer fixed bit width 162. That is, the scale factor 160 is based on the estimated maximum value of the axis to be quantized and the target fixed bit width 162. For example, for 8-bit quantization (i.e., the integer fixed bit width is 8), the integer representation value 410 is 127.0. As another example, for 4-bit quantization (i.e., the integer fixed bit width is 4), the integer representation value is 4.0. After determining the scale factor 160, the model trainer 150 quantizes the input tensor by dividing by the scale factor 160 and projecting it as an integer. Dequantization can be employed by multiplying the tensor by a scale factor of 160.

[0044] Figure 5 is a flow chart of an exemplary arrangement of operations of a method 500 for training an ASR model 200. At operation 502, the method 500 includes obtaining a plurality of training samples 152. Each respective training sample 152 in the plurality of training samples 152 includes a respective speech utterance 154 and a respective text utterance 156 representing a transcription of the respective speech utterance 154. At operation 504, the method 500 includes training an automatic speech recognition (ASR) model 200 on the plurality of training samples 152 using quantization-aware training utilizing native integer arithmetic. At operation 506, the method 500 includes quantizing the trained ASR model 200 to an integer target fixed bit width 162. The quantized trained ASR model 200 includes a plurality of weights 202. Each weight 202 in the plurality of weights 202 includes an integer having a target fixed bit width 162. At operation 508, the method 500 includes providing the quantized trained ASR model 200 to the user device 10.

[0045] Figure 6600 that can be used to implement the systems and methods described in this document. Computing device 600 is intended to represent various forms of digital computers, such as laptops, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions are intended to be exemplary only, and are not intended to limit implementations of the inventions described and / or claimed in this document.

[0046] The computing device 600 includes a processor 610, a memory 620, a storage device 630, a high-speed interface / controller 640 connected to the memory 620 and a high-speed expansion port 650, and a low-speed interface / controller 660 connected to a low-speed bus 670 and the storage device 630. Each of the components 610, 620, 630, 640, 650, and 660 is interconnected using various buses and can be installed on a common motherboard or installed in other ways as appropriate. The processor 610 can process instructions for execution within the computing device 600, including instructions stored in the memory 620 or on the storage device 630, to display graphical information of a graphical user interface (GUI) on an external input / output device (such as a display 680 coupled to the high-speed interface 640). In other implementations, multiple processors and / or multiple buses and multiple memories and multiple types of memories can be used as appropriate. In addition, multiple computing devices 600 can be connected, each of which provides a portion of the necessary operations (for example, as a server group, a blade server group, or a multi-processor system).

[0047] Memory 620 stores information non-temporarily within computing device 600. Memory 620 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. Non-temporary memory 620 may be a physical device for temporarily or permanently storing programs (e.g., instruction sequences) or data (e.g., program state information) for use by computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0048] The storage device 630 can provide mass storage for the computing device 600. In some implementations, the storage device 630 is a computer-readable medium. In various implementations, the storage device 630 can be a floppy disk device, a hard disk device, an optical disk device or a tape device, a flash memory or other similar solid-state memory device, or a device array (including a device in a storage area network or other configuration). In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as the memory 620, the storage device 630, or a memory on the processor 610.

[0049] The high-speed controller 640 manages bandwidth-intensive operations of the computing device 600, while the low-speed controller 660 manages less bandwidth-intensive operations. These assignments of responsibilities are exemplary only. In some implementations, the high-speed controller 640 is coupled to a memory 620, a display 680 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 650 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 660 is coupled to a storage device 630 and a low-speed expansion port 690. The low-speed expansion port 690 (which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet)) may be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device (such as a switch or router), for example, via a network adapter.

[0050] The computing device 600 can be implemented in many different forms, as shown. For example, the computing device can be implemented as a standard server 600a or multiple times in a group of such servers 600a, as a laptop computer 600b, or as part of a rack server system 600c.

[0051] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuit systems, integrated circuit systems, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system that includes at least one programmable processor that can be either special purpose or general purpose and can be coupled to receive data and instructions from and send data and instructions to a storage system, at least one input device, and at least one output device.

[0052] A software application (i.e., software resource) may refer to computer software that enables a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0053] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0054] The processes and logic flows described in this specification can be performed by one or more programmable processors (also referred to as data processing hardware), which execute one or more computer programs to perform functions by operating on input data and generating outputs. The processes and logic flows can also be performed by a dedicated logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). For example, processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as a magnetic disk, a magneto-optical disk, or an optical disk, or be operably coupled to receive data from one or more mass storage devices or to transfer data to one or more mass storage devices or both. However, a computer does not have to have these devices. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0055] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), an LCD (liquid crystal display) monitor, or a touch screen) for displaying information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0056] A variety of implementations have been described. However, it should be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Therefore, other implementations are within the scope of the appended claims.

Claims

1. A computer-implemented method (500) which, when executed on data processing hardware (62), causes the data processing hardware (62) to perform operations comprising: A plurality of training samples (152) is obtained, each corresponding training sample (152) in the plurality of training samples (152) comprising: a corresponding speech utterance (154); and a corresponding text utterance (156) representing a transcription of the corresponding speech utterance (154); Training an automatic speech recognition (ASR) model (200) on the plurality of training samples (152) using quantization-aware training using native integer arithmetic; quantizing the trained ASR model (200) to an integer target fixed bit width (162), the quantized trained ASR model (200) comprising a plurality of weights (202), each weight (202) of the plurality of weights (202) comprising an integer having the target fixed bit width (162); and The quantized trained ASR model (200) is provided to a user device (10).

2. The method (500) of claim 1, wherein the target fixed bit width (162) is four.

3. The method (500) of claim 1 or claim 2, wherein the ASR model (200) further comprises a plurality of activations (204), each activation (204) of the plurality of activations (204) comprising an integer having the target fixed bit width (162).

4. The method (500) of claim 1 or claim 2, wherein the ASR model (200) further comprises a plurality of activations (204), each activation (204) of the plurality of activations (204) comprising an integer having a fixed bit width greater than the target fixed bit width (162).

5. The method (500) of claim 1 or claim 2, wherein the ASR model (200) further comprises a plurality of activations (204), each activation (204) of the plurality of activations (204) comprising a floating point value.

6. The method (500) of any one of claims 1 to 5, wherein quantizing the trained ASR model (200) comprises determining a scaling factor (160) based on an estimated maximum value of the axis to be quantized and the target fixed bit width (162).

7. The method (500) of any one of claims 1 to 6, wherein the ASR model (200) comprises one or more multi-head attention layers (302).

8. The method (500) of claim 7, wherein the one or more multi-head attention layers (302) include one or more conformer layers or one or more transformer layers.

9. The method (500) of any one of claims 1 to 8, wherein: The ASR model (200) includes a plurality of encoders and a plurality of decoders; and Quantizing the ASR model (200) includes quantizing the plurality of encoders without quantizing the plurality of decoders.

10. The method (500) of any one of claims 1 to 9, wherein: The ASR model (200) includes an audio encoder (210); and The audio encoder (210) comprises a cascade of encoders, the cascade of encoders comprising a first causal encoder and a second non-causal encoder.

11. A system (100), comprising: Data processing hardware (62); as well as Memory hardware (64) in communication with the data processing hardware (62), the memory hardware (64) storing instructions that, when executed on the data processing hardware (62), cause the data processing hardware (62) to perform operations, the operations comprising: A plurality of training samples (152) is obtained, each corresponding training sample (152) in the plurality of training samples (152) comprising: a corresponding speech utterance (154); and a corresponding text utterance (156) representing a transcription of the corresponding speech utterance (154); Training an automatic speech recognition (ASR) model (200) on the plurality of training samples (152) using quantization-aware training using native integer arithmetic; quantizing the trained ASR model (200) to an integer target fixed bit width (162), the quantized trained ASR model (200) comprising a plurality of weights (202), each weight (202) of the plurality of weights (202) comprising an integer having the target fixed bit width (162); and The quantized trained ASR model (200) is provided to a user device (10).

12. The system (100) of claim 11, wherein the target fixed bit width (162) is four.

13. The system (100) of claim 11 or claim 12, wherein the ASR model (200) further comprises a plurality of activations (204), each activation (204) of the plurality of activations (204) comprising an integer having the target fixed bit width (162).

14. The system (100) of claim 11 or claim 12, wherein the ASR model (200) further comprises a plurality of activations (204), each activation (204) of the plurality of activations (204) comprising an integer having a fixed bit width greater than the target fixed bit width (162).

15. The system (100) of claim 11 or claim 12, wherein the ASR model (200) further comprises a plurality of activations (204), each activation (204) of the plurality of activations (204) comprising a floating point value.

16. The system (100) of any one of claims 11 to 15, wherein quantizing the trained ASR model (200) comprises determining a scaling factor (160) based on an estimated maximum value of the axis to be quantized and the target fixed bit width (162).

17. The system (100) of any one of claims 11 to 16, wherein the ASR model (200) comprises one or more multi-head attention layers (302).

18. The system (100) of claim 17, wherein the one or more multi-head attention layers (302) include one or more conformer layers or one or more transformer layers.

19. The system (100) of any one of claims 11 to 18, wherein: The ASR model (200) includes a plurality of encoders and a plurality of decoders; and Quantizing the ASR model (200) includes quantizing the plurality of encoders without quantizing the plurality of decoders.

20. The system (100) of any one of claims 11 to 19, wherein: The ASR model (200) includes an audio encoder (210); and The audio encoder (210) comprises a cascade of encoders, the cascade of encoders comprising a first causal encoder and a second non-causal encoder.

Images